Copy Shopdata: Script for upgrading and fixing buggy shops by copying business data and images

[musicmaster]

Copy_shopdata is a script that copies the content (business data) tables from the database of one Prestashop installation to the database of another installation. The products, orders, customers, etc of the new (usually freshly installed) shop will be completely replaced by those from the old shop while its configuration is maintained. Your customers can even keep their old passwords. As some additional settings need to be changed you should read the manual carefully.

By default the database structure and field definitions of the new shop are used. However, the are options to copy (part of) the structure of the old shop. 

You can use Copy_shopdata for fixing broken shops and for upgrading. It will adapt the data to the requirements of the new version. The minimum version of the old shop is 1.5.0.10.

Copy_shopdata should be installed in the same directory as Prestools as it uses Prestools for authorization. It should be installed in the new shop, after which you fill in the database credentials of the old shop in copy_shopsdata's config file.

With the added file copy_shopdata_imghub.php you can copy images from one installation on a server to another. It works even when the files are on a different server.

The added file Copy_shopdata_fix copies all database tables to the database of a new shop of the same version that is on the same server and applies a few fixes. After that you can assign the new database to the shop.

This is mainly meant for shops that used to work but had a bad upgrade. It can be applied very quickly.

Documentation is in the pdf files.

PS. Nearly any shop can be repaired and/or updated this way. If you are afraid to do it yourself you can hire me to do it for you.

PS-2: For upgrading older Prestashop versions up to version 1.6.1.24 you can also try UpgradeHub. That offers an interactive way of executing the Prestashop upgrade process that offers you a way to see and correct the error messages.

You can download from the Prestools website:

 

Edited October 3 by musicmaster (see edit history)

[IceCOOL]

There are some BUGs in the source code of these files...

1. I strongly recommend to use the same DataBase prefixes. (old shop and fresh shop). I had to update all these files manually inserting different prefixes.

2. There is difference between old language codes and new language codes... in ps_lang.

New PS 1.6 have language code for example de-de

Old PS 1.4 -> updated to 1.6 wiil have de language code...

3. some updates to copy_shopdata.php

 

line 62   for($i=0; $i < (sizeof($oldlangs)-1); $i++)

we should reduce count of iterations.

 

line 207  for($i=0; $i < (sizeof($oldlangs)-1); $i++)

same

[musicmaster]

Hi IceCool,

 

Thank you for your reaction:

 - ad 1: I am not sure what you mean. Did you change in copy_shopdata_config.php the setting for the old shop?

 

 - ad 2: I didn't find such problems working with a shop that originally was 1.4. PS update should take care of that.

 

 - ad 3: in line 62 it needs to compare all languages. If you compare "fr-en-it" with "fr-en-ru" you should get an error message.

 in line 207 you need to take into account the case where positions are exchanged. If you change "1=fr 2=en" to "1=en 2=fr" french will temporarily be copied to id 3 and you need a second copy operation to move it back to 2.

[Ray UK]

hey musicmaster,

 

Im trying to copy data from 1.6.1.2 to a fresh install of 1.6.1.3 but getting errors.

 

First error was involving accessories so I remove that from the list, then the next error shows.

13:49:04
1 old languages; 1 new languages. Language check ok.


address 5799 1

MySQL error 1045: Access denied for user '********'@'localhost' (using password: YES)
Generated by Query 'SELECT `id_address`,`id_country`,`id_state`,`id_customer`,`id_manufacturer`,`id_supplier`,`id_warehouse`,`alias`,`company`,`lastname`,`firstname`,`address1`,`address2`,`postcode`,`city`,`other`,`phone`,`phone_mobile`,`vat_number`,`dni`,`date_add`,`date_upd`,`active`,`deleted` INTO OUTFILE '/home/*******/public_html/newshop/*******/triple_edit/copy_shopdata_address.dtx' FROM ps_address' 

This script worked fine copying from an upgraded 1.6.1.2 to a fresh 1.6.1.2 so has code changed in 1.6.1.3 ?

 

UPDATE: I have just managed to update my live shop to 1.6.1.3, yet the script still wont copy from my live store to the fresh install.

 

Cheers

Ray

Edited December 2, 2015 by MerseyRay (see edit history)

[musicmaster]

Hi MerseyRay,

 

This is an "access denied" error. So it has nothing to do with the structure of the database.

 

The problem is in the second part of the command "INTO OUTFILE []copy_shopdata_address.dtx". On Unix servers this very often leads to an error that MySql is not allowed to write that (or any other) file. The immediate solution is to increase your rights. 

 

However, I am considering to remove the option in the script to move data between servers. It gives too much problems. The best way to work is to upgrade on your local computer (localhost) and then upload the files to the server.

[Rudolfo78]

Hi musicmaster,

I'm trying script copy_shopdata on my localhost installations - both 1.6.0.9 and I got stuck at the very beginning of copy process

 

1) old installation - I've downloaded my current installation 1.6.0.9 via FTP & exported database.sql and configured it to run localy (OS X, MAMP, Sequel Pro).

 

2) new installation - I've installed fresh installation same version 1.6.0.9 on my localhost, where I added subfolder "triple_edit" with its files, as well as script copy_shopdata files

13:45:32
1 old languages; 1 new languages. Language check ok.


accessory 1123 1
MySQL error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-new.ps_accessory SELECT `id_product_1`,`id_product_2` FROM maclife-old.ps_acces' at line 1
Generated by Query 'INSERT INTO maclife-new.ps_accessory SELECT `id_product_1`,`id_product_2` FROM maclife-old.ps_accessory'

I've checked manual but I didn't find any useful information.

 

Could you please give me some advice, what should I check or revise. As I'm only basic user with some additional skills, I'm not very good in coding, syntax, etc.

 

Thanks for any advice.

[musicmaster]

I am not totally sure. But my guess is that the hyphens ("-") in the database names are the problem. They might be interpreted as "minus". The solution is to put the server names between backticks.I will have a look at it.

[Rudolfo78]

You've guessed right musicmaster.

 

The problem was in the name of database "maclife-old" was misinterpreted as maclife minus old.

 

I've changed it to "maclife_old" and everything looks ok.

 

Thanks for quick advice.

[musicmaster]

Here an old version as backup

copy_shopdata-v0.17b.zip

[Ed van der Meer]

Hi,

 

I have 1.6.1.2 running and have some database trouble and other problems. My features are messed up and I can't fix it nor I can't get back to a older state. Also i'm struggling too long with this site for almost a year.

So I'm busy creating a new shop with latest version of PS and want to have the products without features and attributes , customers, Categories and orders to import to new one and then recreate attributes and features. I mean I want only the necessary files back to the new site and build it module by module.

 

Then when finished and tested I want to switch over to that.

 

Is that posible to do. Or could someone help me with it? I'm willing to pay for it. I run it at my own server so I can give acces to SQL, BA, FTP and so on.

 

I hope someone can help me.

Thanks in advance.

 

Ed

Edited September 21, 2016 by Ed van der Meer (see edit history)

[phantom1]

I have detected an error the table ps_product_lang is not generated correctly, it is created with the id_lang of the old DB, not with the id that corresponds to it in the new DB

[musicmaster]

Language transformation happens rather late in the code and depends on all the language tables being flagged as such.. Are you sure the script didn't break off somewhere in the middle? 

 

In older versions having a $startnum greater than zero meant the skipped tables were not considered for the language transformation. In the latest versions (several months already) this problem has been repaired.

 

If you believe it is something else please let me know. I am always open for opportunities to improve the script.

Edited November 23, 2016 by musicmaster (see edit history)

[Rudolfo78]

Hi,

I'm not sure If I'm the only one, but after downloading latest version v0.19 of script copy_shopdata it doesn't work properly.

 

I used previous versions multiple times and everything worked perfectly, but now after copy process the source (old one) database is corrupted and the new one is untouched.

 

I've tried to move data on localhost from PS1614 –> PS1614 or from PS1614 -> PS16110 with same bad results.

 

 

Thanks for any advice.

[musicmaster]

Hi,

 

Sorry to hear you have problems with the script. 

 

I will tomorrow have a test run to see whether I can reproduce the issue.

[musicmaster]

 

Hi,

I'm not sure If I'm the only one, but after downloading latest version v0.19 of script copy_shopdata it doesn't work properly.

 

I used previous versions multiple times and everything worked perfectly, but now after copy process the source (old one) database is corrupted and the new one is untouched.

 

I've tried to move data on localhost from PS1614 –> PS1614 or from PS1614 -> PS16110 with same bad results.

 

 

Thanks for any advice.

 

Hi Rudolfo,

 

I tried it for myself and concluded that the script didn't work when the prefix of the two shops differed. In the new version that I have uploaded that is repaired.

 

However, I can't see any sign of changes in the old shop so I am very puzzled by that aspect of your report. Are you sure you put the script with the new shop and entered the references for the old shop in the config file?

 

This time the script put me into the login loop on the new shop. So I can't yet evaluate it further. A good opportunity to finally solve that almost a decade old problem. Unfortunately I don't have much time at the moment so it will have to wait a bit.

 

Regards,

M.

Edited December 8, 2016 by musicmaster (see edit history)

[Rudolfo78]

Hi musicmaster,

 

results are the same with script v0.20.

 

Step by step 1.6.1.4 –> 1.6.1.4 (MAMP, Sequel Pro, macOS Sierra)

 

1) source data installed to root folder on localhost - language ID 2 (only one language active), database name: "1614old"

2) new installation PS 1614 installed to subfolder "1614" with database name "1614new", folder "prestools" with copy_shopdata files copied to "admin..." folder, localization (Slovak) imported and activated as ID 2, default language English ID 1 deactivated and erased

3) script started with .../1614/admin.../copy_shopdata.php

4) after whole process, source/old installation corrupted and the new one untouched, screenshot of localhost attached

 

Thanks

 

Edited December 9, 2016 by Rudolfo78 (see edit history)

[musicmaster]

Hi Rudolfo,

 

Can you switch on the development mode in the old shop so that you can see what error that 500 error hides?

 

Can you export the database of the old shop before and after the copying and look for differences with a program like WinMerge?

 

Are you able to enter the backoffice of the shops after the copying? There may be some problem with .htaccess and the backoffice is not vulnerable to that as it doesn't redirect.

 

Regards,

M

[Rudolfo78]

Hi musicmaster,

 

beforehand I have one more question, whether is it ok to replace one file when I am copying copy_shopdata files to prestools folder or should I keep the original one?

 

Thanks,

R

[musicmaster]

Hi Rudolfo,

 

What do you mean with "one file"? The copy_shopdata files are a package. You cannot replace one of them and keep the old version of another. 

 

Regards,

M.

[Rudolfo78]

Hi musicmaster,

 

Sorry for previous explanation. Mentioned "one file" is exim.php. 

 

Prestools folder contains it, as well as copy_shopdata folder contains it and as I know copy_shopdata files should be copyed to prestools folder. Right?

 

My question: Is it necessary to overwrite files exim.php or not?

 

Thanks,

R.

Edited December 9, 2016 by Rudolfo78 (see edit history)

[musicmaster]

exim.php. 

 

Prestools folder contains it, as well as copy_shopdata folder contains it and as I know copy_shopdata files should be copyed to prestools folder. Right?

 

My question: Is it necessary to overwrite files exim.php or not?

 

Exim.php is irrelevant in this context. It doesn't matter what you do.

 

Regards,

M.

[Rudolfo78]

Ok, today I will check your previous suggestions and I'll give you needed info.

 

Regards,

R.

[Rudolfo78]

Hi master,

 

answers to your questions:

 

1) Development mode see attached screenshot

 

2) According to DiffMerge tool (which is similar to WinMerge), there are 199 changes between both SQL files old vs. new one - there is also significant difference in size of these files (11,8MB vs. 0,6MB)

 

3) I am not able to enter backoffice of old shop, new one works perfect but it is untouched (default data in products catalog etc.)

 

Any suggestions?

 

Thanks,

R.

 

Edited December 9, 2016 by Rudolfo78 (see edit history)

[musicmaster]

Hi Rudolfo,

 

Strange, I don't know how to explain this.

 

First of all, 0.6mb is the size of the new shop's database. Even if you copied the business data from the new shop to the old shop you wouldn't arrive at this number as there are tables like ps_connections and ps_guest that usually are quite big and that are not changed by copy_shopdata.

 

The Prestashop exception suggests that your ps_shop table in the database is empty. Could you have a look?

 

Can you be more precise about not being able to enter the admin of the old shop? What is the message you get?

 

Regards,

M.

[Rudolfo78]

Hi master,

 

Yes ps_shop table is empty (see attached image 1)

 

Error message I've got after entering "localhost/admin..." in browser (see attached image 2)

 

Regards,

R.

 

[musicmaster]

 - Did you make any changes to the newest copy_shopdata_config.php beyond filling in the access data for the old shop?

 

 - Did you make any changes in the new shop before you started the copying?

 

 - if you mail a copy of your old shop to me ([email protected]) I will try it myself. I need only the database dump and the settings.inc.php file. The latter for the Ryndael keys. An export of the database after the copying would also be welcome (it would give me more insight of what is going on).

[Rudolfo78]

- I didn't make any other changes except for filling in the access data for the old shop.

 

- In the new shop I've only imported and activated new (Slovak) language and deactivated and erased default (English) language.

 

- I'll send you PM here shortly.

 

Regards,

R.

[musicmaster]

Fian

 

- I didn't make any other changes except for filling in the access data for the old shop.

 

- In the new shop I've only imported and activated new (Slovak) language and deactivated and erased default (English) language.

 

- I'll send you PM here shortly.

 

Regards,

R.

It turned out that approve.php of Prestools didn't handle the nested shop situation correctly. It included the Settings.inc.php of the shop in the root instead of the shop in the subdirectory under which it had been placed. As a consequence the old shop was also seen as the new shop and its tables were emptied. Things have now been repaired. I have also added an extra check for the case that the old and the new database are the same. In that case you will now get an error message and the script will stop.

 

Thanks,

M.

Edited December 22, 2016 by musicmaster (see edit history)

[bovario]

Can you advise if this script will work for me?

I want to upgrade my shop from 1.6.0.9 to latest version. I do not want to use the 1-click upgrade as it is the risk to end up with an unstable shop.

In the past when want to upgrade:

1 Installed new shop with the same version.

2 Deleted all records from newly installed shop database and imported database from the old shop.

3 Check if all working fine in the back and front office.

4 1-click upgrade of new installed shop to higher version 

5 If new version works fine switch domain (have one domain for tests) to newly upgraded shop.

 

Lately, is not working. Have errors server misconfiguration after import database from the old shop.

 

If your script can copy a database from live shop to newly installed shop in the same version (live shop leave intact)?

[musicmaster]

Can you advise if this script will work for me?

I want to upgrade my shop from 1.6.0.9 to latest version. I do not want to use the 1-click upgrade as it is the risk to end up with an unstable shop.

In the past when want to upgrade:

1 Installed new shop with the same version.

2 Deleted all records from newly installed shop database and imported database from the old shop.

3 Check if all working fine in the back and front office.

4 1-click upgrade of new installed shop to higher version 

5 If new version works fine switch domain (have one domain for tests) to newly upgraded shop.

 

Lately, is not working. Have errors server misconfiguration after import database from the old shop.

 

If your script can copy a database from live shop to newly installed shop in the same version (live shop leave intact)?

 

I don't know what to make from your procedure. It is my impression that instability is often not in the files but in the configuration part of the database. 

 

The big difference between my script and your procedure is that it copies only the "business data" tables of the database: products, orders, etc. It doesn't copy the configuration tables of the database (configuration, module, hook, etc). That means that the chance that the new system will be stable is much bigger. The price is that it will take a bit more work.

[A-Z Hosting]

Hey MusicMaster:  So if I have a 1.6.1.12 old shop and a 1.6.1.12 new shop will it skip modifying the old store?  I would like to be able to run this pulling from a production install over to a fresh install.  Then install theme and extra mods in new shop.  Then run it again just before going LIVE to grab new customers and orders accumulated since the first run.

 

Think it will work?  Backups will be made of course.

 

Wil

Edited April 13, 2017 by A-Z Hosting (see edit history)

[musicmaster]

 - yes, the old shop will not be changed.

 - yes, you can run a last minute run to include the latest orders, etc.

 

Of course you are encouraged to have an extra run somewhere half way your project. Being familiar with how it works is the best way to avoid surprises.

[A-Z Hosting]

 - yes, the old shop will not be changed.

 - yes, you can run a last minute run to include the latest orders, etc.

 

Of course you are encouraged to have an extra run somewhere half way your project. Being familiar with how it works is the best way to avoid surprises.

 

Good idea thanks. One more question while I have your attention.  Once I have things as I want them do you have any ideas on jumping to 1.7.1.0?  Will it still ignore the old and just bring over the updated data?  Or should I upgrade the old, upgrade the new, and then bring over the updated data?

 

Wil

[musicmaster]

Always upgrade Prestashop to the same version. The databases of 1.6.1.12 and 1.7.1 are very similar. So it might work - at least at first sight. But you have no guarantee that the minor differences that are present will not cause problems somewhere in the future.

[A-Z Hosting]

That's kind of what I figured on the side of caution. Just wanted to confirm. Thanks for the scripts these have been much needed.  You ever thought of turning into a full module? Have the settings all configurable from the destination store side? Adding a few other parameters like "skip old store upgrade", or "upgrade old store to version xxxx", etc?  Would be even more awesome for those that don't know how, or don't have the convenience of shell to toss things around.

 

Wil

[musicmaster]

Maybe I will turn it one day into a module. However, I don't think automation of the old store upgrade is desirable. The Prestashop upgrade process is notoriously brittle so I think that it is crucial that someone is actively looking at the messages that are emitted during the process.

[A-Z Hosting]

I thought that I had read that it upgrades the old to the same version of the new already. If not then never mind. I'm done with my import and have some feedback for you.

 

Not all shared hosting packages come with access to INFILE and OUTFILE. Actually the secure ones don't.  Special permission needs to be given to access those files.  Secondly the ~mysql user shouldn't have access to the user folders so the $basepath isn't writable.  Being the host I was of course able to get around this but not everyone has my access.

 

GRANT FILE ON *.* TO 'imako_imakops'@'localhost';

GRANT FILE ON *.* TO 'stagingi_ps1'@'localhost';

flush privileges;

 

That gave the oldshop user and the newshop user access.  However, it still wasn't writable in the users' jailed folder.  I had to take $basepath out and change it to ./ so that the ~mysql user could write the files, which of course went into it's own ~mysql/ folder. Afterwards I just deleted them from there.

 

I think you could get around all of this if you wrote to temporary tables on the destination shop's database.  So instead of writing then reading from copy_shopdata_address.dtx just write and read from a copy_shopdata_address table. After completion drop table. I also think that would do better on benchmarks.

 

Wouldn't that work better or is there something about the dtx formatting that made it necessary?

 

Wil

 

--------------

14:26:14
oldconn = localhost.imako_imakops = 2532436
new conn = localhost.stagingi_ps1 = 2532435
2 old languages; 2 new languages. Language check ok.

Initialization skipped because it had already run
Copying tables

1 accessory 0
2 address 44355

MySQL error 1: Can't create/write to file '/home/xxxxxxx/public_html/exxxx5ckcmg82/prestools/copy_shopdata_address.dtx' (Errcode: 13 - Permission denied)
Generated by URL '/exxxx5ckcmg82/prestools/copy_shopdata.php'
with Query 'SELECT `id_address`,`id_country`,`id_state`,`id_customer`,`id_manufacturer`,`id_supplier`,`id_warehouse`,`alias`,`company`,`lastname`,`firstname`,`address1`,`address2`,`postcode`,`city`,`other`,`phone`,`phone_mobile`,`vat_number`,`dni`,`date_add`,`date_upd`,`active`,`deleted` INTO OUTFILE '/home/xxxxx/public_html/eyxxx5ckcmg82/prestools/copy_shopdata_address.dtx' FROM `ps_address`'

Edited April 13, 2017 by A-Z Hosting (see edit history)

[musicmaster]

Hi Wil,

 

Thank you for the feedback.

 

The recommended method is downloading to your localhost and upgrading there. This also fits with the pattern that people usually simultaneously take a new template and make other changes. As Prestashop's upgrade process is rather buggy and as there haven't been new features for quite some time upgrading for the sake of it is only recommended if there is a specific bug that bothers you and that is fixed in the new version. Copy-shopdata has been mainly developed with this localhost scenario in mind.

 

Copy-shopdata has been developed at a time when there was nothing. So many people for whom the automatic and manual upgrades failed felt they were stuck. That was the problem this software was meant to solve.

 

The problem with on-server upgrades is that there are so many scenarios. In many cases an upgrade involves a move to a new server. Some hosting providers allow remote access to the database. Others not [that's what the dtx files are meant to solve]. Writing rights to files are often a problem. Copy-shopdata could be much improved in this area. But I don't have the time and money. Neither do I have a clear vision on the best way to implement it.

 

So thank you for your feedback. Every bit helps to make the software better.

[lozaria]

Can I move data from 1.6 to 1.7 with this tool?

I'm getting:

 

MySQL error 1054: Unknown column 'id_theme' in 'field list'
Generated by URL '/aaaa/admin3/pst/copy_shopdata.php'
with Query 'SELECT id_theme FROM ps_shop order by id_shop'

 

I checked 1.7 version and there is no id_theme but theme_name field in ps_shop.

[musicmaster]

Can I move data from 1.6 to 1.7 with this tool?

I'm getting:

 

MySQL error 1054: Unknown column 'id_theme' in 'field list'

Generated by URL '/aaaa/admin3/pst/copy_shopdata.php'

with Query 'SELECT id_theme FROM ps_shop order by id_shop'

 

I checked 1.7 version and there is no id_theme but theme_name field in ps_shop.

 

Thank you for reporting the problem. I will have a look into it. For the meantime I advise you to delete ps_Shop from the copied tables.

[musicmaster]

With version 0.24 I have made an update that also works with Prestashop 1.7.

[Rudolfo78]

With version 0.24 I have made an update that also works with Prestashop 1.7.

 

It means that new 0.24 works between different versions 1.6 -> 1.7??

 

Or simply copy from 1.6 to 1.6 and 1.7 to 1.7?

 

Thanks, for info.

Edited June 14, 2017 by Rudolfo78 (see edit history)

[musicmaster]

It means that new 0.24 works between different versions 1.6 -> 1.7??

 

Or simply copy from 1.6 to 1.6 and 1.7 to 1.7?

 

Thanks, for info.

In principle Copy_shopdata is for copying between the same version. The goal is to get rid of defects in the settings of the old shop.

 

However, very little has been changed in the business data between 1.6 and 1.7. So updating this way might work. But at your own risk: I haven't tested it.

[musicmaster]

In the new version I deleted the ps_meta and ps_meta_lang tables from the list of tables that were copied. It turned out that copying those could cause problems with translations of pretty urls. For example for order confirmation the Dutch files should look like the following

http://www.myshop.com/order-bevestiging?id_cart=6&id_module=3&id_order=6&key=f3d234d2476a951b5c80234rr44524804

But instead in one case it looked like

http://www.myshop.com/index.php?controller=order-confirmation?id_cart=124998&id_module=3&id_order=24049&key=c45te4f3t34d65292e44fd5c

If you have such problems you should copy the meta and meta-lang tables from a fresh installation of the PS version that you copied to. It may also be necessary to also install your new theme there first. 

Please note that:

 - this will be only a problem in some shops - not all

 - this will not harm the normal functioning of your shop. Until now the only place where I found it causing problems was in setting goals for Google analytics.

 

Edited November 17, 2017 by musicmaster (see edit history)

[musicmaster]

In version 0.26 a new function (and file) has been added: module-map. It gives a table that shows you which modules are present in the old and the new shops. That way you won't forget to install some modules.

The name of the file is copy_shopdata_modulemap.php. It will be run at the end of copy_shopdata.php and it can also be run independently.

[morepix]

FYI since I found your plug-in investigating an upgrade from 1.6.1.18 -> 1.7.3.0

Quote

26 cart_product 336401 structure different. Extra in new: id_customization,
MySQL error 1366: Incorrect integer value: '' for column 'id_customization' at row 1
Generated by URL '/adm/prestools_suite/copy_shopdata.php'
with Query 'LOAD DATA INFILE 'xxxxxxxxxxxxxxxxx/adm/prestools_suite/copy_shopdata_cart_product.dtx' INTO TABLE `ps_cart_product` CHARACTER SET 'utf8''

 

Example from the dtx file

20    22    0    1    0        1    2011-04-19 20:07:15
23    48    0    1    84        1    2011-04-26 10:48:17

[morepix]

Deleted the new column as a test and that gives duplicate data

 

MySQL error 1062: Duplicate entry '107115-1834-0-0' for key 'PRIMARY'

[musicmaster]

I would love to check this out for you but unfortunately 1.7.3 won't install on my Windows work computer.

[musicmaster]

19 hours ago, morepix said:

Deleted the new column as a test and that gives duplicate data

 

MySQL error 1062: Duplicate entry '107115-1834-0-0' for key 'PRIMARY'

 

I have checked what has changed in the database structure in 1.7.3 and it is a bit too much. If you want to use Copy_Shopdata this way you should do it towards version 1.7.2.4 and then go to 1.7.3 with the 1-click upgrade.

[csoon1992]

Hi!

How can I migrate customers from PS 1.6 to PS 1.7.2 keeping customer passwords?

Thank u!

[DragoshDX]

Hello, and thank you for your work. 

I've managed to migrate the whole of my PS 1.6.1.16 to a new install of 1.7.3.1
It ran FLAWLESSLY. 

It ported over user accounts, user information, user orders, actual products that I can visit in the website (via direct category URL access). It came in with shipping methods and payment methods.
It came in with products, categories, category structure, pricing, tax info, stock levels, you name it. 

I can place an older with my old user from the old website.

However, I have a single problem still. Even though I can create an order and have it seen in the back office, when I go hit products in the left menu, to see my products in the backoffice, the list is empty. But like I said, the actual front office DOES see them, I can even use the ajax search, it's unbelievable how well this worked :)). 

 

If I go through an order in the backoffice and click on any product that has been order, I do get the product form working, it's just that I don't see products in the actual Products section (w perfect stock, and languages!)

Is there anything I can do so my products show up in the backoffice? Has this happened in your experience? I'm ready to start doing tests on the production server as well, but my team won't be able to do much if they don't see the products in the back office. 

Here is the output of the copyshop script on the product section: 

149 product 118 structure different. Extra in old: id_product_redirected, Extra in new: isbn, low_stock_threshold, low_stock_alert, additional_delivery_times, id_type_redirected, show_condition, state, 
150 product_attachment 0
151 product_attribute 133 structure different. Extra in new: isbn, low_stock_threshold, low_stock_alert, 
152 product_attribute_combination 168
153 product_attribute_image 85
154 product_attribute_shop 133 structure different. Extra in new: low_stock_threshold, low_stock_alert, 
155 product_carrier 0

I'll debug this further tomorrow, but I'm thinking that perhaps you might have encountered this before?

[DragoshDX]

On 3/12/2018 at 5:54 PM, csoon1992 said:

Hi!

How can I migrate customers from PS 1.6 to PS 1.7.2 keeping customer passwords?

Thank u!

I could help with that. 

Upgrade to the latest PS 1.6.* ( I did it from 1.6.1.16 though, I think the latest is 1.6.1.18, shouldn;'t matter).
You need to install prestools as per the documentation, then you add the copyshop_data files as well. 

When you download the copyshop files you also get a PDF. You do exactly what it says on the first page. But here is where it is lacking a bit of instructions: 
After everything ran smoothly (hopefully) you need to go to the settings.inc.php file from the old installation  and match the following information with the information in app/config/parameters.php in 1.7

define('_COOKIE_KEY_','');  goes to - >  'cookie_key'
define('_COOKIE_IV_', ''); goes to -> 'cookie_key_iv'
define('_RIJNDAEL_KEY_', ''); goes to  -> 'secret'  

You don't need the '_RIJNDAEL_IV_' key anymore it seems. 

After doing this migration of settings, everything worked both in the backoffice and the front office. All my users worked. 

Hope this helps. 

[musicmaster]

8 minutes ago, DragoshDX said:

Hello, and thank you for your work. 

I've managed to migrate the whole of my PS 1.6.1.16 to a new install of 1.7.3.1
It ran FLAWLESSLY. 

It ported over user accounts, user information, user orders, actual products that I can visit in the website (via direct category URL access). It came in with shipping methods and payment methods.
It came in with products, categories, category structure, pricing, tax info, stock levels, you name it. 

I can place an older with my old user from the old website.

However, I have a single problem still. Even though I can create an order and have it seen in the back office, when I go hit products in the left menu, to see my products in the backoffice, the list is empty. But like I said, the actual front office DOES see them, I can even use the ajax search, it's unbelievable how well this worked :)). 

 

If I go through an order in the backoffice and click on any product that has been order, I do get the product form working, it's just that I don't see products in the actual Products section (w perfect stock, and languages!)

Is there anything I can do so my products show up in the backoffice? Has this happened in your experience? I'm ready to start doing tests on the production server as well, but my team won't be able to do much if they don't see the products in the back office. 

Here is the output of the copyshop script on the product section: 

149 product 118 structure different. Extra in old: id_product_redirected, Extra in new: isbn, low_stock_threshold, low_stock_alert, additional_delivery_times, id_type_redirected, show_condition, state, 
150 product_attachment 0
151 product_attribute 133 structure different. Extra in new: isbn, low_stock_threshold, low_stock_alert, 
152 product_attribute_combination 168
153 product_attribute_image 85
154 product_attribute_shop 133 structure different. Extra in new: low_stock_threshold, low_stock_alert, 
155 product_carrier 0

I'll debug this further tomorrow, but I'm thinking that perhaps you might have encountered this before?

 

Hi,

Nice to hear that it works so well for you.

 

I am not quite sure what you mean with " when I go hit products in the left menu, to see my products in the backoffice, the list is empty. ". Which left column? The first thing what I could think of was language id's. 

[DragoshDX]

8 minutes ago, musicmaster said:

 

Hi,

Nice to hear that it works so well for you.

 

I am not quite sure what you mean with " when I go hit products in the left menu, to see my products in the backoffice, the list is empty. ". Which left column? The first thing what I could think of was language id's. 

 

Well, when I click on Products in the back office the list is empty. 

Just says add your first product. But I know they're there, because I can access them in the front office.

I'm checking language ids now. 

[DragoshDX]

The languages match. 

What is defined in the ps_lang table matches the ps_product_lang table. Furthermore, if I go to a product in the front office, I can switch with the language switcher and the description name and everything changes correctly. I don't think it;s languages. 

I'm only going to be able to fire up the debugger tomorrow and see if I get anywhere. But if you have any other idea it'd be greatly appreciated.

Some more stuff I did was clear cache, stop the caching altogether, but it looks like it's not that. 

Thank you!

 

EDIT: prestools doesn't see the transferred products either. If I create a random product both the Product listing in the backoffice and prestools see it, however, still no trace of the rest of the products. 

I'm thinking that it actually might be easier to debug product-edit.php from prestools and see where my mismatch is, fix it in the DB and see where it takes me. 

LATER EDIT: 
It looks like when I manually created a product in the back office, it actually deleted everything else in the product tables. However I ran copyshop again and now prestools does see the products. 
I'm now thinking that the products might need to belong to the home default category? Which I don't actually have in the old catalog and was never transferred. I need to crosscheck with another install of 1.7. 

Edited April 19, 2018 by DragoshDX (see edit history)

[Minou90]

On 19/4/2018 at 10:46 PM, DragoshDX said:

The languages match. 

What is defined in the ps_lang table matches the ps_product_lang table. Furthermore, if I go to a product in the front office, I can switch with the language switcher and the description name and everything changes correctly. I don't think it;s languages. 

I'm only going to be able to fire up the debugger tomorrow and see if I get anywhere. But if you have any other idea it'd be greatly appreciated.

Some more stuff I did was clear cache, stop the caching altogether, but it looks like it's not that. 

Thank you!

 

EDIT: prestools doesn't see the transferred products either. If I create a random product both the Product listing in the backoffice and prestools see it, however, still no trace of the rest of the products. 

I'm thinking that it actually might be easier to debug product-edit.php from prestools and see where my mismatch is, fix it in the DB and see where it takes me. 

LATER EDIT: 
It looks like when I manually created a product in the back office, it actually deleted everything else in the product tables. However I ran copyshop again and now prestools does see the products. 
I'm now thinking that the products might need to belong to the home default category? Which I don't actually have in the old catalog and was never transferred. I need to crosscheck with another install of 1.7. 

Just have the same problem, did you find a solution?

 

[musicmaster]

23 hours ago, Minou90 said:

Just have the same problem, did you find a solution?

 

I made some test installation upgrade for myself.

Conclusions:

 - I can't find out how the security keys should be copied. But somehow copying the cookie parameters seems enough to get in. When I look in the error log I see lots of "client denied by server configuration"

 - processing takes forever.

 - I can see categories in the backoffice. I can also see the products in Prestools and in the front office. They are not shown in the bacoffice - most likely because the "state" field in ps_product is zero and it should be 1.

 

Edited May 3, 2018 by musicmaster (see edit history)

[Minou90]

2 minutes ago, musicmaster said:

I made some test installation upgrade for myself.

Conclusions:

 - I can't find out how the security keys should be copied. But somehow copying the cookie parameters seems enough to get in.

 - processing takes forever.

 - I can see categories in the backoffice. I can also see the products in Prestools. They are not shown in the bacoffice - most likely because the "state" field in ps_product is zero and it should be 1.

 

yes, I change state to 1

and I find product, but I have to do update for ps_image_shop and an update for ps_product_attribute_shop

Now seems to work, but still problem with version 1.7.3, work well with 1.7.2

 

[musicmaster]

6 minutes ago, Minou90 said:

yes, I change state to 1

and I find product, but I have to do update for ps_image_shop and an update for ps_product_attribute_shop

Now seems to work, but still problem with version 1.7.3, work well with 1.7.2

 

What do you mean with "I have to do update for ps_image_shop and an update for ps_product_attribute_shop"?

What is your problem with 1.7.3? Did you read the recommendation not to use Copy Shopdata for an upgrade beyond 1.7.2? You must go from 1.7.2 to 1.7.3 with the Prestashop upgrade as otherwise you miss the change in how features are handled. 

[Azymo]

Hi !

I also confirm, that script from @musicmaster works as a great tool to transfer all business data from PS 1.6 to PS 1.7.3.2 !!!!

My additional tips / things to do:

- update PS 1.6.x to the newest ver. by 1-click updater (remeber to disable all non-native modules) and overrides.

- mysql configuration:

     disable NO_ZERO_DATE and NO_ZERO_IN_DATE

     set secure-file-priv = "[your_base_path]/prestoolssuite/"

- add php command to     ob_flush(); and   flush();   to copy_shopdata.php file, to get immadiely information if some process stucks.

- after all , run mysql command on new shop: UPDATE `ps_product` SET `state`= 1

- update cookie keys from old to new shop - just like @DragoshDX suggest.

 

Thank you guys for such great tools and help !

Edited May 9, 2018 by Azymo (see edit history)

[DragoshDX]

Hello. 

I did solve my issue eventually as well, however it was not necessarily easy in the end. 
The key was that prestools provides a way to export product data from a prestashop installation but does not allow you to export combinations.
What I did was export products with images from the OLD installation using a prestools installed in there. Then I imported them in the new 1.7 installation (my problem was that products were not showing up in the catalog in the backoffice). To do so I had to do A LOT of CSV magic with Libre Office so that the csv exported from prestools matched the prestashop format. After that I had to do a monster SQL query so I could get the combinations for that import, which I then imported in my new Prestashop. They matched IDs for the newly created products (make sure you tell the importer to also import the specified product IDs instead of doing an autoincrement). After this, all I had to do was upload the new product images to the new installation and my products appeared in the backoffice as well. 

Then I exported my database from the local installation, imported it on my host, copied all the files to my host (ps_shop_url is the table where you configure your shop URL to change it so the site can be moved), also obviously the mysql user name and password for the host. So I had EVERYTHING from my old site in the new one. Now I only need to configure stuff and I"m ready for GDPR.

[Muluku]

Hi, can this script work for presta 1.7.5 ? i mean transfer data from 1.7.5 to 1.7.5 ? 

I have a shop under 1.6.1.23

i want to upgrade it to 17.5

then copy data from that 1.7.5 to a new installation under 1.7.5.

is this possible ? what do you recommend me ?

[Azymo]

hi, @Muluku

I use this script to transfer business data from 1.6.x to 1.7.x 

But I wonder how you are gong to upgrade ps 1.6.1.23 to 1.7.x ?

[musicmaster]

@Muluku: Yes, that should work. There are no new tables in 1.7.5. So for the script nothing has changed.

@Azymo It took a long time, but nowadays Prestashop's updater can update from 1.6 to 1.7. For a long time Copy shopdata was the only way to move data from 1.6 to 1.7 and that worked. But using Prestashop's own functions is always better as it sets new fields to Prestashop's defaults.

 

 

[Muluku]

Thanks for your answers ! but how about the stability of 1.7.5 ? i saw some people advicing to stay with 1.6.x !

and can presta 1.7.4 theme work with presta 1.7.5 ?  thanks 

Edited December 25, 2018 by Muluku (see edit history)

[musicmaster]

9 hours ago, Muluku said:

 i saw some people advicing to stay with 1.6.x !

and can presta 1.7.4 theme work with presta 1.7.5 ?  thanks 

 - in early 1.7 version stability was a big problem. And that lasted quite some time. But by now that has been fixed. What remains is that it is a quite different development environment (with a.o. Symfony, Twig and a different template and file structure). Some people like that. I don't. Neither does the majority of the Prestashop users: there are still much more 1.6 installations active than 1.7. But 1.7 has its fans too: in the end it is a personal choice.

 - I would expect 1.7.4 themes to work with 1.7.5. But there may always be subtle differences: so check with the makes of the theme.

[Azymo]

12 hours ago, musicmaster said:

... nowadays Prestashop's updater can update from 1.6 to 1.7.

I have to admit that I didn't know this.

So, if it works fine, then there is no point to move data to clean installations of 1.7.5 after upgrade from 1.6.1.23 (using PS upgrade module), right ?

 

[musicmaster]

6 hours ago, Azymo said:

I have to admit that I didn't know this.

So, if it works fine, then there is no point to move data to clean installations of 1.7.5 after upgrade from 1.6.1.23 (using PS upgrade module), right ?

Normally not. But if your shop is unstable or if you find that it has become unstable after the upgrade then it is a good way to get it stable again.

[Muluku]

His sir, is there a way to use passwords from 1.6 with 1.7  without changing it ? 

in copy shopdata, it said to copy cookie key, Ryndael keys... to the new one, but in 1.7 parameters file, there is not Ryndael keys,! 

 

[SMA17]

On 6/3/2015 at 10:24 AM, musicmaster said:

The attached script copies the content (business data) of a shop from one installation of Prestashop to another. The products, orders, customers, etc of the new (usually freshly installed) shop will be completely replaced by those from the old shop while its configuration is maintained. Your customers can even keep their old passwords.

Prestashop updates regularly result in unstable situations. This instability is nearly always in the files or the configuration settings.In either case this script can be a solution as it leaves both behind. It is far superior to a csv export as that copies only a part of your data.

Copy_shopdata should be installed in the same directory as Prestools as it uses Prestools for authorization. It should be installed in the new shop, after which you fill in the database data of the old shop in copy_shopsdata's config file.

Copy_shopdata is meant to repair installations. You should leave the upgrade process to Prestashop.

Documentation is in the pdf file.

PS. Nearly any shop can be repaired and/or updated this way. If you are afraid to do it yourself you can hire me to do it for you.

PS-2. Some people have tried to use this tool for updating from 1.6.1 to 1.7. This is to some extent possible as the database structure is very similar. However, nowadays that shouldn't be done anymore as Prestashop's update software now can update from 1.6 to 1.7

copy_shopdata-v0.29e.zip

 

Hello

how do you do this from 1.6 to 1.7 i have try it but not working

[musicmaster]

6 hours ago, SMA17 said:

Hello

how do you do this from 1.6 to 1.7 i have try it but not working

The manual clearly says that you should only use this between shops of the same version. Upgrading should be done with Prestashops upgrade module.

You can use Copy Shopdata when you find upgrading with the official route impossible. But in that case you still should use Copy shopdata between shops of the same version. So if you want to upgrade a PS 1.6.0.5 shop to 1.7 you should install a fresh PS shop of the same version, move the data with Copy Shopdata towards it and then upgrade that shop with Prestashop's module.

[SMA17]

But i dont know how you instal the addon for me not working

[musicmaster]

45 minutes ago, SMA17 said:

But i dont know how you instal the addon for me not working

Inside the zip file there is a manual. Did you read it?

[SMA17]

Yes but i dont understand how i do it

[musicmaster]

On 5/15/2019 at 10:41 AM, SMA17 said:

Yes but i dont understand how i do it

Your answers don't give any sign that you understood even a word of the manual.

Given such a lack of knowledge I can only advise you to hire someone. Using this tool requires considerable background knowledge.

Edited July 18, 2019 by musicmaster (see edit history)

[a-d-g]

Hello, I have an error trying to copy from old shop, both website are in the same server, but i read this messaje when i try to make transfer with COPY_SHOPDATA.PHP, an error 13 when system try to write file on server.

Quote

1 accessory 0DB960674-TRUNCATE prstshp_accessory
DB960675-SELECT COUNT(*) AS mycount FROM prstshp_address

2 address 19DB960675-SHOW COLUMNS FROM prstshp_address
DB960674-SHOW COLUMNS FROM prstshp_address
DB960674-TRUNCATE TABLE prstshp_address
DB960675-SELECT `id_address`,`id_country`,`id_state`,`id_customer`,`id_manufacturer`,`id_supplier`,`id_warehouse`,`alias`,`company`,`lastname`,`firstname`,`address1`,`address2`,`postcode`,`city`,`other`,`phone`,`phone_mobile`,`vat_number`,`dni`,`date_add`,`date_upd`,`active`,`deleted` INTO OUTFILE '/var/www/vhosts/website.es/httpdocs/ps/Backoffice/ptools/copy_shopdata_address.dtx' FROM `prstshp_address`

MySQL error 1: Can't create/write to file '/var/www/vhosts/website.es/httpdocs/ps/Backoffice/ptools/copy_shopdata_address.dtx' (Errcode: 13)
Generated by URL '/ps/Backoffice/ptools/copy_shopdata.php'
with Query 'SELECT `id_address`,`id_country`,`id_state`,`id_customer`,`id_manufacturer`,`id_supplier`,`id_warehouse`,`alias`,`company`,`lastname`,`firstname`,`address1`,`address2`,`postcode`,`city`,`other`,`phone`,`phone_mobile`,`vat_number`,`dni`,`date_add`,`date_upd`,`active`,`deleted` INTO OUTFILE '/var/www/vhosts/website.es/httpdocs/ps/Backoffice/ptools/copy_shopdata_address.dtx' FROM `prstshp_address`'

Anybody knows how to solve it? i tried changing permissions, but issue hasnt resolved. I use CENTOS.

[musicmaster]

The browser user is a different user on a *nix system and for obvious reasons it has very few rights. 

My solution usually is to copy a site to localhost and to process it there.

[ChillMan]

Thank you for this great script

I just used it this way:

- Made a copy of the source PS 1.6.1.24 shop
- Then installed a fresh PS 1.6.1.24 on a subdomain and used Copy Shopdata V0.3 to transfer the data.
-  Then used the native PS 1-Click Upgrade to update the fresh store to PS 1.7.6.1
- So now I have a PS1.7 store / database with all my previous PS.16 business data.
- The upgraded PS1.7 store does have various issue, so I also installed a new clean 1.7.1.6 store. The idea was then to use the script to copy the updated 1.7 database to the fresh PS1.7 store (not done this step yet)

However I experience these problems / have these questions:
1 . I have 4 languages, so for the script to work I created these in the fresh 1.6 store, but ID's were not identical, so had to change /match the ID's (1,6,8,9) in the ps_lang / ps_lang_shop for the script to start. Everything went well and also the 1-click Upgrade afterwards. However I now notice that a few tables in target do not have the correct language ID - mainly ps_meta_lang (all the pages) are 1,2,3,4 (not 1,6,8,9 like source), meaning only the english work. Most other 'Lang' tables were correctly matched, so must be some bug, since a few tables not matched.

2. Since the 1.7 database have these language issue (which in worst case I can manually edit), I kind of fear other issues which would then also be transferred to the fresh 1.7
Is it not possible to transfer some selected Business data only - like orders, cms, reviews, customers related tables only?

3. Does the script actually force overwrite all existing tables, or skip tables/ID with content (if target shop for example already have various data)?

4. If I manage to the get the fresh 1.7 shop ready after while, then how should I get all the changes from the meantime (mainly orders and customers)?
I seem like quite a heavy process if having to do all the initial steps all over again for this (the "source" was 1-click upgraded, so can't just repeat the script now)

Thank you in advance!

 

Edited November 6, 2019 by ChillMan (see edit history)

[musicmaster]

Hi Chillman,

Glad to hear you like the script.

 - you should NOT have adapted the language ids. The script handles adapting the language ids for all the tables that it copies. Once you start changing language ids you also need to change that in all the system tables (those not copied) and then it is easy to overlook something - as it looks you did. The only thing you should take care of is that the same active languages are present in both shops. But it doesn't matter what ids they have.

 - the script overwrites the targeted tables completely. Technically it empties them (so it keeps the table structure of the new shop) and then fills them with data from the old shop. Merging data would be considerably more complicated as you would need to give products new ids and you would need to implement the new ids in all relevant tables - so that is not done.

 - I am not sure how much your language adventures complicate the picture. But installing a fresh shop and giving it the right languages will take maybe half an hour - with a work load of maybe ten minutes. The script runs rather fast and updating a fresh shop shouldn't give complications and be rather fast too. The whole process of getting such a new server ready usually takes me one to two hours (and that includes complications unrelated to the script). And if there were orders in the meantime (I like to do this at quiet hours) I increase the customer and order ids in the new shop so that I can add them manually later on.

Cheers,

M

 

[ChillMan]

11 minutes ago, musicmaster said:

Hi Chillman,

Glad to hear you like the script.

 - you should NOT have adapted the language ids. The script handles adapting the language ids for all the tables that it copies. Once you start changing language ids you also need to change that in all the system tables (those not copied) and then it is easy to overlook something - as it looks you did. The only thing you should take care of is that the same active languages are present in both shops. But it doesn't matter what ids they have.

 - the script overwrites the targeted tables completely. Technically it empties them (so it keeps the table structure of the new shop) and then fills them with data from the old shop. Merging data would be considerably more complicated as you would need to give products new ids and you would need to implement the new ids in all relevant tables - so that is not done.

 - I am not sure how much your language adventures complicate the picture. But installing a fresh shop and giving it the right languages will take maybe half an hour - with a work load of maybe ten minutes. The script runs rather fast and updating a fresh shop shouldn't give complications and be rather fast too. The whole process of getting such a new server ready usually takes me one to two hours (and that includes complications unrelated to the script). And if there were orders in the meantime (I like to do this at quiet hours) I increase the customer and order ids in the new shop so that I can add them manually later on.

Cheers,

M

 

1. Doh! Maybe you could add a note in the manual about the active languages in both stores do not need to be same ID's. This part confused me, since I could not get the script to start (I thought this was related to the language IDs, but was maybe due to different english ISO code in the 2 shops (en vs. en-us).

2. Not possible to copy selected data only? (I already have customers, products and categories transferred which are slightly modified, so would be nice to not have these overwritten)

3. By adding meantime changes (new orders, customers etc.) you manually copy/paste these changes into the various tables in the database?
Especially orders alone affect quite a lot of tables, so this seem a bit scary.

[musicmaster]

10 minutes ago, ChillMan said:

1. Doh! Maybe you could add a note in the manual about the active languages in both stores do not need to be same ID's. This part confused me, since I could not get the script to start (I thought this was related to the language IDs, but was maybe due to different english ISO code in the 2 shops (en vs. en-us).

2. Not possible to copy selected data only? (I already have customers, products and categories transferred which are slightly modified, so would be nice to not have these overwritten)

3. By adding meantime changes (new orders, customers etc.) you manually copy/paste these changes into the various tables in the database?
Especially orders alone affect quite a lot of tables, so this seem a bit scary.

ad 1) en vs en-us is indeed an infamous example. I will look how I can make things yet more clear.

ad 2) My strategy is to implement every business data change in the new database also in the old one. That way they will be there with the new copy. It is a bit more work but much less than trying later to remember what changes you made later on.

Remember that in the old database things are changing too - like customers changing address or password. So merging is complicated.

In addition you are likely to have created new products,orders,etc in the new shop for test purposes. Their id will overlap with real products/customers/orders in the old shop.

Two tactics that you could implement are 1) exporting and importing a csv file with the relevant field(s) and 2) exporting a couple of new databases in Phpmyadmin, running copy_shopdata, then deleting the relevant tables until a certain id and importing the exported tables that go to that number.

ad 3) It works better to use the backoffice. Just make the orders there and watch whether they belong to an existing or a new customer. You can fill the details in later. With order-edit in Prestools orders are easy to change.

[ChillMan]

2 hours ago, musicmaster said:

Hi Chillman,

Glad to hear you like the script.

 - you should NOT have adapted the language ids. The script handles adapting the language ids for all the tables that it copies. Once you start changing language ids you also need to change that in all the system tables (those not copied) and then it is easy to overlook something - as it looks you did. The only thing you should take care of is that the same active languages are present in both shops. But it doesn't matter what ids they have.

 - the script overwrites the targeted tables completely. Technically it empties them (so it keeps the table structure of the new shop) and then fills them with data from the old shop. Merging data would be considerably more complicated as you would need to give products new ids and you would need to implement the new ids in all relevant tables - so that is not done.

 - I am not sure how much your language adventures complicate the picture. But installing a fresh shop and giving it the right languages will take maybe half an hour - with a work load of maybe ten minutes. The script runs rather fast and updating a fresh shop shouldn't give complications and be rather fast too. The whole process of getting such a new server ready usually takes me one to two hours (and that includes complications unrelated to the script). And if there were orders in the meantime (I like to do this at quiet hours) I increase the customer and order ids in the new shop so that I can add them manually later on.

Cheers,

M

 

Hi again

I started from scratch again an this time I did not edit the language ID's (just made the 4 active languages in the fresh install - ID 1,2,3,4). The result is unfortunately the same for some "lang" tables - only the English text is copied into all the 4 languages. Some lang tables are correct though, so not all. Especially ps_meta_lang (pages) is quite messed up and completely different from the source. 

In the Source the 4 active language IDs are 1,6,8,9. However in the database I also notice a ID 3 in some lang tables (Spanish, which was deleted years ago, but apparently still present in some tables). So maybe this is messing up the script since also ID 3.

The script say:
4 old languages: da-DK,en,no-no,sv (6,1,9,8); 4 new languages: da-DK,en,no-no,sv (1,2,4,3); Language check ok.
Doing the following language transformations: 1=>20 6=>1 1=>2 9=>4 8=>3

Edited November 6, 2019 by ChillMan (see edit history)

[musicmaster]

Meta_lang is a system table - not a business data table. So it should not be copied. You won't find it in the original list of tables in the config table.

It is logical that it contains English for all languages when you started with English and then created other languages. It needs to create entries for texts but it has no source. You would need to import a country or language pack to change that.

The presence of deleted languages is rather common. That shouldn't be a problem.

[ChillMan]

1 hour ago, musicmaster said:

Meta_lang is a system table - not a business data table. So it should not be copied. You won't find it in the original list of tables in the config table.

It is logical that it contains English for all languages when you started with English and then created other languages. It needs to create entries for texts but it has no source. You would need to import a country or language pack to change that.

The presence of deleted languages is rather common. That shouldn't be a problem.

Meta / Meta_lang contains all the pages translation/SEO, so if not included with the copy, one would have to translate all these manually again, as they are not part of a language pack.
cms_lang and a few others lang are also copied, which is of course good/needed. So I suggest to include Meta_lang as well.

[musicmaster]

23 minutes ago, ChillMan said:

Meta / Meta_lang contains all the pages translation/SEO, so if not included with the copy, one would have to translate all these manually again, as they are not part of a language pack.
cms_lang and a few others lang are also copied, which is of course good/needed. So I suggest to include Meta_lang as well.

The problem is that that makes assumptions about which pages are present in the new shop. the id_meta numbers in the old shop are not necessarily the same as those in the new shop.

Also I do see different texts for different languages thanks to localization packs.

If you really want to copy this file you are free to add meta_lang to the copied tables in the config file. I advice against it. Check at least that the numbers are the same and if not create your own script.

 

 

[musicmaster]

16 hours ago, ChillMan said:

Meta / Meta_lang contains all the pages translation/SEO, so if not included with the copy, one would have to translate all these manually again, as they are not part of a language pack.
cms_lang and a few others lang are also copied, which is of course good/needed. So I suggest to include Meta_lang as well.

One additionele thought. The reading why the package didn't provider the meta translations was that they would neef tot overwrite existing data. I didn't creatie the languages before i imported the package. IT was the package import that creatief Them.

[bcarron]

Hi musicmaster, this script looks great! Does it work with advanced stock management?

Unfortunately, I can't use the 1-click upgrade (or the manual process) to upgrade my shop since it always fail when updating the database. Therefore, I have to somehow copy the data between shops on 1.6.x before upgrading a clean shop to 1.7.x and copy the data again to a clean 1.7.x shop... Your script seems perfect for that but I am worried that since I use advanced stock management in my 1.6.x shop I will have problems in 1.7.x even by only copying business tables.

[louishand123]

2 hours ago, bcarron said:

Hi musicmaster, this script looks great! Does it work with advanced stock management?

Unfortunately, I can't use the 1-click upgrade (or the manual process) to upgrade my shop since it always fail when updating the database. Therefore, I have to somehow copy the data between shops on 1.6.x before upgrading a clean shop to 1.7.x and copy the data again to a clean 1.7.x shop... Your script seems perfect for that but I am worried that since I use advanced stock management in my 1.6.x shop I will have problems in 1.7.x even by only copying business tables.

I understood what you need. So why you use an automated shopping cart migration tool? There are many popular migration services such as Litextension, Cart2Cart,.. for you to choose

[musicmaster]

It should work with ASM too. But if I remember well 1.7 has reduced ASM functionality. Did you check that out?

[bcarron]

On 11/9/2019 at 4:24 AM, louishand123 said:

I understood what you need. So why you use an automated shopping cart migration tool? There are many popular migration services such as Litextension, Cart2Cart,.. for you to choose

I have quite a lot of products so that kind of service is expensive. Furthermore, I don’t mind doing it myself if possible, I have a background in computer science (unfortunately I don’t know the inner workings of PS that much though).

 

On 11/9/2019 at 9:13 AM, musicmaster said:

It should work with ASM too. But if I remember well 1.7 has reduced ASM functionality. Did you check that out?

I don’t mind using the new “Stocks” interface from 1.7. However, I tried the procedure I outlined and while the shop is now working in 1.7, I have a lot of issues with product stocks and specially product combinations. The existing combinations are not displayed in the product page and I can’t even create combinations with new products. I wonder if this is due to the way product combination stocks were stored in ASM and are now conflicting with the new way stocks are managed.

[musicmaster]

43 minutes ago, bcarron said:

I don’t mind using the new “Stocks” interface from 1.7. However, I tried the procedure I outlined and while the shop is now working in 1.7, I have a lot of issues with product stocks and specially product combinations. The existing combinations are not displayed in the product page and I can’t even create combinations with new products. I wonder if this is due to the way product combination stocks were stored in ASM and are now conflicting with the new way stocks are managed.

The only explanation I can think of is that 1.7 uses some new ASM tables that I have overlooked and so are not included in the list of copied tables. I will look into this. If you have any suggestion please let me know.

[musicmaster]

17 hours ago, bcarron said:

I don’t mind using the new “Stocks” interface from 1.7. However, I tried the procedure I outlined and while the shop is now working in 1.7, I have a lot of issues with product stocks and specially product combinations. The existing combinations are not displayed in the product page and I can’t even create combinations with new products. I wonder if this is due to the way product combination stocks were stored in ASM and are now conflicting with the new way stocks are managed.

Some additional thoughts:

 - the script copies only when the database table is present in both shops. It may be that some database tables are only created when you enable ASM. So you might need to do that in the target shops.

 - if you send me a list of tables in your new 1.7.6 I can can check whether one is missing in the list of copied tables.

 - when you check in the intermediate shops whether ASM works it might help to pinpoint the problem.

[bcarron]

On 11/11/2019 at 11:01 AM, musicmaster said:

Some additional thoughts:

 - when you check in the intermediate shops whether ASM works it might help to pinpoint the problem.

That's a great idea, I have done the process again and the new shop running 1.6.1.1 with copied data from the old shop worked fine. Then I upgraded that shop to 1.6.1.24 in order to use php 7.1 for the upgrade to PS 1.7. I thought it might be the reason why the upgrade to 1.7 failed, and it was the case, this time the upgrade was able to finish successfully with warnings detected during upgrade.

On that shop, I can still see the combinations on the product page but the stocks page is not working, the page is displayed but the content loads forever until I get an Internal Server Error popup. I checked the tables for ps_stock, ps_stock_available and ps_stock_mvt, the tables are similar between a fresh 1.7 shop and the updated one. The only differences I could find are that 1.7 does not use ps_stock anymore and "depends_on_stock" in ps_stock_available is set to 0 by default instead of 1 (both of which are normal since there is no ASM in 1.7). I tried removing all entries from stock and setting all stocks in ps_stock_available to 0 but I still can't load the page.

Since I get no error message it is hard to understand why the query hangs just on the Stocks page but I have noticed that this page is using the BO API instead of querying the DB directly. I know that this is not really related to your script but do you know of any other tables that are involved in the stock management? Once I copy that shop to a fresh 1.7.6.1 shop, I can't even see the combinations on the product's pages.

Edited November 14, 2019 by bcarron (see edit history)

[musicmaster]

"this time the upgrade was able to finish successfully with warnings detected during upgrade."

What were the warnings? My rule for debugging software is to always start with the first error/warning/notice that I don't understand. The rest could all have been caused by this first one. So first solve those warnings. Then get 1.6.1.24 working. And only then go to 1.7.

"On that shop, I can still see the combinations on the product page but the stocks page is not working, the page is displayed but the content loads forever until I get an Internal Server Error popup."

What "stocks page"? Are you talking about the quantities page in the backoffice? Did you have a look at the settings for stock in the backoffice? You can set them yourself and prevent in the settings file of copy_shopdata that they are copied from the old shop.

With an "Internal Server Error" you should enable debug mode and/or look in the error log.

One tip: when debugging a shop always have a fresh copy of Prestashop of the same version nearby. That way you have a reference point of how things should look. And you can create there products with the same combinations/stock/etc as one that doesn't work.

Finally: one thing at a time. If stock and combinations both give problems then don't start with a product that has both.

[bcarron]

 

1 hour ago, musicmaster said:

"this time the upgrade was able to finish successfully with warnings detected during upgrade."

What were the warnings? My rule for debugging software is to always start with the first error/warning/notice that I don't understand. The rest could all have been caused by this first one. So first solve those warnings. Then get 1.6.1.24 working. And only then go to 1.7.

You're right, I kind of glossed over it since it was only warnings and I couldn't find them. I just got the message a the end of the upgrade:

Database upgrade OK
Warning detected during upgrade.

The full log is 22762 lines long (I've attached it to this post) and by searching it more thoroughly I could not find any warnings except these:

[INTERNAL] /var/www/tms17/vendor/ircmaxell/random-lib/lib/RandomLib/AbstractMcryptMixer.php line 75 - Function mcrypt_module_open() is deprecated
[INTERNAL] /var/www/tms17/vendor/ircmaxell/random-lib/lib/RandomLib/AbstractMcryptMixer.php line 76 - Function mcrypt_enc_get_block_size() is deprecated
[INTERNAL] /var/www/tms17/vendor/ircmaxell/random-lib/lib/RandomLib/AbstractMcryptMixer.php line 77 - Function mcrypt_enc_get_iv_size() is deprecated
[INTERNAL] /var/www/tms17/vendor/ircmaxell/random-lib/lib/RandomLib/AbstractMcryptMixer.php line 86 - Function mcrypt_module_close() is deprecated

 

1 hour ago, musicmaster said:

"On that shop, I can still see the combinations on the product page but the stocks page is not working, the page is displayed but the content loads forever until I get an Internal Server Error popup."

What "stocks page"? Are you talking about the quantities page in the backoffice? Did you have a look at the settings for stock in the backoffice? You can set them yourself and prevent in the settings file of copy_shopdata that they are copied from the old shop.

Yes the page where you can edit quantities and see movements (other tab) in the back office. The settings for stock in the product settings page of backoffice? I can't really see any relevant settings except enabling stock management (which is indeed enabled).

 

2 hours ago, musicmaster said:

With an "Internal Server Error" you should enable debug mode and/or look in the error log.

One tip: when debugging a shop always have a fresh copy of Prestashop of the same version nearby. That way you have a reference point of how things should look. And you can create there products with the same combinations/stock/etc as one that doesn't work.

Finally: one thing at a time. If stock and combinations both give problems then don't start with a product that has both.

Debug mode is enabled but the "Stocks" page is using the API to populate a Vue view so the page itself is loading but the content does not load, a popup appear with "Internal Sever Error". Since the page itself loads, I don't get any debug information. Can't find anything useful in the php/apache logs either. From the Symfony toolbar, I can see that the AJAX request that provoque the error 500 is this one: admin/index.php/api/stocks/?_token=XXX&order=product&page_size=30&page_index=1

The log from Apache :

[14/Nov/2019:11:50:17 +0100] "GET /admin/index.php/api/stocks/?_token=XXX&order=product&page_size=30&page_index=1 HTTP/1.1" 500 117784 "XXX/admin/index.php/sell/stocks/?_token=XXX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

I have a fresh 1.7 shop where I tried to play around with new products and changing quantities. That's where I noticed that 1.7 does not use ps_stock anymore and "depends_on_stock" in ps_stock_available is set to 0 by default instead of 1. I also noticed that ps_stock_mvt now uses ps_stock_available id for its id_stock despite still being called id_stock. I don't think that this is my problem though, since I emptied the ps_stock_mvt table to test if it had an influence.

PS17 upgrade log copy.txt

[musicmaster]

You leave me completely puzzled. As far as I know you only get a vendor directory when you upgrade to 1.7. Yet you claim that it happens when you upgrade to 1.6.1.24.

As for "/admin/index.php/api/stocks/?_token=XXX&order=product&page_size=30&page_index=1". You can run such a query - preceded by your domain name - in your browser and see what happens.

 

[bcarron]

23 hours ago, musicmaster said:

You leave me completely puzzled. As far as I know you only get a vendor directory when you upgrade to 1.7. Yet you claim that it happens when you upgrade to 1.6.1.24.

I probably was not very clear but I am talking about the upgraded 1.7 shop, the upgrade to 1.6.1.24 from 1.6.1.1 worked perfectly and the products / stocks / combinations were all working nicely with it.

23 hours ago, musicmaster said:

As for "/admin/index.php/api/stocks/?_token=XXX&order=product&page_size=30&page_index=1". You can run such a query - preceded by your domain name - in your browser and see what happens.

I didn't think of that, thank you so much for your help! The page produced a "Lock wait timeout exceeded" error. I increased the timeout and now the page eventually loads, but it takes forever (over 8 minutes)... Strangely the "Stocks" page works fine with the fresh 1.7 shop where data was copied with your script. I can increase quantities and everything, but I can't see combinations on the product pages and I can't make a new product with combinations. This is all very confusing...

Edited November 15, 2019 by bcarron (see edit history)

[musicmaster]

A long waiting time can mean that the indexing doesn't work correctly. As you mentioned that you have a rather big shop that could be a problem. So I suggest comparing the indexes of product/combination related tables with those of a fresh Prestashop installation of the same version.

You can also switch on profiling (in defines.inc.php) to see which queries take so long. That could provide some idea which tables are a problem.

[bcarron]

On 11/15/2019 at 1:17 PM, musicmaster said:

A long waiting time can mean that the indexing doesn't work correctly. As you mentioned that you have a rather big shop that could be a problem. So I suggest comparing the indexes of product/combination related tables with those of a fresh Prestashop installation of the same version.

You can also switch on profiling (in defines.inc.php) to see which queries take so long. That could provide some idea which tables are a problem.

Which index? You mean ID  product/combination IDs?

Unfortunately, I can't use profiling for that since it's an API call and not a standard page. However I found out that if I empty the ps_order_detail table, the long wait time is reduced to about 15 seconds but I couldn't find any difference between the table from the upgraded shop and the one from a  fresh 1.7 shop.

[musicmaster]

No I mean this: