CSV import questions / duplicate records issues

[Aardvark1971]

I've version 1.5.6 installed and have been trying all kinds of csv import tests with mediocre success.

 

I've got over 1500 products. Early tests showed that anything over 500 records imported would time out.

 

If at any time, there is a timeout, I have to start all over by first deleting all categories and products to avoid duplicate records - the deletion process is itself a multi-step process or THIS will time out too (takes 5 - 6 attempts to get rid of all 1500 products and categories). nginx is set at 300, and fcgid idle timeout at 500. Still not good enough. Not sure my ISP will allow us to go any higher.

 

Using smaller import files of only 400 records at a time "works". This requires 4 imports of 4 seperate files to load the database of 1500 products. Clunky as h*ll, and MUST work perfectly, or I have to start all over (delete everything, reload everything to avoid duplicates).

 

Here's some problems I've found, which I need some help / guidance with:

 

a) if the SAME import file is imported again, it DUPLICATES all the records (versus just updating existing fields such as price or description). 

 

This duplication of records will create a management nightmare for me - our product database is built offline in our ordering system and we change prices constantly.  We need to upload (import) the prices to be updated (versus being duplicated, or updating them all online by hand, one by one) at ANY time. This can happen daily.

 

Does csv import permit "update" or "replace" of field values? If so, none of my test so far have discovered this. If this is possible, how is it done?

 

the csv import has an option to "delete all products" check box. As I understand it, this means ALL products - not just one the ones that match the import file currently being used (which is what is really needed).

 

But what if I don't want to delete all products - just some products, and reimport just those?  Is this possible?

 

c) to get a "fully populated database" under the restrictions I've found so far, I have to have 4 successful imports of 400 or so products (including images), and the database had better be completely empty before starting, otherwise duplicate records will occur, and this must all happen with zero timeouts or I have to begin again.

 

My csv files are now (supposedly) in the exact order required for Products import with all fields filled as required or left empty (unneeded at this time). I read where this will help speed up import and I do not have to set the column headings or use a predefined configuration file (something I had tried earlier).

 

I've modified our ordering system to generate the csv files for me with all fields and values. This means if I change a product price or description, I can easily get a csv file that reflects this.  But I haven't discovered how to do an replace of any fields during import.

 

Prestashop seems pretty cool. I've looked for years for a good replacement cart, but have avoided anything with MySql in the past because of the 500 record limits I've read about during imports. This is now proving to be too true (so far).

 

We would like to avoid doing duplicate manual updates, once in the offline orodering system, and again in the Prestashop online catalog / products, due to the high volume of changes constantly occuring.

 

 

[Chris G 29]

Hi,

 

I'm using 1.4.9 but I think what I've found should still apply.

 

When you import without an ID (i.e. blank ID) you create a new 'product' (or whatever you are importing) hence the duplication.  (unfortunately you will have to delete the duplicates you've created).

 

So if you want to update the product using the same csv file you need to put the product ID in.  This is not as bad as it sounds as IDs are created sequentially so once you've created the products (I always create with a blank ID), you can enter the first ID in the spreasheet and use the auto increment in excel or whatever to fill out the rest of the IDs. All you need to do then is update the required fields.

 

I don't know about partial deleting using CSV, but you could overwrite them using their IDs, this may give you other issues though e.g.you may want the 'new' products in a different category, which can be done with the CSV file.

 

The only way I've seen to get over the upload limitations is to make a number of smaller files, again once you've created all yor products updating changes using the IDs may reduce the problems in that you only load the changes, not the complete catalogue.

 

Although I don't have that many products I break my CSVs down into categories, i.e. one for each category, but this may not work for you.

 

Hope this helps

 

Chris g