Best practice to import 8000+ rows CSV file (each row up to 14k of text) on daily basis (is it possible?)

[Mario_PG]

Hi all, I was asked to import in Prestashop a list of products that are provided as a single CSV file, this file is exported by a dealer and accessible by FTP.

I should read that file and import in Prestashop, that file updates each day and is around 8000 rows and many of them are up to 14k, because each row contains: description, image urls, nested categories up to 5 sublevels, etc. It's a bit of a nightmare...

My idea is to to write a parser in PHP that generates intermediate CSV files (like 1 for categories, 1 for products, etc) and then setup crons to manage all the automated import process.

Since now I have managed to extract all the category tree in a CSV and import in Prestashop, the number of categories are 401 in this moment.

The process is very heavy and slow and and I am afraid that with 8000 products I will end up in a failure.

My question is if (from your experience) to import all that amount of data each day is a feasible task or if is an impossible mission.

I also thought to restart with a different approach: to write all that data in MySql tables and then in some way write them directly to the prestashop tables but is going to be a lot of effort.

I don't know what would be the best solution for automating that import process...

Thanks for any suggestion.

 

 

Edited December 7, 2022 by Mario_PG (see edit history)

[Mediacom87]

Hi,

you can use solutions adapted to this kind of catalog management, such as StoreCommander.

 

[Mario_PG]

40 minutes ago, Mediacom87 said:

Hi,

you can use solutions adapted to this kind of catalog management, such as StoreCommander.

 

Thank you for your answer. I have seen StoreCommander import documentation and seems not really very different from the native Prestashop import tools. 

Also more than managing the products the goal is displaying the products exported daily by the dealer in the data file, so it's a kind of automated reset all / import all job.

Another limit is the monthly fee of the component which is not in the budget.

[Tom Girou]

3 hours ago, Mario_PG said:

Hi all, I was asked to import in Prestashop a list of products that are provided as a single CSV file, this file is exported by a dealer and accessible by FTP.

I should read that file and import in Prestashop, that file updates each day and is around 8000 rows and many of them are up to 14k, because each row contains: description, image urls, nested categories up to 5 sublevels, etc. It's a bit of a nightmare...

My idea is to to write a parser in PHP that generates intermediate CSV files (like 1 for categories, 1 for products, etc) and then setup crons to manage all the automated import process.

Since now I have managed to extract all the category tree in a CSV and import in Prestashop, the number of categories are 401 in this moment.

The process is very heavy and slow and and I am afraid that with 8000 products I will end up in a failure.

My question is if (from your experience) to import all that amount of data each day is a feasible task or if is an impossible mission.

I also thought to restart with a different approach: to write all that data in MySql tables and then in some way write them directly to the prestashop tables but is going to be a lot of effort.

I don't know what would be the best solution for automating that import process...

Thanks for any suggestion.

 

 

That's what we are doing. Actually we export data from our ERP to our Prestashop websites. We manage around 100k products, their images, our customers, orders, categories, and so on. Everything we have inside our Prestashop come from our ERP.
To do that, we both query our ERP database directly, and parse CSV files. Data is imported every 3 hours with cron PHP scripts.
So it's doable. Obviously the first time, it will take forever. But then, the only thing you'll do, is basically check if data is different, if yes => update/remove. If data doesn't exist => insert, otherwise, skip.
So unless your csv data is changing everyday, the script won't take that long on the next iterations.
Choose carefully your CSV parser, some are really heavy and slow. What you need is only to read the CSV, not create complicate ones.

Having a script that will create different CSVs for each "category" (products, categories, images) is a good idea.