Jump to content

NEW PrestaShop Import tool

Hank
 Share

Recommended Posts

Hank Newbie

Hank

Posted
Posted

Hi

I have posted adds on Freelancer.com and rentacoder.com to have a import tool created for PrestaShop.

If you're situation is like mine, I have over 90K items products to import. With the current import tool it is impossible to import that much data. I have installed a test system on my laptop removing all the images except the thumbnail and in 48 hours I only have 20k products imported.

Anyone interested in donating to this project it would be appreciated. I do not have a project bid accepted accepted at this time. I have 2 bids on the project one for $700 and the programmer is creating a project brief for acceptance. The other is for $240, I have messaged the programmer for more details on what they will provide. I will make available through PM to anyone who donates to the project. Once we have 3 or more proposals they will all be reviewed before selecting the winner of the project.

Everyone who donates will become part owners of the end product.

Once we have a completed module it will be put to a vote to release under GPL or sell licenses to the program. All funds that are donated to the project will be put held and final payment to the bid winner will only be made once a working program has been tested and confirmed complete to the specs of the project.

The project as posted reads:

PrestaShop has a data import tool built into the Open Cart software back-end. The import functionality is a necessity with the need to import large amounts of data.

This project is to have this tool redesigned or create a new import tool that will install as a module.

1. Improving the performance of importing data and multiple images per product. Import should be able to import a minimum of 25K items at one time.
2. Import is from csv file to MySql database, all images are Regenerated using the PHP GD Library. The regeneration of the images is extremely slow even with the images local in the tmp directory. As a part of the import function the image import and regeneration needs to be optimized.
3. Ability to add multiple features similar to what is currently available has to be included.
4. Current import tool has categories and and subcategories combined into one field separated by a comma, as well as multiple image paths included into one field separated by a comma. These should be able to be imported from separate fields in the csv file.
5. Optional: If possible image locations should be sorted into sub folders by Vendor/Supplier or by Categories.

If creating this tool in PHP is not possible I am open to suggestions that will work with shared linux web hosting.

Private Message me for more information....

Link to comment
Share on other sites
KIH999 Newbie

KIH999

Posted
Posted

I see one "little" problem here.
If you run that import-script fast on a shared host you can be banned for using too much resources in one go.
There are often a limit on querys /hour, and also high processor load (constant regenerating pictures will use lot of cpu)
Check "acceptable use policy" (AUP) at host before you spend good money on this.

This operation/import should be done on your computer, then upload to host and import the "big SQL-file".

Link to comment
Share on other sites
Hank Newbie

Hank

Posted
  • Author
Posted
I see one "little" problem here.
If you run that import-script fast on a shared host you can be banned for using too much resources in one go.
There are often a limit on querys /hour, and also high processor load (constant regenerating pictures will use lot of cpu)
Check "acceptable use policy" (AUP) at host before you spend good money on this.

This operation/import should be done on your computer, then upload to host and import the "big SQL-file".


I have been running the import on my personal laptop for 48hours now and only have less than 20k items in the database, I also removed all the image formats except the thumbnails. Shared system or not the import utility in PrestaShop is not working well enough to be functional. If no one wants to support the creation of a new tool, I am going to have to go it alone or move to another solution. I have imported 14k items in zencart with no issues on a shared system, with PrestaShop it will not upload at all, images or not.

If you have luck with making it work please share your solution.
Link to comment
Share on other sites
Hank Newbie

Hank

Posted
  • Author
Posted

Update to the Project bidding

The costs that are coming in I would expect that the cost would be $700 - $1000. The end product will be a streamlined import utility that also regenerates the images on the fly.

"All without overloading a Shared Web server."

I would like to have 9 investors to share in the project, 10 including myself. We will use Paypal for the financial transactions for investment of the project. I am expecting that we would need to invest $100 each (to be verified with the acceptance of a project brief). Payment will be expected with your approval of the project brief. For now I am looking for your commitment to invest. When it comes time to invest I will add the project investment as a Product in my store, with the payment being made by paypal we have Buyer and Seller protection built in.

Next Steps

1) Get the 9 additional investors
2) Pick 3 possible programmers to create a project brief for review and approval.
3) Review the project briefs and negotiate the terms.

I am looking forward to your input and support on this project.

Kindest Regards
Henry

Link to comment
Share on other sites
presta-dyr Enthusiast

presta-dyr

Posted
Posted

Hi Henry.

I have done some experiments. I found out that indeed I could speed up the import. I have a CSV file containing 16000 entries. I can now import all entries without pictures in 5 seconds. The image resizing takes a lot of time. My idea was to save the URL for the picture and then when the picture is requested use the URL and resize the image. This means that frequently used products will load fast but rarely seen products will load slowly (at least the first time). What do you think?

After I had imported the 16000 products I tried to rebuild the search index. It took longer time than the allowed 60 seconds. Fortunately the job can be continued. If you have 90000 products you will probably prefer to add a nightly cron job for reindexing.

/Kjeld

Link to comment
Share on other sites
Hank Newbie

Hank

Posted
  • Author
Posted

Did you change anything in the import function or use it as is? How did you upload image URL without regenerating the images. I would prefer to have used the url as the images could reside on my vendors website. I just finished downloading and uploading to my server, nearly 7 days.....

I would appreciate your input.

With the import tool, I was thinking of having it include a bulk update feature as well. Something to load the info in a spreadsheet format for editing. If I could getaway without creating a tool for now it would be better, that way I would be able to use PrestaShop and get something created later that fixes any other issues that I come across...

Regards

Henry

Link to comment
Share on other sites
presta-dyr Enthusiast

presta-dyr

Posted
Posted

Yes, I wrote an import script from scratch. I delete all products before I start the import. But my plan was to have an option if you want to leave the pictures on disk (product IDs are probably the same).

If you put all the unchanged images on your local server a simple substitution of the path in the URL could be the solution. On the other in hand, the images will get cached little by when Google and other bots crawl your webshop. You could also let a cron job resize the pictures at night.

Indeed if you have 90000 items it could be nice with a bulk update.

When I'm done with the delayed picture load I will put it on a server so you can get feeling of load times.


Cheers,
Kjeld

Link to comment
Share on other sites
  • 2 months later...
jaykay2012 Newbie

jaykay2012

Posted
Posted

I have space and bandwidth available on my server for this project for testing purposes. Is dedicated in the UK. I've just been given a csv file by my supplier and I cannot work out how to upload the info into a test presta shop install. The images are served via url's from my suppliers site.

Any help on this in reyurn would be great. I'm sure that it's an easy process but I just need to be shown :)

Hope it all works out for you with this project :)

Jay

Link to comment
Share on other sites
presta-dyr Enthusiast

presta-dyr

Posted
Posted

I have finished working on my tool. I can import XML and CSV files with thousands of products in just a few seconds. The pictures are resized when requested or via a cronjob. I still haven't tried with more than 17,000 products, so I look forward to try it with even more products.

You can send me a PM if you want to know more.

Cheers,
Kjeld

Link to comment
Share on other sites
  • 2 months later...
  • 1 month later...
  • 4 weeks later...
  • 4 years later...
 Share
Go to topic listing
×
×
  • Create New...