Large Volume CSV Import

[Gospodinwizard]

The web stores I am trying to build each contain around 43,000 items.  I fully understand that importing them will take some time, but currently I can't import more than 200 at a time without an error being thrown.  The error is non-specific and literally say something akin to "an error has occured".  At a minimum I need to be able to import at least a few thousand at a time.  Does anyone and suggestions or guidance?

Thank you very much.

[Guest]

This is solved here at least once a week.
It will not work without its own import script.
I've already given instructions on how to run the import repeatedly.
The timeout and number of rows to import are set.
The script imports the number of lines and waits for a tumeout. When the timeout expires, it saves the last line number to a file.
It automatically reloads the import script, which reads the last line number and continues from it, etc.
When it is on the last line, it will end.
It can also be done via Cron and run it, for example, every 5 minutes.
Pictures take the most time.

[Guest]

Sample code for automatic refresh script.

Result before refresh (image).

Save as big-import.php

<?php
    $limit = ini_get('max_execution_time');
    $run_time =  (time() - $_SERVER['REQUEST_TIME']);
    $limit_before_end = 90;
    $max_time = $limit - $limit_before_end; 
    $number = 1;
    $index_ = 0;
    
    if (file_exists('num-import.txt')) {$number = intval(file('num-import.txt')[0]);}
    
     
    $i = 0;
    
    for($i=0; $i < 1000; $i++) {
        $run_time = (time() - $_SERVER['REQUEST_TIME']);
        
        usleep(100000);
        
        $run_time = (time() - $_SERVER['REQUEST_TIME']);
        
        if ($run_time == $max_time || ($run_time > $max_time && $run_time < $limit) || ($i + $number) == 1000) {
            $index_ = $i + $number;
            break; 
        }
    }

    echo '<b>Server max execution time: </b>'.$limit.' seconds<br />';
    echo '<b>Script max limit execution time:</b> '.$max_time.' seconds'.'<br />';
    echo '<b>Processed rows: </b>'.$index_.'<br />';
    echo '<b>Last index line: </b>'.$number.'<br />';
    echo '<b>Current index line: </b>'.$index_.'<br />';
    if ($index_ < 1000) {echo '<br /><b>continue ... </b><br />';} else {echo '<br />IMPORT DONE !!!<br />';}
    file_put_contents('num-import.txt',$index_);
    
    if ($number < 1000) { 
       echo '<script type="text/javascript">setTimeout(function(){window.top.location="big-import.php"} , 1000);</script>';
    }

 

Edited May 10, 2020 by Guest
update script (see edit history)