Smarty cache causing file access loop and crash

[Les Vikings]

Hi!

 

I'm coming to you with a strange one, and hard to diagnose as well...

We are encountering a problem on a 1.6.1.18 shop (but the problem might have been there before this version as we are just now managing this website) : when the Prestashop Smarty Cache is enabled, after some time or after something is triggered, the PHP-FPM instances start to overload the server and finally render the site inaccessible.

 

A bit of context

This shop is running on a dedicated server with:

- 16G of RAM

- PHP-FPM 7.0.32

- MariaDB 10.1.26

It used to be on an other server with a rather complex configuration, but has since been moved. The problem persisted. It also used to use PHP-FPM 5.6, but the problem persisted after we upgraded to 7.0.

The shop has between 7k and 10k page views per day and approximately 15 to 20 orders a day. It also has ~1600 active products in its catalog.

 

The problem

When the shop is using the Smarty Cache (filesystem, not database), at seemingly random moments, the PHP-FPM processes start to overload one after the other until the server gets a load average so high that the website can't be viewed any more. The only solution at this stage is to restart the php-fpm daemon and then the site works again until the next occurrence of this phenomenon. The only way we found to avoid this all together has been to disable the Smarty Cache, but that's not a good solution.

We can't pinpoint what conditions are causing this problem to appear nor why it's happening. We tried to replicate it on an other server simulating a load on the website, adding and editing products, triggering CRON jobs, and so on, without success. From that, we gathered that the problem must be triggered by a load specific to the daily use of the shop (visitors, orders, admin actions, cron jobs, modules calling external APIs for delivery or abandoned carts, ...).

When the problem occurs, what is apparent is that the PHP-FPM processes are trying to fetch or remove template cache files from different modules very rapidly and in a seemingly infinite loop. We found that by running a trace (using `strace`) on the processes. The queried paths vary from one occurrence to another and even during the occurrence, so there is not definite culprit that can be identified among the modules.

The traces show that the systems is trying to remove some cache directories with the `rmdir` system call but that fails as the directory is not empty. Example:
`rmdir("/var/www/vhosts/website.com/httpdocs/cache/smarty/cache/additionalproductstabs/tab/291/1/1") = -1 ENOTEMPTY (Directory not empty)`

 

We don't know anymore where to search for clues. The problem is hard to diagnose as the shop is in production and we can't do everything we want on it. Moreover, as we could not replicate this problem on our staging environment, we can't easily test solutions like deactivating modules to see if it still happens.

 

We are asking for your help to try to find a new perspective or a new approach to diagnosing this problem and solving it. We can provide whatever configuration you need to better understand what could be causing this. Two pieces of trace are attached to this post if you want to have a look at the output (they are just extracts as it runs indefinitely until a daemon restart).

 

Thanks a lot to everyone who can help!

trace_2

trace_1