Too many user cache entries in APC

[peal]

Hello, last few weeks I am trying to configure APC cache for our prestashop on VPS to avoid 100% fragmentation. I have used all hints on internet but no luck yet. Currently I have 1 Segment(s) with 256.0 MBytes, 45.7% is free, fragmentation 100.00% (117.0 MBytes out of 117.0 MBytes in 9493 fragments). *Details below

 

Besides the fact I don't even know how it can fragmentate when there's still so much space left, due to fact that entries should be deleted AFTER ttl an ONLY if there's no more space (deleted by garbage collector then), I really don't know why I have so much user cache entries. I was looking on tutorial from d42 for example and his APC seems pretty static to me.

I observed the cache for last few weeks, deleted it from time to time to test performance (obviously much faster with APC on, very fast when fragmenttaion is low, slower with high fragmentation but still a little faster than without APC).

It always happens during night, so I think google crowler visited my site, checked everything, lot's of variables were cached and for some reason fragmentation raised. My shop has cca 3k products, not so much attributes and combinations, default theme with just slight modifications, but basically just in design, no overrides. So I wonder why I have so many user cache entries while other prestashop sites have no problem? Those entries are just arrays of variables, it seems to me there is one for each product and other things processed by php in presta. Basically all variables. 

Is it worth increasing size more, or it will happen again anyway and there's something wrong with the settings?

 

I would appreciate any hint which will lead me to new ideas how to get rid of this problem.

Thank you guys.

 

My settings is as follows so you can check how it's set:

**General Cache Information**
    APC Version 3.1.13
    PHP Version 5.4.39-0+deb7u2
    APC Host eshop.kouzelna-svatba.cz (kouzelna-svatba) (37.205.11.212)
    Server Software nginx/1.2.1
    Shared Memory 1 Segment(s) with 256.0 MBytes 
    (mmap memory, pthread mutex Locks locking)
    Start Time 2015/04/03 16:22:51
    Uptime  14 hours and 53 minutes
    File Upload Support 1

**Settings:**
    apc.cache_by_default 1
    apc.canonicalize 1
    apc.coredump_unmap 0
    apc.enable_cli 0
    apc.enabled 1
    apc.file_md5 0
    apc.file_update_protection 0
    apc.filters 
    apc.gc_ttl 3600
    apc.include_once_override 0
    apc.lazy_classes 0
    apc.lazy_functions 0
    apc.max_file_size 2M
    apc.mmap_file_mask 
    apc.num_files_hint 1000
    apc.preload_path 
    apc.report_autofilter 0
    apc.rfc1867 0
    apc.rfc1867_freq 0
    apc.rfc1867_name APC_UPLOAD_PROGRESS
    apc.rfc1867_prefix upload_
    apc.rfc1867_ttl 3600
    apc.serializer default
    apc.shm_segments 1
    apc.shm_size 256M
    apc.shm_strings_buffer 4M
    apc.slam_defense 0
    apc.stat 0
    apc.stat_ctime 0
    apc.ttl 7200
    apc.use_request_time 1
    apc.user_entries_hint 8192
    apc.user_ttl 7200
    apc.write_lock 1

**File Cache Information**
    Cached Files 841 ( 76.9 MBytes)
    Hits 1458002
    Misses 849
    Request Rate (hits, misses) 27.20 cache requests/second
    Hit Rate 27.18 cache requests/second
    Miss Rate 0.02 cache requests/second
    Insert Rate 0.02 cache requests/second
    Cache full count 0

**User Cache Information**
    Cached Variables 11433 ( 56.2 MBytes)
    Hits 732005
    Misses 103870
    Request Rate (hits, misses) 15.58 cache requests/second
    Hit Rate 13.65 cache requests/second
    Miss Rate 1.94 cache requests/second
    Insert Rate 3.16 cache requests/second
    Cache full count 0

 

[peal]

I did some further investigation and it's weird. We set size of APC cache to 1GB. That's right, whole gigabyte, we wanted to see what will happen. Unfortunately result is the same.

Cached files are ok as usual, static number of php files is stored (cca 60MB). User cahce has cca 50MB, but broke whole cache again, 100% fragmentation. The only think how to workaround it at this moment is delete just user cache every few hours, but it doesn't solve the problem. Do you have guys any idea why user cache entries are ruining my cache? Is it possible to disable it somehow and keep just the files caching?

[Dh42]

Do you have APC cache turned on in the back of PrestaShop?

[peal]

Yes, I do. Actually I just found very interesting page  (last post with blacklist modification)

It's too soon to say if it helped or not. But so far I can say there are not so many entries inserted in cache now. Before when I opened some product with many attribute combinations, there were many entries inserted. Now it seems much better. Now...

But I wonder if this is really default behavior? According to that bug report, lot's of people experience such problem. My only explanation is that most of them doesn't have access to apc graph so they don't even know they have fragmented cache? Is this reason why apc cache for prestashop is not usable without such tweaks?

Edited April 7, 2015 by peal (see edit history)

[Dh42]

I am not familiar with that. I do know that using APC enabled in PrestaShop will result in more information being cached which will fragment a cache. Honestly I have never worried about it as long as the cache was not filling and dropping. The reason being is that if your server is set up correctly nothing should be swapping. When you start swapping and are getting fragmentation things will slow down as the read write head moves. But with APC you are talking about maybe one clock cycle extra that it takes to access fragmented memory. You are literally talking about an un-measurable amount of time. 

[peal]

Thanks for your opinion Dh42. After 14 hours of APC running fragmentation is on 30% in cca 10k fragments (on 1gb allocated memory).

It seems that putting that table on blacklist for caching helped a lot, it's much better than before (fragmentation within few hours).

But I am still wondered how much variables from prestashop are cached.

User Cache Information
Cached Variables	12025 ( 61.5 MBytes)
Hits	719872
Misses	96331
Request Rate (hits, misses)	16.42 cache requests/second
Hit Rate	14.48 cache requests/second
Miss Rate	1.94 cache requests/second
Insert Rate	13.64 cache requests/second
Cache full count	0

And I think it's prestashop fault, when apc caching in presta was disabled, apc cache was ok.

But even with 30% fragmentation performance is quite good, cca 2s for all pages which is not so bad for presta.

So now it seems only solution is to have that modification set and clean cache from time to time.

 

Does somebody else experience such behavior? How many variables are stored for your shop? There isn't written anywhere what is "normal" value

[Dh42]

Are you not timing out your workers?

[peal]

Hmm, I didn't even thought it can be related to workers.

Related workers pool configuration is below. But thanks for suggestion, I will try to search today what should be set there.

 

There should be some kind of manual from Prestashop provided what to be careful about, there are so many things to be set while they can really cripple the shop performance. If they would do that, Presta could be more competitive :/ 

POOL CONFIGURATION
pm = dynamic
pm.max_children = 50
pm.start_servers = 20
pm.max_requests = 500

; Default Value: 0
;request_terminate_timeout = 0

; Default Value: yes
;clear_env = no

PRESTA SITE CONFIGURATION
fastcgi_read_timeout 1200s;
include fastcgi_params;
proxy_buffer_size 8k;

NGINX CONFIGURATION
worker_processes 64;
pid /var/run/nginx.pid;
events {
	worker_connections 768;
	# multi_accept on;
}
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	# server_tokens off;

Edited April 8, 2015 by peal (see edit history)

[Dh42]

Well the workers are what store the cache, when a worker dies its APC cache dies with it. The way that you have your server set up it can consume more memory than you have I bet and it will swap and or crash. Figure it this way

 

php memory 256mb 

apc memory 128mb

 

Now multiple that by 50 processes. APC memory is not generally reserved like php memory is, but your server should have around 16gb of memory with that set up. 

 

 

Also letting a worker die is a good way to handle fragmentation. When a worker dies, its cache dies, then it starts the cache cycle again. basically your are constantly cycling workers out as they get more fragmented.  

[peal]

I think I finally solved it.

Besides the fact which Dh42 mentioned (thanks for the hints, good server configuration is not so easy...), problem was in WHAT was caching.

 

I found out one 3rd party application on same server which created user cache entries with very low ttl and constantly recreated.

When I turned it off, it was a little bit better than before.

 

Further analysis showed that there are more php scripts which create too many cache entries:

- cron

- blocklayered module, to be more specific the indexers which are croned to reindex some values needed for filtering

 

So I added filter parameter to apc and now it's ok, fragmentation between 2-4%.

If somebody else have similar problem, I suggest to sort user cache entries by creation time, find blocks where many files were created in the same moment and analyze stored entries or check if some php file was cached in the same time. In my case it was indexers which are part of BO run once per night so there is no need to cache it.

Edited April 10, 2015 by peal (see edit history)

[Casper_O]

Peal could you maybe write a little about the APC filters you made?

 

I finally made some changes in our 1.6.X store, so the blocklayered acually show the correct products and not a "static" selection of products, no mather what category was selected.

 

but our fragmentation is hitting 50-70% in a really short time, so i fear to let it run for more than just 30 mins