slow blockcategories.tpl

[Aswin C.]

Hello all,

 

I'm encoutering a problem with performance and I didn't start with my products yet.

 

mod_deflate is on it decrease the size from 980 KB to 90KB

memcached is on, but I don't think it's that usefull if you have a good mysql server.

 

 

At the moment I have one main category, 116 sub categories ( all brands ) and 7236 models spread over those 126 sub cats.

 

- 1 ( one category atm )
 - 116  subcateogries ( all brands )
- 7236 models in total spread over those 126 subcats

 

Smarty debug info:

Smarty Debug Console - Total Time 13.51242
/modules/blockcurrencies/blockcurrencies.tpl (compile 0.00000) (render 0.00073) (cache 0.00000)
/modules/blocklanguages/blocklanguages.tpl (compile 0.00000) (render 0.00124) (cache 0.00000)
/modules/blockpermanentlinks/blockpermanentlinks-header.tpl (compile 0.00000) (render 0.00671) (cache 0.00000)
/modules/blocksearch/blocksearch-top.tpl (compile 0.00000) (render 0.00078) (cache 0.00000)
/modules/blockuserinfo/blockuserinfo.tpl (compile 0.00000) (render 0.00448) (cache 0.00000)
/modules/blockcart/blockcart.tpl (compile 0.00000) (render 0.01011) (cache 0.00000)
/modules/blockcategories/blockcategories.tpl (compile 0.00000) (render 5.75046) (cache 1.65364)
/modules/blockcategories/category-tree-branch.tpl (compile 0.00000) (render 3.99519) (cache 0.00000)
/themes/d1/header.tpl (compile 0.00000) (render 0.00893) (cache 0.00000)
/tools/smarty/debug.tpl (compile 0.00000) (render 3.75241) (cache 0.00000)
/themes/d1/index.tpl (compile 0.00000) (render 0.08027) (cache 0.00000)
/themes/d1/footer.tpl (compile 0.00000) (render 0.00026) (cache 0.00000)

 

how come that he doesn't cache the generated html ?

Isn't it more usefull to cache generated html rather than caching mysql queries?

What can I do to improve the render speed, or let it cache the generated html ?

 

Kind Regards.

[Aswin C.]

Apperently blockcart is the only module that uses the smarty cache.

 

but not at a smart way...

 

$smartyCacheId = 'blockcategories|'.$groups.'_'.$id_lang.'_'.$id_product.'_'.$id_category;

 

this means he created a cache file for EVERY

- group * language * product * category .

 

I have two languages setup, and a total of 7500 categories ( all sub cats inclusive )

2 * 7500 = 15000 possibilities ??

15.000 * 900 KB = 13 GB ?

 

this is'nt even calculated with groups nor products....

 

is this the problem a lot of ppl have with their prestashop filling up space ?

[Aswin C.]

I think you should render the complete tree for each language and each group

but then stop.

 

setting the category "active" should be done afterwards by a regexp or something.

but this is definitely not the way to go.

 

I think it should be:

 

$smartyCacheId = 'blockcategories|'.$groups.'_'.$id_lang;

 

 

then also generate for each li a id, then afterwards with regexp

s/id="cat_125"/id="cat_125" class="open"/;
s/id="cat_125"/id="cat_125" class="selected"/;

or something.

 

 

 

Also:

in blockcategories.php
Tools::enableCache();
if (!$this->isCached('blockcategories.tpl', $smartyCacheId))
{
 // rerender ....
}
$smarty->cache_lifetime = 31536000; // 1 Year  

the cache_lifteime is at the wrong place ...

it should be below Tools::enableCach();

now its on the default time of 3600 seconds.

So the cache will ALWAYS be re-rendered after 1 hour ....

should be:

Tools::enableCache();
$smarty->cache_lifetime = 31536000; // 1 Year  
if (!$this->isCached('blockcategories.tpl', $smartyCacheId))
{  // rerender ...

Edited July 3, 2012 by AswinC (see edit history)

[Aswin C.]

I believe it is really a bug, this cant be from design meant to be..

he generates a cache file for each

language * categories * groups * products...

 

so for each product he makes a cache file ..

just even that is completely wrong.

 

if you start from zero or a recompile... he will always be unexpected slow ...

until all prodcuts AND all categories have been visited.

 

that can't be good ...

+ if you have 20 000 products... its 20 000 cache files at least...

 

the category block only really changes with language and an another group.

so to set the selected.. get the cache for the combination "language_group"

then find the item that should be selected, and string replace it.

 

 

The fix:

 

in category-tree-brach.tpl

 

change line:
<a href="{$node.link}" {if="" isset($currentcategoryid)="" &&="" ($node.id="=" $currentcategoryid)}class="selected" {="" if}="" title="{$node.desc|escape:html:'UTF-8'}">{$node.name|escape:html:'UTF-8'}</a>
to:
<a href="{$node.link}" class="cat_{$node.id}" title="{$node.desc|escape:html:'UTF-8'}">{$node.name|escape:html:'UTF-8'}</a>

 

in blockcategories.php :

 

/* Search for this line, and continue/compare line by line,
 first lines unchanged (snip) until $smartyCacheid */
public function hookLeftColumn($params)  
{
....... <snip> ............... <snip> ............... <snip> ........  /*  NOT changed */
$smartyCacheId = 'blockcategories|'.$groups.'_'.$id_lang; /* CHANGED */

Tools::enableCache();   /*  NOT changed */
$smarty->cache_lifetime = 31536000; // 1 Year	/* CHANGED */
if (!$this->isCached('blockcategories.tpl', $smartyCacheId))
....... <snip> ............... <snip> ............... <snip> ........ /*  NOT changed */
$display = $this->display(__FILE__, 'blockcategories.tpl', $smartyCacheId);   /*  NOT changed */
Tools::restoreCacheSettings();   /*  NOT changed */

  // Fix: Aswin Coolsaet
  // in "category-tree-branch.tpl"
  // <a href="{$node.link}" class="cat_{$node.id}" title="{$node.desc|escape:html:'UTF-8'}">{$node.name|escape:html:'UTF-8'}</a>
  // so each will have his own "cat_id"
  $mySelectedID = 0;
  if (Tools::isSubmit('id_category')){ $mySelectedID = $id_category;}
  if (Tools::isSubmit('id_product'))
  {
if (!isset($cookie->last_visited_category) OR !Product::idIsOnCategoryId($id_product, array('0' => array('id_category' => $cookie->last_visited_category))))
{
 $product = new Product($id_product);
 if (isset($product) AND Validate::isLoadedObject($product))
 {
  $mySelectedID =  (int)($product->id_category_default);
 }else{
  $mySelectedID = (int)($cookie->last_visited_category);
 }
}else{
 $mySelectedID = (int)($cookie->last_visited_category);
}
  }
  $display=str_replace( 'cat_' . $mySelectedID , 'selected' , $display);
 return $display;
} // END public function hookLeftColumn($params)


do the same in:  public function hookFooter($params)

 

this way you only have a limited cache files ... one for each languages * the amount of groups..

so in my case.. just 2 cache files ( languages )

 

and the cache won't fill up your webspace.</snip></snip></snip>

Edited August 24, 2012 by Aswin Coolsaet (see edit history)

[Aswin C.]

More info about the problem:

 

categories on the left side... the HTML is generated.

because it is a recursive query... blablabla ... heavy MySQL query + lot of time to generate all the li and sub li's

so .. they try to cache the generated html output. (performance -> smarty -> cache on )

 

/tools/smarty/cache

go watch inside such a cache file... its just html

 

why are they so many cache files?

- the category html changes for each language

- for each customer group

( - for each selected category ) --> but only class="selected" actually changes

 

here is the bug:

so the programmer who wrote that is caching the output for each

language * customer group * category * product !!

 

 

because I have 7000 categories the RENDER of that category block takes about 10 seconds !!!!

with the HTML cached he doesn't need to render anymore and just output the file ...

 

so your website will always be "unexpected slow" until all languages for all products and categories are rendered ( and cached )

because of a "lifetime" line in the wrong place, the cache expires each hour too ...

my generated html for categories is about 900KB , this times languages * categories * products ...

What was the programmer thinking !!!

Just for the categories only I would need 13 GB of storage

0.9 MB * 7500 cats * 2 languages = 13 GB ! = 15 000 files.

 

clear the folder /tools/smarty/cache

put smarty.config.inc.php -> debug = true

and see how long it takes the block category to RENDER , revisit and it will go fast.

now open a product page... and again the render takes a long time.. ( notice you now have 2 block category cache files )

 

So I think this is what everyone is experiencing and needs to cleanup their cache folder often because of building up block category files.

 

​Apply my fix and you will have just 2 cache files.. and will be lightning fast

Enable mod_deflate to the 900KB doesnt get sended as 900KB to the client but as 90KB ...

 

 

one more small update

this fix makes it lighting fast once the cache files are generated for all languages .

but ! small client computers like those asus 10inch EEE-PC's or simular atom computers they have a hard time to actually handle a 1MB html source file ( over the network it is just 90KB but after *unzip* is is 900Kb again )

the site will be to them quite heavy ( IE, chrome , FF doesn't matter )

their browser will actually hang for a second or two... while 'strong' computers are just fine with it.

the users of those small pc's are used to have such hanging sites ( facebook ) but 7000 categories is little too much for them.

so my advice is to keep it around a 1000 categories. ( by changing the depth of the level's to show )

Edited July 20, 2012 by AswinC (see edit history)

[clayton29657]

Thank you Aswinc that is very useful info

[Mister Denial]

Hi AswinC, thanks for this really great solution!

 

Just a quick question: does this fix change the URL structure of the website?

 

I have about 20 categories and 1K products, and am wondering if this fix would also improve speed for my website, considering the low number of categories.

[Aswin C.]

probably yes because you have at least 20 * 1000 products * 2 languages = 40 000 cache files from blockcategory.

 

and that means just after 40 000 unique page visits your site will show no cpu pikes for rendering / generating the html and the 'heavy' semi recursive query..

but as you can read above... the cache Also gets INVALID each hour

 

set debug in smarty settings on in the config file and look in the popup what takes so long for you.

for me it was the rendering of blockcategories...

and thats how I saw that on each product the rendering took long and after revisit it was zero ...

and I saw the cache files building up...

 

that are those CPU spikes you somethimes see ( meaning someone hit a product or category in a certain language that never has been visited before in the past hour.)

Edited July 20, 2012 by AswinC (see edit history)

[Mister Denial]

Hi AswinC, thanks for your time and detailed reply.

 

I only have one language, three main categories, and a grand total of 20 subcategories max, with a total of about thousand products, so I'd guess the total number of cache files would be between 5000 and 10000 max (which is still impressive enough).

 

The thing I am wondering is, with products that don't change constantly, would it be reasonable (and feasible) to change the cache expiration to once per day, instead of once per hour?

 

But if yes, what happens if you disable or remove a product because it is sold out, I guess because of the cached version, it would still display as "enabled", correct?

[InterAgilité]

Hi AswinC, thanks for your time and detailed reply.

 

I only have one language, three main categories, and a grand total of 20 subcategories max, with a total of about thousand products, so I'd guess the total number of cache files would be between 5000 and 10000 max (which is still impressive enough).

 

The thing I am wondering is, with products that don't change constantly, would it be reasonable (and feasible) to change the cache expiration to once per day, instead of once per hour?

 

But if yes, what happens if you disable or remove a product because it is sold out, I guess because of the cached version, it would still display as "enabled", correct?

 

Hi Mister Denial,

 

I just test Aswinc trick, and it's just amazing ! It seems you don't have to worry about changing catalog .. everything seems fine ( I tested adding new categories, removing products etc.... )

 

The performance is just amazing now !

 

Thanks a lot

[Eddan]

I believe it is really a bug, this cant be from design meant to be.. he generates a cache file for each language * categories * groups * products... so for each product he makes a cache file .. just even that is completely wrong. if you start from zero or a recompile... he will always be unexpected slow ... until all prodcuts AND all categories have been visited. that can't be good ... + if you have 20 000 products... its 20 000 cache files at least... the category block only really changes with language and an another group. so to set the selected.. get the cache for the combination "language_group" then find the item that should be selected, and string replace it. something to start the fix: in category-tree-brach.tpl

 change line: [url="{$node.link}"]{$node.name|escape:html:'UTF-8'}[/url] to: [url="{$node.link}"]{$node.name|escape:html:'UTF-8'}[/url] 

in blockcategories.php :

 public function hookLeftColumn($params) { .......  ...............  ...............  ........ /* NOT changed */ $smartyCacheId = 'blockcategories|'.$groups.'_'.$id_lang; /* CHANGED */ Tools::enableCache(); /* NOT changed */ $smarty->cache_lifetime = 31536000; // 1 Year /* CHANGED */ if (!$this->isCached('blockcategories.tpl', $smartyCacheId)) .......  ...............  ...............  ........ $display = $this->display(__FILE__, 'blockcategories.tpl', $smartyCacheId); /* NOT changed */ Tools::restoreCacheSettings(); /* NOT changed */ // Fix: Aswin Coolsaet // in "category-tree-branch.tpl" // [url="{$node.link}"]{$node.name|escape:html:'UTF-8'}[/url] // so each will have his own "cat_id" $mySelectedID = 0; if (Tools::isSubmit('id_category')){ $mySelectedID = $id_category;} if (Tools::isSubmit('id_product')) { if (!isset($cookie->last_visited_category) OR !Product::idIsOnCategoryId($id_product, array('0' => array('id_category' => $cookie->last_visited_category)))) { $product = new Product($id_product); if (isset($product) AND Validate::isLoadedObject($product)) { $mySelectedID = (int)($product->id_category_default); }else{ $mySelectedID = (int)($cookie->last_visited_category); } }else{ $mySelectedID = (int)($cookie->last_visited_category); } } $display=str_replace( 'cat_' . $mySelectedID , 'selected' , $display); return $display; } // END public function hookLeftColumn($params) do the same in: public function hookFooter($params) 

this way you only have a limited cache files ... one for each languages * the amount of groups.. so in my case.. just 2 cache files ( languages ) and the cache won't fill up your webspace.

 

AswinC, When you put <snip> in your above mentioned solution, do you mean that I should erase everything between, and just keep your changes... for example you wrote

 

public function hookLeftColumn($params)
{
.....<snip>.........<snip>.........<snip>.........<snip>....
$smartyCacheId = 'blockcategories|'.$groups.'_'.$id_lang; /* CHANGED */

 

does this mean that I must erase all rows between 142-150?

[Aswin C.]

I have changed it and added /* NOT changed */ after it as the first snip snip snip ...

it means unchanged until you see that line of code

[alex2c]

Hi all

I wrote before, that my case doesnt work but it was my fault. Your solution is fantastic, menu shows immediately

 

greetings!

Edited August 17, 2012 by alex2c (see edit history)

[indus]

My smarty cache folder is always empty, wonder why that is. I have cache turned on in preferences. But CCC CCC for CSS first option is off since it breaks my theme.

Edited August 22, 2012 by indus (see edit history)

[Aswin C.]

which folder are you referring to ?

[indus]

Tools > smarty > cache

[bellini13]

Aswin, you have documented your solution above in post #4. Is that the final solution? Have you made any other changes?

[Mister Denial]

I have just implemented the changes as per post #4 and it works fine. I have not noticed any real speed difference yet, but I have a quite optimized website anyways and it usually is snappy fast. I'll monitor speed changes over the next days and see if I see noticeable differences. One things is sure already though, my cache folder is MUCH smaller now with only two files!

 

So thank you Aswin for this fix, cool stuff! And it's much appreciated!

 

Dan

[Mister Denial]

Hi again, quick question: setting the cache to expire after only a year, does this affect how categories are displayed, or only the category tree / block in the left column?

[DDD]

Hi Aswin,

 

I don't really understand what and where to replace/change/delete. My blockcategories.php doesnt look the same as in post #4???

 

//Hans

[Aswin C.]

* this has only affect on the module blockcategories in the left / footer.

if you recompile, or add categories or anything.. the cache for this module expires.

it can be cache for 10 years... doesn't matter, IF anything changes he will regenerate.

* DDD, it's chronological, so try with find: "public function hookLeftColum" and start from there in your document.

[bellini13]

Aswin, why not just post the final code changes that you have made?

[Aswin C.]

it is the final ...

[bellini13]

no i mean the entire code without chunks removed, it makes it confusing. just post the entire file somewhere so people can download it.

 

i think you did a good job here, so I'm not complaining, well i guess I am a little bit, but I think some folks could get lost in your post.

a simple concise posting of the final version of code would be helpful.

[Aswin C.]

the reason why I don't do that is because this fix is also working for v1.5 for instance. Probably for earlier versions too...

If you do it line by line... then this is a 100% fix for many versions. If I just give the whole file.. probably only works for that version then.

 

I filed this all as a bug, and they only changed the year thing. the cacheid remains like that because the developer said it needs to be like that for very old themes... I don't see how a theme has anything to do with it... but ok... 1.5 and future versions will because of this always be very slow as soon as you have lot of products and/or lot of categories...

Edited August 25, 2012 by Aswin Coolsaet (see edit history)

[bellini13]

their response makes no sense. to avoid fixing a bug because of "very old themes" conflicts with fixing the bug in v1.5. "very old themes" would not work with v1.5 and would need to be upgraded.

 

this developer should be shot, @Mike @Ben, if you guys are following this thread, is there something you could do about this item?

[Aswin C.]

I reported it in 1.4 http://forge.prestashop.com/browse/PSCFI-6023

and in 1.5 http://forge.prestashop.com/browse/PSCFV-3061

 

in 1.4 ludvig doesn't understand it I think, he is saying something about clearcache.

I think blockcategories doesn't or didn't used to clear the cache files if asked for it. don't know.

 

in 1.5 the year cache lifetime is fixed and then the comment about old themes...

"We have fixed on the SVN the TTL issue, however we can't improve the number of cached files for now due to retrocompatibility with older themes."

TTL = time to live for the cache

 

I don't have a single clue what number of cache files has to do with a theme ...

the cache files are just the generated html output of this module ... but once again, I'm not a presta developer.

[Aswin C.]

Personally I think the best option to have the maximum performance

is having a cache file for each language * displayed categories in the block.

Butt ! there should be an option to generate all the cache files at once.

so all +600 files ( or more ) are generated at once. ( Which is quite easy to do. )

so the heavy query + expensive time consuming generation of html only has to happen once.

something like the compiling thing, a button to generate all cache html files.

[bellini13]

i would agree with having a generation script, the java (JSP) language has this option, which would pre-compile all JSP pages.

This is probably something you could do with a custom cron job, invoking each combination off hours nightly/weekly/monthly, to account for any changes that would require re-generation.

[sarafian]

Thank you! Thank you! Thank you!

 

I am running 1.5.2. I have tried all previous suggestions to improve web page load time (misc performance settings, using nginx instead of apache, using Amazon CloudCache for media content, etc).

 

I have applied your change to blockcategories.php and my Time To First Byte went from over 8 sec to 2.6 sec! I then applied the same change to blocktopmenu.php, which also uses the smarty cache, and I was able to reduce TTFB to 1.45 sec.

 

Thank you again.

[sarafian]

Once more performance tweak, I have modified blockmanufacturer.php to use the smarty cache (it doesn't by default), and this has further increased my performance. I now have < 1 sec Time to First Byte for web site loading.

[Okidoki]

Cant understand the fix.

Couldnt locate the codes.

My files are diferent

My version is from 1.5.1.3

[Okidoki]

What about hookRightColumn?

[aduarte]

does this applies to 1.5.4.0 as well??

[Peda]

Dear Friends,

 

Could you please tell the solution for Prestashop 1.5.4.x versions?

Could not find the codes in new version

[1andy1]

Hello,

 

Does anybody know the solution for version 1.5.4.

I think that the code has been changed a lot and the only valid code I have found is in function getCacheId in file blockcategories.php. Other changes couldn't be applied, because I couldn't find the corresponding code.

[maximo88]

So dissapointing. If you repair something, you should put the code for us. 

So so bad friends.

[maximo88]

Deprecated for 1.5.6 . It's a pity

I need help´!

Edited December 2, 2013 by maximo88 (see edit history)

[laziali]

ok, did work on a 1.5.3.1

 

however i need help with the blocktopmenu fix.

who said he did it?

[CumaSegini]

At first I'd like to say thanks to Aswin for this great solution, though your way is not easy to understand, but after read and read your code a couple times I thing it's not that difficult either.

 

I have 2 categories, 20 sub-cat and about 4000 products. It cause my hosting size increase everyday.

 

I use prestasop 1.5.6.2 and this Aswin way is working great on my site. Smarty cache not increase, but I don't know about the performance, still testing it.

 

note-

- if you want to try this file, you should backup or just rename blockcategories.php and category-tree-branch.tpl then upload modification files (unzip first)

- I don't know exact location of category-tree-branch.tpl file, so I have copy both on /modul/blockcategories folder and also on themes folder

- Please report if you have try this modification, working or not.

 

 

 

 

 

blockcategories.zip

[silenec]

sarafian: could you please share your modified blockcategories.php and blockmanufacturer.php files?

Thank you and thanks CumaSegini for the blockcategories.zip, it is working.

Edited March 3, 2014 by silenec (see edit history)

[Dashe]

I found this old post.

I've the same problem with 1.6

Category is too much slow and if I'll enable smarty cache, the category doesn't load completly

 

Do you find a solution for 1.6?