Page 1 of 1

Page cache non-stop growing (+500Go per day)

Posted: 28 May 2019, 13:34
by nz-photo
Hey everyone.
Coming back to Imagevuex recently to check the latest version, I've noticed an issue (at least I'm supposing it is) :

The page cache is non-stop growing everyday (500Go to 1.5Go) and filling all my server space (which is 5Go allowed)
I've tried to change the cache options but it's still adding files in the cache page folder day after day..
Did I miss something in configuration?

Has anyone noticed that or knows how to stop it adding new files ?
Thanks in advance!

Re: Page cache non-stop growing (+500Go per day)

Posted: 28 May 2019, 23:00
by mjau-mjau
First of all, you specifically speak of the PAGE cache in the /_cache/pages/ directory? Nothing to do with your /_cache/images/ directory? If so, this should be clarified, as there is a big difference between caching pages and caching images.

As for your PAGE CACHE:
  • X3 will create TWO cache-fragment files for each page. One for the page when loaded directly (full document), and one for the ajax json request, when the page is navigated to from the menu.
  • In addition, if search engines are indexing your image-landing pages, or if you have disabled POPUP, each of your images on your website will create two cache fragments (html + json) for the image's own landing page (normally not visited by humans, but if cache elements are created, it is definitely visited by either humans are bots).
  • If you have 100 pages and 1000 images, you might therefore get something around 2200 + cache files. The cache files might be between 20 kb - 50kb. In example above, that would require 110 MB.
  • Just as an example, our X3 demo, which of course is relatively small, contains 217 page cache files, in total of 7.4MB [screenshot].
  • Needless to say, if there was an option to disable page cache, your pages would load MUCH slower. This is perhaps the single-most important option for speeding up page-delivery from your server shared server. Because of this, there definitely should NOT be an option to disable page cache.
I don't know how you managed to gather that amount of page cache, but either you are created an abnormal amount of changes within a short time period, or you are accessing from different paths/domains ... I don't know. It is entirely safe to delete your page cache. You can delete all the files, and then access one page after another, and you will see that cache-files are created only ONCE for each page/json request and/or image-landing page in the /_cache/pages/ directory. These files represent one json- or page request, and are crucial to speed up delivery of that request. X3 does not add random junk into here obviously, and will manage outdated cache fragments for existing paths.

You might consider deleting your the page cache once your website is more or less "ready", and then you will find that it will eventually populate to saturate all requests (as they are made), nothing more nothing less. It could be that there are loads of orphan files there right now if you for example have been copying, moving or renaming main folders.

The page cache is non-stop growing everyday (500Go to 1.5Go)
The page cache will definitely NOT continue growing, unless you inevitably continue to make extensive changes to folder paths and/or image structure. It will always saturate at the point where all requests are cached. It could be your cache might contain orphan cache files, in which case there is no problem deleting all cache (considering you must have been making lots of structure changes), but it won't continue growing inevitably. 

5GB?
Having said all the above, and although your page cache seems abnormally large, I really don't understand what kind of hosts offer only 5GB these days. Many hosts offer unlimited storage, the cheapest Godaddy package, is 100GB, and we even offer unlimited storage with our flamepix.com hosting, where we also offer price-match. Thus, in 2019, I really don't see why anyone would need to spend their time worrying about how much storage that cache requires for a fast, modern website with extensive content.

To conclude:
  • Page cache is important for speed, and should never be disabled.
  • All newly created cache elements are unique. If they get created, it means they don't exist yet.
  • It is safe to delete the cache files, as it will eventually re-populate from existing pages.
  • Even at 500MB, cache storage should really not be a concern. If you want to host a modern website on your own hosting that is.
If you want full diagnosis, I would need FTP login. I would need to check the amount of folders and files in your content, and would delete the page cache, and then keep it under surveillance. It will eventually saturate, and definitely not grow inevitably.

Re: Page cache non-stop growing (+500Go per day)

Posted: 29 May 2019, 05:46
by nz-photo
First thanks A LOT for your detailed answer.
That's overall how I thought page cache should be working and why I knew I have a problem with it.

It concerns only the page cache, image cache folders seems ok (around 350Mo) with 2 folders rendered and request.

I have already deleted all the page cache to see what would happen then and it grows every next days to eventually reach the 5Go limit within 5 days (I receive an alert from the host when the 5Go are about to be reached or reached, so I know)
And I really don't know why.
For instance I've deleted everyfiles yesterday and since (so today so far) it has reached 635Mo with 29405 files which could be normal BUT it keeps on growing...

Years ago when I tried your first X3 version I didn't have that issue, it started last week when I finally came back to imagevue, installing the latest version...

For you to know I didn't make any changes to the website after reinstallation and just kept my layout from my older version.
So I thought it would simply recreate a cache at beginning not more.
Maybe I should clear all the imagevue and reinstall it from zero (just need to be sure to keep the files used for my layout because I don't want to start it from scratch)

If you need to check my ftp how could I send you access to what you need safely ?
Thanks again would love to understand what the trouble is !

Re: Page cache non-stop growing (+500Go per day)

Posted: 29 May 2019, 09:58
by mjau-mjau
Ok, so I can only check this with FTP. I need to see how much content/images you have, which will be proportional to the cache amount. Then I will delete the cache, and will have to put the cache under surveillance. Each cache file represents a request, and the whole point of the cache is that once cache is created for a request, it won't cache that request again. Basically I am wondering what these files are then ...

You can send me a private message with FTP details if you like.

Re: Page cache non-stop growing (+500Go per day)

Posted: 31 May 2019, 19:15
by nz-photo
Allright, sorry been quite busy these last 2 days, but I'm sending you ftp access by PM.
Thank you if you could check.
I've deleted all cache page on the 28th when we were talking about it here, so you could see how it has grown since, everyday day...
29th:  41080 files - 885 Mo
30th:  16682 files - 363,3 Mo
31st:  30711 files - 661 Mo

Re: Page cache non-stop growing (+500Go per day)

Posted: 01 Jun 2019, 01:30
by mjau-mjau
Answered by private message. Some kinda previous exploit on your website, with bots hammering URL's that don't exist on /index/ and index.php.

Re: Page cache non-stop growing (+500Go per day)

Posted: 01 Jun 2019, 06:13
by nz-photo
mjau-mjau wrote: Answered by private message. Some kinda previous exploit on your website, with bots hammering URL's that don't exist on /index/ and index.php.
I've been just reading all that.
A very huge thanks for your time and efforts !