Since we already covered all aspects of X3 frontend performance, I figured it was time to see see how fast we could get X3 to create and output pages from your server. Before starting, I predicted that speed could be increased perhaps twofold [2x]. However, by cleaning oh so many skeletons from the closet and adding several new smart features, it seems we may have a performance boost of [5-10x] compared to earlier. There are also a few important changes and new features, and a few compromises.
Users who are updating their existing websites, please read the CHANGES and UPGRADE CHECKLIST section below to avoid any issues. Also, if you are using FTP to manage content folders and images, you will need to follow some new procedures.
Download and Upgrade
1. Download latest version x3.latest.nocontent.zip
2. Update ALL files in your X3 folder (also root files), EXCEPT folders /content/ and /config/.
3. See the CHANGES and UPGRADE CHECKLIST sections below.
<!-- PERFORMANCE IMPROVEMENTS -->
What "performance" improvements are you talking about?
First of all, let me explain performance and speed in context of this release. X3 was already optimized for frontend loading and "best practices" a long time ago, as noted in this post. This release however, addresses the server creation- and output speed for X3 pages and data. As you may be aware, like any other app, X3 needs to "compile" pages based on data, images and folders, before the page document is output.
Why is this speed important?
The speed of which your server manages to output a page, is the first thing that happens before anything else. This is basically the time visitors are left "waiting" for a page to start displaying. Obviously, it affects the website-admin also, who is affected by the processing time for pages each time they make a change. In some cases, it could mean the difference between make- or break. Furthermore, page output speed is an important signal for SEO, as search engines like fast-loading documents ... and so do your visitors!
Just for reference, there are TWO states of page outputs: 1. pre-cache, when building the page, and 2. post-cache, when outputting a cached page. Both are important, and both have received massive speed boosts.
* The below can be quite technical, so don't feel compelled to understand it all.
"Touch" Cache Invalidation
When an X3 page starts loading, it needs to check if any changes were made in your content, to decide if it should output a cached page or build a new one. Now, instead of having to loop through all your content folder (directories and images) to see if any changes were made, it only checks the timestamp of the /content folder. This is obviously MUCH faster, and will affect page speed both before and after cache. Now, when you make any changes from the panel, it will "touch" the /content folder, update the timestamp, and X3 pages will by this know if any changes were made.
New Dedicated Menu Creation
Often, creation of the main menu is the heaviest process in page creation, because it requires looping through all pages, loading page data and reading images. We created a new dedicated script for this process, which is optimized on so many levels compared to before. It massively improves the speed of the page-creation process.
Another problem with the menu, is that it had to be re-created every time you made ANY change from the X3 panel. This is because there may be changes that affect the menu, so it always got re-created. However, often you are making hundreds of changes to your website that do NOT affect the menu at all, so why always re-create it? Now, you can use the "refresh menu" button to manually re-fresh the menu when required. This will make new pages load faster, as menu is always cached, and you can choose when to re-create it. This feature may be especially beneficial on websites with heavy content and/or on slow servers. * This will not affect performance of cached pages.
* Manual menu refresh is not always required, and can be disabled in settings -> settings -> Menu Processing -> Manual Refresh. If you disable manual refresh, menu will be re-created automatically when you change something.
New Simple Menu Mode * Blazing fast!
Although most of you won't want or need to use this feature, it is available for websites with EXTREMELY heavy content structure. Simple menu mode will drastically speed up the creation time of heavy menu structures. In simple mode, the menu is created by simply reading the directory structure of your content, instead of having to load page data for each page in the menu. The compromise is that it 1) does not support mega menu, 2) uses capitalized folder name instead of page label and 3) does not support menu links. * Use this setting if you plan on having thousands of pages and don't require advanced menu behavior. This setting may be especially useful for the sidebar layout.
* Simple menu mode option is available from Settings -> settings -> Menu Processing.
URL Route Data Cache
When X3 attempted to process an URL from browser, it needed to compare the input url recursively with the content directory structure to see if a matching path exists. Now, page paths are stored in a lookup file /app/_cache/data.json, which X3 uses to match a ROUTE with a PATH. The data.json file is managed by the panel when any changes occur. This will improve page speed both before and after cache. The route data cache is automatically managed by the panel.
Article Nav simplified
"Article Nav" are the arrows that appear on some pages to navigate between sibling pages. The process of including these buttons required us to load page data also for sibling pages when loading a single page, which was quite unnecessary. The article-nav buttons are still there, but now they just use the name of the sibling folders in the link. For example /my-page/ will create a link name "My Page".
Imagevue X3 Twig templates are now pre-compiled, and moved to app/templates. This means you avoid heavy processing immediately after upgrading X3, or setting up a new installation. Furthermore, X3 app does not check to see if templates are updated. Warning! This means you can't edit files in /templates, although that is not something you should be doing anyway.
Other Performance Improvements
- Preview images are not searched for from the menu create process.
- Application file-cache is built only with files and folders required by the created page.
- Application file-cache doesn't check app/ or templates/ folder any more.
- Removed unnecessary is_readable() from file_cache build.
- Site_updated variable now only checks a few files, synchronized with the new "touch" system.
- Page_updated loop now only checks time-stamp of folder (instead of all files within).
- Protection AUTH script is only loaded if there are passwords + improved performance.
- Faster and cleaner JSON output by using native PHP, without pre-parsing data.
- Many PHP include scripts are now only included if page is re-created, creating faster cache output.
- Optimized twig templates for less PHP processing [pre-cache].
- Images get width/height from EXIF if set.
- Don't get file date if already set from EXIF.
- More effective redirect for page links, does not render page collections first.
- Slightly faster panel menu.
- Faster Preload Site Object from panel, created and saved directly on server.
<!-- BUGS -->
- Added strtotime(date) to fix ill-configured date strings from EXIF.
- Fixed potential double-slash// index path flaw.
- Fixed file.json bug, when popup disabled.
- Fixed a bug with timeago in mega menu carousel.
- Fixed inconsistent urls in image landing-page article-nav.
- Fixed so that site preloader never caches stale site objects in browser.
- Fixed bug when using a 3rd party CDN service when pages are loaded in JSON [forum].
- Fixed bug in panel gallery if file exists with filesize 0 (empty file).
- Fixed bug in panel rename file if file name includes multiple dots (ex. "this.is.filename.jpg").
- Panel language always defaults to English if no language was stored in session.
- Tooltips don't show empty bubble if nothing to display.
<!-- NEW FEATURES -->
- Added Panel Tools -> Clear Page Cache. * This is not a necessary process.
- Re-introduced folders_amount, to display amount of "albums" inside a child-page https://d.pr/i/125ui
- Pages will return error if page cache is not writeable (cache is crucial for performance!).
- Panel automatically sets preview-image when you upload images.
- Atom feed VALID, and more effective.
- Updated help for many panel options.
- Width and height added to EXIF data in popup, hidden by default.
- Folders without a page.json data file will now work, although this would only apply when using FTP.
<!-- CHANGES -->
Page Titles- and Descriptions
We used to enforce page title- and description by requiring default values from global settings. This feature was added to make sure you have valid page titles and descriptions. However, it is incorrect to inherit them from a default value, as they should always be unique. Therefore, default title- and description is now empty, and page title is created from page slug name unless specifically set for a page.
- Title can now be empty, in which case capitalized folder name (slug) will be used instead.
- Default page title- and description changed to empty.
- Description can be empty, although it is recommended to set a description.
- * Users upgrading, should empty their default titles and descriptions from settings -> page -> details.
Hidden files and folders with _underscores
Previous, all files with _underscore were hidden from the gallery. This was not really a good thing, because many users have valid image file names that start with _underscore. Therefore, image names starting with _underscore (for example _file.jpg) are no longer hidden. Instead, files with DOUBLE __underscore are hidden (for example __file.jpg).
PS! There is new settings -> settings -> Hide images, where you can set how images are hidden, and revert to the former if required [see post].
- Images are no longer hidden with single _underscore.
- Images are now hidden with double __underscore, for example __filename.jpg.
- Hidden __images can be used as preview images, or for plugins.
- Folders/pages are still hidden with single _underscore, for example /_pagename/.
- * Users upgrading will need to rename their hidden files from _file.jpg to __file.jpg.
As you may know, each page has option for a preview-image, which is used in the mega menu, category page listings, and when sharing the page. Earlier, this image was set automatically by checking for a valid image in the folder, or by searching a child folder for images, or searching an "assets" folder. However, with the performance improvements, we had to rid this on page-creation, because it is such an ineffective process just to set a "preview image". Instead, the panel will automatically set a preview-image from uploaded images. You can also use the new Tools -> Update preview images, to check all your pages, and attempt to set a preview image where possible (unless already set and valid).
- Menu creation does not need to traverse multiple pages for each page just to locate an image.
- Page preview image does not need to traverse multiple folders to locate a preview image.
- Page preview image is now automatically set for a page in the panel once you upload images.
- Preview-image used in og:image (sharing) meta tag now always uses the original image size. It is pointless to try to serve a down-sized image, since Facebook/Google can easily handle large images. This will speed up delivery, and requires less processing from server.
- New Tools -> Update preview images. * Make sure to use this tool once if you are upgrading.
- New: Images with name preview.jpg are automatically set as preview image for a page, and will not display in the page gallery. This specific file name is hidden from the page layout, and is exclusively reserved as preview image. Useful for setting a preview image for a page when you don't want a gallery to display.
Page article nav (arrow links on sides of screen for some pages) now uses page slug (from folder name), instead of label. This is because the process of loading full page-data from sibling pages just to get the page label is too slow. If you are using article-nav, you will just need to accept this change for the greater good, and it is unlikely it will change anything for most of you anyway.
<!-- UPGRADE CHECKLIST -->
Although the upgrade process itself is simple as usual, there are some changes to the core (see above) that may require your attention. To make sure you avoid any issues, I suggest following these points after upgrading:
1. FTP: Delete /app/_cache/templates
Although not strictly required, I recommend deleting the folder /app/_cache/templates by FTP. As stated above, templates are now pre-compiled, so this folder is no longer being used.
After update, re-check your diagnostics just to make sure necessary folders are writeable. Simply go to yourwebsite.com/?diagnostics.
* You may see an error: "Default Preview Image does not exist". Don't worry, this is covered in point #7.
3. Login to Panel
Since a few of the new cache mechanisms depend on the panel, I recommend logging in to the panel before anything else. This will make sure the new data.json cache is created, and you can perform the following tasks:
4. Are you using passwords anywhere?
If not, go to panel -> protect, and delete the DUMMY passwords originally set as sample content. Removing passwords will prevent the AUTH script from triggering unnecessarily to check if folders are passworded. * If you ARE using passwords, this process can be disregarded.
5. Empty your default page title- and description
Because of the changes noted above, it is now recommended to NOT use default page titles- and descriptions. Go to settings -> page -> details, empty both "Default Page Title" and "Default Page Description", and click "save". This will not affect your pages, because likely you have set unique titles and descriptions for all your pages anyway. Even if you have NOT set unique titles and descriptions for each page, it is better that these values remain empty, in which case title will get set from unique page slug name, and description will remain empty (which is OK) ...
6. Do you have hidden files with single _underscore?
As of changes stated above, image names with single _underscore are no longer hidden. If you are using hidden _image.jpg anywhere, you will need to rename the files to __image.jpg. * It is UNLIKELY you have any hidden images unless you are aware of it.
PS! There is new settings -> settings -> Hide images, where you can set how images are hidden, and revert to the former if required [see post].
7. Change default preview image
In this release, we changed the default preview image from 1.index/thumb.jpg to 1.index/preview.jpg, because "preview.jpg" is the new default for preview images. Unless you have changed this value, it means default preview might be pointing to a file that doesn't exist, since we renamed the default thumb.jpg to preview.jpg in new sample content.
Go to Settings -> Page -> Default Preview Image, and check if this value shows "1.index/preview.jpg" (it should, unless you changed it). If it does, then navigate to the 1.index page -> gallery, and you will likely see a file "thumb.jpg". Rename this file to "preview.jpg".
8. Set all Previews
Because of the changes to preview images (see "changes"), you may have missing preview images for many of your pages. This process is now automatic from the panel when you create NEW pages, but your existing pages may need to get updated also. Navigate to Tools -> Previews, and click "Update preview images". * I recommend you read the help section for this feature.
9. Refresh Menu
Now that necessary changes are made, you can click "Refresh Menu" from the main menu, to refresh- and cache your first menu. This process is required every time you want to re-create the menu.
* Note: If this process is very fast, for example below 0.3 seconds, you could easily disable manual menu refresh by going to settings -> settings -> Menu Processing, and unchecking "Manual Refresh". The menu will then get re-created more frequently (after any change in panel), but it will not affect page-creation time to much degree, and cached pages will not be affected at all.
10. Check your website and pages
Likely everything will be working fine, but better check your pages and menu.
<!-- MANAGING CONTENT BY FTP? -->
Since some processing is now handled automatically by the panel, you cannot simply make changes to your X3 pages in FTP without following some new procedures. After making updates to content by FTP, you may need to run the following scripts, with are otherwise handled automatically from the panel:
This script will tell X3 to recreate new pages, instead of serving cached files. It will also update the necessary data.json cache file.
If you have made changes that affect the main menu (creating new folders etc), you will need run this script to refresh the menu. *If you run this script, you don't need to also run the touch.php script, because the same process is included. This script is heavier, and should only be used when you make changes that affect the menu.
* Both processes above require you to be logged in to panel, as we can't risk having these scripts accessed by the public or bots.
* You may still need to update your preview images from panel tools -> previews.
<!-- TESTS -->
For those interested, I made some quick tests of page output speeds. Tests were made from a standard shared server environment, from a page with 28 images and default release sample content:
Page creation time before anything has been cached, including menu creation and new data cache file:
<!-- Page creation output time: 0.29689502716064 seconds -->
<!-- Page creation output time: 0.15347194671631 seconds -->
<!-- Cache output time: 0.0056369304656982 seconds -->