Search…

X3 Photo Gallery Support Forums

Search…
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

404 Errors

30 Dec 2015, 23:27

I installed the latest version of X3 on www.merelyafleshwound.com yesterday.

When I crawl my site using the ScreamingFrog SEO Spider I am getting 5,586 404 Page Not Found errors. I think there is one for every image on the site.

Here is one example...

Type: AHREF
From: http://www.merelyafleshwound.com/mounta ... /columbia/
To: http://www.merelyafleshwound.com/mounta ... mboldt-13/

Any ideas why this is happening?
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: 404 Errors

31 Dec 2015, 00:35

alexhenes wrote:Any ideas why this is happening?
Yes. It seems there was an issue because all your images end with *.JPG in uppercase, and these files were not detected properly. The issue is fixed already, and you can just update the following file:
/app/stacey.inc.php
alexhenes wrote:When I crawl my site using the ScreamingFrog SEO Spider I am getting 5,586 404 Page Not Found errors. I think there is one for every image on the site.
On an unrelated note, I'm impressed that your server doesn't buckle under from this process. Some X3 users have problems just getting a single new page to re-process. In your case, you are sending a spider to process 5586 pages as fast as possible ... All these pages require processing (menu, content, page-creation-to-cache). How many seconds does it take to create a single un-cached page for you after you change something in the panel? Multiply that result with 5586, and you have the task your server is asked to process. Keep in mind, the image-pages aren't normally visited by humans, unless they "open link in new tab", and is mostly reserved for landing-pages for sharing and SEO. Therefore, the pages will likely get processed progressively over time in real-life.

To be honest, the task might be much more heavy after the fix. Why? Because in your current case, X3 is just returning a cached 404 page for each result. In the case it works, it will need to process the actual page.

Just a heads-up, so you know what is going on ... ;)
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

31 Dec 2015, 02:12

Thanks... all my 404s are gone!!!

When I initially started crawling the site I did crash the server... ha! Now I crawl with just 1 thread.... don't want to piss off my server mate. :wink:

I have crawled with up to 3 threads and things seem to work fine.... more than that is dicey.

ScreamingFrog is a great tool to see site issues, response times, etc.

I also use it to create my cache file for various image sizes. I crawled the site with various user agents... ie. iPhone, Android Phone, iPad, Android Tablet, Googlebot Desktop agent, etc. A great way to quickly build your cache.

Site performance has significantly improved since the X3 beta first started. Here are the stats for the crawl I just did with the bug fix.

Pages Crawled = 12,498
Average Response Time = 0.39 secs
99.77% of pages were sub second

Here are some more details
Capture.JPG
Capture.JPG (44.53 KiB) Viewed 7219 times
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

31 Dec 2015, 13:18

After building cache...

Avg Response Time = 0.23
Capture.JPG
Capture.JPG (28.46 KiB) Viewed 7213 times
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: 404 Errors

01 Jan 2016, 00:18

Nice. We did some improvements in the code, and I'm happy to see it's faster. I think primarily it's related to the "menu fragment cache" we now use. Instead of each page having to create/process the menu from scratch, it gets created and cached on the first page visited (after change), and the other pages can use the cached menu fragment. The menu-creation process is the heaviest process, so it should help a lot.

Honestly, it seems your server is very fast considering your relatively heavy structure. We have plans for further performance improvements in the code, because there are still some slow processes that run when validating the cache.
alexhenes wrote: I also use it to create my cache file for various image sizes. I crawled the site with various user agents... ie. iPhone, Android Phone, iPad, Android Tablet, Googlebot Desktop agent, etc. A great way to quickly build your cache.
Just a comment about this ... Keep in mind, there is no difference between the cached pages across different devices. The page output is the same, and instead it's the "responsive" frontend (javascript+css) that modifies layout based on device. However, different devices will likely request different image-sizes, but I doubt the crawl bots load images in the first place ... The bots would have to have javascript enabled, which would be highly unusual. Therefore, I would conclude it unnecessary to crawl the website with multiple agents.
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

02 Jan 2016, 11:33

Cool... thanks Karl.

The only performance related issue I am seeing is that it takes about 30 seconds for the three bars in the mobile menu to load. In addition the first image to appear in mobile also takes about 30 seconds. When the image does appear it is about 3-4 images into the slideshow. This happens if I am on wifi or on 4G
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: 404 Errors

02 Jan 2016, 12:10

Alex, why do you have multiple insanely massive images in your start page slideshow?
Image

Each image is around 8MB. That's a whopping 30MB to load just for the first images in the slideshow for the start page, so no wonder it loads slowly on mobile. This will obviously steal all network resources, partially blocking other scripts (for example icons in menu) from loading effectively also. It's slow on desktop also because of this, but you may not have noticed if some resources were cached.

You need to use the upload-resizer, or resize images prior to upload.
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

02 Jan 2016, 12:21

will do...

What is the max size you recommend?
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: 404 Errors

02 Jan 2016, 12:47

Btw, you have disabled the CDN option? May I ask for what reason? No wonder the icons load even slower, since you are trying to load everything from your own local server in addition to the heavy images. If you used the CDN, many resources (scripts, css, icons) would be loaded in PARALLEL with images, instead of being queued from the same server.
alexhenes wrote:What is the max size you recommend?
Without going into detail: 1600px, as we have set as default in the uploader resizer. I was hoping that those who turn OFF the upload resizer, do it because they have already resized their images, and/or are technically aware of the implications involved.

From our demo gallery, with images around 2-400kb, which is 4% the size of your images:
Image

Resizing images is very important, and Im surprised you were not aware that all these massive images would basically block the network. Imagine the visitors who try to visit the site on 3G, or from mediocre internet speeds.
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

02 Jan 2016, 21:01

All fixed! The only images that were that large were the main slideshow. All others were re-sized. I also enabled CDN and pre-load site. Mobile performance is much better! Thanks
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

06 Jan 2016, 00:46

I just crawled another domain I have X3 installed on... www.39thavenueapartments.com and am getting 7 404 errors. One example is...

From: http://www.39thavenueapartments.com/
To: http://www.39thavenueapartments.com/39th-02/

I installed the same fix to this domain as you recommended for www.merelyafleshwound.com
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: 404 Errors

06 Jan 2016, 02:00

Where can I find that page in the menu? I need to see that this is a valid page ... if not then obviously it will return 404 page not found.
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

06 Jan 2016, 02:21

I don't think it is a page... it is an image in the slideshow minus the .JPG
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13997
Joined: 30 Sep 2006, 03:37

Re: 404 Errors

06 Jan 2016, 02:53

alexhenes wrote:I don't think it is a page... it is an image in the slideshow minus the .JPG
Are you using the updated file /app/stacey.inc.php as noted earlier in this thread? I am not 100% sure what version we are dealing with here to be honest. Can you try to rename the file to *.jpg (lowercase)? This will allow us to know exactly what the issue is.
 
User avatar
alexhenes
Experienced
Topic Author
Posts: 568
Joined: 28 Sep 2006, 16:13

Re: 404 Errors

06 Jan 2016, 10:35

images renamed to *.jpg (lower case). Still seeing the same 404 errors in the crawl. All errors stem from the www.39thavenueapartments.com and point to a directory with the name of the images in the slideshow minus to .jpg
Alex
https://www.merelyafleshwound.com
https://www.goldenbikeshop.com