Page 1 of 1

Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 05:38
by roseline
Hello Dears,

I have strange URL errors and I would like to know if all x3 users have the same.

When I go under Crawl > Crawl Errors
I see 26 Access denied.
When I see URL (see screen copy) it's "internal" URL (I mean it's folders I have on the serveur and it's my panel menu).
Capture-d’écran-2016-06-10-à-11.28.19.jpg
Google Search Console URL crawl errors
Capture-d’écran-2016-06-10-à-11.28.19.jpg (178.85 KiB) Viewed 7258 times
Of course it's a 403 error and this is normal as it's not the "public" URL !

I don't understand how Google "find" those URL and - for me - it's a bug.

Important to know
- sitemap is fine > http://www.ozimages.ch/sitemap/
- robots.txt is fine > http://www.ozimages.ch/robots.txt

So I would like to know if somebody else have the same errors....

Thank you in advance
:-)

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 10:38
by mjau-mjau
roseline wrote:When I go under Crawl > Crawl Errors
I see 26 Access denied.
When I see URL (see screen copy) it's "internal" URL (I mean it's folders I have on the serveur and it's my panel menu).
The only part of X3 that creates URL's like this, is from the panel manage section: For example, the href (link) may be "../content/2.galleries/1.landscapes", although the click is intercepted by javascript so that it loads DATA from that path into the panel. It does NOT navigate to the specified link, as it is not a valid URL that can be accessed.

HOW Google webmaster is even able to detect these links, I don't know. I can only guess that you are using some kinda global google analytics? Or some other webmaster tracking added to panel? This would allow Google to detect links behind a protected area. Unless you have some tracking system from within the panel, then bots (including Google) cannot access your panel.

I checked on our website mjau-mjau.com which uses X3, and it doesn't report any crawl errors:
Image

Finally though, I would ask: Is there any reason you can't just ignore the crawl errors? Google webmaster is just reporting everything, regardless of the importance. These links it "finds" are not real links, and are not supposed to be indexed ... so if Google finds them and sees they go "nowhere", then that is not really a problem. Just for the record.

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 10:49
by roseline
Hi Karl,

Thank you for your feedback :-)

Of course I can ignore this as - as you mention - that don't have "real" effects (sorry for my very bad English).
But I don't like to see errors :lol:
We never know when Google will do a "google dance" and perhaps once it will be some importance in the ranking...

Yes, I use Google Analytics.
But I already do this in x2 and I never had this kind of URL (I know x2 and x3 are totally different).
This is why I ask ;-)

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 12:26
by mjau-mjau
Generally speaking, I don't know how Google is seeing these links, but one can assume they are from the panel manager menu. These links are not links to pages though ...
roseline wrote:Yes, I use Google Analytics.
My question was more, HOW have you added Analytics? Google Analytics isn't by default added to the panel. In your case, it seems Google can read from behind the panel login somehow.
roseline wrote:But I already do this in x2 and I never had this kind of URL (I know x2 and x3 are totally different).
Yep. As mentioned, this is only Google reporting broken links in the panel that are not supposed to be traditional links.
roseline wrote:But I don't like to see errors :lol:
We never know when Google will do a "google dance" and perhaps once it will be some importance in the ranking...
Of course not, but truth is, nothing is perfect, including Google webmaster reporting. ALL x3 galleries have panel manage menu's with these links, and they are not supposed to be followed, indexed, or even seen by anybody (or any bots), except admin user, in the first place.

I will look into fixing this in next release, although it would only be a "cosmetic" adjustment so that Google doesn't mistake the panel menui links for links. Google shouldn't be seeing your panel in the first place ...

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 12:53
by roseline
[quote="mjau-mjau"]My question was more, HOW have you added Analytics? Google Analytics isn't by default added to the panel. In your case, it seems Google can read from behind the panel login somehow.

I just put the tracking code in the panel under Settings > Accounts
Capture d’écran 2016-06-10 à 18.41.52.png
Settings > Accounts
Capture d’écran 2016-06-10 à 18.41.52.png (40.14 KiB) Viewed 7243 times
Thank you Karl for your real good job on v 0.18 and the new Newsletter how is very usfull :-)

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 13:01
by mjau-mjau
That would be the correct way to add Google Analytics, and we do not add analytics to the panel. Unfortunately, that also means I simply don't know how/why/where Google webmaster is detecting these links.

Is there no information about where the broken links are?

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 13:15
by roseline
mjau-mjau wrote: Is there no information about where the broken links are?
Absolutely nothing :-(
I have create this afternoon a post on Google forum but without success for the moment. BUT it's the French forum... so it will be perhaps a good idea to do the same in English ;-) Could you do it ?

Have a nice weekend :-)

Re: Who use Google Search Console (Webmaster tools)?

Posted: 10 Jun 2016, 16:57
by mjau-mjau
roseline wrote:BUT it's the French forum... so it will be perhaps a good idea to do the same in English ;-) Could you do it ?
Sorry, I have to say no ... I would need links and logins to your panel. Furthermore, this is not a bug or an issue in any sense. Just because these links exist in HREF tags in your panel, and just because Google has managed to sniff them out, doesn't mean anything else than exactly what google reports: The links are not valid links.

These links are not part of anything your human visitors can access, and will not be indexed in search.

Re: Who use Google Search Console (Webmaster tools)?

Posted: 26 Jul 2018, 18:13
by elpd
Hi guys,

It's two year later....and today i discovered the exact same issue in my Search Console.
@roseline do you still have these crawl errors or did you found the cause? I can't unfortunately.

Thanks, Erik

Re: Who use Google Search Console (Webmaster tools)?

Posted: 27 Jul 2018, 02:04
by mjau-mjau
Ok let me explain what is happening here, why it is harmless, and what the solution is (even though not required). I recently came across this for our own X3 websites, so I did some digging some time ago.

Let's look at what Google search console is reporting:
https://www.flamepix.com/content/5.faq
Image

The above is entirely CORRECT by Google of course. The url /content/5.faq is NOT a website page and should NOT be indexed. When accessed, it correctly returns an error "404 page not found".
Image

The url is actually a folder, and contains the data that creates the page www.flamepix.com/faq/ and images on that page.


Ok, but why is Google listing these pages in my search console under crawl errors?
That is a good question, but first of all, let me be 100% crystal clear here: X3 does NOT link to these 404 URL's under any circumstance. I believe Google is finding these folders from IMAGE files located in the folders, and simply thinking "ok if there is an image /content/5.faq/image.jpg, there must be an URL /content/5.faq/". That is correct of course, but that folder is not a page, something Google will learn when it tries to access it.

Furthermore, I would like to emphasize what Google says when you CLICK any specific crawl error like this:
Google search console wrote:Generally, 404s don't harm your site's performance in search, but you can use them to help improve the user experience.
With the above, I would just like to emphasize that this does not mean there is anything wrong with your website or it will hurt your performance. Google search console will seemingly try to browse ANYTHING it might consider an URL, and sometimes report them as crawl errors regardless. Doesn't mean you have to fix anything. In this case, "you can use them to help improve user experience" ... Ok, but keep in mind, no users are actually viewing those pages ... So thanks for reporting Google, but this is not a problem.


X3.25.0
Since I found this annoying myself, we have solved it in X3.25.0 (released yesterday, available from Panel > Tools > X3 updates) by adding a rule to robot.txt.
X3.25.0 Release wrote:Robots.txt

For some reason, Google Bot attempts to crawl parent folders of images (for example “/content/folder/image.jpg”), falsely assuming there is a PAGE at “/content/folder/“. X3 content folders are NOT pages, so it will only find a “403 Forbidden”. We have added a rule Disallow: /content*/$ in robots.txt to prevent Google bot from crawling content folders, without blocking access to files.

Re: Who use Google Search Console (Webmaster tools)?

Posted: 29 Jul 2018, 10:46
by elpd
I upgrade to X3.25.0 Beta today so this issue is solved, thanks!