Search…

X3 Photo Gallery Support Forums

Search…
 
User avatar
Dane
Experienced
Topic Author
Posts: 121
Joined: 04 Dec 2020, 15:05

Problem with google indexing

23 Dec 2020, 02:21

Hi!

I received an e-mail from Google saying "Coverage problems detected on linsenschuss.de". Do I still have to make special settings so that Google can index the page without problems? Do I still need to edit the robots.txt specifically?
2020-12-23 08_14_05-E-Mail - Dane Vetter - Webmailer Pro.jpg
2020-12-23 08_14_05-E-Mail - Dane Vetter - Webmailer Pro.jpg (54.03 KiB) Viewed 85490 times
 
User avatar
Dane
Experienced
Topic Author
Posts: 121
Joined: 04 Dec 2020, 15:05

Re: Problem with google indexing

23 Dec 2020, 02:23

I found this:

Supposedly the crawling is blocked by the robots.txt?
2020-12-23 08_22_58-URL-Prüfung.jpg
2020-12-23 08_22_58-URL-Prüfung.jpg (44.1 KiB) Viewed 85488 times
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Problem with google indexing

23 Dec 2020, 04:11

What website is that for? Link please ... You recently removed the website from "dane-vetter.com", so if it's for that site, it would be expected. I checked your other website linsenschuss.de and the file linsenschuss.de/robots.txt is NOT blocked, unless you have something in your server that is blocking Google bot (X3 does not do that).

No, you do not need to edit robots.txt or change any special settings.

You can also check spider simulator below, nothing reported:
https://smallseotools.com/spider-simulator/

I don't know what Google webmaster is sending you, but it sends lots of junk. I don't read German, but it seems like it says "can't access robots.txt"? Perhaps your website was down? If it is blocked, only your server can do that, but I doubt that very much. X3 doesn't do anything to block your website, and Google doesn't require any settings to index your website. 
 
User avatar
Dane
Experienced
Topic Author
Posts: 121
Joined: 04 Dec 2020, 15:05

Re: Problem with google indexing

23 Dec 2020, 14:21

No, its not for dane-vetter.com, its for linsenschuss.de. The message is: Indexed, although blocked by robots.txt.

But its good to know that i dont need to change any settings and that the robots.txt is correctly. So i ask my hoster where is the problem. 

Thank you!
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Problem with google indexing

24 Dec 2020, 00:16

Dane wrote:No, its not for dane-vetter.com, its for linsenschuss.de. The message is: Indexed, although blocked by robots.txt.
So the question is, what exactly are the url's being blocked that should not be blocked? X3 DOES block some url's which should NOT be accessed and indexed by Google. For example when Google tries to access "pages" within the /content/ dir.
Dane wrote:But its good to know that i dont need to change any settings and that the robots.txt is correctly. So i ask my hoster where is the problem. 
First please figure out if it is a problem, which I am almost 100% sure it is not. All url's blocked by X3's "robots.txt" are meant to be blocked. Why else would I add them?

The entire point of robots.txt is to tell Google to NOT index some url's. And that's what's happening. If google wants to report that to you by email (yes I also receive lots of emails like this), then that's fine. But it's not a problem unless it's a wrong.
 
User avatar
thinkingpixels
Posts: 14
Joined: 06 Jul 2016, 18:27

Re: Problem with google indexing

07 Jan 2024, 21:33

Hi,
My new gallery umbraldreams.com went live three days ago and yesterday morning I received the same message from search console that Dane mentioned four years ago.
I created an image sitemap because I want Google to index all my 1000+ images. But search console reports that 1000+ entries (which are all the image pages) from the sitemap.xml could be indexed because of restrictions in the robots.txt file.

This is what my robots.txt file looks like:
User-agent: *
Allow: /
Disallow: /app/
Disallow: /panel/
Disallow: /content*/$

I suspect that the image urls from the sitemap.xml somehow get interpreted as ending with a "/" (weird server config of my hoster or some weird Google weirdness?) and therefore get disallowed...?

Will adding this line fix it?
Allow: /content/*/*.jpg$

But then again, maybe it should be
Allow: /content/*/*.jpg/$

I'm not quite sure what's going on. All I can say for sure is that Google does not want to index my image pages without changes to the robots.txt files. Any suggestions?

Cheers,
André
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Problem with google indexing

07 Jan 2024, 22:55

That's interesting, but wasn't there any link/response from the search console that explicitly told you why a specific page/url couldn't be indexed?

There is no logical reason why your robots.txt is blocking Google from accessing neither your images, or the image landing pages that contain the image (or even the gallery where the images are contained for that matter). There is no /content/ path segment in your image landing pages, and your direct image urls definitely do not match /content*/$. Already tested this on https://logeix.com/tools/robots-txt, for this image on this image landing page.

Image

Image

Google search console does report many weird things, but surely there must be a better explanation than this available.
 
User avatar
mjau-mjau
X3 Wizard
Posts: 13993
Joined: 30 Sep 2006, 03:37

Re: Problem with google indexing

07 Jan 2024, 23:00

Even without your robots.txt modifications, indexing should be allowed by all tests. Therefore, I can't really comment on your modification.