.Google's John Mueller addressed a question regarding why Google.com indexes webpages that are actually forbidden from creeping through robots.txt and why the it's risk-free to overlook the relevant Look Console records regarding those creeps.Robot Visitor Traffic To Question Criterion URLs.The person asking the concern documented that bots were making hyperlinks to non-existent concern criterion URLs (? q= xyz) to pages along with noindex meta tags that are likewise blocked in robots.txt. What motivated the inquiry is that Google.com is creeping the hyperlinks to those webpages, getting obstructed by robots.txt (without noticing a noindex robots meta tag) at that point acquiring shown up in Google Explore Console as "Indexed, though blocked out through robots.txt.".The person inquired the complying with concern:." But below is actually the big inquiry: why would certainly Google.com index web pages when they can not even view the material? What's the advantage in that?".Google.com's John Mueller verified that if they can not crawl the webpage they can not view the noindex meta tag. He additionally produces a fascinating acknowledgment of the internet site: search driver, urging to overlook the results due to the fact that the "normal" individuals will not observe those outcomes.He composed:." Yes, you're correct: if our company can't crawl the page, we can't view the noindex. That mentioned, if our team can't crawl the web pages, after that there's not a whole lot for our team to index. So while you could observe several of those webpages with a targeted site:- question, the ordinary individual will not see all of them, so I wouldn't bother it. Noindex is likewise alright (without robots.txt disallow), it just means the Links will certainly find yourself being crawled (and end up in the Search Console document for crawled/not indexed-- neither of these statuses lead to concerns to the remainder of the web site). The vital part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the restrictions in operation the Website: hunt progressed hunt operator for analysis explanations. Some of those factors is actually since it is actually certainly not connected to the regular search mark, it is actually a different thing completely.Google.com's John Mueller commented on the internet site search operator in 2021:." The quick answer is actually that a web site: inquiry is certainly not suggested to become comprehensive, nor utilized for diagnostics functions.An internet site inquiry is a particular type of search that confines the outcomes to a certain web site. It's basically merely the word web site, a bowel, and then the web site's domain name.This query confines the end results to a specific website. It's not indicated to become a detailed collection of all the webpages coming from that internet site.".2. Noindex tag without making use of a robots.txt is actually great for these type of circumstances where a bot is actually linking to non-existent web pages that are getting found out by Googlebot.3. URLs along with the noindex tag will certainly create a "crawled/not indexed" entry in Look Console and that those won't possess a bad result on the remainder of the website.Check out the concern and also respond to on LinkedIn:.Why would certainly Google index web pages when they can't even find the web content?Featured Picture by Shutterstock/Krakenimages. com.