.Google.com's John Mueller addressed an inquiry regarding why Google marks webpages that are actually disallowed from crawling by robots.txt as well as why the it's secure to disregard the relevant Browse Console records regarding those crawls.Crawler Traffic To Concern Parameter URLs.The individual asking the inquiry chronicled that crawlers were actually developing links to non-existent query criterion Links (? q= xyz) to web pages with noindex meta tags that are additionally shut out in robots.txt. What caused the question is that Google is crawling the links to those pages, obtaining blocked out through robots.txt (without seeing a noindex robotics meta tag) then acquiring turned up in Google.com Search Console as "Indexed, though blocked by robots.txt.".The individual inquired the following question:." However listed below's the large inquiry: why would certainly Google mark pages when they can not also view the web content? What is actually the perk in that?".Google.com's John Mueller verified that if they can't creep the webpage they can't find the noindex meta tag. He likewise produces an exciting reference of the internet site: hunt operator, encouraging to ignore the end results considering that the "typical" users will not observe those results.He composed:." Yes, you're right: if our company can't crawl the page, our experts can not see the noindex. That mentioned, if our team can't crawl the pages, then there's certainly not a lot for our team to mark. Thus while you may observe several of those webpages with a targeted website:- question, the common individual will not find them, so I wouldn't fuss over it. Noindex is actually additionally great (without robots.txt disallow), it just implies the URLs will certainly end up being actually crawled (and find yourself in the Browse Console report for crawled/not listed-- neither of these standings create concerns to the rest of the internet site). The integral part is actually that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's response validates the constraints being used the Internet site: search evolved hunt driver for diagnostic causes. Some of those causes is actually since it's certainly not attached to the regular hunt index, it is actually a separate trait altogether.Google's John Mueller talked about the internet site search operator in 2021:." The quick solution is actually that an internet site: question is certainly not implied to become comprehensive, nor made use of for diagnostics functions.An internet site question is actually a particular type of hunt that limits the outcomes to a particular site. It is actually generally merely the word internet site, a colon, and then the website's domain name.This concern restricts the results to a details website. It is actually not suggested to be an extensive compilation of all the webpages from that site.".2. Noindex tag without using a robots.txt is alright for these kinds of scenarios where a bot is actually connecting to non-existent web pages that are actually getting found through Googlebot.3. Links along with the noindex tag will definitely create a "crawled/not recorded" entry in Explore Console and also those will not have an adverse effect on the rest of the website.Check out the inquiry and also answer on LinkedIn:.Why would Google index pages when they can't even observe the material?Included Photo through Shutterstock/Krakenimages. com.