GoogleOther: Google introduces new web crawler to streamline various tasks
GoogleOther can be monitored in various ways including tracking website performance and analyzing server logs in Google Search Console.
Google has introduced a new web crawler- GoogleOther in order to streamline various tasks and reduce the burden on Googlebot. Google’s main crawler is called Googlebot. Basically the primary job of the GoogleOther is to handle non-essential tasks like R&D crawls. This means that the Googlebot will now focus mostly on building the search index.
The new crawler is also expected to help Google optimize and streamline its crawling operations. For those who are unknown, the function of Web crawlers (otherwise known as robots or spiders) is to automatically discover and scan websites. GoogleOther can be used by various product teams in order to get publicly accessible content from various websites.
Google Search Analyst Gary Illyes has shared more details about GoogleOther in his latest LinkedIn post.
“We added a new crawler, GoogleOther to our list of crawlers that ultimately will take some strain off of Googlebot. This is a no-op change for you, but it’s interesting nonetheless I reckon.
As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot’s crawl jobs are only used internally for building the index that’s used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot’s other jobs like R&D crawls to free up some crawl capacity for Googlebot.
The new crawler uses the same infrastructure as Googlebot and so it has the same limitations and features as Googlebot: hostload limitations, robotstxt (though different user agent token), http protocol version, fetch size, you name it. It’s basically Googlebot under a different name, “said Gary IIyes.
The uses who run their own website or work for a website need not worry about the introduction of GoogleOther. This is because GoogleOther works using the same infrastructure and parameters as the Googlebot.
GoogleOther can be monitored in various ways including tracking website performance, analyzing server logs and monitoring crawl stats in Google Search Console.
Ms Kalinga