Why does Googlebot attempt to crawl /admin/install.php?

Question

On one site I own, I recently started seeing Googlebot checking for non-existing URIs:

66.249.76.89 - - [23/Feb/2020:10:18:48 +0100] "GET /robots.txt HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
66.249.76.87 - - [23/Feb/2020:10:18:49 +0100] "GET /admin/install.php HTTP/1.1" 404 181 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

This would all be well and good, if not for the fact that it has never done so before, this URI never existed (I own the domain for 10+ years) and looks suspiciously like casually scanning for possible security issues.

89.76.249.66.in-addr.arpa domain name pointer crawl-66-249-76-89.googlebot.com. is also indeed a Googlebot address.

Can anyone shed more light on this?

Evgeniy · Answer

For other sites (as for me) this is a kind of pretty usual behavior of Googlebot - to crawl non-existing pages.

But don't worry - it will stop it, definitely, with the time. Just care about your non-existing pages have correct status code (404, 410).

Why Googlebot does it? There are some causes:

It finds something in your javascripts, what looks similar to paths - and tries to crawl them to maybe find some more (~truffles~) content, greedy pig.
It tries to get new content from adding different paths and file names together.

Why does Googlebot attempt to crawl /admin/install.php?

One Answer

Add your own answers!

Ask a Question