Bot check script.

[Torniquet]

Here is a nice little script i found which tells you when an important bot/spider/crawler visits your site.

Code: Select all

    $botlist = array(   
                "Teoma",                   
                "alexa",
                "froogle",
                "inktomi",
                "looksmart",
                "URL_Spider_SQL",
                "Firefly",
                "NationalDirectory",
                "Ask Jeeves",
                "TECNOSEEK",
                "InfoSeek",
                "WebFindBot",
                "girafabot",
                "crawler",
                "www.galaxy.com",
                "Googlebot",
                "Scooter",
                "Slurp",
                "appie",
                "FAST",
                "WebBug",
                "Spade",
                "ZyBorg",
                "rabaz");


    foreach($botlist as $bot) {

      if(ereg($bot, $HTTP_USER_AGENT)) {

          if($bot == "Googlebot") {
            if (substr($REMOTE_HOST, 0, 11) == "216.239.46.") $bot = "Googlebot Deep Crawl";
            elseif (substr($REMOTE_HOST, 0,7) == "64.68.8") $bot = "Google Freshbot";
          }
          if ($QUERY_STRING != "") {
            $url = "http://" . $SERVER_NAME . $PHP_SELF . "?" . $QUERY_STRING . "";
          } else {
            $url = "http://" . $SERVER_NAME . $PHP_SELF . "";
          }

// settings
$to = "torniquet@project-blackwidow.com";
$subject = "Detected: $bot on $url";
$body = "$bot was deteched on $url\n\n
Date.............: " . date("F j, Y, g:i a") . "
Page.............: " . $url . "
Robot Name.......: " . $HTTP_USER_AGENT . "
Robot Address....: " . $REMOTE_ADDR . "
Robot Host.......: " . $REMOTE_HOST . "
";

mail($to, $subject, $body);

      }

    }

It does work, it informed me today i had googlebot visit twice and slurp bot visit once.

you could change it to store into a database easy enough.

if anyone knows of anymore bots, pleas mention them so the list can grow for everone

[PaxBritannia]

The problem with relying on their user-agent to determine their origin is that some malicious bots will fake their identity.

I normally adopt a more agressive stance to bots because I prefer my sites to fall under the radar and not appear in search engines. And since most spider bots are invasive and don't obey the no-index no-follow tags, I've devised several little "bot traps" to counter against them.

pax.