At the bottom of each of my PHP pages, I have a few lines of code that saves "hits" to a log file.
$fp = fopen('name of the file you want to save.txt', 'a');
$FWwritethis = $date("YmdHis").', '. $_SERVER['PHP_SELF'].', '.$_SERVER["REMOTE_ADDR"].', '.$_SERVER['HTTP_REFERER'].', '.$_SERVER['QUERY_STRING'].'^ '.$_SERVER['HTTP_USER_AGENT'].PHP_EOL;
fwrite($fp, $FWwritethis);
fclose($fp);
I then read that file with a script, that looks for keywords I have picked out, for "Bots". The file is TEXT, so you can also just open it and look it over! Eyeball it!
Most of my sites are getting about 55% BOT traffic!
Here's a few lines from that file: (The first entry shows that bing likes to read gfy forum posts!) You can see some of the different search engine spider/crawler names and IPs.
20160601113239, /index.php, 157.55.39.212, , q=gfy^ Mozilla/5.0 (compatible; bingbot/2.0; +
Meet our crawlers) |
20160601113240, /TheBritishBeef.php, 157.55.39.212, , pg=1^ Mozilla/5.0 (compatible; bingbot/2.0; +
Meet our crawlers) |
20160601113258, /TheBritishBeefBooks.php, 157.55.39.227, , ^ Mozilla/5.0 (compatible; bingbot/2.0; +
Meet our crawlers) |
20160601113258, /MeetThem.php, 157.55.39.227, , pg=2^ Mozilla/5.0 (compatible; bingbot/2.0; +
Meet our crawlers) |
20160601113308, /index.php, 157.55.39.122, , q=ygroup^ Mozilla/5.0 (compatible; bingbot/2.0; +
Meet our crawlers) |
20160601121733, /TheBritishBeef.php, 66.249.69.162, , pg=4^ Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html) |
20160601122856, /TheBritishBeefBooks.php, 66.249.69.156, , ^ Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html) |
20160601131022, /TheBritishBeef.php, 66.249.69.150, , pg=2^ Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html) |
20160601134745, /TheBritishBeef.php, 66.249.69.162, , pg=3^ Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html) |
20160601135957, /TheBritishBeef.php, 66.249.69.162, , ^ Mozilla/5.0 (compatible; Googlebot/2.1; +
http://www.google.com/bot.html) |
20160601141145, /TheBritishBeefBooks.php, 92.145.71.253,
http://images.google.fr/imgres?imgur...2Fthe-british-
20160614222010, /index.php, 180.76.15.19, , ^ Mozilla/5.0 (compatible; Baiduspider/2.0; +
http://www.baidu.com/search/spider.html) |
20160614222029, /index.php, 180.76.15.23, , ^ Mozilla/5.0 (compatible; Baiduspider/2.0; +
http://www.baidu.com/search/spider.html) |
20160603020924, /index.php, 68.180.228.233, , ^ Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp) |
20160603015205, /MeetThem.php, 5.9.73.227, , pg=1^ Mozilla/5.0 (compatible; MJ12bot/v1.4.5;
http://www.majestic12.co.uk/bot.php?+) |
20160603015208, /index.php, 5.9.73.227, , q=ygroup^ Mozilla/5.0 (compatible; MJ12bot/v1.4.5;
http://www.majestic12.co.uk/bot.php?+) |