Bots/Cheating
 

So, I get this email today from a trade partner saying "you send me a lot of bots".

How do I detect bots? How do I get rid of them so they don't get passed on to my trades?

Any help is appreciated. I'm using TE3 btw...

At the bottom of each of my PHP pages, I have a few lines of code that saves "hits" to a log file.

$fp = fopen('name of the file you want to save.txt', 'a');
$FWwritethis = $date("YmdHis").', '. $_SERVER['PHP_SELF'].', '.$_SERVER["REMOTE_ADDR"].', '.$_SERVER['HTTP_REFERER'].', '.$_SERVER['QUERY_STRING'].'^ '.$_SERVER['HTTP_USER_AGENT'].PHP_EOL;

fwrite($fp, $FWwritethis);
fclose($fp);

I then read that file with a script, that looks for keywords I have picked out, for "Bots". The file is TEXT, so you can also just open it and look it over! Eyeball it!

Most of my sites are getting about 55% BOT traffic!

Here's a few lines from that file: (The first entry shows that bing likes to read gfy forum posts!) You can see some of the different search engine spider/crawler names and IPs.

20160601113239, /index.php, 157.55.39.212, , q=gfy^ Mozilla/5.0 (compatible; bingbot/2.0; +Meet our crawlers) |
20160601113240, /TheBritishBeef.php, 157.55.39.212, , pg=1^ Mozilla/5.0 (compatible; bingbot/2.0; +Meet our crawlers) |
20160601113258, /TheBritishBeefBooks.php, 157.55.39.227, , ^ Mozilla/5.0 (compatible; bingbot/2.0; +Meet our crawlers) |
20160601113258, /MeetThem.php, 157.55.39.227, , pg=2^ Mozilla/5.0 (compatible; bingbot/2.0; +Meet our crawlers) |
20160601113308, /index.php, 157.55.39.122, , q=ygroup^ Mozilla/5.0 (compatible; bingbot/2.0; +Meet our crawlers) |
20160601121733, /TheBritishBeef.php, 66.249.69.162, , pg=4^ Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
20160601122856, /TheBritishBeefBooks.php, 66.249.69.156, , ^ Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
20160601131022, /TheBritishBeef.php, 66.249.69.150, , pg=2^ Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
20160601134745, /TheBritishBeef.php, 66.249.69.162, , pg=3^ Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
20160601135957, /TheBritishBeef.php, 66.249.69.162, , ^ Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
20160601141145, /TheBritishBeefBooks.php, 92.145.71.253, http://images.google.fr/imgres?imgur...2Fthe-british-
20160614222010, /index.php, 180.76.15.19, , ^ Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) |
20160614222029, /index.php, 180.76.15.23, , ^ Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) |
20160603020924, /index.php, 68.180.228.233, , ^ Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
20160603015205, /MeetThem.php, 5.9.73.227, , pg=1^ Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+) |
20160603015208, /index.php, 5.9.73.227, , q=ygroup^ Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+) |

The issue of BOT traffic has been raised for years, It is a factor to consider when buying traffic. It is a factor others consider when traffic is sold. Is a Million hits a Million hits, of a half Million hits?

If you want more info about your hits, just look at the manual page PHP: $_SERVER - Manual

When someone buys something, like a DVD or Digital Download, I can trace his or her IP and their steps on my site, to see how long they spent, what they looked at that they bought or didn't buy, even if they visited other sites of mine.