View Single Post
Old 10-12-2017, 06:58 PM  
Barry-xlovecam
It's 42
 
Industry Role:
Join Date: Jun 2010
Location: Global
Posts: 18,083
Your problem is your PHP script and the MySQL daemon (server). Software for your application; or,
If you look at the times of the peak usage and grep those times in the server access logs you may find that bing is indexing too many pages too fast -- you can place a directive in the robots.txt
User-agent: bingbot
Crawl-delay:$v
5
10

see
https://www.siteground.com/kb/how_to..._eng ine_bot/
https://www.bing.com/webmaster/help/...ntrol-55a30302

Slow bing down -- don't Disallow Bing they bring good converting traffic and the sell their PSaaS or indexed database to Yahoo and other search engines.

You may find Baidu is indexing too many pages too fast -- block them at your firewall I have had luck that way
Porn is illegal in China and you won't sell legit Chinese buyers either.
# Free IP2Location Firewall List by Search Engine
# Source: Whitelist Robots by Search Engine | IP2Location

Code:
whois -h v4.whois.cymru.com " -c -p 183.131.32.0/20"
AS      | IP               | BGP Prefix          | CC | AS Name
4134    | 183.131.32.0     | 183.128.0.0/11      | CN | CHINANET-BACKBONE No.31,Jin-rong Street, CN
 -c -p 12.1.72.32/27"
7018    | 12.1.72.32       | 12.0.0.0/9          | US | ATT-INTERNET4 - AT&T Services, Inc., US
" -c -p 104.193.88.0/22"
55967   | 104.193.88.0     | 104.193.88.0/24     | US | CNNIC-BAIDU-AP Beijing Baidu Netcom Science and Technology Co., Ltd., CN
or use Ruby gem nicinfo
https://github.com/arineng/nicinfo

that will give you full RDAP/whois information

Third way is just $ whois <ip address>

If you are generating many dynamic pages search engines may be causing this problem

Scrapers and *bad bots* may be the issue too.

This is what server logs are for to search for problems and find patterns.
A firewall is the way to go -- just do not answer -- drop the packet.
Barry-xlovecam is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote