GoFuckYourself.com - Adult Webmaster Forum - View Single Post

Barry-xlovecam · 10-12-2017, 06:58 PM

Your problem is your PHP script and the MySQL daemon (server). Software for your application; or,
If you look at the times of the peak usage and grep those times in the server access logs you may find that bing is indexing too many pages too fast -- you can place a directive in the robots.txt
User-agent: bingbot
Crawl-delay:$v
5
10

see
https://www.siteground.com/kb/how_to..._eng ine_bot/
https://www.bing.com/webmaster/help/...ntrol-55a30302

Slow bing down -- don't Disallow Bing they bring good converting traffic and the sell their PSaaS or indexed database to Yahoo and other search engines.

You may find Baidu is indexing too many pages too fast -- block them at your firewall I have had luck that way
Porn is illegal in China and you won't sell legit Chinese buyers either.
# Free IP2Location Firewall List by Search Engine
# Source: Whitelist Robots by Search Engine | IP2Location

Code:

whois -h v4.whois.cymru.com " -c -p 183.131.32.0/20"
AS      | IP               | BGP Prefix          | CC | AS Name
4134    | 183.131.32.0     | 183.128.0.0/11      | CN | CHINANET-BACKBONE No.31,Jin-rong Street, CN
 -c -p 12.1.72.32/27"
7018    | 12.1.72.32       | 12.0.0.0/9          | US | ATT-INTERNET4 - AT&T Services, Inc., US
" -c -p 104.193.88.0/22"
55967   | 104.193.88.0     | 104.193.88.0/24     | US | CNNIC-BAIDU-AP Beijing Baidu Netcom Science and Technology Co., Ltd., CN

or use Ruby gem nicinfo
https://github.com/arineng/nicinfo

that will give you full RDAP/whois information

Third way is just $ whois <ip address>

If you are generating many dynamic pages search engines may be causing this problem

Scrapers and *bad bots* may be the issue too.

This is what server logs are for to search for problems and find patterns.
A firewall is the way to go -- just do not answer -- drop the packet.

10-12-2017, 06:58 PM
Barry-xlovecam It's 42 Industry Role: Join Date: Jun 2010 Location: Global Posts: 18,083	Your problem is your PHP script and the MySQL daemon (server). Software for your application; or, If you look at the times of the peak usage and grep those times in the server access logs you may find that bing is indexing too many pages too fast -- you can place a directive in the robots.txt User-agent: bingbot Crawl-delay:$v 5 10 see https://www.siteground.com/kb/how_to..._eng ine_bot/ https://www.bing.com/webmaster/help/...ntrol-55a30302 Slow bing down -- don't Disallow Bing they bring good converting traffic and the sell their PSaaS or indexed database to Yahoo and other search engines. You may find Baidu is indexing too many pages too fast -- block them at your firewall I have had luck that way Porn is illegal in China and you won't sell legit Chinese buyers either. # Free IP2Location Firewall List by Search Engine # Source: Whitelist Robots by Search Engine \| IP2Location Code: whois -h v4.whois.cymru.com " -c -p 183.131.32.0/20" AS \| IP \| BGP Prefix \| CC \| AS Name 4134 \| 183.131.32.0 \| 183.128.0.0/11 \| CN \| CHINANET-BACKBONE No.31,Jin-rong Street, CN -c -p 12.1.72.32/27" 7018 \| 12.1.72.32 \| 12.0.0.0/9 \| US \| ATT-INTERNET4 - AT&T Services, Inc., US " -c -p 104.193.88.0/22" 55967 \| 104.193.88.0 \| 104.193.88.0/24 \| US \| CNNIC-BAIDU-AP Beijing Baidu Netcom Science and Technology Co., Ltd., CN or use Ruby gem nicinfo https://github.com/arineng/nicinfo that will give you full RDAP/whois information Third way is just $ whois <ip address> If you are generating many dynamic pages search engines may be causing this problem Scrapers and bad bots may be the issue too. This is what server logs are for to search for problems and find patterns. A firewall is the way to go -- just do not answer -- drop the packet.