Abusive Hits Per Day

[barelist]

I've been noticing some really high hits from certain ips over the last few weeks.

Mainly from Germany, Russia, and China... in 24hrs I had two ips hit 74,181 and 18,696 hits.

I've writen an app before which will block IPs based on hits in a certain timeframe, but not for adult.

What would you consider excessive hits from one IP in a 24hr period?

[RayBonga]

any idea why this is happening?

[barelist]

Originally Posted by RayBonga

any idea why this is happening?

No idea... I noticed a 20mbps increase in bandwidth... I knew something was up.

[SmokeyTheBear]

isolate what they are hitting first.

[barelist]

At that level of hits it doesn't matter to me. If it's not a search engine spider nothing needs to hit the site near that many times.

If a valid surfer is blocked, I'll redirect blocked IPs to a friendly error page with contact info if they think they've been blocked by mistake.

[EdgeXXX]

Normally I would say they are trying to run U:P lists against your site, but seeing as you have free registration that wouldn't make a whole lot of sense.

[barelist]

I have a lot of buffer in terms of bandwidth, so that's not a problem yet... mainly it's just wasted bw and it's slowing my page down. After speeding a few weeks on page speed that's pretty annoying :/

[DateDoc]

If you host any of the videos on your site see which ones are being hotlinked.

[1200mics]

Why would they do that ?

[barelist]

Here's a few of the IPs I've noticed hammering the site the last few days:
61.145.136.114
77.88.26.27
79.203.75.91
88.64.52.20
125.85.15.25
221.174.16.60

[woj]

like smokey said you need to figure out what they are hitting first, look in the log files..

[mkx]

they are probably jacking your website content to make their own

[barelist]

Originally Posted by woj

like smokey said you need to figure out what they are hitting first, look in the log files..

Looks like web scrapers... pulling a bunch of .aspx and .html pages without any content on them.

Then all of a sudden there will be a bunch of thumbs with no page opened...

[barelist]

Wrote an app to discourage abuse and block IPs... I'm just not sure if I'm being too liberal or too strict. I guess I'll find out soon enough.

Here's a graph of where my bandwidth was spiking 15-30mbps over the last week or so:

[SmellyNose]

What language? Is it a cronjob to check your access log sort of like:

cat /var/log/apache2/access.log | awk '{print $1}' | wc -l

I will be looking at implementing something similar soon so would be interested in your thoughts on your current setup.

[barelist]

Originally Posted by AdultStoriesNow

What language? Is it a cronjob to check your access log sort of like:

cat /var/log/apache2/access.log | awk '{print $1}' | wc -l

I will be looking at implementing something similar soon so would be interested in your thoughts on your current setup.

.net

I'm grabbing low level ip hits (no client needed) over all my sites and if hits exceed a certain number over a set interval it's flagged as abuse and the app blocks the IP in IIS.

Had my wife and myself browse the site pretty heavy, but stopping like a surfer would, then figured out a number of hits to start with. I'll probably look over the blocked ips for the next week and see if I'm blocking any legit looking IPs. I'll also see how the bandwidth changes.

They are then directed to a 403 page with contact information if they feel they were blocked in error

[raymor]

For Throttlebox we graphed the derivative of IP versus hits.
The knee in the graph clearly shows normal versus abusive behaviour.
Alternatively, slight less accurate is to graph to top X IPs where
X is high enough to show the knee. Here's an example of that from
Throttlebox:

https://bettercgi.com/throttlebox/ma...oosing_limits/

You see in the graph that the second highest, third highest etc. are roughly
linear with a near horizontal slope. That's indicative of normal usage. In the
case of the graph illustrated, only the #1 top user is far from being linear with
the others - that's the abusive one.

For another example, let's say the graph looked like this:

20 #
19 #
18 ##
17 ###
16 ###
15 ###
14 ####
13 ##### 
12 #####
11 #####
10 #####
9  ######
8  ######
7  ########
6  #########
5  ############
4  ##############
3  #####################################################
2  #################################################################
1  ####################################################################

In this ASCII graph, the top three are way out of line from the others, which is
indicative of abuse.

That will tell you where the cut off line should be, but that's the easy part.
There are much more difficult issues to work out before you have something
truly effective.

You have to be careful since you're working with IPv4 addresses.
You should expect that AOLs proxies and DTAGs proxies, for example, are going to
have a LOT more hits than any normal IP, on a site with a broad user base.
If the site has 12 AOL users on at different times of the same day, six of those
users may show up as the same IP.

On the other side, an cracker going through a zombie web server may use all sixteen
IPs on that server, so you really want to look at ranges of IPs as well.

I know I'm throwing a lot out there at you, but only because there are a lot of things
to consider.
We've been working on "detect and stop abuse" for a decade and half and still need
to do updates all the time in order to remain optimally effective.