Forum has been very slow 2 days now...

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

User avatar
hgm
Posts: 28381
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Forum has been very slow 2 days now...

Post by hgm »

There has been a period last night during which we were running entirely without mod_evasive, and I could not connect to talkchess at all. This was my fault; I had tried to install a modified version that contained a syntax error. The apxs command used to compile and activate a new module also enables that module, built only works when you disable the module with the same name first. So I had disabled mod_evasive, and then the compile error prevented a new version to be enabled without me noticing it, and going off to do something else.

When I got back to my computer it turned out I could not connect at all to talkchess.com, and I fixed things.

The logs do show the response header to all requests, so we can see whether this is a 200 (success) or 403 (forbidden). But it doesn't work completely as designed yet; only 80-90% of all requests for index.php get a 403. For reasons I do not understand IPs in a range that received a 403 on the previous request get a 200 in the same second. I will still have to investigate this. But I have no time before the weekend.

I am not sure that giving a 403 response to the attackers will be enough to make the attack ineffective. It still requires a significant amount of processing to give the response. And if we respond faster it might just intensify the attack, when the attackers wait for a response before sending a new request. It would be more effective to just delay responses to attackers, giving priority to legitimate requests.
User avatar
Tibono
Posts: 137
Joined: Sat Aug 01, 2015 6:16 pm
Location: France
Full name: Eric Bonneau

Re: Forum has been very slow 2 days now...

Post by Tibono »

Any time I gave a try to browse the forum today, it has been unusually fast - a big thank you to hgm for the efforts and the cure.
Much appreciated!
User avatar
hgm
Posts: 28381
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Forum has been very slow 2 days now...

Post by hgm »

I managed to crank up the fraction of rejected requests from the pool of DDoS attackers. The main problem was that the mod_evasive did not recognize its configuration file (probably because now it calls itself mod_evasive20), and was using the default settings. But these settings are no good, and should probably be considered a bug. The default time intervals within which two requests have to come to be considered part of a burst is 1 sec. But it compares the actual time difference as <, not <=. And the clock ticks in steps of 1 sec. So every clock tick terminates the burst, and you really have to exceed the maximally allowed burst length within the same second to get blocked. Not all attackers achieve that, and these were consequently not blocked.

I now changed the defaults to what I wanted to configure, and the longer intervals make the problem disappear. The output I used for debugging suggests that multiple CPU threads are used to process the requests, though, and that they each run mod_evasive independently of each other. So clients only get blocked when they satisfy the burst criterion on a single thread, and then only that thread blocks further requests. The other thread might still continue to service that client, until it also detects a thread.

For the current attackers this is no problem; they quickly get blocked in all threads.
User avatar
towforce
Posts: 12500
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK
Full name: Graham Laight

Re: Forum has been very slow 2 days now...

Post by towforce »

Tibono wrote: Wed Sep 10, 2025 9:00 pm Any time I gave a try to browse the forum today, it has been unusually fast - a big thank you to hgm for the efforts and the cure.
Much appreciated!
+1
Human chess is partly about tactics and strategy, but mostly about memory
PaulieD
Posts: 239
Joined: Tue Jun 25, 2013 8:19 pm

Re: Forum has been very slow 2 days now...

Post by PaulieD »

towforce wrote: Thu Sep 11, 2025 8:41 am
Tibono wrote: Wed Sep 10, 2025 9:00 pm Any time I gave a try to browse the forum today, it has been unusually fast - a big thank you to hgm for the efforts and the cure.
Much appreciated!
+1
+1