Push Do Mitigation

For some time now I have been observing and attempting to resolve the slowest Distributed Denial of Service (DDoS) attack I have ever seen. Searching for similar scenarios online you keep coming back to the name PushDo – of which there are plenty of other articles about – and a number of suggested solutions that will ‘help’ but not stop on the whole. I appear to have found a workable solution to stop this consuming resources on an Apache2 based server within the shared environment – so thought I would take a moment to share. In doing so if it makes your life easier – great.

Usually when I think of DDoS – I think of high packets per second, or lower rates initiating actual connections against services, or being on the recipient end of an amplification attack – where a small request is made to a number of remote services, which results in a large response, and the spoofed address means you get the answer. All – equally lovely. All – thankfully also the kind of thing that we have a very expensive piece of hardware to mitigate these days.

The effect of which is an initial hit, a hesitation while a pattern is spotted, and then mitigation. Result.

So lets turn that model on its head for a moment. Imagine its slower – imagine its pretty much slipping under the radar maybe only 20 connections per second. Every second. For four months at this point.

Each connection is from an IP address you will see maybe four times a day. Each connection opens a genuine service port… it opens a connection, attempts to POST (as this is HTTP), starts to speak … and then just waits – waits until the connection times out. What is the harm in that? Well – okay – back to 20 times a second for four months, and within the shared environment. Suddenly this becomes quite an effective attack – resources are consumed, hardware is slowed down. Taking the site down – well the requests are still processed, return an error code, still processed. You will even start to see Apache unable to open more ports. There is no winning here.

Looking at the logs it doesn’t take long to realise there is an epic consistency with requests. Whoever owns this network has enough live attackers to keep this up day in day out. Given the nature of the target (academia), and the assumed most likely reason for the attack (religious/political) – I should imagine that each of the hosts I see visiting two or three times a day, do not sit idly – but have a list of targets to work their way through. The traffic per host is minimal – and they are likely to go un noticed on their host network, or even machine.

[attacker] - - [03/Feb/2015:11:23:57 +0000] "POST / HTTP/1.0" 302 448 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

[attacker] - - [03/Feb/2015:11:23:57 +0000] "POST / HTTP/1.0" 302 448 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

[attacker] - - [03/Feb/2015:11:23:57 +0000] "POST / HTTP/1.0" 302 448 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

[attacker] - - [03/Feb/2015:11:23:57 +0000] "POST / HTTP/1.0" 302 448 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

[attacker] - - [03/Feb/2015:11:23:57 +0000] "POST / HTTP/1.0" 302 448 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Hello MSIE 6.0 – nice to see you – do sites really still cater for your needs? Probably not no. So there are no tears shed when you look to prevent this from spoiling our collective days.

So first up I tried iptables.

iptables -A INPUT -d [site address] -p tcp --dport 80 -m string --algo bm --string "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" -j DROP

As this is not over HTTPS (thank heavens) – then this is easy enough – right? Sure. You can even tie it down to the single IP involved so just their traffic is effected.

However either locally or at the border firewall – this is not cutting it. Why – is a very good question.

Using a local .htaccess – equally seemed to fail to deliver the results – with either a redirect or a forbidden.


RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (MSIE\ 6.0) [NC]
RewriteRule ^(.*)$ http://www.non-existant-domain-name.com/ [L,R=301]

Suggestions I also found were to use a CDN. So the client signed up with a popular CDN free service, and moved their DNS to them. The attack diligently moved with the domain name, and now was coming in from the CDN’s IP addresses. FORTUNATELY for us the website in question doesn’t require a global reach, in fact, it is pretty local. So shutting off access to all the usual suspects, and restricting what locations could reach the resource at a DNS level means that the bulk of the traffic that needs processing is hit on the head. Immediately. Go team. Great success.

So then – when looking and wondering about the implementation of Nginx as a reverse proxy, using an OWASP rule-set, and custom rules to prevent and protect against these kinds of occurrences where the usual tools simply are not cutting it.

Having started to invent a wheel with a box racked up and an OS good to go – I realised that there was probably going to be a middle ground.

The service was running on Parallels Plesk. This has a configuration that allows you to run Nginx as a front end in front of Apache2.

Once installed, pull up the configuration for just that user.

vim /var/www/vhosts/[target]/conf

An inclusion was then made to the location section within the server declaration thus:

location / {
proxy_pass http://[backend apache2 and port];
if ($http_user_agent ~ "MSIE 6" ) {
return 403 "Browser Not supported" ;
}
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Accel-Internal /internal-nginx-static-location;
access_log off;
}

Here the IF section has been added – where it is now returning a ‘403 not supported’ for all MSIE 6 instances.

Restarting the Nginx front end – and witness the glory of the Apache2 service no longer getting bogged down with requests of that kind.

This can be further tested at various points along this journey with the following incidentally:

:~$ curl -i -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1" http://[target]

HTTP/1.1 403 Forbidden
Date: Tue, 03 Feb 2015 15:28:44 GMT
Content-Type: application/octet-stream
Content-Length: 21
Connection: keep-alive
Set-Cookie: __cfduid=d18af4137fc99f65088d1655b15ac61d31422977324;
expires=Wed, 03-Feb-16 15:28:44 GMT; path=/; domain=[target]; HttpOnly
Server: cloudflare-nginx
CF-RAY: 1b2fa8f34cde136b-LHR

Browser Not supported

The above shows that this is returned as a 403 Forbidden Browser Not Supported before taking up resources with Apache2 that is delivering the actual content. Small, lightweight in terms of work and resources used, not too invasive, takes the load off of the busy Apache2 process, doesn’t effect any other users on the same host. Winning.

This is of course with the caveat of the attacker NOT changing the agent type it is reporting. We can cross that bridge when we come to it. In the interim – back to the OWASP Nginx reverse proxy – especially as from the test above it is pretty apparent that is what CloudFlare do… or did, and report as now.

Other articles I have read around:

http://tabbles.net/blog/pushdo-how-we-are-mitigating-the-ddos-attack/

http://markswanderingthoughts.nl/post/102675319945/mitigating-a-pushdo-d…

http://www.sheffieldforum.co.uk/showthread.php?t=1262374

http://www.distilnetworks.com/is-pushdo-screwing-you-details-of-the-botnet/

http://www.shadowserver.org/wiki/pmwiki.php/Calendar/20100129

Please Note – PushDo is a far more capable beast with a number of profiles and payloads from my limited understanding (I have no formal security training – I simply live on this side of the fence, and work with what I see daily) – it is just a pattern of references you will have found if you are looking for solutions to a torrent of POST from a distributed huge range of addresses. As such it makes a convenient handle for these notes.

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this:
Skip to toolbar