Yesterday w had a funny (I say that now that I’ve been able to relax a little) situation on one of our hosted servers. The server’s RAM usage went bang to the ceiling in a matter of minutes, and it was then practically impossible to get it back (had to hard-kill Apache and the monit process that was restarting it).
What happened? Basically, a DDoS on a single server which is, as most knowledgeable people will tell you, practically impossible to manage.
Just a few definitions:
DoS = Denial of Service = a high load generated on a server which gets the server into a resources starvation (either no RAM left, or disk over-used or CPU overloaded or anything really essential to your system).
A simple DoS attack is originating from one single source and is pretty easy to stop: locate the origin IP and block it. Done.
A more advanced version of a DoS is a DDOS.
DDoS = Distributed Denial of Service = a high load generated on a server … originating from a large number of sources at the same time
The detail is in this last part. Whereas it was easy in DoS to detect where the attack came from, in DDoS, it is made impossible to distinguish one attacker from a normal user, because no single attacker is making “a lot” of requests. They just sum up to a massive number of requests, which eventually make your server get into resource starvation.
So now the concept is clear, how can you do to make it work? Well, that might require a very high skills set if you want to do that on purpose, but surprisingly if you don’t even know what DDoS means and you’re a simple Flash banners designers, it’s apparently dead easy: just make sure you put a flash banner on the homepage of a moderately visited website (let’s say 6000 visits per hour) which has a mechanism to recommend pages when a user tries to find a page that doesn’t exist (technical translation: a custom 404 page which generates a SQL query), on which users stay for a while (put some full-size breaking news or something like that). Then try to load, from your Flash, an URL that doesn’t exist. Oh and wait, while you don’t get the expected answer, just repeat that call, over and over, once every second.
That’s it, you’ve built yourself a nice 1000-users DDoS in less than 10 minutes. Prepare to see your RAM usage soar if you have a CMS with that custom 404 recommendation page, then (if you use swap memory), your disk fill up with swapping pages, and then finally your processor burn its “waiting” states.
The beauty of it is, once you find the cause of the mess, you can’t really stop it straight away, because your users will still be on that page with the banner, so as long as they don’t change page, it will continue querying your website. To make it even more perfect, the Flash might use a lot of the clients’ CPU, which will possibly end up in freezing their browser, which means they won’t be able to get out of the page for a while…
If you’re lucky enough to have a Varnish reverse proxy installation (or any other reverse proxy), one (emergency) solution is to just tell the proxy to *not* pass the request to the webserver, and just answer directly (from cache) a default error page. This will effectively (but it might take a few minutes) highly reduce the server load.
My previous post explains how to do that, just in case.