Where Were You During the Great DDoS Cybergeddon of 2013?

Published: 2013-03-28
Last Updated: 2013-03-28 13:59:06 UTC
by John Bambenek (Version: 1)
8 comment(s)

We've had a few e-mails come in to the ISC now that the popular media has picked up the story of the distributed denial of service attack on CloudFlare and SpamHaus.  For instance, here is the New York Times article on the subject.  CloudFlare has their own write-up here.  I was peripherally involved (very peripherally) as were some other handlers.

Let's start with some truth.  The attack did reach upwards of 300 Gb/sec and is the largest recorded DDoS to date.  It combined already known issues on DNS open resolvers but combined it with specific targetted at a choke point which did have a real impact for SpamHaus and CloudFlare.  There also were many people who spent many hours helping deal with this problem.  A good number of those had no real connection to SpamHaus or CloudFlare, they are just fellow members of the information security community who came together to deal with a threat.  This is a Very Good Thing <tm> that this level of cooperation has built up over time and we respond to these threats as a community.

Here is what did not happen: the Internet did not come close to coming down, not much real impact was felt outside the victims and those in close Internet-proximity to them and we were all still able to get to pinterest and see cat pictures online.  The attack was significant, but not globally so despite the media reports to the contrary.  When news of the attack reached the Internet Storm Center, we did have a brief moment of panic and contemplated resorting to cannibalism.  However, we quickly decided against this option (due to a combination of calmer heads prevailing and a lack of consensus on whether people could be turned into bacon).

That's not to say it was a non-event.  It exposed some real problems, problems that each of us should take steps to help remediate so this doesn't happen again.  And by us, I mean those of you reading this that may maintain networks that unknowingly participated in this attack.  More on that shortly.  The attackers were part of a group dubbed StopHaus who decided to take down SpamHaus.  For our purposes here, we'll leave the politics surrounding the attack out of it and focus on the technical.

DNS Amplification Attacks

Accomplishing a successful denial of service attack with straight up network flooding is difficult to accomplish.  In the general case, you have to have more bandwidth than your target.  While it might be somewhat easier to take a gaming server offline over an MMO grudge match, going against a protected target requires you to have a greater amount of bandwidth than your victim which is usually not the case.

You can get around this problem two ways: control lots of machines all over the Internet (i.e. a botnet) or find ways to amplify your attacks to make the attacks much bigger in size than it took you to generate.  Those of you who remember the Smurf attack knows how this can work with ping (spoof your source and ping a broadcast address and every machines on that network will send an echo reply back to your victim).  We've fixed Smurf with default configurations.

Enter DNS resolvers.  Sending a DNS query is not generally a large request.  However, due to the security advances in DNS (such as DNSSEC), responses to requests can be quite large in comparison. Interestingly enough, CloudFlare has a pretty good write-up on DNS amplification attacks here. In their example, they have a 64 byte query that generated a 3,223 byte response.  That means they can amplify their bandwidth by ~50 times.

In short, here is how it works.  The people who were upset at SpamHaus (and by extension CloudFlare) picked a choke point inside CloudFlare that would hurt, the spoofed DNS requests to known open resolves "from" that victim IP address and they were able to generate a 300 GB/s attack.  Estimates ranged from a 30x - 100x amplification of their own bandwidth use.  When they were keeping their peak DDoS up, that's what CloudFlare was seeing.  (To see the progression to this point, you can read CloudFlare's write up linked above).  To achieve a DDoS attack of 300 Gb/s you would need access to 3-10 Gb/s of bandwidth.  Not insignificant, but also not unachievable for someone with motivation and some money.

The important takeaway is that DNS amplication has been known for some time now and that this DDoS attack was entirely preventable.  Not by CloudFlare, mind you, but by the rest of us who maintain networks and DNS servers.

So what can you do about it?

For these DDoS attacks to work, there needs to be two different components and the presence of either is not a best practice.

The first is that networks where these attacks are being launched are not filtering spoofed traffic.  In order for spoofed traffic to leave a network, the perimeter devices need to allow packets with a source address not on the internal network to be routed out.  This is a bad thing and not good network neighborhood behavior.  Everyone that has netspace should make sure all traffic leaving their network does not have a source address that is not their internal network.  The second component of this is that networks are not doing ingress filtering per BCP 38.  Namely, traffic should not be passed by upstream providers unless it is coming from a known and advertised IP spaces of their clients.  If this were adopted universally, spoofed IP traffic would all but disappear.

The second portion of this is having open recursive DNS resolvers on your network.  Rarely is this a good thing and in most cases, they are unknowingly present and being used to generate attack traffic.  The Open Recursive Project has a tool to check for Open DNS Recursive servers in your netspaces and some advice on what to do if this is an intentional choice (namely rate limiting).  Generally speaking though, most DNS servers do not need to perform recursive queries and many of the rest don't need to do it for the entire Internet. Turning off recursion is as easy as putting "recursion no;" into your named.conf file and if you need to recurse for local clients, restricting it to just your own netblocks.

If those of you who maintain networks do the above two things, this DDoS (and those that will follow) would be non-events.  So please, implement some flavor of BCP 38 and turn off open recursive DNS servers.

John Bambenek
bambenek \at\ gmail /dot/ com
Bambenek Consulting

8 comment(s)


I was on nanog pushing for BCP38 (lookup bcp38.org soon(tm)).

Using bindguard - http://bindguard.activezone.de/ - too log some of the informations from 2 of my customers DNS Servers.

They were already rate limited to 1Mbps...

All in all, a fun few days.

But nothing compare to those in deep during that event.

But it is nothing new... from the IRC Flood days (*wave* to HE), to the SNMP amplification DDoS, etc.

Until the offending Peers keep allowing spoof IP to get out of their network... we'll have to deal with that.
I have a simple patch:

In a future version of DNS protocol, enforce that a IP packet cannot be larger than the IP packet containing the request, or the acknowledge packet if there was any.
See https://www.sans.org/dosstep/roadmap.php from 13 years ago, particularly problems #1 and #2...

On the bright side I think that as more people figure out how to work this 300 Gbps weapon, its effects will be diluted among multiple targets. Recursive resolver owners may start to question why their servers max out their upstream link. And networks not implementing BCP38 may eventually notice (although relatively much smaller) the outgoing traffic volume from source IPs they don't even host.
Politics? - SpamHaus are the good guys, compiling info on cybercriminals in the ROKSO (Register Of Known Spam Operations) system, and StopHaus the bad guys, most likely the very people listed in in ROKSO. The StopHaus site is even located on a local broadband connection in Moscow, Russia.

I will continue to block any and all mail originating or passing through systems listed by SpamHaus. The guys in ROKSO are evil and their mails are not welcome.
One other interesting aspect of this attack against Spamhaus is that one of their IP addresses got hijacked by the alleged organizer of the attack . The same guys also hijacked DoD address space and used it to send spam earlier this month. Details here:
Unless I'm mistaken, all stock distributions of BIND that have been released over the past several (many?) have been configured to be close by default. So where are all these open resolvers coming from? When vendors of DNS server software are still shipping **** that is open by default?

Somebody needs to find these turkeys, publicly shame them, and give them a wedgie.

In a similar vein, who is keeping the public "hall of shame" list of all the ASes that still are allowing forged packets to flow out of their networks? There really ought to be one of these on the web someplace. (Yes, like everything else, it could be a double-edged sword, and could help the bad guys, but they already don't seem to be having any trouble finding "loose" networks, so it is better to publicly out and shame these networks.)
Sorry. I type too fast. I meant to ask "WHICH vendors of DNS server software are still shipping **** that is configured to be open by default?"
To perform this type of attack open resolver isn't necessary. Any authoritative DNS server will do. Query them about the domain they're authoritative for with the query type set to ANY and you'll get your job done. Query sans.org DNS servers about sans.org domain or isc.org DNSes about isc.org. Query a lot of other authoritative servers around the globe simultaneously with the spoofed addresses and do this a thousand times per second and you'll create this kind of attack. So there is no need in open resolvers to perform this attack, though this attack can benefit from them, because then you don't have to stick to a single domain name per DNS (or per several DNS) server(s).
This important point that also missed in CloudFront article.
That's why this attack isn't or wasn't "entirely preventable".
AFAIK all the popular DNS server software today have no options to rate-limit replies for the particular address by default.

Diary Archives