On February 28th, 2018, GitHub reported that it experienced the largest DDoS attack on record, with a maximum volume of 1.35Tbps.
The GitHub Attack Process
Many applications use a memory caching system called “memcached“. Administrators set up a memcached server in order to make frequently used content or data easier to access, reducing strain on individual instances. By submitting a query for previously used data, the user will get the information faster without additionally burdening the system. Since memcached doesn’t require authentication, most admins set it up only for internal use. However, misconfigurations means that these servers can often reach the internet.
An attacker can trigger a DoS by purposefully making the memcached server store large quantities of data. Since memcached previously enabled UDP communications by default, an attacker can send a single byte of data to the server and get large quantities of data back, up to 51,000 times as much. Again, the attacker can do this continuously since no authentication is necessary. By spoofing the source IP, pretending the victim sent the request, the victim will get the reply, and the attacker can trigger a DoS with very little work. By causing hundreds or thousands of machines to do the same thing, the amount of traffic the victim gets can easily hit the hundreds of gigabits or terabits per second, exactly what happened to GitHub.
What This Means
So how can you prevent or protect against an attack like this?
Developers
- Don’t use the UDP protocol if you can help it.
- If you need it, disable it by default or require the user to actively make the application use UDP.
- Always try to send less data than you get per packet. If one packet is 1KB, only send 0.5KB back.
- Always require authentication to use your application, within reason.
Administrators
- Run this command to check if you’re vulnerable:
echo -en "\x00\x00\x00\x00\x00\x01\x00\x00stats\r\n" | nc -q1 -u 127.0.0.1 1121
- If you see any output, you are.
- If you’re vulnerable, do the following: disable UDP for memcached. On startup you can specify “-U 0” to disable UDP and “–listen 127.0.0.1” to listen to localhost exclusively.
- memcached uses port 11211 in both TCP and UDP, so use your firewall-fu to lock that access down (we can help if you need it). Use nmap to verify:
- If it looks like the above, that’s a problem.
- Also make sure your disaster recovery and incident response policies are up-to-date and have been tested recently. Build or strengthen your relationships to protect against outages – GitHub’s strong relationship with Akamai meant that this attack was mitigated within 10 minutes.
More Reading
Secure Compliance Solutions is the trusted security advisor for Chicagoland’s small-to-medium businesses. We offer a variety of services that promote a strengthened security posture and a culture of compliance. Our solutions include: risk advisory services, strategic cybersecurity planning, security and privacy awareness, regulatory guidance, penetration testing, and managed security services. We tailor our engagements and solutions to align with your cultural needs and business objectives; not the other way around. We keep your appetite for risk, budget constraints, and timeline in mind to define strategy and operational tactics that maximize your return on investment. At SCS, we help you navigate the course of your cybersecurity journey.
