The next BIG thing: ‘small’ DDoS Attacks are often hardest to block
December 9, 2016
Would it surprise you to learn that the most enduring and toughest to fix DDoS vulnerabilities RedWolf testing exposes occur at very low bandwidth levels between 1 and 10 megabit/sec? For years RedWolf has confirmed that it is possible for a botnet of as few as 10 to generate a specific set of packets that significantly impacts firewalls, load balancers and sometimes even web applications at very very low attack levels. Even if sites are protected by on-premise anti-DDoS, stateful firewalls, and load balancers there are many varieties of slow DDoS attacks that can wreak havoc.
State-of-Firewalls: Surprisingly, stateful firewalls are the biggest architectural weakness in most enterprise architectures RedWolf tests. This runs counter to normal thinking where stateful firewalls provide an additional layer of security – they do but not verses DDoS 
<1 Megabit/sec attacks: Once a state table is full an up the attack can become even slower as it only needs to keep the firewall state tables ‘topped-up’ as the sessions expire. Below is a chart of a DDoS test where the majority of outages were caused at sub 1-megabit/sec DDoS test:
The above chart shows latency and outages. The lines show latency from different monitors. Pink sections are >= 10 second outages. All outages after 12:15 were caused by DDoS attacks from 1 to 10 megabit/sec – too low to trigger common Anti-DDoS mechanisms. The pink sections show regions where RedWolf web monitors could not reach the site from at least 3 monitors within a 10 second period.
RedWolf testing can be used to:
- Ensure firewall is not vulnerable to low level state attacks (including Black Nurse, see below).
- Ensure the upstream anti-DDoS mitigation systems activate properly before firewall is impacted.
- Find out how firewall performs as state table is filled up.
- Determine if the firewall logs any ‘state table maximized’ errors / alert messages.
- Optimize timeouts are correctly set up (they often hang around too long)
- Test advanced firewall features to block things like DNS Reflection or invalid protocols (vendor dependent)
- How fast the firewall recovers when upstream mitigation.
- Verify firewall drop policy works to performance levels.
- Verify firewall accept policy works to performance levels – often firewall policies degrade maximum theoretical performance.
- Ensure firewall is counting real TCP session states accurately. E.g. if RedWolf creates 20,000 TCP connections exactly many firewalls might report a vastly different number.
SYN-COOKIES – Every TCP connection starts with a ‘SYN(chronize)’ packet. Stateful firewalls can either choose to pass a SYN packet because their policy allows it, or drop the packet. If a packet is dropped a firewall can optionally inform the client via an ICMP packet that the destination is unreachable. If the packet is passed a stateful firewall will begin ‘keeping track’ (aka state) of the new connection. Even SYN Flood DDoS, at low levels, has been shown to often impact stateful firewalls. Most firewalls have a SYN-Flood mechanism but often it is not enabled. Even when it is enabled it often causes a fair amount of CPU load on the firewalls at very low bandwidths (<10 megabit/sec). A firewall’s SYN COOKIE throughput should be benchmarked and …
RedWolf testing can be used to:
- Verify firewalls and load balancer SYN-cookie mechanisms work as expected and do not put undue CPU load on the devices.
- Verify only one SYN-cookie or SYN-ACK is sent. Sometimes network configuration problems cause multiple responses to a single SYN. RedWolf can measure returned packet rate which should never exceed the transmitted packet rate. If it does a network configuration problem likely exists.
- Verify the correct operation of DDoS mitigation systems and their impact to different types of clients (API users, Google bots, real web browsers). RedWolf monitors are intelligent enough to be able to detect different types of SYN mitigation/authentication strategies used by modern defense systems.
DNS Attacks – UDP DNS attacks are surprisingly difficult to defend against at low levels. Often as little as 5 to 10 megabit/sec of DNS requests can overflow a corporate DNS server. Most organizations have never benchmarked their DNS servers to determine how many queries/sec they can deliver but the number is often less than 50 thousand queries/sec. A botnet of even 10,000 only needs to send 5 packets/sec (queries/sec) – a level that most anti-DDoS systems don’t trigger against. Most DNS DDoS defense systems rely on mechanisms to force UDP DNS to TCP DNS. This mechanism works if the DNS mitigation device is tuned and your DNS servers are set up to handle TCP DNS. Still, even with TCP DNS DDoS attacks DDoS mitigation platforms often are configured to allow too much TCP DNS traffic through and the server can still be overwhelmed.
RedWolf testing can be used to:
- Benchmark your DNS server against UDP and TCP DNS to find the 70% CPU utilization point and max throughput
- Help tune your mitigation so it activates at the right levels before your DNS server is saturated
- See how your mitigation system performs for both UDP and TCP DNS attacks – they are quite different and ensure your policies protect against both types of attacks.
BLACK NURSE – there are vectors like ‘Black Nurse’ and other attacks that can cause the firewall CPU to spike and possibly trick it into responding to a number of packets.
This seems counter-intuitive but the reality is that most abuse detection systems are designed to detect abuse only when it crosses certain thresholds. Today’s defense systems typically detect abuse if the attack crosses some sort of threshold. For instance, if a source-IP creates too many TCP connections per second, sends too many packets/second, generates too many HTTP(s) requests/sec, etc… If the activation threshold of a mitigation device is set too high then attack traffic will leak through and can consume enough resources to cause an outage.
RedWolf testing can be used to:
- Test against “Black Nurse” (ICMP Type 3 – Code 3) attacks.
- Test against many other types of ICMP attacks as well.
- Test other types of firewall state table attacks, like RedWolf’s “Hit and Run”, “Hanging Connection flood” and about 10 other variations of TCP attacks. There is a surprising variety of attacks beyond Black Nurse that RedWolf has identified.
APPLICATION ATTACKS – Many organizations that start out doing DDoS testing are first loathe to test specific application features like authentication and search. The common rationale is prioritizing testing defenses against the larger scale attacks like volumetric or very bold and brazen HTTP attacks. In practical terms this is a focus on everything ‘above’ the application beyond the root page. This is a good strategy but at some point the testing should turn towards the application itself. Indeed, RedWolf often recommends testing the broad protection features up to the load balancer first since there is only so much that can be covered within a single test window.
RedWolf testing can be used to:
- Simulate simple and targeted application attacks. A targeted attack is where an adversary spends some time crafting specific attacks against your application.
- Perform load/stress tests (not DDoS) with precise control over HTTP/HTTPS request rate, # of TCP sessions, # of TLS/SSL handshakes, etc… – this is not DDoS: it is strictly a load/stress test and can start as low as 1 request/sec.
- RedWolf can supply no cookies, static cookies, or dynamically learn cookies as a browser would. This ability makes it possible for RedWolf to launch the most sophisticated types of application attacks.
- RedWolf can fill out forms and supply carefully constructed static or random values. It is possible to stick to specific user-name/password combinations (real or fake), pass in proper data values like banking ID’s.
- Requests can be targeted against a single URL, in isolation, like most botnets, or run within real browsers, or both simultaneously.
- RedWolf has mechanisms to bypass many types of JavaScript challenges seen from modern anti-DDoS defenses.
In conclusion, while DDoS is usually pictured as huge floods, there are a great many cases where small attacks can cause a lot of trouble. As we start 2017 RedWolf would like to remind organizations with anti-DDoS defenses that many windows-of-opportunity probably exist and should be closed off. The main benefits of low-level DDoS testing are:
- Identify any infrastructure systems that are vulnerable to attacks at very low levels – too low for on-premise Anti-DDoS systems to consider the traffic an attack.
- Identify application logic particularly prone to failing at very low levels. Typically this would be authentication and interactive search features presented by your web site.
- Ensure monitoring systems are working as expected.
- Improve operational root-cause ability in identifying the impacted system(s). In practice this means monitoring every element the traffic passes through: routers, anti-ddos firewalls, IDS/IPS, load balancers, web servers, application servers.
- Optimize DDoS activation thresholds are appropriate and activate before the target systems are impacted.
- Verify 3rd party in-the-cloud Anti-DDoS systems (routed and proxy types) can identify and/or defend against these types of attacks. Manual intervention is typically required as automatic thresholds are not likely to identify these attacks. It is a great test of the operational capability of a 3rd party cloud defense system if they can zero in on tricky attacks like these.
