DDoS Testing – Common Questions
July 30, 2017
Q: What are the benefits of DDoS testing?
• Demonstrable capability
• Improve Operational Performance
• Optimize configuration
• Extract full value of infrastructure and services
• Manage 3rd party risk
• Confirm technical controls
• Confirm SLA’s
• Keep up to date with emerging threats
Q: How can DDoS testing be done safely?
RedWolf DDoS testing is quite safe, as we manage the risks by:
1. Ensuring permissions are granted for every asset and network being tested.
2. Slowly ramping up traffic levels from very low levels.
3. Ramp-ups are done when client says to increase, not automatically or on a timer. People are always in full control of traffic levels.
4. Emergency stop all traffic in 2 seconds.
5. >99.9% of the time the applications recover in at most a few minutes when the test stops. In rare circumstances an application might need to be restarted by the application teams. If this happens it is also a finding that the application has some critical vulnerabilities that only could be uncovered with these tests.
That said, RedWolf recommends the customer have application team and load balancer teams be available during the test in case as system needs to be restarted.
Q: What should I expect if a system is overloaded?
Almost every RedWolf test can result in some sort of system limit being reached. It is very important that production systems recover effectively.
1. Slightly Degraded performance – RedWolf suggests slowly increasing attack intensity from very, very low levels. As the intensity increases operations teams should monitor the increased load and performance of the full-stack (router to application) – any of these elements may fail first. RedWolf can monitor the performance from the Internet, and from on-site monitors and identify the exact point where systems become degraded. Even with a DDoS appliance or upstream WAF some attack traffic can leak through and degrade the site. It is okay to live with degraded performance under a DDoS attack so long as the user experience is not too bad.
2. Moderately Degraded Performance – When performance degrades and regular users will know the site is operating slowly, but still usable. It may take many seconds to open a page, but a business transaction can still function. This condition occurs if the attack-mitigation leaks too much attack traffic through. One positive outcome of DDoS testing is to optimize DDoS mitigation activation thresholds to begin protecting services before this point is reached.
3. Severely Degraded Performance – When a site has long periods of failed requests and can not service.
4. Servers Hang or Crash — Servers taken out of pool – Enterprise application servers
5. Auto-scaling events
Broadly, across all test vector types, during a 6 hour window, normalized to ‘10 tests’, the following are observed:
Q: Where can traffic be sourced from?
Traffic can originate from the following:
• Canada
• USA
• Mexico
• Netherlands
• Finland
• Iceland
• England
• Ireland
• Germany
• Italy
• France
• Belgium
• China
• Taiwan
• India
• Japan
• Korea
• Singapore
• Australia
• Brazil
• Chile
And within these countries over 140 separate data centers in total. Each ‘agent’ has an IPV4 address. If IPV6 is required the list is smaller.
Q: How many source addresses are involved in the DDoS testing?
We recommend using 200 attackers and 4 monitors for an initial test. This is the number included in managed test bundles. Additional agents can be purchased if required. In subsequent tests, depending on how the first test went, the number of agents should be increased from between 500 to 1000. The more agents used the more realistic the DDoS attack.
The number of attackers used in a test is variable, up to 20,000. Each attacker costs a fixed amount per hour and can be placed in any supported cloud provider (Amazon, IBM, Microsoft, Google, Rackspace, Upcloud, Oracle, and others).
The following table suggests current (2017) best practices for # of agents to use.
Describe the DDoS vectors that can be run. Do they target layers 3, 4 and 7 of the OSI model? Are there layer 7 tests for HTTP and HTTPS?
The list of DDoS tests is extensive (over 300) and include all layers 2 through 7, the latest in-the-wild attacks (e.g. Mirai), IPV4 and IPV6, and all popular protocols.
For the above attacks the following options can be changed in real-time:
• Real time control over: Bandwidth, Request Rate, TCP Concurrency
• Emergency stop (5 seconds or less)
• Attackers can be added/removed dynamically
• Pure or blended attacks
• Single or multi-target
• Modification of any attack parameter in real-time
Q: What is the maximum bandwidth / throughput of the DDoS test in Gbps?
A: As large as you need. The only practical limit to attack size is permission and budget. The highest bandwidth generated by the platform was 550 Gigabit/sec. The largest Layer 7 attack was over 2 million HTTPS requests/sec.
A single agent is actually capable of sending over 500 megabit/sec (large packets) and at least 50k packets/sec. So 200 agents is capable of sending 100 Gigabit/sec! In practice it is not wise to run agents this aggressively. It is better to simulate traffic levels generated from real cloud-based botnets.
Instead of super-high levels of traffic from a single attacker it is recommended to start attacks at lower levels, and ramp-up progressively to whatever maximum limit that the test plan indicates. At each ramp-up step systems are checked for impact. By sending a variety of traffic levels from each attacker it is possible to learn if systems are vulnerable to low traffic levels before an ‘Anti-DDoS’ mechanism activates.
Further, by varying traffic levels across agents makes an attack appear more realistic. The platform allows individual attackers to shape traffic to arbitrary levels to each target tested.
Note that for Layer 7 attacks, the network traffic is ultimately limited by server capacity. Once it is exhausted traffic will likely begin to falter vs. increase.
Q: How long it does it take to stop the DDoS test traffic?
Under 5 seconds including human reaction time. Once the ‘stop’ button is pressed traffic typically stops in 1-2 seconds.
Managed tests: A test can be stopped in three ways:
1. By asking for it to be stopped on the bridge. RedWolf will stop the test. This typically takes 5 seconds.
2. By clicking the ‘emergency stop’ button in the portal. Any user can click the emergency stop button. This stops traffic for all actions (attacks and monitors) in about 2 seconds.
3. By clicking the ‘stop’ button for a specific test. Any user has permission to stop a test. This stops traffic for a given action.
Self-serve portal: A test can be stopped in two ways:
1. By clicking the ‘emergency stop’ button in the portal. Any user can click the emergency stop button. This stops traffic for all actions (attacks and monitors) in about 2 seconds.
2. By clicking the ‘stop’ button for a specific test. Any user has permission to stop a test. This stops traffic for a given action.
Q: How do you monitor systems being tested?
Surprisingly, RedWolf offers extensive and class-leading monitoring capabilities that far exceed the capabilities of almost all commercial monitoring services. The simple reason for this is that the RedWolf monitors are designed to capture maximum-information, at maximum-resolution to support detailed forensic diagnosis.
The following are the types of monitoring available:
From the Internet:
• Web site performance
• User experience monitoring
• Application performance monitoring
• Global DNS performance & change detection
• Global CDN performance
• Global BGP routing changes
• Network route performance
• Network latency monitoring (Internet to local network)
From your network:
• Web site performance
• User experience monitoring
• Application performance monitoring
• Local DNS performance
• Network latency monitoring (local to Internet)
• Firewall Monitoring
• DDoS Appliance Monitoring
• Load Balancer Monitoring
• Direct to Service Monitoring
• OS Performance Monitor (Linux, Windows)
• IIS Monitor
• Web Log Monitor
• Packet Analyzer
• Syslog Collector
• Custom Metrics Collector
