By Jesse Blazina, Director Edge Services
When operating a large global network that supports thousands of web applications and streaming media services, dealing with distributed denial-of-service (DDoS) attacks is part of our day-to-day operations. Having a resilient and intelligent DDoS mitigation platform is essential to the operation of our network and to the web services that depend on it.
To provide that protection, we developed Stonefish, our anti-DDoS detection and mitigation platform that prevents layer 3/4 attacks from impacting our customer’s web applications. Stonefish works 24 x 7 x 365, analyzing millions of packets per second, scoring them for threats, automatically taking action when necessary, and keeping our network operations center (NOC) informed so they can perform additional analysis and take mitigative action quickly.
Stonefish was purpose-built to be a massively scaled, DDoS mitigation platform. Before we take a deeper dive into the Stonefish architecture, let’s first look at the evolution of DDoS protection and see why a new approach to DDoS mitigation was necessary.
The evolution of DDoS protection
DDoS mitigation products and services have been in use for nearly two decades. But as web application infrastructure has shifted to take advantage of the cloud, so have DDoS mitigation requirements. The evolutionary steps can be grouped into four categories:
- Specialized hardware
- Scrubbing centers
- Cloud-based protection
- CDN-based protection
Specialized hardware
Specialized hardware has been the choice for DDoS protection since the early days of web applications. The value proposition is straightforward — place an appliance in a data center, connect it to the network, configure it, and let it do its job in identifying and mitigating attacks. However, in many instances, manual intervention is required, pulling critical staffing away from normal operations. And appliances can be very expensive to maintain as they require large amounts of standby capacity and network to handle today’s cloud scale attacks.
Scrubbing centers
Scrubbing centers are often located in PoPs. Compared to specialized hardware, scrubbing centers scale on demand, in some cases to multiple Tbps. Traffic is analyzed in a centralized data cleansing station, and malicious traffic is removed. The advantage of scrubbing centers is their ability to support multi-protocol data center defenses.
Cloud-based protection
Cloud-based protection, such as those offered by AWS, Azure Cloud, and Google Cloud Platform, is ideal for companies using the vendor’s other services, such as compute, database, storage, functions, and so on. DDoS protection is enabled natively, but as is the case with most cloud-based services, security is one of many services, and getting specialized security support often requires a service contract.
CDN-based DDoS protection
CDNs are similar to cloud providers in that services are distributed across many edge locations in the cloud. CDNs, however, often specialize in offering additional security services, support, and tools that improve responsiveness and accuracy. Coupled with a massive frontline network architecture, CDNs can provide automatic protection on highly configurable stacks that afford customers a great deal of visibility and control.
Stonefish design goals
Stonefish is a DDoS mitigation platform purpose-built to protect our delivery network. We developed our DDoS security stack using a mix of open source and custom software that runs on our commodity hardware, allowing us to provide a highly scalable and automated DDoS platform that enhances the ability of our frontline NOC to provide DDoS mitigation support. We designed Stonefish to deliver to the following minimum specifications:
- Defend against a broad range of DDoS attacks, from volumetric to state exhaustion.
- Protect against layer 3 and 4 attacks.
- Leverage our existing commodity server hardware and network capacity.
- Build a security stack that is software-centric with a control plane that acts as the brain.
- Be cloud-based, automated, and intelligent.
- Deploy a control plane that manages the DDoS rulesets and creates rules on the fly in an automatic response to attacks and enforces our policies globally.
- Create a data engine that detects and filters out bad traffic within seconds.
Our efforts resulted in a fully automated system that detects and blocks 99% of denial of service attacks. Stonefish is the control plane that acts as the brains of the system enforcing security policies to every PoP in our network in real time.
Stonefish architecture
Taking a software-defined approach to Stonefish enables us to house our DDoS mitigation on our distributed server infrastructure, enabling every PoP in our global network to function as a scrubbing center that can detect and filter out bad traffic. Stonefish is built with a modular software architecture, which allows us to easily add functionality to the system against an ever-evolving threat landscape.
Stonefish leverages our massive global Internet Protocol (IP) Anycast network. IP Anycast routes malicious traffic to the PoP that is closest to the infected device or botnet. This allows us to block any attack at the edge before it can reach a customer’s origin server and data center. Most of the time, customers are unaware they are being attacked. Our services, meanwhile, are always on, using values such as the source IP address/port, destination IP/port, and packet fields to identify potential attacks and stop them before they can cause any damage.
Sampling and scoring
All incoming network traffic is sampled and analyzed by Stonefish. A scoring system is used to determine the severity of maliciousness and automatically block traffic. The results of the analysis are also sent to our 24 x 7 NOC and evaluated if further action is needed. Here’s how it works.
- The browser sends a request for content to an Internet-facing application.
- The router receives the request and sends it to our load balancing infrastructure.
- A sample of the traffic is sent from the load balancer to Stonefish.
- Stonefish analyzes and scores the traffic.
- If bad traffic is identified, it sends instructions to the load balancer to drop the traffic.
- The NOC is notified of an attack and will follow-up if further action is needed.
How Elasticsearch powers the brain of Stonefish
The distributed RESTful search and analytics engine, Elasticsearch, powers the brain of Stonefish. Elasticsearch data is continually analyzed for changes to packet metrics. This is done via a custom software application. Our software retrieves scores for time intervals and compares them to previous intervals for anomalous or out-of-bounds changes. Each protocol type has a custom query and detection logic for the most accurate identification possible, such as TCP, UDP or ICMP packets.
How our NOC and Stonefish work together
Stonefish is one of many tools our NOC uses to monitor our applications from a security and performance point of view. It is built into a dashboard that alerts staff of sophisticated attacks in real time. While Stonefish blocks DDoS attacks automatically, it is also configured to alert for anomalies, which engages our NOC specialists to investigate and take action.
DDoS mitigation is included with every one of our service plans. Customers can access our NOC personnel for DDoS assistance by phone or email 24 x 7 x 365. Enhanced support and escalations for DDoS attacks do not require specialized security service rates or tiers, including proactive mitigation and customer support in the case of DDoS ransom.
Conclusion
As one of the largest global content delivery networks, that processes between 10–15% of all Internet traffic, we defend against and mitigate DDoS attacks against thousands of customer websites daily.
DDoS mitigation is only one layer in an effective security defense, but it remains an essential one. We built Stonefish to automatically defend our customer’s web applications from layer 3 and 4 attacks by integrating an intelligent software stack on our massive network edge that can detect and mitigate these threats. Working in conjunction with our service teams, customers have proactive DDoS support that can work with them to block 99% of denial of service attacks, from both the network and transport layer.
Visit verizondigitalmedia.com to learn more about our Cloud Security Solution.
