By Jake Hilton, Sr. Manager, Software Dev Engineering, Verizon Media
Content delivery networks (CDNs) are well-established as an integral part of streaming media workflows, enabling a high-quality video experience that scales globally. While most streaming services leverage the CDN to enhance video performance, they may be missing an opportunity to leverage the CDN’s full power in securing their OTT streaming infrastructure. In this article, we’ll review how a CDN can be deployed as a security layer in an OTT streaming infrastructure to mitigate DDoS attacks and other vulnerabilities. We also share CDN configuration best practices that enhance the performance and resilience of the streaming infrastructure.
The manifest server
In a streaming workflow, once a client authenticates and pushes play, the client/player establishes a session with a manifest server. The manifest server directs the player to a video store, or CDN, to retrieve the video files. The manifest server is in constant communication with the client throughout playback. And in some streaming workflows (as is the case with Verizon Media Platform), our manifest servers create a session for every viewer.
In a 1 to 1 streaming workflow, each user gets their own session. Since the manifest is personalized and continuously changing, when directing the player to fetch a video file that changes depending on the stream bit rate and ad break, the manifest doesn’t benefit from CDN caching. As we will discuss later in this article, you must take care to configure CDN caching so that it does not negatively impact the manifest server’s performance.
High-performance manifest servers depend on horizontal scaling. For example, we’ve architected the manifest server infrastructure in our streaming service to scale up in real time across multiple geographical regions to deliver millions of sessions for popular live streams such as the NBA Finals and Super Bowl.
Can adding a CDN layer in front of the manifest workflow still offer advantages for both performance and security? That is what we sought to confirm when we moved our manifest servers behind our CDN. We discovered three benefits of this workflow.
Figure 1. A CDN is commonly used to enhance video file delivery (Diagram A), but it can also be leveraged to enhance the manifest server’s security and performance (Diagram B).
Benefit 1: Automated DDoS protection
Because web servers are publicly accessible online, they are an open, attractive target for DDoS attacks. While manifest server URLs are not typically advertised, they are publicly accessible, and it takes little effort for someone with a knowledge of networking and some basic probing of a browser’s web developer tools to discover your URL.
Given the relative ease of identifying the attack surface, DDoS attacks are one of the most common tools in a hacker’s arsenal. Using low-cost services found on the dark web, attackers can harass any web server in the world, including manifest servers. Despite the relative ubiquity of DDoS countermeasures, Verizon counted more than 13,000 DDoS attacks in 2020.
Many web services have deployed DDoS defense technology. Specialized hardware in the data center and third-party scrubbing center services are common. But as applications have shifted to the cloud, it’s becoming increasingly common to move DDoS protection to a cloud-based DDoS provider.
Our CDN incorporates Stonefish, a resilient and intelligent DDoS mitigation platform that automatically blocks 99% of layer 3 and 4 attacks. Stonefish was purpose-built to offer DDoS protection on a massive scale. Built into our 126 Tbps network across 165 PoPs, Stonefish offers the cloud-scale capacity required to match the largest DDoS attacks. Stonefish analyzes millions of packets per second, scoring them for threats and automatically taking action when necessary or referring attacks to the network operations center for escalation.
Figure 2. Stonefish samples and scores traffic traversing our global network and automatically blocks DDoS attacks before they can impact the customer’s web infrastructure.
Benefit 2: Request distribution with IP Anycast
DDoS protection is also enhanced through IP Anycast, a networking technique built into the Verizon Media Platform Delivery network. It allows for multiple servers to share the same IP address. Based on the location of a user request, routers send it to the closest endpoint, reducing latency and increasing redundancy. IP Anycast uses the CDN’s scale to protect against large volumetric or DDOS attacks. Each server within the CDN absorbs portions of the attack resulting in less strain on the server and network.
Benefit 3: Reducing manifest-client latency
Despite the uncacheable nature of manifest server sessions, a CDN still offers some performance advantages. A typical manifest to client to server path could have as many as 20 hops through the public internet. In contrast, CDN’s leverage their dispersed edge servers to close this gap, eliminating hops, which reduce the number of links where congestion can occur, connecting to the closest point of presence (PoP), which is likely just one or two hops at the most. The CDN then routes traffic over its highly-optimized connections between POPs.
Optimizing the CDN for manifest server performance
To validate these benefits, Verizon Media’s performance engineering team created a test environment to ensure results were the same or better when behind the CDN — including error rates, response times and time behind live — for both HLS and DASH manifests.
The test compared:
- non-CDN fronted AWS zone with a CDN fronted AWS zone
- non-CDN fronted Azure zone with a CDN fronted Microsoft Azure zone
Each zone had a target of 250,000 concurrent simulated live viewers for a total of 1 million, with 500,000 going through the CDN. Clients were given a 10 to 1 ratio of HLS vs. DASH, meaning that for every 10 HLS viewers generated, there would be one DASH viewer generated. The channel viewers used had frequent, higher-than-normal ad breaks designed to over-stress the system — 30 second ad breaks once a minute, resulting in 30 seconds of content followed by 30 seconds of ads. The initial viewer ramp-up was more than 700 viewers per second, simulating a fast start to a live event.
Our initial testing revealed some performance degradation on the zones behind the CDN, resulting in clients experiencing increased response times and timeout errors. To resolve these issues, we made two changes.
First, we configured the CDN not to distribute the manifests. As described above, since manifests are individualized at a 1 to 1 session level, CDN caching of manifests is unnecessary and can degrade performance.
Second, we looked at configuring the HTTP keep-alive setting so that the CDN established a more optimal handshake frequency with the manifest servers. Using the manifest server’s keep-alive setting as the baseline, we set the CDN keep-alive setting just below 12 seconds. Why not keep the connection open indefinitely? This has to do with striking an optimal balance between efficiency and performance. Just like a meeting over Slack can only maintain a maximum number of threads before becoming overloaded, so too does a manifest/CDN communication need to be configured for what the servers can handle. A setting of 12 seconds optimized the interaction frequency, allowing the CDN and manifest to communicate at optimal levels.
Following these changes, we found little difference between manifest performance behind the CDN and not behind the CDN. Both AWS and Microsoft Azure performed comparably in both setups. The CDN didn’t report any issues with the performance and load.
Bringing it all together
The CDN plays a vital role in any media service’s success, delivering a high-quality viewer experience at scale. While virtually all OTT services rely on the CDN for content distribution, many miss an opportunity to tap into the CDN’s power to guard their services against DDoS attacks. A CDN can help in two powerful ways. First, the CDN’s massive scale can match the scale of the largest of DDoS attacks. Second, the use of IP Anycast spreads any DDoS attack across multiple servers. Along with improved security, the CDN can also play a role in reducing manifest-client latency.
The number and severity of DDoS attacks is growing yearly. Taking a comprehensive look at all your infrastructure will likely reveal opportunities to improve performance and increase security. OTT services must take action to defend against a service-disrupting DDoS attack while maintaining optimal performance. Moving manifest servers behind the CDN can accomplish this goal.
Let us evaluate your security needs across your OTT infrastructure and suggest ways to increase your protection and performance levels. Connect with us now to learn more.
Read our latest security report, “Protecting your OTT streaming service from cyberattacks,” to learn about proven security technologies that help you confront the new set of cyberspace challenges specific to OTT platforms.
