- API security in the cloud
- Configuring the GigaOm API workload test
- Test results
- Appendix: Recreate the test
- About NGINX
- About William McKnight
- About Jake Dolezal
Data, web, and application security have evolved dramatically over the past few years. Just as new threats abound, the architecture of applications – the way we build and deploy them – has changed. We’ve swapped monolithic apps for microservices running in containers and communicating through application programming interfaces (APIs), and all deployed through automated continuous integration / continuous deployment (CI / CD) pipelines. The frameworks we have established for building and deploying applications are optimized for time to market, but security remains of the utmost importance.
The challenge of securing and innovating runs deep and requires a lightweight, integrated security solution that won’t hamper performance and delivery. For example, DevOps teams need security controls that work in distributed environments without slowing or invasively slowing down the release cycle. The maturation of these controls and processes ultimately moves into the realm of DevSecOps, where security is built into the CI / CD pipeline.
The multitude of deployed applications, APIs and microservices produces a constant flow of communication and data between applications that require active management, both internally and externally. The applications themselves can vary widely in terms of protocols, allowed methods, authorization / authentication schemes, and usage patterns. Perhaps more importantly, IT departments need granular control over the entire application ecosystem to prevent security breaches and attacks, whether they are man-type attacks. -in-the-middle, distributed denial of service or script / code / SQL injection attacks.
While security is of the utmost importance, the pace of modern businesses demands high performance, and this is especially true in businesses using applications and microservices. The conventional approach of deploying a perimeter Web Application Firewall (WAF) to protect applications by filtering and monitoring traffic between the application and the Internet is no longer sufficient. Even internal communication between applications and microservices on the trusted corporate network can be compromised and must be corrected. A defense-in-depth strategy is needed with multiple WAFs.
This report focuses on web application security mechanisms deployed in the cloud and closer to your applications. The cloud enables businesses to differentiate and innovate with microservices at a rapid pace and allows microservice endpoints to be cloned and scaled in minutes. The cloud also offers elastic scalability over on-premises deployments, enabling faster server deployment and application development and less costly compute. However, the cloud is just as vulnerable, if not more vulnerable, to attacks and breaches than APIs and on-premises applications.
We specifically focus on approaches to securing applications, APIs, and microservices optimized for high performance and availability. We define “high performance” as companies that experience heavy workloads. more than 1000 transactions per second (tps) and require a maximum latency less than 30 milliseconds across the landscape.
For many organizations, performance is a big issue – they need to guarantee secure transactions at rates that keep pace with their business. A WAF or an application security solution can not be a performance bottleneck. Many of these companies are looking for a solution that can balance the load between redundant microservices and allow high transaction volumes.
The numbers add up. If a business records 1,000 transactions per second, that translates to 3 billion API calls in a month. And it’s not uncommon for large companies with high-end traffic levels to experience 10 billion or more API calls in a 30-day period. Make no mistake, performance is a critical factor when choosing an API security solution.
In this report, we tested the performance of security mechanisms on NGINX, AWS, and Azure: ModSecurity, NGINX App Protect WAF, AWS Web Application Firewall (WAF), and Azure WAF. The latter product has been tested as a fully managed security offering. Note: ModSecurity is commercially distributed by NGINX and will be referred to as “ModSecurity” throughout the remainder of this report.
In our benchmarks, NGINX App Protect WAF outperformed ModSecurity at all attack rates tested. NGINX App Protect WAF produced 4.7 times lower latency than NGINX running ModSecurity at 99e percentile at 1,000 transactions per second (tps) on the 5% Bad Query test. In our tests, the latencies for App Protect and ModSecurity diverged at the upper percentiles, becoming pronounced at the 95e percentile and above.
For fully managed offerings, NGINX App Protect WAF produced 128 times lower latency than AWS WAF at 1000 fps on the 5% bad request test at the 99th percentile. Additionally, NGINX App Protect WAF produced 82 times lower latency than Azure WAF at 1000 rpm on the 5% bad request test at the 99th percentile. Because the AWS and Azure WAF are fully managed, we don’t know which underlying compute resources are running in the background, making it difficult to compare performance between apples. Again, the latency differences were minimal until the 90e percentile, with a significant difference observed at the 99e percentile and above.
On a single small 2 CPU instance and 5.25 GB of EC2 RAM, we captured the maximum transaction throughput achieved with 100% success (no 5xx or 429 errors) and maximum latency of less than 30ms. NGINX App Protect WAF produced approximately 5,000 requests per second, compared to just 2,000 requests per second with ModSecurity. App Protect provides the same level of throughput as direct API access without WAF in between.
Testing hardware and software in the cloud is very difficult. Configurations can favor one vendor over another in terms of feature availability, generations of virtual machine processors, amounts of memory, storage configurations for optimal I / O, network latencies, software versions and operating systems and the workload itself. Even more difficult is testing fully managed as-a-service offerings where the underlying configurations (processing power, memory, networking, etc.) are unknown. Our tests show a narrow range of configurations and potential workloads.
As a sponsor of the report, NGINX opted for a default NGINX installation and out-of-the-box API Gateway configuration – the solution was not tuned or modified for performance. GigaOm has selected identical hardware configurations for App Protect and ModSecurity. Fully managed AWS WAFs and Azure WAFs have been used “as is” because, by being fully managed, we have no access, visibility or control over their infrastructure.
We leave it to the reader to determine the question of fairness. We strongly encourage you to look beyond the marketing messages and see for yourself what is of value. We hope this report will be informative and useful in uncovering some of the challenges and nuances of security architecture selection.
We have provided enough information in the report for anyone to repeat this test. We encourage you to compile your own representative workloads and test compatible configurations applicable to your needs.