I Survived A DDOS

A startup’s app was hit by a sustained denial-of-service (DoS) campaign that first showed up as sudden, unexplained slowness—then escalated into massive request floods, connection exhaustion, and repeated “back again” waves over roughly two weeks. The core lesson wasn’t just that DoS attacks happen; it was that mitigation depends on fast operational muscle memory, tight logging/forensics, and the willingness to iterate rules as attackers adapt.

The trouble began in mid-June 2023 on a Wednesday evening when the team noticed the app becoming unusable shortly after a recent release. Slack activity and load balancer alerts pointed to a traffic spike: the system was receiving around 10x peak traffic, even though no new deployment explained it. Investigation in AWS tooling revealed the API Gateway scaling out, and the team suspected a DoS. They turned on a Web Application Firewall (WAF) rule set that had been under consideration during security certification planning. Once enabled, the attack dropped quickly—initially filtering out about 10% of traffic still left enough volume to hurt, but the team then used WAF sampling and access logs to identify the dominant sources.

Early samples showed most suspicious traffic originating from North Korea, Russia, and China—regions the team didn’t serve. They created a geographic rule and the DoS stopped. Afterward, the team planned follow-ups: export logs for querying in AWS Athena and add alerts tied to WAF allow/block outcomes. For a short window, the incident looked contained.

Then the attack returned. A second wave brought staggering numbers—hundreds of millions of requests within an hour—and attempts to block non-US traffic failed because the attacker redirected traffic and kept probing. The team found a consistent “keep-alive” header pattern across sampled requests that didn’t match their own front-end behavior. That clue led to identifying a Slowloris-style approach: overwhelming the target by maintaining many simultaneous HTTP connections. Even when WAF alerts showed heavy blocking, enough traffic still slipped through to keep the database and endpoints overwhelmed.

Mitigation shifted toward broader containment: blocking non-authenticated traffic, adding rules for known malicious IP ranges (including “onion network”/Tor-class traffic and other bad-reputation sources), and patching a front-end/back-end issue that allowed requests to reach the API Gateway and backend in ways that worsened the flood. After a hot fix removed the problematic path and tightened enforcement, the DoS ended—only to continue for nearly two weeks in a pattern of daily morning surges. Eventually, the campaign stopped as abruptly as it began.

In the aftermath, the team emphasized practical preparedness: use a cloud provider with a WAF, keep load balancer/access logs enabled, learn Athena-based log querying, and rehearse incident response so rules can be applied during an attack. The takeaway was blunt—there’s no guaranteed way to prevent DoS, but there are ways to survive it without burning out or losing the ability to operate when the next wave hits.