Scaling One Million Checkboxes

A one-million-checkbox website launched on June 26 quickly turned into a mainstream, real-time stress test—hitting hundreds of millions of checkbox updates (passing 650 million) before being shut down two weeks later. The core insight wasn’t about checkboxes themselves; it was about how to engineer “shared state at massive scale” when every user interaction must appear instantly to everyone else. The project also became a practical lesson in cost control: bandwidth and update frequency—not just server CPU—ended up driving the hardest scaling decisions.

The site started with a deliberately simple architecture: the checkbox state lived as a compact bitset (1 million bits, one bit per checkbox). Clients rendered only what was visible (using React Window) and relied on updates rather than re-rendering the entire million-item DOM. When a user toggled a checkbox, the server flipped the corresponding bit and broadcast the change to all connected clients, avoiding the need to send a million checkbox objects over the wire.

Early load was dominated by the mismatch between expected traffic and reality. Tens of thousands of users arrived within hours from Hacker News, Mastodon, and Twitter; the site crashed repeatedly until the system stabilized by day two. The initial approach used JSON and base64-encoded snapshots for full state recovery, plus incremental updates for normal operation. That choice created a heavy serialization and payload overhead, and the system eventually ran into Redis connection pressure and WebSocket throughput limits.

Scaling required a series of targeted engineering pivots. The most important were (1) batching updates to reduce per-message overhead, (2) moving away from socket.io toward raw WebSockets to cut framing and inefficiency, and (3) adding connection pooling—though it initially conflicted with the deployment model and still required careful tuning. Redis was treated as the shared state store: it held the bitset and used Pub/Sub to fan out events. To recover from missed WebSocket updates (e.g., backgrounded tabs), the system periodically sent full snapshots, but snapshot frequency and incremental update size had to be reduced to keep bandwidth—and therefore cost—under control.

A key debugging moment came from state synchronization: without ordering metadata, clients could apply stale incremental updates after receiving a newer full snapshot, producing incorrect checkbox views. The fix was to introduce timestamps/counters so clients could drop out-of-date update batches and only apply increments that matched the latest snapshot baseline.

As traffic kept climbing, the project also faced adversarial behavior and operational failures. A bug allowed checkbox counts to jump into the 100 million range, effectively corrupting the intended bitset size and forcing a rollback to a truncated, validated state. Later, bots drove extreme traffic, leading to a denial-of-service situation that was mitigated by putting the site behind Cloudflare and tightening rate limiting. When Redis replicas and connection pooling didn’t behave as expected, the solution became pragmatic: spin up replicas, route traffic to private IPs, and add automated process restarts when Redis connections caused Flask workers to crash.

Finally, the shutdown plan turned the same real-time engineering into a lifecycle feature. Checkboxes were “frozen” if they weren’t toggled within a time window; frozen state was tracked in Redis and distributed like the main bitset. This allowed the system to gradually stop accepting meaningful changes and wind down safely. The project concluded with a Go rewrite that delivered major performance gains over Python, and a broader takeaway: building fast, hackable systems can work—if scaling constraints (especially bandwidth, serialization, and ordering) are treated as first-class design problems from day one.