Google takes down the internet! (The Standup)

A Google Cloud outage last week briefly knocked out core quota/authorization checks across regions, turning many Google Cloud API calls into 503 errors and effectively taking large parts of the internet offline. The immediate trigger, according to Google’s published account, was a null pointer in code tied to a policy change: a missing-field condition produced a null pointer, and the resulting crash collapsed the cloud management plane. The outage mattered far beyond Google’s own services because countless websites and platforms route through Google Cloud components—so even non-Google businesses experienced failures and cascading “internet down” reports.

The discussion zeroed in on how a simple robustness gap can become a system-wide failure. The crash path appears to have been reachable only under a specific configuration for the quota system, meaning the problematic code likely sat dormant until a particular policy update exercised it. That raised questions about testing and fuzzing practices. Google sponsors OSS-Fuzz, a large-scale fuzzing effort aimed at finding memory-corruption bugs, yet the incident reportedly came down to a null-pointer dereference—something that, in principle, could be caught by straightforward input-structure validation or fuzzing that mutates configuration data. Participants argued that a memory fuzzer is essentially a coverage-seeking test that mutates inputs to force edge cases; if the relevant “missing fields” scenario had been exercised, the crash should have surfaced earlier.

Another theme was the opacity of dependency chains. Even when a single provider fails, downstream services can fail too—Cloudflare was cited as reporting disruptions after the Google Cloud incident, attributed to Cloudflare Workers relying on Google Cloud. That sparked a broader point: companies often lack a clear, shared map of which internal systems depend on which external services, making it hard to compute real risk. Without knowing whether two “different” vendors share the same upstream dependency, redundancy can be illusory.

The conversation compared this outage to other high-profile incidents, like CrowdStrike’s, where configuration-driven behavior caused production failures. In both cases, the failure emerged when production data finally fed the bad path. Participants also discussed how large organizations test resilience: Netflix’s “Chaos Monkey” and “Chaos Kong” approaches can kill instances or entire regions to validate failover, but they can’t easily simulate an external provider outage without also collapsing everything that depends on it.

Finally, the group debated “fail open” versus “fail closed” design. Google’s post-incident notes reportedly suggested the quota/authorization system should have failed open—allowing requests to proceed in a constrained way rather than denying everything and cascading into an outage. The counterpoint was that “fail open” in authorization is not straightforward: allowing requests blindly can create security risk, and physical systems (like car locks) show why fail-open behavior can be constrained by real-world mechanics. The takeaway was that reliability engineering is inseparable from threat modeling and from the practical limits of how systems degrade under failure.

Alongside the technical debate, the episode drifted into lighter commentary about smart-home frustrations, paranoia about connected devices, and the practical reality that modern systems can fail due to incompetence as much as due to adversarial attacks. But the central story stayed consistent: a null pointer in a cloud control component exposed how tightly the internet is coupled to a handful of upstream services, and how difficult it remains to test, model, and design for those dependencies before they bite.