Get AI summaries of any video or article — Sign up free
The Internet Was Weeks Away From Disaster and No One Knew thumbnail

The Internet Was Weeks Away From Disaster and No One Knew

Veritasium·
5 min read

Based on Veritasium's video on YouTube. If you like this content, support the original creators by watching, liking and subscribing to their content.

TL;DR

A widely used dependency can become a single point of failure: XZ’s compromise enabled a backdoor into OpenSSH via shared-library and linker behavior.

Briefing

A single, long-maintained compression library—XZ—was quietly weaponized to open a backdoor into OpenSSH, putting millions of Linux systems at risk. The danger wasn’t theoretical: the exploit targeted the SSH authentication path, meaning a successful compromise could enable spying, ransomware, or even large-scale disruption—potentially at the scale of “millions of internet servers” within weeks.

The chain of failure began with how Linux software is built and maintained. Linux’s dominance—running on Android, most internet servers, supercomputers, and sensitive government and defense systems—rests on the assumption that open-source code is secure because it’s widely inspected. That belief is often summarized as “Linus’s Law,” but the transcript stresses a structural weakness: open source is not one big project. It’s an ecosystem of thousands of small dependencies, many maintained by individuals doing unpaid work. When one of those “small” components becomes a weak link, the blast radius can be enormous.

XZ illustrates that fragility. The project’s maintainer, Lasse Collin, had spent about two decades keeping XZ compatible with evolving systems, but burnout and pressure from the community left the project vulnerable to takeover. Into that gap stepped Jia Tan, a contributor who appeared helpful and responsive—exactly the kind of person a stressed maintainer might rely on. Over time, Jia Tan gained enough influence to land changes that would later prove malicious.

The backdoor itself was engineered through three stages. First came a Trojan-horse insertion: Jia Tan hid a payload inside binary test artifacts that typically go unread, then used build-time tricks to unpack it into the XZ library. Second came the timing attack (“Goldilocks”): the payload needed to hijack a narrow window during program startup when the system’s Global Offset Table (GOT) is being populated and before it becomes read-only. To do that, the exploit leveraged Linux’s IFUNC mechanism and a dynamic audit hook—tools meant for performance or debugging—repurposed to swap in malicious behavior at the exact moment the SSH code would trust it. Third came the “cat burglar” phase: once inside, the backdoor waited for a hidden master key and used its own mini authentication exchange so it would only activate for the attacker, while also erasing traces to reduce the chance of detection.

The plot nearly succeeded. Jia Tan pushed the compromised XZ into major Linux release pipelines, including Fedora pre-release builds, and worked to ensure the payload would reach OpenSSH via dependency chains. A critical turning point arrived when Andres Freund, a Microsoft employee working on Postgres, noticed consistent connection slowdowns in Debian’s unstable release—about 400 to 500 milliseconds. That timing anomaly led him to trace the delay back to XZ, then to uncover the backdoor’s meticulous memory-hunting and obfuscation.

After Andres reported the issue to Debian security and publicly, Red Hat rolled back Fedora and the community scrambled to audit the ecosystem. The episode became a warning about modern cyber risk: even when code is open and scrutinized, attackers can still exploit human bottlenecks—burnout, dependency chains, and the assumption that “enough eyeballs” will always catch the one critical piece before it reaches production.

Cornell Notes

The XZ compression tool was compromised in a way that enabled a backdoor into OpenSSH’s RSA authentication path. The attack relied on the open-source ecosystem’s dependency structure: many systems trust a vast chain of libraries, and a single weak link maintained by one person can become a system-wide vulnerability. Jia Tan gained influence over XZ during a period when maintainer Lasse Collin was under pressure, then inserted a Trojan payload into binary test artifacts and activated it during a narrow startup window using Linux mechanisms like IFUNC and dynamic audit hooks. The backdoor only triggered when an attacker-provided hidden master key matched, and it attempted to reduce detection by wiping evidence. The threat was stopped when Andres Freund noticed consistent SSH connection slowdowns in Debian unstable and traced them back to XZ, prompting rapid rollbacks and community investigation.

Why did a compression library like XZ end up threatening SSH authentication across the internet?

Because OpenSSH depends on shared libraries, and those libraries depend on other libraries—forming a dependency chain. XZ was widely used in major Linux distributions, and the exploit targeted a point in the SSH authentication process by hijacking a shared crypto function (RSA decryption) indirectly through the dynamic linker’s Global Offset Table (GOT). Once the malicious XZ code was present, it could influence OpenSSH behavior without directly modifying OpenSSH source code.

What made the XZ backdoor hard to detect before it reached real systems?

It was designed to be both stealthy and precisely timed. The payload was hidden inside binary blobs associated with test data—materials that typically aren’t reviewed. It also activated only during a “Goldilocks” window: after the GOT entry for RSA decryption was filled legitimately but before the table became read-only. That timing reduced the chance of crashes or obvious behavior during normal testing.

How did the exploit use Linux-specific mechanisms to redirect execution?

The transcript describes using IFUNC resolvers and a dynamic audit hook. IFUNC can select optimized implementations early in program startup, and the exploit repurposed that early execution to set up an audit hook. When the dynamic linker later wrote the real RSA decryption address into the GOT, the audit hook fired and swapped in the attacker’s payload at the exact moment the system would trust the function pointer.

Why didn’t the backdoor simply open access for anyone who installed the update?

Activation required a hidden master key and an additional authentication step. The backdoor checked for a shared secret and then authenticated the user before running the payload. That meant most installations would behave normally, while attackers with the correct key could gain control.

What clue stopped the attack, and why was it persuasive?

Andres Freund noticed consistent SSH connection slowdowns—around 400–500 milliseconds—in Debian unstable. The delay matched results from testing the XZ compromise in a controlled way, suggesting the slowdown wasn’t random. Tracing the timing anomaly back to XZ led to discovery of the backdoor’s memory-hunting and obfuscation logic.

How did open-source “many eyeballs” fail in this case?

The transcript argues that open source is an ecosystem, not a single project. Many dependencies are maintained by individuals doing unpaid work, and a malicious takeover can propagate through downstream projects before anyone notices. The XZ case shows how a single compromised maintainer-controlled component can quietly become a critical part of security infrastructure.

Review Questions

  1. What specific part of the SSH process did the XZ backdoor target, and how did it reach that target through dependency chains?
  2. Explain the role of the GOT and the “Goldilocks” timing window in the exploit’s activation.
  3. Why did Andres Freund’s observation of connection slowdowns matter, and what did it lead him to investigate next?

Key Points

  1. 1

    A widely used dependency can become a single point of failure: XZ’s compromise enabled a backdoor into OpenSSH via shared-library and linker behavior.

  2. 2

    Open-source security depends not just on code visibility, but on maintenance capacity; burnout and small-team dependency chains create exploitable gaps.

  3. 3

    The XZ payload was hidden in binary test artifacts and unpacked during the build process, reducing the chance of human review.

  4. 4

    The exploit required precise timing during program startup, activating only after legitimate GOT population but before the table was locked read-only.

  5. 5

    Linux mechanisms like IFUNC and dynamic audit hooks were repurposed to swap function pointers at the moment they were written.

  6. 6

    The backdoor was designed to trigger only with a hidden master key and to reduce detection by wiping evidence.

  7. 7

    The attack was detected through behavioral signals—consistent SSH connection slowdowns—rather than obvious crashes or overt malicious output.

Highlights

XZ’s compromise threatened SSH at scale by hijacking the RSA authentication path through dependency and linker mechanics, not by directly altering OpenSSH source code.
The “Goldilocks” window—between GOT entry setup and read-only locking—was central to making the backdoor both reliable and stealthy.
Andres Freund’s discovery hinged on a subtle, repeatable slowdown of about 400–500 milliseconds, which traced back to XZ in Debian unstable.
The backdoor’s activation required a hidden master key and included steps to erase evidence, aiming to keep most systems looking normal.

Topics

  • Open Source Security
  • Dependency Chains
  • XZ Backdoor
  • OpenSSH
  • Linux Exploitation

Mentioned