Get AI summaries of any video or article — Sign up free
Why Making a Debugger is So Hard! | The Standup (ft. Ryan Fleury) thumbnail

Why Making a Debugger is So Hard! | The Standup (ft. Ryan Fleury)

The PrimeTime·
5 min read

Based on The PrimeTime's video on YouTube. If you like this content, support the original creators by watching, liking and subscribing to their content.

TL;DR

Debuggers and printf logging both collect runtime information; the difference is the workflow cost and how quickly the information can be gathered.

Briefing

Debuggers are only “hard” because they sit across two worlds at once: they must control a running program at a low level, yet present massive amounts of state in a fast, interactive UI. That mismatch helps explain why many developers fall back on printf-style instrumentation—and why a debugger can feel slower or unreliable when the tooling isn’t up to the job.

Ryan Fleury (of the RAD Debugger project) draws a direct line between logging and debugging: both collect information from a running program. The difference is the amount of work required to get that information. A debugger effectively runs alongside the program, gathering logs without permanently modifying the target—until it hits practical limits, at which point developers still add manual logging. So “printf vs debugger” isn’t a philosophical split; it’s mostly a performance and usability split.

The conversation then zeroes in on why debuggers often lose to printf in practice: many debuggers are slow or clunky, especially on Linux. Fleury argues that if debuggers were fast enough, they would beat instrumentation because they can collect the same data without the compile–run–edit loop. But when the UI and workflow are inefficient, the debugger’s theoretical advantage evaporates. He describes RAD Debugger’s goal as removing that choice—making data collection fast enough that developers don’t have to decide between “instrumentation” and “interactive inspection.”

A major technical bottleneck is stepping—advancing execution “over” a line of source code. Stepping isn’t a simple mapping from source lines to CPU instructions. Compilers can turn one line into many instructions, reorder or interleave code under optimization, and even stop in the middle of a line. To implement stepping, a debugger must translate source-level intent into instruction-level control while coordinating with the operating system’s scheduling and control-transfer mechanisms.

Fleury explains how the common approach uses trap instructions like x86’s int3 to force the program to interrupt and return control to the debugger. But each trap can trigger expensive round trips between user space and kernel space, making conditional breakpoints and fine-grained stepping dramatically slower—especially inside tight loops. He also outlines why recursive stepping is particularly nasty: stepping “over” a call in recursive code requires tracking stack behavior and return points, often turning the debugger into a state machine rather than a straightforward “next line” feature.

The project history behind RAD Debugger adds context: it began as a Linux-focused effort funded by Valve and RAD Game Tools, driven by the lack of strong debugging tools for game developers. Over time, the team built not just an engine but also a UI capable of visualizing complex runtime data.

That UI is where RAD Debugger tries to justify itself beyond raw breakpoints. Instead of forcing developers to dump data and re-parse it, it supports watch-window visualizers—turning pointers into tables, rendering memory as hex views, and even showing 3D geometry, textures, and bitmap previews live as execution steps. The key claim is speed and integration: a debugger feature only matters if it’s faster than printf-style workflows.

By the end, the central takeaway is pragmatic. Debuggers are “awesome if they’re good” and “useless if they aren’t,” and the hardest part isn’t just finding bugs—it’s building a system that can control execution precisely while presenting the right information quickly enough to beat manual instrumentation.

Cornell Notes

Debugging and printf-style logging aren’t opposites; both collect information from a running program. The real difference is the cost of getting that information: debuggers aim to gather it automatically and interactively, while printf requires manual edits, recompiles, and reruns. Stepping is especially difficult because source lines map messily to optimized machine instructions, and each trap-based step can require expensive kernel/user round trips. RAD Debugger targets these pain points by speeding up data collection and building a UI that can visualize complex state (images, memory, geometry) directly in the watch window. When a debugger is fast and reliable, it can beat printf; when it’s slow or unreliable, developers revert to instrumentation.

Why does “printf debugging vs debugger” often feel like a real choice for developers?

Because debuggers can be slower in practice when their UI/workflow is inefficient. Fleury argues that debuggers should collect the same information faster than manual instrumentation, but many tools—especially on Linux—are “too freaking slow” or clunky. When adding printf lines requires a compile–run–inspect loop, a good debugger can win by collecting data without that loop. When the debugger’s interaction cost is high, printf becomes the faster path, even though it’s conceptually redundant.

What makes stepping over a source line so hard?

Source-level stepping requires mapping a source line to a set of instruction addresses, then arranging execution so the thread stops after leaving that line. Compilers can generate many instructions per line and optimization can interleave instructions from different lines. Stepping also must handle cases where execution stops mid-line (e.g., stepping out of a function may land in the middle of a caller’s line). A naive “single-step one instruction” approach is accurate but extremely slow due to repeated kernel transitions.

How do trap instructions like int3 help stepping, and why can they still be slow?

On x86, int3 is a one-byte trap instruction that can be written into the code stream so that when the CPU executes it, execution interrupts and control returns to the debugger. This avoids the repeated single-instruction round trips of CPU single-step. However, each trap hit can still cause expensive context switching between user-level debugger code and kernel-level handling, so conditional breakpoints inside tight loops can become prohibitively slow.

Why is stepping “over” recursive calls harder than stepping through linear code?

In recursive code, a trap placed for “the next line” may be hit again in a deeper call stack frame, so the debugger can’t just rely on instruction addresses. The debugger must track stack behavior (e.g., stack pointer changes) and determine when the call returns to the correct frame. Fleury describes this as requiring a state-machine-like approach rather than a simple trap placement strategy.

What does RAD Debugger add beyond breakpoints that makes it more than a “fancy printf”?

It emphasizes visualization inside the watch window using visualizer rules. Examples include viewing bitmap/pixel data with width/height/format parameters, rendering memory as hex views, and showing 3D geometry and textures. The watch-window integration matters because it updates live as execution steps, and it can automatically associate types with visualizers so developers don’t have to reconfigure the display every time.

How did RAD Debugger’s origins connect to Linux and game development needs?

The project traces back to a Valve-and-RAD Game Tools relationship and a desire to ship Linux support with better debugging tools for game developers. Valve wanted Linux debugging capabilities, and RAD’s cross-platform experience made it a natural partner. The effort aimed to create a viable debugger solution on Linux, including specialized visualization for graphics-related debugging.

Review Questions

  1. What specific technical reasons make stepping over a source line slower than many developers expect?
  2. Explain how trap-based stepping (int3) reduces overhead compared with CPU single-stepping, yet still suffers from performance costs.
  3. Why do recursive calls force debuggers to track more than just “next line” instruction addresses?

Key Points

  1. 1

    Debuggers and printf logging both collect runtime information; the difference is the workflow cost and how quickly the information can be gathered.

  2. 2

    Many developers avoid debuggers not because debugging is impossible, but because debugger UI/workflows and performance can be too slow or unreliable to beat instrumentation.

  3. 3

    Stepping is hard because source-to-instruction mapping is messy under optimization and execution can stop mid-line.

  4. 4

    Trap-based stepping (e.g., x86 int3) can reduce the overhead of single-instruction stepping, but trap hits can still incur expensive kernel/user round trips.

  5. 5

    Conditional breakpoints become especially slow when each condition check requires returning to the debugger after every trap event.

  6. 6

    Recursive stepping requires tracking call/return context (often stack behavior), turning stepping into a more complex state-management problem.

  7. 7

    RAD Debugger’s differentiator is not only control-flow debugging but also fast, integrated visualization in the watch window (images, memory, geometry) that beats manual dump-and-inspect loops.

Highlights

“printf vs debugger” isn’t a real dichotomy—both are ways to collect logs; the deciding factor is how much work and time it takes to get the information.
Stepping over a line isn’t just “next line”: compilers can reorder and interleave instructions, and stepping can land mid-line or require complex mapping.
Trap-based stepping with int3 avoids some overhead, but conditional breakpoints can still be dramatically slowed by repeated kernel/user transitions.
RAD Debugger’s watch-window visualizers aim to make debugging faster than dumping data and re-parsing it—especially for images and 3D geometry.
Recursive stepping forces debuggers to manage stack context, not just instruction addresses, because “the next trap” may occur in deeper frames.

Topics

  • Debuggers
  • Stepping
  • RAD Debugger
  • Debug Info Formats
  • Conditional Breakpoints

Mentioned