40x Faster Binary Search

High-throughput searching over a static, sorted array can beat classic binary search by reorganizing data for cache prefetching and then squeezing the remaining work through SIMD, batching, and careful memory layouts. The core target is a “static search tree” (an S+ tree variant) that returns, for each query Q, the smallest stored value ≥ Q (or u32 max when no such value exists). Instead of optimizing per-query latency, the work measures reciprocal throughput—how many independent queries can be answered per second—because modern CPUs can hide long memory waits when many queries are in flight.

The project starts from a baseline: Rust’s standard library binary_search over a sorted Vec<u32>. It then compares against an “Eytzinger” (implicit heap) layout, which reorders the array so that the next few binary-search steps land in nearby cache lines. That locality enables hardware prefetch to pull in values for several iterations ahead, letting the CPU overlap cache misses with useful work. When the dataset fits in cache, Eytzinger looks similar to binary search; once the array grows beyond L3, it can become several times faster—roughly matching the idea that prefetching four iterations ahead hides traversal latency.

From there, the S+ tree design focuses on cache-line efficiency rather than pointer chasing. The tree is stored as contiguous 64-byte-aligned nodes (16 u32 values per node), packed so that each cache line contains enough information to advance multiple logical levels. The layout duplicates keys across internal nodes (S+ / B+ style) so searches can avoid extra comparisons while descending. A key performance lever is branchlessness: rather than stopping early when a comparison fails, the search counts how many lanes are less than the query and uses popcount to compute the first index where the value is ≥ Q. SIMD vectorization turns the “linear scan within a node” into parallel comparisons, and subsequent hand-tuned AVX2 intrinsic work reduces instruction count and fixes unsigned-compare limitations by mapping u32 values into an i32-safe range.

The biggest throughput gains come from systems-level tactics. Batching multiple queries lets the CPU issue many memory reads concurrently; prefetching the next tree node for each query further smooths stalls when data lives in L3 or RAM. Interleaving work across tree levels—processing layer i for many in-progress batches—helps balance CPU-bound early steps with memory-bound later steps, improving overall throughput even when single-thread latency is not the focus.

The transcript also tests variants: changing node branching factor (e.g., 15 vs 16 values), reversing or fully expanding layouts, and partitioning the keyspace by prefix bits into multiple smaller trees. Partitioning can help on skewed real-world inputs (like DNA k-mers from human genome data) but often adds complexity and extra lookups, and it doesn’t consistently beat the simpler interleaved batching approach. Multi-threading then pushes the system toward the real bottleneck: aggregate RAM bandwidth. With enough threads, throughput approaches near-optimal behavior, and the overall speedup relative to baseline binary search reaches roughly 40x.

The takeaway is less about one magical data structure and more about aligning algorithms with CPU realities: memory ordering, cache-line packing, SIMD-friendly branchless logic, and throughput-oriented batching can dominate asymptotic “O(log n)” expectations in practice—especially for static search workloads like suffix-array searching in bioinformatics.