Gzip is all You Need! (This SHOULD NOT work)

A surprisingly effective sentiment classifier can be built from a simple recipe: compress text with gzip, convert those compression results into normalized compression distances (NCD), and then run K-nearest neighbors (KNN) on the resulting distance vectors. On common sentiment data, this “parameter-free” style approach reaches roughly the mid-70% accuracy range—far above random guessing—despite using no neural network and only a few dozen lines of straightforward code. The core takeaway is that statistical compression behavior can encode enough information about language patterns (like common phrasing tied to positive vs. negative sentiment) for KNN to exploit.

The method starts by representing each text string numerically. For any two strings X1 and X2, gzip compresses them individually and together, and NCD is computed from the combined compressed length relative to the smaller and larger individual compressed lengths. Each training example becomes a feature vector: it stores its NCD values against every training sample. That means classification is driven by how similarly a new text compresses relative to the entire training set’s “compression neighborhood,” not by token counts or embeddings.

Training then uses standard KNN. For a test string, the model compresses it, computes its NCD vector against the training set, and finds the closest neighbors under the NCD distance structure. Those neighbors’ sentiment labels vote the final prediction. The computational bottleneck is the all-pairs NCD calculation: every sample must be compared to every other sample, producing a quadratic workload. The implementation speeds this up with multiprocessing while preserving the correct ordering of the NCD vectors so each training row aligns with the right sample.

Results are mixed when compared to the original research claim that this compressor-based method could beat BERT on sentiment. Using a small dataset (500 samples), accuracy lands around 70% in the author’s tests, with noticeable variance depending on which samples are drawn. The method stabilizes as dataset size grows: around 10,000 samples, accuracy settles near 75.7%. That’s still below the paper’s reported performance, but it remains strong for a baseline that uses only gzip, NCD, and KNN.

Part of the discrepancy is traced to a likely evaluation issue in the original paper: using K=2 in KNN can create ties, and the paper’s tie-handling reportedly counted ties as correct without a tie-breaker. Correcting that kind of mistake can materially change reported accuracy. Even after accounting for that, the approach still performs meaningfully, suggesting the compression-distance representation is not just a fluke.

The analysis also highlights practical limitations. Accuracy depends on text length: very short strings tend to be misclassified, and the classifier appears to work best when inputs are within a length range common to the training data (roughly 200+ characters, and more reliably around 600+ in these experiments). Attempts at adding dimensionality reduction via PCA didn’t reveal extra structure to exploit. Overall, the work functions as a reminder that non-neural baselines can still capture real signal—and that revisiting “first principles” methods can uncover useful, lightweight alternatives to deep learning in some NLP settings.