Second-generation PLINK: rising to the challenge of larger and richer datasets

This paper addresses a practical bottleneck in genome-wide association studies (GWAS) and population genetics: the original PLINK codebase (PLINK 1.x) is widely used, but it was designed around a data representation and computational patterns that do not scale well to today’s “larger and richer” genetic datasets. The authors’ research question is essentially engineering-focused: how can PLINK be redesigned to (i) run substantially faster on modern multicore hardware, (ii) handle datasets larger than available RAM, and (iii) prepare for future compatibility needs arising from imputation outputs (probabilistic genotype calls), phasing information, and multiallelic variants.

This matters because the field has shifted from modest, hard-call genotype matrices toward whole-genome sequencing and imputed datasets that are both larger and more information-rich. Many downstream analyses—identity-by-descent (IBD), genomic relationship matrices (GRMs), linkage disequilibrium (LD) pruning, haplotype block estimation, and permutation-based association tests—are computationally heavy and are often executed repeatedly inside pipelines. When software cannot exploit modern CPU parallelism or cannot operate directly on compressed/packed representations, researchers either lose time, lose statistical power (e.g., fewer permutations), or are forced to use high-end computing resources.

Methodologically, the paper is a technical note describing the design and implementation of “second-generation” PLINK. The first major release from this effort is PLINK 1.9, which is positioned as a drop-in replacement for PLINK 1.07 in most workflows. The authors evaluate performance using timing benchmarks across multiple machines and datasets, comparing PLINK 1.9 against PLINK 1.07 (and sometimes against other tools such as GCTA, Haploview, and PERMORY). The paper does not present a randomized clinical/biological study; instead, it uses empirical computational experiments.

The core methodological changes in PLINK 1.9 include: (1) extensive use of bit-level parallelism by rewriting inner loops to operate on packed genotype data; (2) improved “bit population count” (popcount) implementations using SSE2-based algorithms for portability; (3) multithreading and optional distributed computation via a cluster-splitting flag; (4) memory efficiency by avoiding storing the full genotype matrix in RAM for many operations; and (5) algorithmic improvements in several statistical tests and analyses.

A key example of the bit-parallel approach is the identity-by-state (IBS) computation. The authors replace per-marker, per-pair loops with operations over blocks of markers using XOR and bit population counts, masking missing calls. They report that PLINK 1.9 can process a block of 960 markers in less than twice the time that PLINK 1.07 takes to handle a single marker—an illustration of how bit-parallelism changes the effective scaling of inner-loop work.

They also introduce an early-termination strategy to reduce the computational complexity of Hardy–Weinberg equilibrium (HWE) exact tests and Fisher’s exact tests. For SNP-HWE, they argue that while a naive exact method is size- $O(n)$ in the total contingency table entries, only $O(\sqrt{n})$ likelihood terms meaningfully affect the final p-value because likelihoods decay super-geometrically. They terminate when additional terms become too small to change the p-value under IEEE-754 double precision.

For haplotype block estimation, PLINK 1.9 accelerates the existing confidence-interval-based method (from Gabriel et al., via Haploview) by streamlining diplotype frequency computations, reducing the number of likelihood evaluations needed to classify variant pairs as “strong LD,” and avoiding large intermediate tables by updating only sliding-window counts. They also incorporate pruning logic that can skip classification of many variant pairs once it becomes impossible for a block to satisfy the required “strong LD vs recombination” ratio.

The paper’s performance comparisons are organized into tables. Across datasets and machines, the authors report speedups frequently exceeding two orders of magnitude and sometimes reaching three orders of magnitude for common operations. For example, in Table 1 (initialization and basic I/O, $--freq$), the ratio PLINK 1.07/PLINK 1.90 ranges from modest to very large depending on dataset and machine. On the largest chromosome dataset (chr1), PLINK 1.9 runs $35.0$ times faster on Mac-2 and $5.3$ times faster on Mac-12, while on Linux64-512 the ratio is $5.3$ and on Win64-2 it is $8.1$. For the full 1000 Genomes phase 1 chromosome 1 SNP-only subset (chr1snp), ratios include $10.2$ on Linux32-8 and $2.2$ on Win64-2.

More striking improvements appear in compute-heavy tasks. Table 2 (IBS matrices and clustering) shows that for synth2p on Linux64-512, IBS matrix calculation drops from $\sim 98k$ seconds in PLINK 1.07 to $25.3$ seconds in PLINK 1.90 (ratio $\sim 3.9k$). For synth1p on Linux64-512, the ratio is $400$ (PLINK 1.07 $1492$ seconds vs PLINK 1.90 $3.7$ seconds). For chr1snp on Linux64-512, the ratio is $320$ ($\sim 14k$ seconds vs $43.1$ seconds). These results are consistent with the authors’ claim of 1–4 orders of magnitude speedups for several operations.

Table 3 (GRM calculation, $--make-grm-bin$) compares PLINK 1.9 to GCTA. On synth1p for Linux64-512, GCTA takes $4.4$ seconds while PLINK 1.90 takes $8.3$ seconds (ratio $0.53$, meaning GCTA is faster there). However, on other platforms PLINK 1.9 is much faster than GCTA or avoids platform limitations (GCTA cannot run on OS X or Windows in the authors’ comparison). For example, on synth1p for Win64-2, GCTA takes $367.2$ seconds vs PLINK 1.90 taking $6.6$ seconds (ratio $56$).

Table 4 (LD-based pruning, $--indep-pairwise$) shows that PLINK 1.9 can be dramatically faster, but also that some steps remain single-threaded in PLINK 1.9 (notably $--indep-pairwise$ as of the paper). For synth1 on Linux64-512, PLINK 1.07 takes $462$ seconds and PLINK 1.90 takes $0.60$ seconds (ratio $770$). For synth2 on Linux64-512, PLINK 1.07 is $\sim 120k$ seconds vs PLINK 1.90 $26.4$ seconds (ratio $4.5k$). For chr1 on Linux64-512, PLINK 1.07 is $\sim 950k$ seconds vs PLINK 1.90 $1553.3$ seconds (ratio $610$).

Table 5 (haplotype block estimation, $--blocks$) indicates that PLINK 1.9’s rewrite can turn previously infeasible runs into feasible ones. Many PLINK 1.07 runs are marked “nomem” (ran out of memory). Where PLINK 1.07 completes, PLINK 1.9 is faster; for synth1 with $--ld-window-kb=500$ on Mac-12, PLINK 1.07 takes $\sim 45k$ seconds vs PLINK 1.90 $3873.0$ seconds (ratio $1.3$). For chr1 with $--ld-window-kb=500$ on Win64-2, PLINK 1.07 is “nomem” while PLINK 1.90 takes $1037.2$ seconds (ratio not applicable due to failure).

Table 6 (association analysis permutation tests) demonstrates that PLINK 1.9 extends fast permutation strategies (PERMORY) and makes Fisher’s exact tests practical in permutation contexts. For synth1 with $--trend$ and $--mperm=10000$, on Linux64-512 PLINK 1.07 is $\sim 17k$ seconds vs PLINK 1.90 $285.0$ seconds (ratio $2.8$). On Win32-2, PLINK 1.07 $\sim 35k$ vs PLINK 1.90 $1444.2$ (ratio $61.5$). For Fisher’s exact permutation tests on synth1, on Linux64-512 PLINK 1.07 $\sim 120k$ vs PLINK 1.90 $35k$ (ratio $35k/120k$ reported as $3.4$). For QT association ($--assoc(QT)$) on Linux64-512, PLINK 1.07 $\sim 29k$ vs PLINK 1.90 $29.2$ seconds (ratio $990$).

The authors acknowledge limitations that follow from their engineering choices. The most important is that PLINK 1.9 still uses the PLINK 1 binary file format, which cannot represent probabilistic genotype calls, phase, or multiallelic variants. They therefore propose PLINK 2.0 with a new core file format that can represent probabilities, phase, and multiallelic variants efficiently, while also maintaining an equivalent representation to the PLINK 1 format to avoid performance regressions. They also note that PLINK is not designed for analyses where a single coordinate system is inadequate (e.g., structural variation in whole-exome/whole-genome sequencing) and recommend PLINK/SEQ for such tasks.

Practical implications are clear: users can upgrade to PLINK 1.9 to obtain major speedups and better scalability, including the ability to handle datasets too large for RAM. This benefits researchers running GWAS pipelines, population genetics analyses, and permutation-heavy association tests, especially those without access to high-end computing clusters. The paper also signals that future compatibility with modern imputation and sequencing outputs will require PLINK 2.0’s richer data model.

Overall, the paper’s contribution is not a new statistical method per se, but a comprehensive systems-and-algorithms update to a foundational genomics tool, with extensive empirical evidence that careful bit-level and memory-aware engineering can yield order-of-magnitude performance improvements while preserving usability as a drop-in replacement for most existing PLINK workflows.