Looking For Grammar In All The Right Places | Alethea Power | OpenAI Scholars Demo Day 2020

Alethea Power’s project targets a practical, high-stakes question: where inside GPT-2 does grammar information live, and how can that location be measured well enough to improve interpretability and safety. Power frames interpretability as “mind-reading” for neural networks—opening the model to see how it represents and processes information—because AI systems now shape daily life and can amplify bias, waste energy, and enable misuse. Her specific focus is grammar: understanding how GPT-2 handles English structure could help reduce harmful outputs, make models more efficient, and even support longer-term efforts to connect machine representations to human thought.

Power begins by contrasting traditional software with deep learning. In software engineering, humans write explicit logic from inputs to outputs. In deep learning, humans instead provide training data and mathematical objectives; the resulting “software” is harder to inspect because it’s encoded in parameters rather than readable code. She then motivates why interpretability matters personally and socially: airport scanning systems can flag transgender people for additional screening; other AI-driven systems (including those used in transportation) have been linked to higher risk for people of color. Better understanding of model internals, she argues, can reduce bias and improve efficiency by enabling smaller networks that do the same job with less compute.

To study grammar inside GPT-2, Power narrows the problem to a tractable slice: how GPT-2 represents English grammar. She uses GPT-2 (including a smaller variant that fits on a home GPU) and builds a “grammar modeling layer” on top of the transformer stack. Instead of predicting the next word, the modified model predicts grammatical annotations—three granularities: simple part of speech, detailed part of speech, and syntactic dependencies. She constructs datasets of 300,000 sentences each, tagged using spaCy, and trains the grammar layer as a probe for information available at different depths.

Her analysis uses attention-head behavior to infer how information is being mixed. She measures entropy of attention matrices across layers: higher entropy suggests more complex mixing of information across token positions. The entropy patterns show that lower layers perform more restructuring for the grammatical task than upper layers. She then tests where grammar becomes “readable” by training the probe on top of each layer and tracking performance (loss/score). Grammar tagging improves markedly from early layers and peaks around layers 5–6 for syntactic dependencies, while information about incoming tokens is strongest in lower layers and information about expected outgoing structure becomes more accessible higher up.

Power interprets this as information being “rotated” into an abstract space by attention heads—then “stretched” or “warped” so that grammatical features become linearly accessible at specific depths. To go further, she truncates GPT-2 to the top half and performs head-level ablations to identify which attention heads matter for each grammatical structure. For some structures, removing nearly all heads still preserves strong performance; for others, only a small subset is needed. She concludes that grammar is not uniformly distributed across the network: different grammatical phenomena rely on different sub-networks, and mapping those head dependencies could eventually enable more targeted interpretability and model understanding.