Using Semantic Trees In Place of Sentences | Munashe Shumba | OpenAI Scholars Demo Day 2018

Semantic trees—specifically dependency parse trees—can outperform plain sentence sequences for natural-language tasks like semantic relatedness when paired with an LSTM-style model. In a demo built around 10,000 sentence-pair comparisons, the dependency-tree approach reached lower mean squared error (0.35) than a baseline trained on regular sentence word sequences (1.3), reaching the target loss in far fewer training steps. The result matters because it suggests that preserving grammatical structure as an explicit tree can make learning more sample- and compute-efficient, even when the downstream model is sequence-based.

The work started with a semantic relatedness quiz: pairs of sentences were scored from 1 to 5 based on how closely they matched in meaning. Examples included near-paraphrases about “dogs fighting” and “dogs wrestling and hugging,” plus unrelated cases where only the presence of an actor overlapped. Instead of treating sentences as undifferentiated word sequences, the project treated meaning as layered: a high-level “essence” (e.g., fighting/wrestling) with additional details (who is fighting, what objects are involved, and when/which participants).

To generate the tree structure, the project used a syntactic parser (described as a “syntactic net,” producing dependency trees) and manually corrected parsing mistakes. Word embeddings came from GloVe, converting each token into a vector representation. The key technical challenge was feeding trees into an LSTM, which expects sequences. The solution was to linearize the dependency tree using a depth-first search traversal, producing a bracketed sequential form. In this representation, the parent-child structure becomes explicit via bracket tokens placed around subtrees, with the “main idea” appearing earlier in the traversal and more specific details appearing deeper in the resulting sequence.

Two models were trained for comparison: one consumed the linearized dependency trees, and the other consumed the original sentences as standard word sequences. Both achieved similar training loss, but the tree-based model converged dramatically faster—about 150 steps to reach the target level versus roughly 1.06 million steps for the sentence baseline. That convergence gap translated into the lower mean squared error reported for semantic relatedness.

Looking ahead, the project planned to move beyond generic LSTMs toward tree-structured recurrent models (noted as “three LSTMs” / an LSTM variant designed for trees) that can operate directly on tree inputs rather than relying on bracketed sequence conversion. There was also interest in applying the same dependency-tree manipulation to question answering using SQuAD, leveraging the fact that dependency trees are easier to edit: for example, reordering a set of children in the tree to generate new, structurally valid training examples. The demo also clarified that the relatedness scores were curated by people with consensus rather than derived from an automatic empirical metric, and it described how special bracket symbols were embedded separately from normal word vectors to prevent confusion with ordinary tokens.