LLaMA2 Tokenizer and Prompt Tricks

LLaMA 2’s behavior hinges less on “magic prompting” and more on two concrete levers: the tokenizer’s limited vocabulary size and, especially, the exact system prompt wrapped in the model’s special chat tokens. The result is that the same model can swing from overly cautious, refusal-prone answers to direct, useful responses—simply by changing the system prompt text and how it’s packaged.

On the tokenizer side, LLaMA 2 uses a 32,000-token vocabulary (the transcript notes it matches the LLaMA-1 tokenizer). That smaller vocabulary can be efficient for many Romanized languages, where words and subwords tend to break into predictable chunks. But it becomes a poor fit for languages like Thai and likely Arabic, because the tokenizer may not form the right “clumps or groups of tokens” that correspond to words. The transcript contrasts this with alternatives such as OpenLLaMA, which uses a much larger tokenizer vocabulary (50,000+ tokens), implying better coverage for non-Roman scripts.

The more dramatic control comes from special tokens and the system prompt. The transcript points out that LLaMA 2 chat-style models rely on tokens such as SYS (system prompt) plus instruction framing tokens like INST, along with begin/end markers for each section. In practice, the model appears trained to treat the system prompt as strict policy. The default system prompt shown is heavily constrained: it demands helpfulness while forbidding harmful or illegal content, requires social bias avoidance, and instructs the model not to fabricate answers.

When that strict system prompt is left intact, the model often over-corrects even benign requests. Examples include adding unnecessary caveats (e.g., treating “England is not a country” as a reason to expand beyond a simple capital question), refusing or blocking tasks like “convert the following to JSON” by claiming the text contains personal identifying information, and declining to provide opinions or emotional responses (“I don’t have feelings”). Even when the user’s intent is clear—such as asking about Homer Simpson or Hogwarts—the model repeatedly inserts “fictional character/school” disclaimers, making outputs feel less cooperative than expected.

Changing the system prompt flips the behavior quickly. A revised system prompt in the transcript instructs the assistant to be helpful, never refuse, avoid correcting the user, always answer opinions, and—crucially—stop adding information the user didn’t ask for. With that change, the capital-of-England question becomes a straightforward “London,” the tone shifts away from “as an AI” phrasing, and tasks like JSON conversion succeed. The model also becomes willing to express preferences, such as openly liking The Simpsons and naming Homer Simpson as a favorite.

The takeaway is practical: system prompts aren’t just safety text—they’re a steering mechanism. By using the SYS and instruction tokens correctly and crafting a system prompt aligned with the desired interaction style, users can make LLaMA 2 feel substantially more useful. The transcript also notes that larger variants (e.g., 13B and 70B) may sometimes follow formatting tasks like JSON conversion even without prompt changes, but smaller models may require the system prompt adjustment to behave as intended.