XGen-7B: Long Sequence Modeling with (up to) 8K Tokens. Overview, Dataset & Google Colab Code.

Salesforce’s XGen-7B is positioned as an open 7-billion-parameter language model built for long-context work, with an input sequence length that scales up to 8K tokens using standard dense attention. That matters because most widely used open models historically cap context around 2K tokens, limiting how much source material they can read before losing coherence. XGen’s design targets tasks where users need to ask questions or extract summaries from long documents—an area where longer context windows can translate directly into better results.

The training setup combines two stages and a multilingual-first approach. In the first stage, XGen is trained on a large RedPajama-style mixture drawn from Common Crawl, GitHub, books, and other sources, with coverage across 22 languages (including Bulgarian, French, Spanish, German, English, and Japanese). This is a notable contrast to some earlier open models that were overwhelmingly English-focused, which can make multilingual fine-tuning less effective. The second stage shifts toward code-generation capability via additional fine-tuning, using a separate dataset mix rather than continuing with the original large code corpus—an intentional change that aims to teach coding behavior without fully rebalancing the earlier data distribution.

Training is also described as computationally expensive at long lengths because attention cost grows quadratically with sequence length. To manage that, the model is trained progressively: first at shorter contexts (up to 2K), then 4K, and only later at 8K, with different token budgets allocated to each stage. The process includes discussion of training “spikes” (temporary instabilities) and how longer sequences up to 8K can trigger them, while reporting that these issues did not ultimately derail performance.

On evaluation, XGen-7B is reported to perform strongly on multitask understanding benchmarks, including a weighted average on the Massive Multitask Language Understanding (MMLU) suite, where it reportedly beats other open models such as Llama-family baselines and MPT/Falcon variants. For code generation, results are more mixed: the model appears comparatively stronger than some English-only open alternatives, but it still does not match top code-focused systems. The clearest reported strength is summarization and question answering over long text.

To test long-form QA, an in-house method generates questions from Wikipedia-derived passages across domains (physics, engineering, history, entertainment), asks the model to produce answers up to 256 tokens, and then uses GPT-4 to judge answer quality across metrics. The transcript also notes that this kind of evaluation isn’t fully standardized and may carry bias, even if the reported ranking favors XGen.

Finally, the transcript walks through practical usage in a Google Colab notebook using the Salesforce checkpoint repository. The 8K instruct variant is loaded with Transformers in 16-bit and then quantized to 8-bit for a T4 GPU, and inference is run with a chat-style prompt format. In live prompting, the model produces clean summaries and generally coherent answers, but it struggles with some coding tasks and makes license-related mistakes when asked to interpret a table of model licensing—highlighting that long-context competence doesn’t automatically translate to perfect reasoning on structured or legal details.