Reza Shabani - How Replit Trained Their Own LLMs (LLM Bootcamp)

Replit’s Ghostwriter code-completion model is built through a tightly engineered pipeline designed to make smaller, cheaper, and more specialized LLMs—without sacrificing control over data, evaluation, or deployment. The core idea is that general-purpose models often miss what code-completion needs (especially across specific languages and coding styles), so Replit trains its own models using permissively licensed code at scale, then customizes tokenization, evaluation, and production serving to fit real user workflows.

The motivation for training in-house centers on customization, cost, and control. Replit argues that broad models like GPT-4 or “Chachi Turbo” aren’t well suited for code completion, and even within code-focused models (like Codex), language specialization matters—Replit’s users heavily use web stacks such as JSX and TSX, so its training is weighted toward those languages. Dependency reduction is another driver: Replit wants developers to access models without relying on a small set of external providers. Cost efficiency is framed as existential for scale: with roughly 20 million users, API pricing for tools like Copilot (noted as around “10 bucks a month”) would be too expensive, pushing Replit toward smaller models that can be hosted at drastically lower cost. Data privacy and security also factor in, along with keeping IP and controlling updates.

Technically, Replit’s stack is organized into three stages: data processing, model training, and production deployment. Data pipelines run primarily on Databricks, which handles preprocessing, analytics, transformations, and distributed scaling. The starting corpus comes from a large collection of permissively licensed code, deduped and made available on Hugging Face in Parquet or Hugging Face dataset formats. Sources include the GitHub archive and many repositories, filtered by file extensions and licenses, with near-deduplication across roughly 358 programming languages.

Before training, the pipeline scrubs and normalizes the data. Replit anonymizes sensitive content by removing or replacing emails, IP addresses, and secret keys with sampled placeholders. It removes auto-generated code using heuristics and regex, and it filters out minified or non-parsable code—using language-specific checks such as AST parsing for Python to avoid training on Python 2 artifacts. Quality filtering is attempted via signals like GitHub stars or issues, but Replit reports little evidence that those metrics reliably improve outcomes. A major differentiator is tokenizer training: instead of relying on an off-the-shelf Codex/codegen tokenizer, Replit trains a custom tokenizer and vocabulary from its own code data, aiming for a smaller vocabulary that better reflects code tokens. The result is faster training and inference and more efficient context usage.

Model training uses MosaicML, described as a GPU-focused platform for distributed training across multiple cloud providers, with pre-tuned LLM configurations and infrastructure for fault tolerance. Training runs are monitored with Weights & Biases, and the goal is stable loss curves without spikes.

Evaluation is treated as the hardest part. Replit uses “human eval” for code by generating code from prompts (e.g., writing a Python function) and then executing tests to verify correctness—leveraging Hugging Face code inference to run code quickly. But runtime cost, multi-language complexity, and the need for decontamination (ensuring test cases weren’t present in training data) make evaluation increasingly difficult as models saturate. Deployment into production uses FasterTransformer and Triton Inference Server, with batching, request cancellation, autoscaling, and handling of GPU constraints.

Lessons learned emphasize that data pipelines dominate effort and iteration speed, evaluation blends art and science, and real-world “model vibes” from user testing can reveal issues that offline metrics miss. Collaboration across the stack is essential because training-time features may not translate cleanly to inference-time libraries or latency budgets.