Mistral OCR - Multimodal & Multilingual OCR

Mistral has launched a non-open-source OCR system delivered through an API that turns scanned documents, PDFs, and images into structured, multimodal outputs—text interleaved with extracted tables and preserved images—supporting multiple languages and math. The core value is practical: the service returns content in a form that can be fed directly into downstream LLM workflows like RAG or visual question answering, without forcing users to manually convert layouts, tables, or figures.

The API accepts either full PDFs or single images (including base64-encoded images). For each page, it returns an object containing extracted markdown for text and tables, plus image elements where the layout calls for it. In the example of an academic paper, tables are converted into markdown while figures and other non-text elements remain as images, preserving the document’s structure and positioning. That “interleaving” matters because it keeps context intact when the output is later used for retrieval or for question answering over the document’s content.

Mistral’s pitch centers on two capabilities: multilingual OCR and multimodal extraction. The OCR output includes math converted from the page, and it handles scripts beyond English—demonstrations include Hindi and Arabic, with the system still extracting usable text even when the source is messy or misaligned. Benchmarks shown in the release materials place it near the top against other OCR competitors, including results that extend across languages such as Chinese, Hindi, and Russian.

Pricing is set at $1 per thousand pages, with batch inference described as roughly half price (at the cost of slower turnaround). For organizations with privacy or data-upload constraints, Mistral also markets an on-prem option, claiming throughput of up to 2,000 pages per minute on a single node. The monetization model is consistent with that: license the OCR for on-prem deployments rather than distributing the model weights.

A key usability feature is structured output. Users can supply a document plus a custom prompt and request JSON-like results that can drive automation—such as triggering function calls in a workflow once fields are extracted. In the provided code walkthrough, helper functions assemble the OCR results into readable text or return images for further processing. The workflow also demonstrates how to enforce schemas using Pydantic classes so that repeated document processing yields consistent keys (e.g., file name, topics, language, and OCR contents).

The practical experiments go further: a receipt image is processed into a structured JSON response, and another test targets Thai, showing that the system can identify the language and extract structured fields even when the Thai characters are represented in a specific Unicode form. Batch processing notebooks show how to queue large volumes for lower cost, trading speed for efficiency.

Overall, the OCR system is positioned as a production-oriented tool for extracting document content while preserving layout cues—especially tables and figures—at a price that’s framed as reasonable for an API service. The trade-offs are clear: it’s not open-source, and like any LLM-based OCR pipeline, it can still produce occasional structural errors or hallucinations, so accuracy needs validation against specific document types and quality requirements.