Building a Vision App with Ollama Structured Outputs

Structured outputs in Ollama make it practical to turn both text and images into validated, schema-shaped data—locally—using Python classes (Pydantic) or JavaScript schemas (Zod). That shift matters because it replaces brittle “free-form” extraction with outputs that can be checked, indexed, and fed directly into apps and downstream automation, without needing an agent framework.

The core workflow is straightforward: define a schema for the expected result, pass that schema to the model, and validate the returned JSON against it. In Python, the approach centers on Pydantic models; in JavaScript, it uses Zod. The transcript emphasizes that this isn’t just about getting “some text back.” It’s about reliably extracting entities, nesting structured objects, and pulling specific fields out of model responses—then using those fields programmatically.

A first example demonstrates named-entity recognition (NER) via a simple class that can hold organizations, products, people, and locations. Results vary depending on the model and prompt quality. Using an off-the-shelf Llama 3.1 setup yields mixed accuracy—some organizations and products land correctly, but certain mentions (like specific model names) and some people/organizations can be missed. Switching to a different Llama variant (Llama 3.2) and adding a system prompt improves consistency, especially for organizations and people. The transcript also notes an iterative path: treat the initial extraction as a starting point, then refine prompts, choose better models, and even build datasets (for example using stronger models) to fine-tune for the exact entity types needed.

The next example shifts from entity extraction to image understanding. A bookstore photo is processed with a vision model using a schema that nests a list of “book” objects (title, author, confidence score), plus scene-level fields like a summary and number-of-books category. The results show the promise and the limits: the system can identify some book titles and authors correctly, detect that the scene is a bookshelf with rows of books, and even pick up some text and colors. But it also misses side text and struggles when prompts are generic or when the image quality is limited.

Finally, a mini app extracts track listings from the backs of album covers. Using Llama 3.2 Vision, the app reads raw text from images and returns structured album details: album title plus a list of songs, with optional durations only when they appear in the source. The implementation runs as plain Python—no agent framework—processing a list of image files and saving results as Markdown. The transcript highlights practical outcomes: one album cover yields a correct track list, another includes times that align well with the printed durations, and blurry low-resolution images lead to partial transcription rather than fabricated times (helped by the schema design).

Overall, structured outputs are presented as a way to build small, task-specific “vision-to-database” pipelines that run locally for privacy and cost control. Once the prompts and schemas are tuned—and optionally fine-tuned for a domain—the same extraction logic can be reused repeatedly as a background job, feeding databases, RAG systems, or other agent workflows without the overhead of complex orchestration.