Build Private Chatbot wtih LangChain, Ollama and Qwen 2.5 | Local AI App with Private LLM

A fully local “private chatbot” workflow can be built by combining LangChain’s message orchestration (via LangGraph), Ollama for on-device model serving, and Qwen 2.5 for the actual language reasoning—then grounding every answer in a filtered set of mobile app reviews stored in a local SQLite database. The practical payoff is immediate: the app can answer questions like “What are the top three most common issues users have?” by analyzing only the reviews that match user-specified filters (package, rating range, and review count), and it can keep conversational context so follow-up prompts build on earlier questions.

The demo starts with a simple command to run the app (filtering reviews with ratings of 1–2). A welcome screen appears, then the chatbot analyzes the selected reviews in real time on an M3 Pro machine. One example question asks for the top three recurring problems users report; the model returns a ranked list derived from the review text. A second prompt then asks for an implementation suggestion—summarized in two to three sentences—for a notification fix tied to the previously identified problem. That follow-up works because the system maintains chat history and reuses it when constructing the next model call.

Under the hood, the project uses UV as a fast package manager to install dependencies, then relies on Ollama to download and run the Qwen 2.5 model locally (the walkthrough uses the 14B parameter variant, quantized to 4-bit by default in the Ollama setup). The core application is a single main file (app.py) that wires together: (1) a review data model (package name, review text, rating), (2) a SQLite query that fetches only relevant reviews, and (3) a LangGraph state machine that manages conversation state.

The SQLite layer filters reviews by rating bounds (ratings are treated as 0–5 inclusive), excludes empty review text, and limits the number of rows to a maximum count. Each review is formatted into a structured XML-like block (review text plus rating) and inserted into a system prompt. That prompt instructs the model to be helpful, accurate, and brief, and it frames the task as a “mobile app review analyzer.” LangChain’s chat prompt template then builds the message list by combining the system prompt content with the evolving chat history.

Model calls are made through a LangChain chat model wrapper pointed at Ollama, with temperature set to 0 for consistent outputs. The LangGraph “cyclic” workflow repeatedly invokes the model whenever the user submits a new message, appending the model’s response back into the in-memory message history. For this prototype, checkpointing is kept in RAM via a memory saver; the walkthrough notes that production deployments would likely persist checkpoints elsewhere.

The result is a compact (about 146 lines) local app that streams responses to a console UI using the Rich library, validates CLI arguments (min/max rating, max reviews), and loops indefinitely for interactive Q&A. The next steps hinted include adding a web interface, exposing an API, and extending the review-grounded chatbot into an agentic system using a newer framework (PydanticAI).