Build 100% Local Chatbot with Gemma 3, Ollama and LangChain | AI Assistant with Memory and Tool Use
Based on Venelin Valkov's video on YouTube. If you like this content, support the original creators by watching, liking and subscribing to their content.
The chatbot runs locally by combining Gemma 3 for responses with Qwen 2.5 for tool-calling decisions about saving memories.
Briefing
A fully local chatbot can now keep both conversation history and long-term “memories” across separate chats—without sending data to a hosted service. The system stitches together a local Gemma 3 model for responses, a local tool-calling model (Qwen 2.5) for deciding when to save new memories, and a local SQL database (SQLite + SQLVector) to store both chat context and vectorized memory entries. The result is an assistant that can remember what a user said earlier, retrieve the most relevant saved memories for the current conversation, and selectively store new details when they matter.
The workflow is built around three stages. First, each user message is combined with prior conversation history pulled from a local database (the transcript references SQLite for conversation storage). Second, the chatbot retrieves up to five relevant memory records for the current user using vector similarity search over stored memory embeddings (generated with FastEmbed). Those memories are injected into the prompt so Gemma 3 can respond with continuity.
Third, after generating a reply, the system decides whether to save new memories. A dedicated “save memory” tool is exposed to the model. The tool-calling logic uses Qwen 2.5 to evaluate the conversation and existing memories, then returns a structured memory payload only when the model judges something should be stored. Saved memories include a content field plus an importance score from 1 to 10; the transcript notes an importance threshold (importance of 3 or below should not be saved). This keeps the memory store from filling up with low-signal details.
Implementation details reinforce the “local-first” goal. The project is organized with a Streamlit UI (app.py) and a LangChain/LangGraph backend workflow (chatbot workflow and task-decorated functions). A configuration file initializes model providers (the transcript mentions Groq API support as an option, but the core models used are Gemma 3 and Qwen 2.5) and sets up the SQLite database and the vector store table for memories. The LangGraph checkpointing is wired to the SQLite connection so conversation state can persist across runs.
The memory system is formalized with a Pydantic-style Memory model containing memory_id, content, user_id, created_at, and importance. A MemoryManager handles saving memories by converting them into vector-store documents with metadata, retrieving memories via similarity search filtered by user_id, and also includes a SQL-based method to fetch memories directly from the table using JSON metadata extraction.
On the UI side, the app supports multiple conversation IDs (thread_id) and a user_id. The sidebar lists retrieved memories for that user, and the main chat view streams assistant output chunk-by-chunk. In the demo behavior described, the assistant stores the user’s name with high importance (10), then later recalls it when starting a new conversation thread. It also stores other interests (like wanting to buy a motorcycle) with lower importance, demonstrating that memory selection is selective rather than automatic.
Overall, the core insight is practical: memory persistence and retrieval can be implemented as a local, tool-driven loop—retrieve relevant memories, generate a response, then conditionally store new memories—using LangChain/LangGraph orchestration plus SQLite-backed vector search and embeddings.
Cornell Notes
The system builds a local chatbot that remembers across separate chats by combining three pieces: (1) Gemma 3 for generating answers, (2) Qwen 2.5 for tool-calling decisions about what to store, and (3) a SQLite database with SQLVector for storing and retrieving memory embeddings. For each new user message, it pulls prior conversation history and retrieves up to five relevant saved memories for the current user via vector similarity search using FastEmbed embeddings. After responding, the chatbot uses a “save memory” tool that only stores new items when they meet an importance threshold (importance 3+). Memories include content, user_id, created_at, and an importance score, enabling selective long-term personalization.
How does the chatbot decide which memories to include in a new conversation?
What mechanism prevents the memory store from growing with low-value details?
Where are conversation history and memories stored locally?
How does the system persist state across runs and support multiple chat threads?
What does a saved memory record contain, and how is it used later?
How does the UI stream responses while keeping the message list consistent?
Review Questions
- What are the exact steps in order for handling a new user message (from history retrieval to memory retrieval to response generation to conditional memory saving)?
- Why does the system use a separate tool-calling model (Qwen 2.5) instead of relying on Gemma 3 alone for memory decisions?
- How does the memory retrieval process ensure memories belong to the correct user, and what limits are applied to the number of memories returned?
Key Points
- 1
The chatbot runs locally by combining Gemma 3 for responses with Qwen 2.5 for tool-calling decisions about saving memories.
- 2
Conversation history and long-term memories persist via SQLite, with SQLVector used to store memory embeddings for similarity search.
- 3
For each message, the system retrieves up to five user-specific memories using FastEmbed-generated vectors and injects them into the prompt.
- 4
After replying, a “save memory” tool is called only when the tool-calling model judges new information is worth storing based on an importance score.
- 5
Saved memories include content, user_id, created_at, and an importance rating from 1 to 10, enabling selective personalization.
- 6
The Streamlit UI supports multiple conversation IDs (thread_id) while keeping a sidebar of retrieved memories for the same user.
- 7
LangGraph checkpointing is wired to the SQLite connection so workflow state can persist across sessions.