Run your own AI (but private)

Local “private AI” is becoming practical: a person can run an LLM entirely on a laptop or workstation, keep data off third-party servers, and then extend it to answer questions about personal notes or company documents using RAG (retrieval-augmented generation). The core setup is built around downloading an open model and running it through a local inference tool, so the model runs without internet access and without shipping prompts or files to a remote provider.

The walkthrough starts by demystifying what an AI model is—an LLM pre-trained on large datasets—and points to Hugging Face as a catalog of hundreds of thousands of models, including Llama 2 variants. Llama 2 is highlighted as a large language model trained on trillions of tokens and instruction data, with training described in terms of massive compute (thousands of GPUs, millions of GPU hours, and an estimated tens of millions of dollars). The key takeaway isn’t the training bill; it’s that the resulting model can be downloaded and run locally.

To run those models, the guide centers on Ollama (Olama.ai). After installing Ollama on macOS or Linux—or on Windows via WSL (Windows Subsystem for Linux)—the user can pull a model like “Llama 2” and start chatting immediately. Performance depends heavily on hardware: GPUs speed things up, while CPU-only setups are slower. The transcript also notes that Mac systems with M1/M2/M3 chips can work well, and that Nvidia GPU drivers may be needed in WSL for full acceleration.

Running a local model solves privacy, but it doesn’t automatically know personal facts or internal business information. The transcript illustrates this with “hallucination” examples—questions about identities or current events can come back wrong because the model’s training data is incomplete or outdated. That gap leads to the next step: teaching the AI the right context.

Two approaches are presented. Fine-tuning adapts the model by training it on new, proprietary examples—useful for internal procedures, help-desk knowledge, or product documentation. The transcript contrasts the enormous resources needed for original pre-training with the smaller scale of fine-tuning, describing an example workflow where only a small fraction of parameters are updated using a few thousand labeled examples. The other approach is RAG, which avoids retraining by connecting the LLM to a vector database (PostgreSQL is mentioned) so it can retrieve relevant passages from an internal knowledge base before answering.

VMware’s role is framed as packaging the infrastructure and tooling needed for private AI in enterprise environments. The transcript references VMware Private AI with Nvidia, plus supporting ecosystems involving Intel and IBM, and describes a typical enterprise stack: vSphere for virtualized infrastructure, deep learning VMs preloaded with tools, GPU assignment via passthrough, and a data scientist workflow using Jupyter notebooks and datasets of prompt/answer pairs. RAG is positioned as the practical bridge between a general LLM and an organization’s private documents.

Finally, the transcript demonstrates a more hands-on “Private GPT” side project (separate from VMware) that uses RAG to let a user upload documents and chat with them through a web interface. In the example, a user ingests markdown journal entries and asks questions like what happened in specific places and when—showing the promise of private, document-grounded Q&A even if results aren’t perfect. The overall message: local inference plus RAG is a credible path to AI that stays private while becoming useful for real work.