Create Your Own Microsoft Recall AI Feature with RAG?

A practical “Recall”-style system can be built by combining automated screen capture, AI-based interpretation of what’s on-screen, and a retrieval layer that lets users search past activity by natural language. The core idea is to avoid saving every frame: the setup only records a new screenshot when the screen changes enough, then uses a vision-capable GPT model to extract actions and URLs from that image, storing both the screenshot and a text “history” entry. Later, a RAG (retrieval-augmented generation) pipeline embeds that history locally and retrieves the most relevant past events—along with the exact screenshot—when someone asks questions like what they did on a specific site.

The workflow is split into three phases. In the record phase, a script continuously monitors pixel differences between the current screen and the previous screenshot. If at least 5% of pixels change, it captures a new screenshot and saves it with a custom, AI-generated filename meant to function like a searchable label. That filename is produced by GPT-4o and is derived from the analyzed on-screen description, so later queries can match keywords rather than random IDs. The analyze phase then sends each saved screenshot to GPT-4o to extract the most important information: what interactions occurred, any visible URLs, and the associated screenshot name. The extracted text is appended to an archive (“history text”), and the screenshot itself is stored in a dedicated folder so the user can “rewind” to a frozen moment.

The retrieval phase turns the archive into a searchable knowledge base. The system creates embeddings from the history text using local embedding models, then uses Llama 3 to search within that RAG space. A user can ask a question such as “Did I visit Discord yesterday?” and the system returns the relevant history entry plus the screenshot filename, enabling the user to open the exact captured image. The transcript emphasizes that this approach mirrors the conceptual mechanism behind Microsoft Recall: store time-stamped evidence, summarize it into retrievable text, and connect queries to the underlying artifacts.

A key implementation detail is chunking: each extracted action is broken into segments capped at about 1,000 characters and appended to the history text so the embedding step can index the full sequence of user activity. The system also includes a “compare screenshots” step using OpenCV to compute pixel diffs, and it runs in a loop with a short startup delay. In a live demonstration, the script captures activity while the user browses sites and opens content; later, the RAG layer successfully answers questions about reading posts and identifies related screenshots by matching query terms to the AI-generated filenames.

Despite working as a prototype, the builder warns against active use because the vision analysis relies on GPT-4o via an API, which can transmit proprietary or private data. The project was initially intended to run fully locally, but performance and stability issues with local vision models forced a hybrid approach. Still, the result shows a feasible path toward a more privacy-preserving, local Recall-like system—provided a sufficiently strong open or local vision model becomes available.