Create Your "Small" Action Model with GPT-4o

A “small action model” framework turns a short burst of user screen activity into a repeatable automation script by using GPT-4o’s vision to infer what happened and then GPT-4o again to generate code that replays it. The core idea is simple: record screenshots while a person performs actions on a computer, analyze those images to reconstruct the action sequence, convert that sequence into a step-by-step plan, generate Python code to execute the plan, and then run the code to recreate the same workflow.

The workflow is split into two phases. In the recording phase, the system captures screenshots at a fixed rate—set to two frames per second in the example—over a short window (the transcript mentions a 15-second duration). Each screenshot is stored in a folder. In the execute phase, every saved image is converted to base64 and sent to GPT-4o for vision-based analysis. The goal is to determine the sequential order of the user’s actions across the screenshot timeline, effectively turning visual snapshots into an ordered “what happened” description.

Once the action sequence is extracted, GPT-4o is prompted to produce a step-by-step plan for recreating the actions. That plan then becomes input to another GPT-4o call that generates Python code designed to reproduce the same behavior. The generated code uses operating-system controls for mouse and keyboard input rather than browser automation tools like Selenium. The prompt also instructs the model to output only the code, and the project includes a “clean generated code” function to strip away extraneous formatting so the result can be saved as a runnable script (for example, action.py).

A practical test demonstrates the pipeline end-to-end. The system records a brief interaction: opening the start menu, launching Chrome, navigating to google.com, searching for “Taylor Swift down bad,” and clicking through to the resulting YouTube video. After analysis, the system prints the inferred action sequence and the generated plan, then produces Python code. When executed, the script successfully repeats the same steps—opening Chrome, visiting google.com, entering the query, and selecting the video—matching the recorded behavior closely enough to be considered a “good pass.”

The project is positioned as a starting framework rather than a finished product. The author notes that improvements are likely, and invites others to experiment and report ideas. Future directions hinted in the transcript include expanding capabilities with additional inputs (such as video or “voices”) and potentially using local vision models to reduce reliance on hosted services. Overall, the approach reframes “small” action automation as a vision-to-plan-to-code pipeline, where short visual evidence becomes executable instructions.