Gemini 2.5 Computer Use MCP | On The Edge #7

Google DeepMind’s Gemini 2.5 “computer use” model is being put to work through a custom MCP server that can control both a Mac and a browser. The core takeaway from the demo is practical: the model can translate high-level goals—like finding a specific file, opening it in the right app, or filling out a web form—into sequences of UI actions, and it does so reliably enough to complete tasks, even if speed and edge cases still lag.

The setup splits control into two toolsets: browser use and macOS app control. In the MCP server, the browser side provides tools such as taking screenshots and executing actions, while the macOS side adds screenshotting plus actions for interacting with the desktop (including clicking, typing, and launching apps). The author uses Cloud Code to run the MCP integration, then demonstrates four tools per environment (browser and Mac), with separate TypeScript implementations referenced from a single index.

On macOS, the first test targets a real-world workflow: locating an MP4 file named “Elizabeth” (created earlier by Suno) and playing it in QuickTime Player. The model begins by searching the Finder, then types the filename, right-clicks the result, chooses “Open with,” selects QuickTime Player, and starts playback. It’s not fast—there’s a visible delay and at least one attempt where the model needs to adjust (including switching to smaller screenshots)—but it succeeds at the pixel-level UI navigation required to find the correct file and launch the correct application.

Next comes browser automation using a “robo form filler” page. The goal is to fill out a multi-field form by role-playing as “Neo from Matrix” and generating plausible answers. The agent takes a screenshot of the page, plans the form-filling steps, and then proceeds to populate fields. It handles cookie prompts and begins entering data (starting with a first name consistent with Thomas Anderson / Neo). The run ends before every field is completed, not because the approach fails, but because the session hits a turn limit; increasing max turns would likely extend completion.

A final test pushes the model into a more brittle workflow: using computer control to open Cursor, create a Python file named “hello YouTube.py,” write code that prints “hello YouTube,” save it, and run it in a terminal. The model manages to create and edit the file and save it, but the execution is inefficient and error-prone—file naming and keyboard/layout issues cause confusion, and the terminal/run step takes a long time and appears to involve missteps (including a “No such file” moment and a rename to “hello.py”).

Overall, the demo frames Gemini 2.5 computer use as an incremental but meaningful step forward when paired with MCP tools: it can perform useful UI tasks across apps and websites, yet it still struggles with speed, turn budgeting, and precise developer workflows. The combination of richer context via MCP and a computer-use model is positioned as promising—while also carrying obvious risk when granting agents control of a real machine.