4 HARD Challenges for Claude Computer Use: Very Promising Results for AI Agents!

Claude-style “computer use” agents can follow multi-step instructions across real web interfaces—watching a video, taking image-based tests, and even completing an email workflow—though speed and reliability still limit performance.

In the first challenge, the agent was tasked with learning from a YouTube video by observing it end-to-end, taking incremental notes with time stamps, and producing observations tied to what appeared on screen. Running in a Tesla Optimus-themed setup, it repeatedly paused playback, captured screenshots, and wrote notes rather than trying to “guess” content in one pass. When the run finished, the notes included time-stamped observations describing the robot shown in a factory setting and its autonomous exploration/navigation behavior. The results weren’t perfect, but the structure—pause, screenshot, note, repeat—matched the instruction set closely enough to make the approach feel usable for instruction-following rather than passive transcription.

The second and third challenges tested whether the same agent could handle interactive, image-driven assessments. For an IQ test, it was given a prompt to answer step-by-step and submit responses to advance through questions. The agent selected answers, scrolled to reach new items, and continued through the test with a surprisingly high success rate. It ultimately scored at the 93.7th percentile (smarter than 937 people in a room of 1,000), with the most notable weakness being occasional missed questions and a crash near the end tied to memory instability. The key operational detail was that it didn’t just click randomly—it navigated the page, selected an option, then moved on to the next question.

A similar quiz test using a history category from a daily trivia site pushed the agent through 10 multiple-choice questions. It again achieved 10 out of 10 correct answers on the first attempt, including specific picks for questions about the Battle of the Nile, World War I dates, U.S. presidential marriage details, the Romanov surname, and other historical trivia. The tradeoff was speed: the agent was correct but slow, which would cap scores on timed or high-volume quizzes even when accuracy is strong. Switching to science and nature produced the same pattern—correct answers, but sluggish pacing that prevented a higher overall score.

The final challenge moved from “answering” to “acting” in a workflow: the agent was given its own Gmail account and asked to retrieve the top five Hacker News headlines from an email task, then reply with the results. After the user sent an instruction email, the agent opened the message, navigated to Hacker News, extracted the top five headlines, and composed a response back to the sender. It clicked send successfully, and the reply arrived in the user’s inbox with the headlines and links—formatting wasn’t polished, but the end-to-end automation worked.

Across all four tests, the agent demonstrated promising real-world capability: it can observe, reason through interactive UI tasks, and complete multi-step actions across video, quizzes, and email. The remaining constraints are practical—memory crashes, occasional misses, and especially slow execution when the interface requires many sequential steps.