OpenAI DevDay 2024 | Community Spotlight | Swyx

AI agents are best built by treating them as a stack—LLMs plus memory plus planning plus tools—then mapping those capabilities to concrete engineering jobs as model abilities advance. The core message is that developers don’t need to be PhD-level researchers to ship state-of-the-art agents; they need a clear mental model of what an agent requires and where new model capabilities create new opportunities.

The talk frames “agents” with a practical definition attributed to Lilan Wang: agents are “LLM + memory + planning + tools.” The presenter then refines that idea into an engineering checklist. First comes the infrastructure layer: teams need a gateway and operations tooling, plus retrieval-augmented generation (RAG) frameworks to connect models to data. Open-source tools are highlighted as the default starting point, and the talk points attendees toward Singapore-based work, including Feedist AI, as evidence that local builders are already contributing to the ecosystem.

Second is memory and knowledge. ChatGPT-style systems can be improved dramatically when memory is added, and the talk treats memory as a core agent capability rather than an optional feature. For knowledge storage and retrieval, vector databases remain common, but knowledge graphs are presented as an increasingly interesting direction. A standout example is “Graph RAG,” which was described as the most surprising popular talk at the AI Engineer Conference the presenter runs.

Third is planning and multi-agent behavior—described as the most active research area. The talk recommends reading OpenAI’s “Let’s Verify Step-by-Step” work for process-style reasoning, and also points to the SW F project as another reference point. Multi-agent systems are positioned as a straightforward path to better performance, and the presenter promises a demo showing how splitting tasks across agents improves outcomes.

Finally, tools and orchestration tie the stack together. The talk argues that many agent systems converge on the same building blocks: a code interpreter/sandbox for deterministic computation, browser control for searching and reading from the web, and a self-ask or ReAct-style loop that cycles through observing and reacting to the environment. These components are presented as the practical substrate behind agent frameworks and products, including Microsoft’s “magentic” and “cognition,” which the presenter says aligns closely with OpenAI’s architecture.

The demo turns the theory into a workflow. Starting from a fork of Bolt.new, the presenter generates a Space Invaders-style game from a short prompt, then adds a second agent to iteratively improve gameplay: aliens arrive in waves, power-ups trigger on kills, stars appear in the background, and visuals become more “alien” using emoji styling. The second agent effectively performs QA and feature requests in the background, demonstrating how planning and multi-agent collaboration can transform a rough prototype into a more engaging experience. The closing takeaway is a call to action for Singapore: use foundation models to build an “AI engineering nation,” leveraging the agent stack as a roadmap for what to build next as capabilities keep improving.