A vast amount of meaningful work still occurs within web interfaces. To resolve a ticket, support teams rely on admin consoles; operations teams monitor dashboards for alerts; sales representatives check CRM systems before engaging with customers.
For LLM agents to be truly useful in these contexts, they must operate directly through the browser. This post will guide you through building a browser-using agent using the OpenAI Agents SDK and Playwright MCP. We’ll first explore the core loop behind browser-based agents, then apply it to a real-world case study.
1. The Mental Model
At a high level, a browser-using agent can be understood as an LLM placed in an interactive loop with a browser. The process begins with the agent receiving a task and the current state of the browser. The agent interprets that state, decides on the next step, and sends an action back to the browser. That action generates a new browser state, which becomes the input for the next decision. This cycle repeats until the agent determines that the task is complete.
To enable this loop, two connections between the agent and the browser are necessary:
- An observation channel: allows the agent to receive the current browser state.
- An action channel: enables the agent to interact with the browser.
- Screenshots + coordinate-based mouse/keyboard actions
- Structured page state + element-targeted browser actions
For the observation channel, common approaches include screenshots, structured page information (e.g., DOM or accessibility tree), or a combination of both. For the action channel, the agent might use mouse and keyboard controls, target specific page elements, or issue higher-level browser commands.
While observation and action choices are often independent, two common pairings emerge in practice:
The first pairing leans toward general computer use, extending beyond browsers to other desktop applications. The second is more specific to browser use, leveraging the structure available within a web page. This post focuses on structured page observations combined with element-targeted browser actions.
2. Case Study: Resolving a Customer Support Request
For our case study, we’ll build a browser-using agent that resolves a customer request through a web-based support console.
2.1 Preparing the Support Console
To create the browser environment, I built a small customer support console using plain HTML, CSS, and JavaScript. It’s a static web app with no backend or database—all data resides in the browser.
Serve the console locally with Python’s built-in HTTP server:
cd toy_app
python -m http.server 8000 --bind 127.0.0.1
This makes the console available at http://127.0.0.1:8000, which we later pass to the agent as APP_URL.
Figure 1. Screenshot of the customer support console. (Image by author)
The left side of the console functions as a support inbox. When a case is selected, the rest of the console displays the related order, customer context, and resolution policies. From there, the case can be resolved with an internal note. The console also includes an audit log to track updates.
The agent’s task is straightforward: investigate an incoming support case and carry it through to resolution.