Teaching Agents to Explore

When I build with coding agents, the agent writes most of the code and most of the tests. The tests pass. But that green checkmark means less than it used to.

The same agent wrote both, from one picture of how the feature should work. So the tests mostly check that picture. Nobody asks whether the picture is right.

So after an agent builds something, I ask it to explore what it built. That is usually when it starts questioning its own assumptions.

I wanted to know whether this habit really works, so I ran an experiment on two buggy APIs. The short answer: a coding agent can explore well, and a written mission helps most where bugs hide in state and in data you have to follow across the system. Exploring finds bugs that reading and guessing can't, with evidence for every claim.

I turned what I learned into a skill any coding agent can use. The skill, the experiments and every result are in a public repo.

Use the skill

The skill works with any agent that supports the open Agent Skills format, including Claude Code, Codex, Cursor and Gemini CLI.

In Claude Code, install it with:

/plugin install exploratory-testing --marketplace mahdavipanah/exploratory-testing-skill

For other agents, use the skills CLI:

npx skills add mahdavipanah/exploratory-testing-skill

Then start your software and ask the agent to explore it:

Explore the orders API you just built. It runs on localhost:3000, and the docs are in docs/api.md.

The skill makes the agent work like a tester. It will:

1. Check that the environment is what it seems.
2. Map the system: its entities, its states, and everything it can vary.
3. Write down the rules that must never break, like "money is never created or lost", then rank the risks.
4. Write three to five short missions, called charters, and use its whole budget on them.
5. Follow the data after every change, try every action in every state, and test both sides of every boundary.
6. Back every claim with the command and its output.

You get a report that keeps bugs, behaviour that needs your decision, and open questions apart. Every bug comes with the request and the response that prove it.

For an API, a shell and curl are enough. For a web UI, give the agent a browser it can drive, such as Playwright, Google's Chrome DevTools MCP or Vercel's agent-browser.

What is exploratory testing?

James Bach defined exploratory testing as simultaneous learning, test design, and test execution. You try something, look at what happened, and let that decide what you try next.

Elisabeth Hendrickson's book Explore It! explains why you need both tests and exploring. Tests check that the software does what we intended. Exploring asks a different question: what other risks are there? An agent's own tests only ask the first one.

Exploring isn't random clicking. Hendrickson gives each session a charter, with a simple template: Explore (target) with (resources) to discover (information).

Jonathan and James Bach's session-based test management puts each charter in a time box, with notes. James Whittaker's tours add themes, such as following one piece of data through every path. Together they turn curiosity into a method you can hand to an agent.

Why agents need a method

Agents are good at the mechanics. Simon Willison points out that a coding agent's defining ability is running the code it writes.

Agents also fix exploratory testing's oldest weakness: it is hard to show what was tested, and hard to repeat it. An agent logs every command and every response.

But left alone, agents explore badly. Maaret Pyhäjärvi tested this on a to-do app with 73 known problems. An unguided AI prompt found 5% of them, autonomous Playwright agents 16%, and a skilled human tester 62%.

The researchers behind GUITester name one reason: agents care more about finishing the task than about reporting what looks wrong. Hendrickson tells of a job candidate who listed a few test ideas and declared he was done. He didn't get the job. One of my agents did the same thing.

The experiment

I gave Claude Opus 5.5, running in Claude Code, two practice APIs:

• Restful-booker, Mark Winteringham's deliberately buggy hotel-booking API. It is famous, so the model may remember its bugs.
• claimdesk, a small expense-claims API no model has seen. A separate agent built it and planted 15 bugs, and their list stayed sealed until the scoring was done.

Every agent got the API docs, curl, 80 requests, and one rule: report only what you saw and reproduced. Only the mission changed:

• A got one line: "Find bugs in this API."
• B got four charters I wrote.
• C wrote its own charters.
• D got the skill.
• M got only the docs, no API, and had to predict the bugs.

Each one ran five times, in separate sessions, in random order. On claimdesk, a separate agent matched every finding to the bug list without knowing which agent wrote it. A second scorer agreed with 96 of 100 of its decisions.

On Restful-booker, a mission helped: B and D found clearly more than A. On claimdesk, all four exploring agents found about two-thirds of the bugs. That API is small and its docs state every rule, so even "find bugs" covered most of it.

Writing its own charters worked most of the time. But one run of C wrote five good charters, used only 37 of its 80 requests, and stopped. That's Hendrickson's job candidate.

The surprise was M. Without touching the API, it found as many known bugs as the explorers. On Restful-booker it remembered them. On claimdesk, the bugs sat exactly where the docs made promises.

But M got there by guessing. On claimdesk, three out of four of its guesses were wrong. The exploring agents reported one wrong finding in 486.

What only exploring finds

On both APIs, some bugs were found only by exploring. No prediction named any of these:

• Restful-booker answers one documented response format with status 200 and an error message as the body.
• It labels its XML responses as HTML.
• It turns 30 February into 2 March without a word.
• One update that sets the dates to null breaks a booking for good: every later read returns status 200 with a JavaScript error as the body.
• A date sent with a timezone offset is saved as a different day.
• claimdesk's default page size is 10, but its docs say 20. Every exploring run caught it; no prediction did.
• A rejected claim can't be resubmitted, although the docs say it can.
• Ordinary amounts lose a cent: 19.99 is saved as 19.98.
• An unknown sort value crashes the server.

You can't find these by reading. They live in what the system actually returns: its defaults, its arithmetic, its headers, and the state that earlier requests leave behind. Hendrickson's advice applies to agents too: when the map and the territory disagree, trust the territory.

Reading the code doesn't replace exploring either. I had Claude review Restful-booker's source first, and it found 30 problems. The exploring agents then found 9 more, one of them on a line the review had read.

Two problems from the review were never found by exploring. Reading and exploring find different bugs, so use both.

What a mission adds

A mission helped most with bugs hidden in state and in data you have to follow. In claimdesk, removing an item from a claim updated the claim itself, but not its total in the list or the amount finance paid later.

B's charter said to follow one claim through every view, and B found this bug in all five runs. The skill says to follow the data after every change, and D found it in four. A, with no mission, never found it.

A mission can also hide things. B's charters pointed it at views and lifecycles, so B found a bug that let a user remove an item from someone else's claim in only one run of five. A and D found it every time.

The misses followed patterns, and each one became a rule in the skill:

• Agents test one path and generalise. On Restful-booker, they saw an id ignored when creating a booking and assumed it was ignored everywhere. In an update, it silently moves the booking to another id. Only D, told to send fields like ids on every write path, found that.
• They check one side of a boundary. One agent saw a date included at one edge and called an inverted filter correct.
• They trust the docs. When the docs and the API agreed on a wrong status code, most agents filed it as a question.
• They stop early.

What stays human

Two jobs stay with you. The first is deciding what counts as a bug. Agents lean on the docs and park real problems as questions. Someone has to decide whether a 200 should be a 201, or whether a case-sensitive search is intended.

The second is the sandbox. An exploring agent reads whatever the system sends back, and a page or a response can carry instructions aimed at it. Anthropic calls prompt injection one of the biggest security challenges for browser agents.

Within those limits, the split works well. The agent brings stamina, evidence and repeatability. You bring judgement about what matters. Exploring stops being a phase someone schedules and becomes the step after every change.

Before you trust the green checkmarks, ask your agent to explore.