Case study

An AI tester that goes from a ticket to proven test results

I designed my company's first AI-powered QA agent. It prepares the testing for a new feature and runs it, while a person approves every stage. Testing starts 70% faster and the paperwork shrinks by 85%.

The problem

Before any test could run, every request for a new feature went through the same slow routine.

  1. Someone read the request and worked out what it really meant.
  2. Someone hunted for gaps and unclear points, then chased answers.
  3. Someone wrote a test plan, an effort estimate and the test cases by hand.
  4. Someone loaded the cases into the test tool, ran them and collected proof.

The first three steps were repetitive and varied from person to person. They also produced a lot of documents.

What I built

An agent that takes a ticket number and carries the work as far as executed tests. People stay in charge of every judgment call. It runs inside Claude Code and is made of a few simple parts.

  • Commands for the main jobs: write tests from a ticket, upload them to the test tool, run them in a browser.
  • Helper agents for the autonomous work: gathering requirements, planning, investigating production bugs, organising evidence.
  • Written procedures the agent follows, such as how to find gaps or how to estimate effort.
  • Scripts that produce the documents in a fixed format, so the output never depends on the model's mood.
  • Connections to the ticket system and to a real browser, so it uses real tools instead of guessing.

How a ticket moves through it

  1. AskIt asks two to five short questions to understand the scope.
  2. ReadIt reads the ticket, the app's code, the product documents and the change history.
  3. Find gapsIt works out what is missing from the code first, and only asks a person about real business questions.
  4. SummaryA plain-English summary of the scope. A person approves it.
  5. Test planScope, approach and risks in a document. A person approves it.
  6. EstimateAn effort workbook with three-point estimates. A person approves it.
  7. Test casesFull steps and expected results, including negative, boundary and role-based cases. A person approves them.
  8. Hand overA file ready for the test tool. It stops here.

Uploading to the test tool and running the tests in a browser are separate commands that a person chooses to run. They are not chained on, because browser runs need real logins and an upload writes to another system.

Decisions I would defend

Code first, questions second
The agent works out how the app behaves from its code, and only bothers people with genuine business questions. Every requirement points back to evidence in the code.
Approvals are real stops
The summary, plan, estimate and test cases each need an explicit yes, even if someone asks for "just the test cases quickly".
Double confirmation before writing to the ticket system
It drafts the comment, a person approves the words, then it asks once more before posting. Reading is free.
Passwords stay in memory
Logins are never written to a file, a report or a screenshot.
Scripts over free text
Documents come from generators with enforced formats, such as valid priorities and no repeated login steps.
Plain language
Summaries are written from a tester's point of view, so non-engineers can approve them.
A person can steer a run
While it tests in the browser you can type "skip this test" or "click Next, not Sign In". It stops, follows, and your instruction wins.

What went wrong first

Early on, the agent wrote test steps and expected results that were too vague to run. The fix was not a cleverer instruction. I gave it the product's own help documentation as reference material, and I rewrote its skills, the written procedures it follows, so that it had to be concrete. After that, the steps were specific enough for another tester to follow.

What it produces

  • A test plan document and an effort workbook with the estimate calculations.
  • Test cases covering normal, negative, boundary and role-based situations.
  • A file ready to load into the test tool.
  • An execution report with screenshots taken at the checks that matter and at failures. Bugs are reported in the chat and never filed unless someone asks.

Results

70%faster to start testing a new feature
85%less documentation effort

Figures from my resume, measured on the project.

Back to portfolio