AI Test Automation Workshop: Coding Agents, Specs and CI Gates

Learn test automation with coding agents: write specs first, generate meaningful tests, avoid tautological tests and enforce quality with CI gates.

Short answer

Coding agents can write large numbers of tests quickly, but volume is not quality. A test automation workshop should teach teams to start from specifications, review generated tests for what they actually verify, use mutation and coverage signals sensibly, and set CI gates that keep agent-written code honest.

On this page
  1. Why testing is the foundation of agentic coding
  2. The core problem: tests that test nothing
  3. Module 1: specifications before tests
  4. Module 2: generating tests that matter
  5. Module 3: measuring test quality
  6. Module 4: end-to-end and integration tests
  7. Module 5: CI gates for agent-written code
  8. Exercises we use
  9. After the workshop
  10. FAQ
  11. Tools from the jocapps ecosystem

Why testing is the foundation of agentic coding

Tests are what make agent output trustworthy. An agent that can run a meaningful test suite can check its own work, iterate on failures and hand over a change that has already passed a real bar. Without tests, every agent change has to be verified by a person reading code, which quickly becomes the bottleneck.

The irony is that many codebases where teams want to use agents have weak test coverage. The good news is that agents are also useful for fixing that. The workshop we run at neooutput teaches both directions: using agents to build a test foundation, and using that foundation to make agent-written features safe.

The core problem: tests that test nothing

When you ask an agent to "write tests for this module", you often get a lot of tests that:

  • Assert that a function returns what it currently returns, without asking whether that is correct
  • Mock so much that only the mocks are tested
  • Check implementation details that break on any refactor
  • Cover the happy path many times and edge cases never

Coverage numbers go up while protection does not. The first part of the workshop makes this visible: participants generate tests for a module, then deliberately introduce bugs and see how many the new tests catch.

Module 1: specifications before tests

Good tests start from intended behaviour, not from code. We practise writing short specifications in a form both humans and agents can work with:

Feature: invoice rounding
- Amounts are rounded per line, not on the total
- Rounding uses half-up to two decimals
- Negative lines (credits) follow the same rule
- A zero-amount invoice is valid and produces no tax line

The agent then writes tests from the specification, and only afterwards looks at the implementation. Disagreements between specification and code are surfaced as questions rather than silently encoded in tests. This works especially well when product owners help write the specification.

Module 2: generating tests that matter

Participants work on their own repositories with a set of techniques:

  1. Test lists first. Ask the agent to propose a list of test cases with rationale before writing any code. Review and prune the list.
  2. Boundary and error cases explicitly. Require cases for empty inputs, limits, invalid data, timeouts and permissions.
  3. Behaviour over implementation. Tests use public interfaces; internal helpers are tested through them.
  4. Fail first. For new features and bug fixes, the test must fail before the change and pass after. The agent shows both runs.
  5. Readable names. Test names describe the behaviour, so a failing test tells a reviewer what broke.

Module 3: measuring test quality

Coverage tells you which lines run, not whether tests would detect a fault. We introduce additional signals:

SignalWhat it showsHow to use it
Line and branch coverageWhich code is executed at allFind untested areas, not as a target in itself
Mutation testingWhether tests detect small injected faultsRun on critical modules to find weak tests
Flakiness rateTests that pass and fail without code changesQuarantine and fix; flaky tests destroy trust
Test runtimeHow long the suite takesAgents run tests often; slow suites slow everything

Mutation testing is particularly revealing for agent-generated tests. Participants run it on one module and ask the agent to strengthen tests that let mutants survive.

Module 4: end-to-end and integration tests

Unit tests are only part of the picture. Agents can also write integration and end-to-end tests, but these need more care: stable test data, isolated environments and clear waits instead of arbitrary sleeps. We practise one end-to-end scenario for a critical user flow and discuss when such tests are worth their maintenance cost.

Module 5: CI gates for agent-written code

The final module connects everything to the pipeline. We design gates that apply equally to human and agent changes:

  • All tests pass, with no newly skipped tests
  • Coverage on changed lines meets a threshold the team agrees on
  • New code paths in critical modules include tests
  • Linting, type checks and security scans pass
  • Pull request descriptions state what was tested and how

Teams that run agents headless in CI, for example to backfill tests overnight, also need gates on those runs. Our agentic DevOps workshop covers that part in detail.

Exercises we use

  • Backfill tests for a stable but untested module, then measure with mutation testing
  • Reproduce a recent production bug as a failing test, then fix it with the agent
  • Turn a vague ticket into a specification and test list together with the product owner
  • Refactor a module with tests as a safety net and check that no test needed changes

After the workshop

We recommend a test backfill sprint as the first follow-up: pick the modules where the team most fears changes and let agents build a test foundation under human review. That investment pays back every time an agent works on those modules later. Our article on evaluating the quality of AI-generated code gives a broader framework for reviews.

For organizations that want an external team to build that test foundation or own quality engineering for a product, OnlineAppCreator offers done-for-you engineering by AI Code Engineers with SLAs. The workshop itself is delivered by neooutput, remote or on site, in English or German, typically as one or two days depending on whether end-to-end testing is included. Teams modernizing old systems can combine it with the legacy modernization workshop.

FAQ

Can coding agents replace QA engineers?

No. Agents speed up writing and maintaining tests, but deciding what should be tested, judging risk and exploring a product for unexpected behaviour still need human expertise.

Is high coverage from AI-generated tests a good sign?

Not on its own. Generated tests can produce high coverage with little protection. Combine coverage with mutation testing and reviews of what tests actually assert.

Which test frameworks does the workshop support?

The approach is framework-independent. We work with whatever the team uses, for example JUnit, pytest, Jest or Vitest, XCTest or Playwright.

Should agents write tests and code in the same task?

For new features, it is often better to agree on the tests first, then implement. That keeps tests anchored to the specification instead of to the code the agent just wrote.

Tools from the jocapps ecosystem

Want this done right, and fast?

Talk to our AI Code Engineers and AI architects. In a free 30-minute call we look at your idea, prototype or roadmap and tell you honestly what it takes to ship it.

24/7 call support in English and German.

Free call

Free intro call

Tell us which task an AI agent should take responsibility for. We show how our platform, AI Code Engineers and operations plug into your stack.

  • 30 minutes, no obligation
  • Honest feasibility check
  • Concrete next step
Rather talk now? 24/7 call support in English and German+49 761 519 751 50