The Agent Wrote the Code and the Tests. How Do You Know They Are Right?

But who decided that the value was correct?

When an agent writes both code and tests, check them against an accepted requirement and expected results that were established independently of the implementation. Exercise meaningful failure cases and real boundaries, then check that the tests reject a relevant wrong implementation. A second agent or a high coverage number cannot supply that confidence by itself.

I do not think the purpose of a test changes because an agent wrote it. We still want to know when behavior is wrong, where it failed, and what needs to be fixed.

The risk is that code and tests can share the same mistaken interpretation. A test suite may run successfully while proving a rule nobody agreed to build.

Establish the expected result before accepting the implementation

Consider an illustrative pricing feature with these accepted rules. An Enterprise customer receives a 20% discount on orders of at least $500 and a 10% discount below that threshold. A Pro customer receives a 15% discount on a $500 order. The amounts here are before tax and other charges.

The expected subtotal for the Enterprise customer’s $500 order is $400.

Suppose the agent implements a 15% discount and writes a test expecting $425. The arithmetic is consistent. The behavior is wrong.

There is a subtler version too. The test calls the same production helper to calculate the expected amount, then compares that result with the checkout response. If both paths use the wrong discount rule, agreement tells us little about whether the requirement was met.

I would start with a few independently worked examples tied to the accepted rule.

These examples do not settle every pricing question. If the feature also changes coupon stacking, rounding or tax calculation, those rules need their own accepted examples. Do not let the agent invent them because a fixture is missing.

Inspect what the test actually observes

I would ask the reviewer to read the assertion before counting the test.

Does it check the subtotal, the invoice line and the relevant account? Or does it check only that the response was successful?

A test that checks HTTP 200 can catch a request failure. It cannot establish that a successful response contains the correct amount. A browser test that checks a confirmation message has a similar limit if the invoice behind it is wrong.

The assertion needs to match the claim. For this pricing feature, checking the accepted $400 result is stronger than checking that the subtotal is positive. Checking the right invoice and customer is stronger than finding the number 400 somewhere on the page.

Keep essential assertions mandatory. If a setup error causes a test to skip its main check and still report success, the result can hide the very path we wanted to verify.

Check that the test rejects a relevant mistake

On a disposable branch or sandbox, introduce a small, controlled defect that represents a risk you care about. Change the Enterprise discount to 15%, for example. The relevant pricing test should fail for the expected reason.

Then restore the accepted implementation and run the check again. Record the revisions so the negative experiment cannot be confused with the release candidate.

This is one practical way to evaluate a test’s ability to detect a specific fault. Mutation-testing tools can automate families of small code changes, but a surviving mutation needs interpretation. It may expose a weak assertion, an untested path, or a change with no meaningful effect on behavior. One score does not establish complete correctness.

Also check the ordinary regression path. If the test is intended to reproduce a bug, it should fail on the affected earlier revision and pass on the corrected one, where that comparison can be reproduced. A test for a genuinely new feature may need a different baseline because the old application did not support that behavior.

Do this according to risk. A controlled defect can be valuable in a payment rule; it does not mean every text change needs an extensive test-quality experiment.

Keep a real check at the boundary that matters

Mocks help isolate behavior and give fast feedback. They also define the world the test sees.

If a mocked invoice worker accepts a field the real worker does not understand, the test may confirm only that the mock agrees with the new code. Keep a contract or integration check against the actual consumer when that boundary is part of the change.

In my earlier software-factory account, I described a producer emitting created_at while the consumer expected createdAt. The repositories looked reasonable in isolation. An integration check caught the mismatch when the services ran together.

That does not make unit tests unhelpful. My testing feedback-loop post explains why a focused pricing test can identify a wrong percentage much more directly than an end-to-end checkout failure. The checks answer different questions.

Use narrow tests to identify local rule failures. Use integration checks to inspect the interactions that narrow tests cannot establish. If a dependency is simulated, keep that fact beside the result.

Give another agent an independent basis for checking

A QA or review agent can challenge the implementation, but the instructions matter. Giving it the implementing agent’s explanation and asking whether it looks correct can preserve the same assumption.

Give the evaluator the accepted requirement, domain rules, relevant contracts and expected examples. Ask it to identify ambiguous expectations, missing failure cases, weak assertions and boundaries hidden behind mocks. It should inspect the implementation and tests, but it should not derive correctness solely from their agreement.

Anthropic’s application-development experiments describe separating generation from evaluation to address overly generous self-assessment. The authors also note that separation alone does not remove that tendency. A second agent still needs clear criteria and useful observations.

For the pricing example, QA should be able to explain why $400 is expected before seeing that the implementation returns it. If the accepted rule and an older contract conflict, return that decision to the owner.

The implementation agent can propose changes to a test or fixture. It should explain whether the original expectation was wrong or the code failed to meet it. An accountable reviewer should approve a changed behavioral expectation rather than allowing the agent to weaken it simply to get a pass.

Review confidence in the behavior rather than test volume

Before accepting agent-written tests, ask a few concrete questions.

• Can the expected result be traced to an accepted rule?
• Would a relevant wrong implementation cause the check to fail?
• Do the assertions inspect the result we are claiming to verify?
• Are denied operations and important failure cases exercised?
• Are the significant service boundaries checked against more than a convenient mock?
• Are changes to expectations, skipped checks and blocked scenarios visible?

Coverage can help locate code that tests did not execute. It does not establish that executed code was checked against the right business rule. Keep coverage, defect-detection experiments and scenario evidence as different observations.

This is where the verification layer around coding agents matters. The sandbox guide explains how to give agents somewhere to run the checks and return findings. Prinevo’s delivery workflow describes how those findings return to the owner and how people steer the decisions.

For your next change, pick one important rule and ask the agent to show the accepted result, the check that establishes it, and a controlled wrong implementation that the check rejects. Then inspect the boundaries the test does not cover. That gives a reviewer something concrete to assess before deciding what can move toward production.