What Should a QA Agent Actually Verify Before a PR Reaches Review?

What did those checks actually establish?

A QA agent should verify the accepted behavior, the affected service boundaries, relevant failure and permission cases, and the existing behavior the change could disturb. It should record the revisions and environment it checked, return failures to the implementation owner, and leave the reviewer evidence and explicit gaps. It cannot establish production readiness from a sandbox pass alone.

The first question I would ask is where the expected result came from. If QA gets the answer only from the implementing agent’s summary, both agents can agree on the wrong behavior.

Start with a requirement that can be checked

Consider an illustrative invoice-export feature. A billing admin can request a CSV for their organization and a selected date range. A background worker creates the file. The admin downloads it when it is ready. A viewer cannot request or download it.

This sounds straightforward until the agent has to decide which invoices belong in the file.

Does the date range use invoice issue time or payment time? Which timezone applies? Are cancelled invoices included? What happens if the admin loses access while the export is queued?

Those answers belong to the requirement’s accountable owner. QA should surface missing decisions before declaring a pass. It can proceed with other checks, but the unresolved behavior remains unverified.

Give QA access to the accepted requirement, relevant domain rules and contracts, implementation diff, affected components, and known risks. Keep the source and version of those inputs so a later reviewer can see what the checks were based on.

Follow the user outcome across the boundaries

For this example, a successful API response establishes that a request was accepted. It does not establish that the export contains the right records or that downloading it is authorized.

The verification path should follow the work.

Billing admin requests export
    → API checks role and organization
    → Job carries the accepted filters
    → Worker selects the invoices
    → File is stored
    → Download checks the user's access
    → UI shows completion or a useful failure

The actual path must come from your application. Some flows will have fewer components. A change to a local calculation may need a focused test and a small regression check rather than the whole stack.

For the export feature, I would start with this verification plan.

These are proposed checks, not results from a Prinevo run. Their value is that each check has a reason tied to the feature.

Verify the result behind the screenshot

A screenshot can show that the interface displayed an export button or a completed message. It does not show which database the worker queried or which invoices ended up in the file.

QA should use the observations appropriate to the claim. That can mean browser assertions, API responses, the downloaded file, sandbox data and correlated worker logs. Read state through approved interfaces with the access needed for the task.

Suppose the UI says the export completed but the file is empty. The useful finding connects the visible result to the job ID, request filters and worker output. A reviewer should not have to search unrelated logs to discover whether these observations came from the same attempt.

For asynchronous behavior, wait for an expected state with a deadline. A fixed pause followed by a screenshot can miss a slow worker or capture the previous state. Playwright’s retrying assertions provide one way to wait for a stated browser condition. The assertion still needs to describe the correct behavior.

Select checks by risk and feedback quality

Running everything on every edit can make feedback expensive. Running only the new unit tests can miss the boundary that matters.

Use focused checks to localize a rule failure, contract checks to inspect exchanged data, and integration or browser checks to establish the combined behavior. Widen the selection when the impact map is uncertain, and retain the broader CI checks your team requires.

In my testing feedback-loop post, I separated time to detect an issue from time to find and fix its cause. That is useful for QA agents too. A slow test that says only “export failed” leaves a much larger investigation than a check identifying the wrong organization filter.

The sandbox setup guide covers provisioning, fixtures, permissions and diagnostics. Here, the decision is which behavior QA must establish once that environment exists.

Give Implement a finding it can act on

If the export includes another organization’s invoice, QA should return a concrete finding to the implementation owner.

The finding should identify the accepted isolation rule, tested revisions, fixture identities, reproduction steps, expected invoice IDs, observed IDs and the relevant evidence. Share synthetic identifiers and redact credentials or sensitive data.

The implementation owner fixes the change. QA then checks the corrected revision, repeats the failure scenario and runs the checks affected by that correction. If the fix changes a shared invoice query, the affected checks may extend beyond the new export.

The loop can repeat. Keep unresolved findings visible, preserve failed attempts, and set a retry or time limit. A missing product decision goes back to its owner. An unavailable dependency is a blocker. Neither becomes a pass because the agent has already spent an hour trying.

Use four distinct outcomes.

A pass after retry should retain the earlier failure and any remaining instability. A green final summary should not erase the history needed to judge it.

Hand over a bounded verification claim

The reviewer needs to know what was verified and what remains open. Include the accepted behavior, component revisions, fixture and configuration versions, selected checks and rationale, outcomes, findings, fixes and reruns. Mark simulated dependencies and untested boundaries.

Then separate that claim from release approval. A migration, old and new worker versions running together, production permissions or a staged rollout may need additional checks and an accountable release owner.

The Prinevo delivery walkthrough places verification inside a workflow with named responsibilities and human steering. QA contributes evidence to that decision. It does not acquire authority to release because its checks passed.

For a first evaluation, give the QA agent a known scenario and a deliberately introduced defect in a disposable environment. Can it expose the wrong behavior, send a useful finding, and check the correction? Record the human help it needs along the way using the real-change scorecard.

That is a more useful starting point than counting how many tests the agent generated.