How to Set Up a Verification Sandbox for Coding Agents

The agent has written the code. Now the QA agent needs to check it.

Where does it run the app? Which database does it use? How does it sign in? Can it start the worker that consumes the API’s output? Where does it look when something fails?

If someone has to answer those questions manually on every change, verification still depends on that person being available.

A verification sandbox should let an agent start the relevant system from recorded code revisions, prepare the required state, exercise the changed behavior, and collect evidence. When a check fails, the agent should have enough information to diagnose it or send a concrete finding back to Implement. After the fix, the checks run again.

I would start with one real flow your team already understands. Make that path repeatable before trying to reproduce the whole company’s infrastructure.

1. Define what the sandbox needs to verify

Start with the acceptance criteria. The agent’s implementation cannot be the only source of truth for what its tests should expect.

Consider an illustrative pricing change. An Enterprise customer placing a $500 order should receive a 20% discount, giving a $400 subtotal before tax. The UI, pricing API, and invoice worker all need to agree on that amount. A customer without permission should not be able to change the pricing rule.

The sandbox needs to establish more than whether the API returns a number. It needs to check the agreed rule, the payload the worker receives, the invoice it creates, and the amount the customer sees.

Write a small verification plan before bringing up services.

These are proposed checks for this example. Your actual plan should follow your product’s rules and the change’s blast radius.

Ready when someone other than the implementing agent can explain the expected result and which check would detect a wrong one.

2. Choose the environment and record what runs together

The right setup depends on your architecture and the change.

For a small application, an isolated local environment may be enough. A change involving several repositories may need a disposable cloud workspace with its own services, data, and network configuration. A feature relying on managed infrastructure may need an approved test instance of that dependency.

Choose a setup the agent can recreate with the isolation your organization requires. A preview URL may be part of it, but the backend, worker, and data behind that URL also need to be identified.

For the pricing flow, I would start with the frontend, API, invoice worker, database, and queue. Record the exact code revision or image digest for each component, along with configuration, migration level, and relevant feature flags. Include the test and fixture revisions too.

Across repositories, “test the latest branch” is too vague. The reviewer needs to know which combination actually ran. If verification uses uncommitted edits, capture the diff or an equivalent immutable snapshot as well as the base commit.

You do not need every external service running for every check. A payment stub can help test the application’s behavior, but it cannot establish that the real provider accepts the request. Use its approved test environment when that boundary matters, or record the boundary as unverified.

If the rollout allows old and new service versions to coexist, add the relevant compatibility check. Testing only the new API with the new worker leaves that rollout question unanswered.

Ready when another agent can identify every component behind the tested flow, reproduce the same revisions, and see which dependencies were simulated.

3. Put startup and reset into the repository

The setup should survive the person who first got it working.

Keep the environment definition, prerequisites, migration commands, seed routines, health checks, and troubleshooting instructions in the repository. For work across repositories, keep one integration setup that records how the compatible components come together.

A simple layout could look like this. These are suggested repository files, not a Prinevo command interface.

verification/
  README.md
  compose.yaml
  fixtures/
  scenarios/
  collect-evidence.sh

The README should explain how to provision the environment, apply migrations, seed data, check readiness, run a scenario, collect evidence, reset state, and tear down the run. Scripts should return a failure status when a required step fails.

Pin the runtime and dependencies sufficiently to reproduce failures. Keep configuration templates in version control and inject credentials through your approved secret mechanism.

Separate starting a process from establishing that the service is ready. A database container can be running before it accepts queries. A worker can be alive without consuming from the expected queue.

If you use Docker Compose, configure health checks and dependency conditions where needed. Compose does not wait for service readiness by default. Its startup-order documentation explains service_healthy and service_completed_successfully for dependencies and setup jobs.

Then add a small application smoke check. Can the API reach the database? Can a harmless test event reach the worker? Set a startup timeout and report which prerequisite failed rather than waiting indefinitely.

Ready when a fresh agent can create the environment, pass the smoke checks, reset it, and repeat the process without undocumented manual fixes.

4. Seed the states the feature needs

A running application with an empty database may still be impossible to verify.

For the pricing example, seed an Enterprise account, an account on another plan, the required products, and users with the relevant permissions. Give each fixture a known expected state so a failed assertion has a useful reference.

Build scenarios around the requirement. Include the successful path, a denied operation, and the failure or retry behavior that matters for this change. If the result depends on time, control the clock where possible or record the test time explicitly.

Give concurrent runs separate accounts, queues, database namespaces, or equivalent isolated state. Otherwise, one agent can change the state another agent is trying to check. Reset routines should affect only resources owned by that run.

Prepare authentication as part of setup. For browser checks, saved test authentication can reduce repeated login work, but the state may contain credentials. Keep it out of source control and shared evidence. Playwright’s authentication guide also explains when tests that change server-side state need separate accounts.

Use synthetic data or data approved and prepared for testing. Do not make a production database copy the default way to unblock a QA agent.

Ready when the agent can reach each required scenario from a clean state, and two runs can execute without changing each other’s results.

5. Scope what the agent can access

The QA agent needs tools that match its job. That may include a browser, API client, sandbox logs, test runner, and a way to inspect test data. It does not follow that it needs production credentials or unrestricted access to the host.

Give the run an identity, an owner, a limited lifetime, and explicit permissions. Restrict the repositories, filesystem mounts, network destinations, and operations it can use. Route email, payments, and webhooks to controlled test destinations.

Put those boundaries in the environment and underlying services. An instruction saying “do not touch production” cannot enforce a permission boundary.

A container is useful packaging, but its configuration still matters. Avoid giving agent-written code privileged access or the host’s Docker socket as a shortcut. Docker’s security documentation explains the daemon’s authority and how mounts and capabilities affect isolation.

Keep verification expectations reviewable too. If Implement changes a contract, fixture, or assertion to resolve a failure, the reviewer should see why. Passing by weakening the expected behavior is not the same as fixing the change.

Set time, retry, and compute limits. When the agent reaches one, it should leave the evidence and blocker with an accountable owner.

Ready when the agent can perform the required sandbox checks, a controlled unauthorized operation is denied, and expiry or cancellation removes its access.

6. Make failures easy to diagnose

The purpose of a test is to give useful feedback.

For the pricing example, a focused test can show immediately that the Enterprise rule returns 15% instead of 20%. A browser flow that ends with the wrong invoice amount tells you something broke, but the cause could sit in several places.

I discussed this in my testing feedback-loop post. There are two costs to consider. How long does it take to detect the issue, and how long does it take to find and fix the cause?

Use focused checks for specific rules, contract checks at interfaces, and integration or browser checks for the complete behavior. Widen the selection when impact is uncertain. Keep the broader CI checks your team requires.

Give the agent access to useful observations. Carry a run or correlation identifier through the API request, event, and worker logs. Preserve the expected value, observed value, response status, and relevant state change. Redact credentials and sensitive payloads before retaining or sharing evidence.

For a browser failure, a trace can help explain what happened before the screenshot. Playwright’s Trace Viewer supports inspecting actions, page snapshots, and network activity.

If a worker is delayed, wait for the expected state with a deadline and report what was observed. Repeatedly adding sleeps can hide a timing problem without explaining it.

Ready when an intentionally introduced defect on a disposable test branch produces a finding that identifies the broken behavior and gives the next agent a useful starting point.

7. Connect Verify back to Implement

This is where the environment becomes part of an agent workflow.

Verify should return a finding with the accepted requirement, tested revisions, expected result, observed result, reproduction steps, and evidence. The finding goes back to the implementation owner. The owner returns a corrected revision, and Verify reruns the failed check and any checks affected by the fix.

1. Accepted requirement and code revisions
2. Start sandbox and seed data
3. Verify behavior and collect evidence

Required checks pass?

No

Finding to Implement

Corrected revisions

Return to Verify

Yes

Evidence for human review

If the fix changes configuration, migrations, or fixtures, refresh the environment and record the new setup before rerunning. A successful result from the earlier revision does not verify code changed afterward.

Keep these outcomes separate.

If the database cannot start, verification is blocked. If authentication is missing, the browser scenario has not passed. If a flaky test passes on retry, retain the failed attempt and investigate the instability rather than treating the first failure as if it never happened.

The loop can repeat, but it needs a limit and an escalation path. A missing product decision belongs with its owner. An unavailable dependency belongs with someone who can restore it. More agent attempts will not necessarily resolve either.

The coordination backbone guide explains how to preserve these handoffs and outcomes across agents.

Ready when a deliberate failure goes from Verify to Implement, returns as a correction, and is checked again without someone manually relaying the context between chats.

8. Keep the evidence after the sandbox is gone

The environment can be temporary. The reviewer still needs to understand what happened.

Attach a compact verification record to the PR or delivery run.

Export the approved evidence before destroying resources. Give it the right access controls and retention period. Then remove temporary data, workspaces, and credentials. Configure expiry and cleanup for abandoned runs too.

A sandbox pass establishes the recorded scenarios under the recorded conditions. Production data, traffic, permissions, and mixed versions may still differ. Keep release approval, rollout checks, and production observation as separate responsibilities.

Finally, review what made the run difficult. If a missing seed user blocked QA, fix the seed routine. If an event contract was unclear, update the authoritative contract and add the relevant regression check. If agents repeatedly skip a consumer, propose a change to impact analysis and evaluate it before making it shared.

This is the link between the sandbox and a compounding software factory. The correction should have a source, an owner, and somewhere useful to live for the next run.

Ready when the sandbox can be removed while another engineer can still inspect the result and recreate the scenario from its recorded inputs.

Start with one change and grade the whole loop

Take a representative change from your backlog. Choose one that crosses a boundary your team worries about, rather than a task that cannot exercise the setup.

Ask the agent team to start from a fresh environment, prepare the state, run the checks, diagnose a controlled failure, fix it, and return evidence on the corrected revision. Watch where a person still has to supply missing context or repair the environment.

Track setup effort separately from recurring work. Record provisioning time, time to detect and diagnose failures, human interventions, verification outcomes, and reviewer rework. Include blocked attempts. Otherwise, a green final PR can hide the manual work needed to reach it.

Use the real-change evaluation scorecard to decide whether the setup reduces engineering work while preserving the evidence your reviewers need.

This is the verification loop we are building around Prinevo’s agent team. Shared context informs the checks, coordination returns findings to the right agent, and people steer the decisions that need an owner. The Prinevo delivery walkthrough shows where verification fits in that workflow. The wider software factory implementation guide covers the layers around it.

For a first pilot, I would ask one question. Can the agent bring up your app, show you what failed, fix the change, and hand back evidence you can actually review?