How Do You Know AI-Generated Software Is Actually Correct?

A coding agent can produce hundreds of lines of plausible code before a human has finished reading the task.

That creates a new kind of engineering problem.

The bottleneck is no longer:

Can we write this implementation?

It is increasingly:

What evidence should make us trust this implementation?

This question matters whether the code was written by an agent, a junior developer or the CTO.

AI simply makes it more urgent because generation is so fast.

Code review is necessary but insufficient

Reading code is useful.

A reviewer can spot:

• obvious security issues;
• bad abstractions;
• incorrect assumptions;
• maintainability problems.

But many software failures are behavioural.

The code may look reasonable while producing the wrong state under a sequence nobody considered.

Verification therefore needs more than aesthetic review.

Begin with invariants

An invariant is a property that should remain true.

Examples:

Billing

The same successfully processed provider event cannot grant the same entitlement twice.

Permissions

A member without the required permission cannot perform the privileged operation, regardless of UI state.

Scheduling

Two confirmed bookings cannot occupy the same exclusive resource for overlapping time.

E-signatures

A completed document cannot silently return to an editable unsigned state.

These statements are powerful because they are implementation-independent.

The agent can choose different internal designs while still having to satisfy the same property.

Test transitions, not just endpoints

Many bugs occur between states.

Suppose a subscription can be:

trialing
active
past_due
cancel_at_period_end
cancelled

Testing each state in isolation is useful.

Testing transitions is more revealing:

trialing → active
active → past_due
past_due → active
active → cancel_at_period_end
cancel_at_period_end → active
cancel_at_period_end → cancelled

Then add external disorder:

duplicate event
retry
late event
out-of-order event
partial failure

This begins to resemble real production behaviour.

Write the acceptance criterion before trusting the implementation

The order matters.

If the agent writes the code and then invents tests that match the code, there is a risk of verifying its own assumptions.

A stronger approach is:

1. define required behaviour;
2. define representative acceptance tests;
3. implement;
4. run tests;
5. inspect failures;
6. review sensitive logic.

The agent can still help create the tests.

But the behaviour should not be reverse-engineered from whatever implementation happened to emerge.

Use different levels of evidence

Not every requirement belongs in a unit test.

A useful verification stack may include:

Static checks

• types;
• linting;
• schema validation.

Unit tests

• pure business rules;
• state transitions;
• validation.

Integration tests

• database behaviour;
• API routes;
• service boundaries.

Acceptance tests

• end-to-end product behaviour;
• externally meaningful invariants.

Manual review

• security boundaries;
• destructive migrations;
• complex concurrency;
• unusual data handling.

Production observation

• structured logs;
• metrics;
• alerts;
• reconciliation.

Good verification is layered.

Test the failures the architecture claims to handle

If a design says:

webhook processing is retry-safe

then there should be a test that fails mid-processing and retries.

If a design says:

permission changes take effect immediately

then there should be a test with an existing session.

If a design says:

scheduling prevents double booking

then there should be a concurrency test.

Architecture claims should have evidence.

This principle is especially important for AI-generated code because agents are very good at producing implementations that look like familiar patterns.

The pattern itself is not proof that the required behaviour exists.

Ask the agent to attack its own work

Coding agents can be useful adversarial reviewers.

After implementation, ask a fresh context or separate agent to inspect:

• race conditions;
• trust boundaries;
• missing states;
• partial failures;
• rollback;
• replay;
• data migration;
• assumptions about ordering.

This is not a substitute for testing.

It is an efficient way to generate hypotheses about where testing should be stronger.

Verification should scale with risk

A marketing banner does not need the same ceremony as billing.

A sensible team spends more verification effort where failure has greater consequences.

High-risk areas often include:

• money;
• permissions;
• authentication;
• irreversible data changes;
• privacy;
• compliance-sensitive workflows.

Coding agents make it cheaper to add tests.

That is an opportunity to raise the baseline rather than lower it.

Correctness is not a binary state

No realistic production system is "proved correct" in the mathematical sense merely because a test suite passes.

A better mindset is:

What classes of failure have we considered, and what evidence do we have against them?

That keeps the process practical and honest.

As What Does Production-Ready Mean When AI Writes the Code? argues, production readiness is not the absence of all possible bugs.

It is deliberate behaviour plus evidence appropriate to the risk.

The coding agent changes who does the work

Historically, writing comprehensive tests competed directly with implementation time.

Now the same agent that implements the capability can often generate:

• fixtures;
• integration cases;
• boundary cases;
• migration tests;
• failure simulations.

The human job increasingly becomes choosing which properties actually matter.

That is a very good trade.

Generate the code quickly.

Spend the saved time deciding what must be true.

Then make the machine prove as much of it as practical.