The AI Unit Test “Problem”
Agent-written tests check the code the agent wrote. Deleting them is the wrong fix; a separate context that sees only the requirements & entry points is the right one.
Why Agents Write the Wrong Tests
When the agent that wrote the code also writes the tests, the tests check the code as it is, not as it was meant to be.
You ask an agent for a feature. It comes back with the code, a pile of unit tests & a green build.
The agent wrote those tests by reading the code it had just written. Every assumption in the implementation went straight into the assertions, including the wrong ones. If the code has a bug, the agent writes a test that expects the bug (the agent is grading its own homework, & it's a generous marker).
The build is green & the feature is still wrong.
Let's look at why deleting those tests is the wrong fix, what the right tests look like & how to get an agent to write them.
Why Deleting the Tests Is the Wrong Fix
The common fix is to tell the agent to stop writing tests & delete the ones it wrote. It does more harm than good.
I understand the impulse. A test file twice the size of the change, passing no matter what the code does, is noise, & reviewing it feels like a wasted afternoon.
Deleting it treats the symptom. The problem is who wrote the tests & what they could see while writing them. Remove the tests & you keep the problem & lose the safety net.
An agent writes more code in a day than you can read line by line. Tests are how you check the part you didn't read, so cutting them cuts your oversight along with the noise.
The more code an agent writes, the more weight the tests carry.
What Do the Right Tests Look Like?
A right test comes from the intent: the requirement, the acceptance criteria or the bug report. It fails when that intent breaks.
Here is what that looks like in practice:
- Written from the requirement - The expected value comes from what the feature should do. If you can't point to the line in the requirement an assertion checks, the assertion came from the code.
- Tested at the public boundary - Call the code the way its callers do & check what they get back. Private methods & internal state are the implementation's business.
- Edges & failures included - Boundary values, empty input & error paths are where bugs hide. An agent copying its own code gets them wrong the same way twice.
- Able to fail - Break the behavior the test covers & it goes red. A test that can't fail is decoration.
The wrong tests have tells of their own. They mirror the implementation line for line, mock everything the code touches, assert on internals or work out the expected value with the same formula the code uses.
If the code were wrong, would this test fail?
One Bug, Two Tests
Here is a requirement small enough to hold in your head: orders of $100 or more ship free, & everything else pays a flat $5.99.
The agent writes this:
public class ShippingCalculator
{
public decimal ShippingFor(decimal subtotal) =>
subtotal > 100m ? 0m : 5.99m;
}> where the requirement says >=.An order of exactly $100 should ship free & pays $5.99 instead. Then the same agent reads that method & writes its tests:
public class ShippingCalculatorTests
{
[Fact]
public void Order_over_100_ships_free() =>
Assert.Equal(0m, new ShippingCalculator().ShippingFor(150m));
[Fact]
public void Order_at_100_pays_shipping() =>
Assert.Equal(5.99m, new ShippingCalculator().ShippingFor(100m));
}The second test is worse than no test. It records the bug as expected behavior, so whoever fixes the bug later (person or agent) gets a red build & a good reason to put it back.
A test written from the requirement only knows that $100 or more ships free:
public class ShippingCalculatorTests
{
[Theory]
[InlineData(100)]
[InlineData(150)]
public void Orders_of_100_or_more_ship_free(decimal subtotal) =>
Assert.Equal(0m, new ShippingCalculator().ShippingFor(subtotal));
[Fact]
public void Orders_under_100_pay_flat_rate() =>
Assert.Equal(5.99m, new ShippingCalculator().ShippingFor(99.99m));
}The $100 case fails, & that failure is the whole point of the test. It caught the bug because it never saw the bug.
A Separate Context for Tests
Write the tests in a separate context from the one that wrote the code.
Start a new agent session (or a subagent) for the tests & give it two things: the requirements & the top-level entry points it should test through. It doesn't get to read the implementation or follow the code path down through every class.
It can't copy the implementation's assumptions because it never sees them. When one of its tests fails, either the code is wrong or the requirement is unclear, & you want to know about both before a customer finds them.
It costs a second session & a little more setup. The entry points also need to be clear enough to test without reading what's behind them. If they aren't, that's worth knowing too.
Test from the requirement, through the front door.
Agents are going to write most of the code. I want more tests, not fewer. I just want them written by something that didn't write the code.