← All posts

Why I Stopped Letting AI Write My Tests And Started Making AI Write My Tests

Why I Stopped Letting AI Write My Tests And Started Making AI Write My Tests

There is a fundamental flaw baked into how most software gets built, and it existed long before anyone ever prompted an LLM to write a unit test.

It goes something like this: a product manager hands you a requirement. You translate that requirement into code. Then, you write a test to verify your code works.

On the surface, this looks like responsible engineering. In practice, it is an echo chamber. The code you have writen is not the actual requirement, it is the representation of it that you created in code, your test will only be able to verify that representation. Subsequently, the test is only as good as the original code. How many edge cases did the code cover for? How many code paths crossed with other features and they were taken care of, and how many “do not touch this code” were left untouched?

The test will turn green but it will share the exact blind spots as the code writen.

When you hand this workflow over to AI, you don’t fix the problem, you just let someone else do it, faster and potential with even more blind spots. Asking an AI agent to “write tests for this function” produces the ultimate confirmation bias. The model reads your narrow implementation, infers what the code is trying to do, and generates tests that faithfully prove your bugs work as intended.

This is how invisible regressions quietly take over a codebase. You cannot write tests for scenarios you haven’t imagined. Of course you always have your production users to take care of that.


The Illusion of 100% Coverage

When teams get burned by regressions, the standard reflex is to demand higher test coverage. But 100% line coverage only proves that every line of code you wrote executes without errors, it says nothing about the code you forgot to write.

Consider a seemingly straightforward domain rule from our own invoicing platform: calculating VAT for a German small business (Kleinunternehmer).

At a glance, the requirement is simple: if the business is under the revenue threshold, apply 0% VAT. Then reality sets in:

  • What if they crossed the warning threshold this year, but haven’t hit the hard cap?
  • What if they crossed the hard cap mid-year and now have to charge standard VAT?
  • What if they crossed the warning limit last year, changing their status for the current fiscal year?
  • What if they crossed the cap last year?
  • What if they crossed the cap last year, should they even be labeled as a small business?
  • What if they are issuing an invoice to a client outside the European Union?

No product requirement document will ever exhaustively list every permutation of these rules. Not because product managers are careless, but because human brains aren’t wired to hold multi-dimensional state matrices in working memory. Even the six questions above barely scratch the surface.

If you sit down to manually write unit tests for every possible intersection of these rules, you will eventually surrender to the sheer volume. I know this because I tried. Driven by a deep aversion to Sunday-night anxiety and Monday-morning hotfixes, I attempted to write out every test case by hand. Halfway through, my natural laziness kicked in, and I quit. There had to be a better way.


Why TDD Fails Where BDD Succeeds

That frustration led me to rethink the relationship between specifications and tests, bringing me straight to Behavior-Driven Development (BDD).

Over the last decade, the software industry went from writing no tests at all to treating Test-Driven Development (TDD) as a silver bullet. While TDD is great for designing clean interfaces, it remains fundamentally code-centric. The developer is still thinking about functions, mocks, and implementation details.

BDD flips the perspective from the inside-out to the outside-in. Instead of testing how the code works, you define how the system must behave in plain, human-readable language (like Gherkin) before touching the implementation.

In the era of AI, this separation isn’t just good practice, it is mandatory. Whoever writes the tests cannot be the same entity that writes the code.

Look at the current wave of autonomous coding harnesses. Teams are spending $5,000 a month on agentic loops: feeding a prompt to an AI, letting it write the code, running it against a database, and looping until the generated tests pass. It looks impressive if you are looking to raise VC money, but architecturally, it is the exact same closed loop. The moment you introduce a new dimension to an existing feature, the agent will miss the combinatorial side effects unless you explicitly map the entire state space for it upfront. AI is good at doing what it is told and telling it “find the gaps” does not cut it in large codebases.


Flipping the Script: Combinatorial BDD

Instead of asking AI to guess what to test, I went back to first principles and broke our domain down into its atomic building blocks. In an invoicing system, everything revolves around three core entities:

  1. The Seller (the user’s tax profile)
  2. The Recipient (the client being billed)
  3. The Catalog Item (the product or service)

Rather than writing individual test cases by hand, I decided to let code and math generate the entire problem space.

Take the user profile. If you look at German small-business tax rules as a series of boolean flags, Is small business? Crossed warning limit? Crossed hard cap?, you start with a simple $2 \times 2 \times 2$ matrix. Once we included every relevant tax parameter on a profile, the script generated 12,288 possible combinations, and now you know why life in germany is complicated.

Of course, mathematical possibility is not the same as real-world validity. Many of those only 1,584 states are legally or logically impossible (for example, crossing the hard cap without having crossed the warning threshold).

So, we built a lightweight rule filter to strip out the impossible states, then turned those combinations into explicit Gherkin scenarios: this state is valid, this state is invalid.

  Scenario: de-valid-regular
    Given isSmallBusiness is false
    And hasTaxId is true
    And nonUnionOssRegistered is false
    Then the profile is Valid

Those 12,288 raw permutations distilled down into 3 valid and 11 invalid scenarios. For the first time, we weren’t guessing whether our profile validation logic had blind spots. We had mathematically accounted for every state a user profile could ever inhabit.


How the Pipeline Actually Works: Building a Combinatorial BDD Engine

Instead of asking AI to guess what to test, I went back to first principles and broke our domain down into its atomic building blocks. In an invoicing system, everything revolves around three core entities:

  1. The Seller (the user’s tax profile)
  2. The Recipient (the client being billed)
  3. The Catalog Item (the product or service)

Rather than testing everything at once, we built a repeatable pipeline to systematically map out each entity, starting with the user profile, using four distinct steps:

1. Strip the Model Down to Behavioral Properties

An entity has dozens of fields, but most of them don’t affect business logic. Whether a user’s legal company name is “Acme GmbH” or “Beta GmbH” changes nothing about how VAT is calculated. We stripped away all non-functional metadata and created a stripped-down test model containing only the properties that actually alter behavior (e.g., small business status, warning thresholds, hard caps, One-Stop Shop registration).

2. Partition by Jurisdiction and Generate the Matrix

Tax laws don’t exist in a vacuum; they change per country. One jurisdiction has a warning threshold and a hard cap; another has no warning threshold at all; One-Stop Shop (OSS) rules apply in some regions and are invalid in others. Because the rules shift by country, we split the entire pipeline per jurisdiction.

For a single jurisdiction, we fed our stripped-down profile model into a generator to compute every possible parameter permutation. Starting with a simple $2 \times 2 \times 2$ boolean concept and scaling up to all active tax parameters, the script spat out 3,702 raw combinations.

3. Filter Impossible States & Let AI Expose the Grey Areas

Mathematical possibility is not the same as real-world validity. Many of those 3,702 combinations are legally or logically impossible, for instance, you cannot have crossed the small-business cap last year and still legally register as a small business this year.

Nobody wants to manually review 3,700 rows, so this is where we brought in AI. We gave the AI our domain documentation and the raw combinations, tasking it with grouping the patterns and separating valid states from invalid ones.

Two things happened:

  • Automatic pruning: It quickly grouped and excluded impossible combinations (e.g., if above the cap, small business status is automatically invalid regardless of other flags).
  • The “Aha!” moments: Whenever the AI encountered a combination that wasn’t explicitly answered by our documentation, it stopped and asked us: “Can this state actually happen? What is the expected behavior here?” That forced us to dig into the edge cases we hadn’t thought of, turning 3,702 raw permutations into 100 concrete, verified Gherkin scenarios covering both valid states and invalid rejections.

4. Test the Entry Points, Without Ever Showing the Code

We took those 100 Gherkin scenarios and ran them directly against the only two places where a profile’s state can actually change: the Create and Update service endpoints.

Here is the most critical rule of the entire workflow: neither the generator nor the AI ever looked at our implementation code. They only ever saw the stripped-down domain model and the business rules.

Because the tests were generated completely blind to the implementation, any failure meant one of two things: either our Gherkin scenario had a mistake in the legal rule, or our code’s validation wasn’t defensive enough to catch an invalid state. Either way, we had deterministic proof of a mismatch—zero confirmation bias.


Where AI Actually Shines

Once the profile matrix was locked down, we applied the exact same pipeline to recipients, then to catalog items, and finally to the cross-product of all three: invoice creation.

This is where AI finally earned its keep, not as a blind code generator, but as a pattern-recognition engine.

Instead of asking an LLM to invent test cases out of thin air, we fed it our raw, unmapped combinations and gave it a strict analytical job: “Group these parameter permutations together, and based on our tax logic, classify which combinations are valid and which are invalid.”

Running those AI-grouped scenarios against our invoice creation engine, allowed us to map out the whole surface area the cross-section of these individual components and immediately paid off:

  • It surfaced edge cases we hadn’t realized were valid.
  • It exposed state combinations we didn’t know were broken.
  • It exposed state combinations we did not even think someone would apply.
  • It caught several real bugs hiding in production logic.

The Payoff: Deleting Half Our Tests

The entire shift took three days. In 72 hours, we went from crossing our fingers on edge cases to having deterministic, domain-level coverage across the core entities of our business.

To be clear, this doesn’t replace everything. You still need practical integration checks to verify that external dependencies, like the EU’s VIES VAT validation service, behave properly in the wild. And if your team misinterprets a tax law when reviewing the Gherkin files, the code will still reflect that mistake. The difference is that the Gherkin feature files are now the single source of truth, and reviewing plain English is infinitely easier than auditing thousands of lines of mock-heavy test code.

When I started this experiment, we had 1,473 unit tests, and bugs were still slipping through every week.

Today, we have deleted more than half of those unit tests. They were just noise testing implementation details. In their place, we have a lean, combinatorial BDD suite that executes in a few seconds—and lets me actually sleep on a Sunday night.

Interested in these topics?

Contact me on LinkedIn or X.

Need help with your process?

Book a call

More posts