What Everyone Gets Wrong About AI Code Generation
 |
You installed Claude Code or Codex. You describe a feature. The agent writes the code.
Great. Until you try the feature.
The code does something. Just not quite what you intended.
So you investigate, explain what went wrong, wait for another attempt, and check it again.
Rinse and repeat.
The agent writes the code. You are still the feedback loop.
You have a tool that can keep working toward a goal, but you are still moving it forward one correction at a time.
Familiar?
That is a surprisingly manual way to use a coding agent.
Why does it happen?
Actually, it is a combination of missing context, ambiguity, and hallucinations. Yep, not just one.
Lack of context
LLMs do not work the same way as your brain if you use off-the-shelf models like Fable or GPT-6. They are certainly capable. But they just don’t know your project the way YOU know it.
- They didn’t talk to the project manager who described to you their vision and how things are supposed to work and why your code is not what they intended.
- They didn’t walk through the codebase to familiarize themselves with the code.
- They didn’t spend countless hours writing features and fixing bugs understanding all the nitty-gritty details of the code.
- They didn’t have conversations with the lead engineer on the project.
- They don’t know the challenges of your architecture in production.
- They probably haven’t read ALL of the project’s documentation either.
- And they definitely didn’t receive the detailed code reviews that shaped your understanding of the project.
Can LLM read your project or documentation – surely it can. But when you deal with it for a long period of time your understanding is deeper and you can see some consequences that can’t be predicted by just looking into the code the first time. Since you learned. And LLM just accessing the information at hand, it can’t fit it ALL in the context window.
The result? LLMs don’t have intuition and clear understanding how things SHOULD work in your particular project.
Can you spell it all out? In theory. But you’ll miss something for sure. And the amount of typing would be more than the amount of code you need to write.
Ambiguity
If you think human language is precise, you have no idea how far from reality you are. Even in the most precise area in the banking industry the ambiguity of specifications lead to 31% of the code needing rework. This is why people invented BDD in the first place.
Even a simple written instruction can be ambiguous:
Consider this:
`Open the file with Chrome`
Is it “use Chrome to open the file”? Or open the file that is associated with/labeled “Chrome” or containing Chrome (like an installer)?
A requirement can feel completely precise to the person writing it while still leaving several reasonable ways to implement it.
This happens between product managers and engineers. It happens between engineers. And it happens when we ask a coding agent to build something.
For example, imagine a product manager creates a ticket:
“Add automatic retries for failed tests.”
To the product manager, this may seem perfectly clear because they already have a mental model of the feature. They may assume everyone knows that a “retry” means retrying the entire test once, only for infrastructure-related failures, without counting the retry as a separate execution.
But an engineer reading the ticket may reasonably interpret it in several different ways:
- Retry the entire test or only the failed step?
- Retry once or several times?
- Retry every failure, including failed assertions?
- Should retries happen immediately or at the end of the test suite?
- Should a test that passes on retry be reported as passed, flaky, or failed?
- Does this apply to scheduled tests, CI/CD executions, manual executions, or all of them?
The PM didn’t intentionally leave these questions unanswered. They simply forgot which parts of their own context were not shared with the engineer.
So this is more than ordinary ambiguity. It is a combination of missing information and ambiguity caused by assumed context: the author believes the requirement is precise because the missing information already exists in their head, while the reader has to fill those gaps with their own assumptions.
Hallucinations
Frankly, it is hard to find a direct scientific comparison to humans. The last comparison was done by OpenAI in 2021 and cites humans as 6% vs AI at 42%.
The most recent I could find is AA-Omniscience: Knowledge and Hallucination Benchmark which cites Opus 5 on Max setting hallucination at 23% of the time! That means that even if everything is perfect and you spent a month building a perfect prompt there is still no guarantee that you’ll get a working result in return.
Why 23% of the time is a big deal? Because, based on our statistics the average length of an end-to-end test case is 56.1 steps. That means that if you give a go-through to AI it won’t even be able to get to the end of the test without hallucinating.
The result?
Not consistently what you expected.
How to deal with all of that?
Validation. Validation. Validation.
What if you had a way to check whether the feature you requested actually works as expected?
This would allow you to enable Claude Code/Codex to keep iterating with the `/goal` feature until the feature is delivered and working as expected!
As you have probably already guessed, I’m talking about tests.
Depending on the feature’s complexity, these could be unit, integration, or end-to-end tests. And I firmly believe that if it touches the UI and is complex enough it probably should be end-to-end.
But how do you write end-to-end tests if you don’t have the HTML/XML structure yet? Would that be impossible/waste of time?
Yes for PlayWright and Selenium it would, but not if you can create end-to-end tests without reliance on details of implementation. Imagine you can do it with plain English. Wouldn’t that be wonderful?
The improved workflow
Well, great news, you can now! With testRigor.
Imagine the process where you can generate your tests and get them reviewed if needed by either QA team OR the product manager! And then use those tests as part of the `/goal` feature for code generation to make sure you can generate the code that works as you need, instead of somehow maybe! On top of that you get reliable repeatable end-to-end tests without reliance on expansive hallucination-prone LLMs for execution.
Plain English is not magically unambiguous. The value is in turning a broad request into concrete examples that make the assumptions easier to see.
Once those scenarios reflect what we want, they become part of the agent’s /goal workflow. It implements the feature, runs the tests, inspects the failures, adjusts the code, and tries again.
The target remains the intended behavior—not simply making two AI-generated outputs agree with each other.
You are not replacing manual coding with manual test writing. The agent can generate both. The team contributes the product context and judgment, while the test results provide feedback during implementation.
Passing the tests does not answer every engineering question or replace code review. It does give the agent concrete evidence about the behavior those tests cover.
And when the feature is finished, the tests stay. They become repeatable checks for the next change, not another conversation you have to recreate.
That is the opportunity: not just generating code faster, but spending less time personally moving it from “looks done” to “works as intended.”
See the full walkthrough, including a video demonstration.
To explore it with your own application, start a free trial or book a demo.
| Achieve More Than 90% Test Automation | |
| Step by Step Walkthroughs and Help | |
| 14 Day Free Trial, Cancel Anytime |
