What Is Right-Sized AI in Test Automation?

Megana Natarajan

AI in software testing has been marketed as the next big thing in test automation for the past few years. It is considered to be the most significant table turner in QA engineering. Dependent on whom you ask, AI can apparently build entire test suites, maintain flaky automation, put manual QA testers out of work, and eventually run software testing with little human involvement.

The above were the expectations; the actual reality inside engineering teams is quite different.

Most of the QA teams experimenting with AI quickly come to accept two things. First, AI does help improve productivity if it is used correctly. Second, using excessive AI in the wrong scenarios leads to maintenance problems, instability, and a dangerous sense of false confidence.

Another issue with bringing in AI in test automation was that a lot of organizations followed the “bigger is better” strategy. This led to big models like GPT-4 and Gemini Ultra, which have several billion parameters. While this is powerful for generic tasks, this massive scale often generates unnecessary computational burden without offering relevant gains in specialized domains such as software testing.

The Size Paradox

There is an efficiency gap that explains why just leveraging the biggest AI is not the best strategy for productive AI testing processes. Rather, it is advised to use the right-sized AI models for the most efficient test automation solutions.

Right-sized AI doesn’t mean that you avoid AI. It is about using the right level of AI for the issue you are facing. In some areas, AI is absolutely successful in improving testing efficiency. In some scenarios, it is better to stick to traditional deterministic automation.

The experienced QA teams are not focused on making AI run the complete testing lifecycle independently. They are creating systems where AI enables engineers rather than replacing engineering judgment.

And that difference is more critical than most people understand.

The Issue with “AI Everywhere” in QA

The majority of the current AI testing conversation is fueled by hype rather than engineering reality.

If you observe product demos from AI testing vendors, everything looks miraculous. A prompt creates tests immediately. An AI agent repairs selectors automatically. Another AI model analyzes failures and decides which tests to run.

• Requirements change frequently
• UI structures evolve
• Flaky tests start multiplying
• Generated assertions miss business logic
• Teams stop trusting the automation suite

Typical automation tools such as Playwright, Cypress, and Selenium became famous as they are deterministic. A test either passes or fails, depending on explicit logic written by engineers.

Gen AI transforms that model fully. Large language models are probabilistic systems. They build outputs based on patterns, not certainty. This makes them extremely useful for accelerating iterative work, but much less dependable when accuracy becomes important.

Read: Generative AI vs. Deterministic Testing: Why Predictability Matters.

Think of something as trivial as generating a checkout validation test.

test('checkout flow', async ({ page }) => {
  await page.goto('/checkout');

  await page.fill('#card-number', '4111111111111111');
  await page.fill('#expiry', '12/30');

  await page.click('#submit');

  await expect(page.locator('.success')).toBeVisible();
 });

• pricing validation
• tax verification
• failed payment handling
• fraud edge cases
• retry behavior
• backend consistency checks

The AI created a happy path, not a dependable business validation strategy.

This is the main problem with overusing AI in testing. AI is often good at producing plausible automation, but plausible automation is not the same thing as trustworthy automation.

What is the Actual Meaning of Right-sized AI

Right-sized AI is basically a balancing strategy. Instead of enquiring: “How much testing can we automate with AI?”

Experienced QA teams ask: “Where does AI create the most engineering leverage without reducing reliability?”

That small shift modifies how teams approach automation entirely.

• writing boilerplate test code
• summarizing failures
• catching duplicate defects
• repairing broken locators
• recommending edge cases
• prioritizing regression execution

These are tasks where speed and pattern recognition matter more than perfect deterministic reasoning.

• compliance validation
• financial correctness
• complex business rules
• release gating decisions
• security-sensitive workflows

In such scenarios, determining automation is still relevant.

The goal of right-sized AI is not to maximize AI usage. The objective is to maximize confidence while reducing engineering effort.

Why QA teams are Shifting Toward Hybrid Automation

A very interesting change taking place in modern QA is that teams are slowly walking away from the idea of “fully autonomous testing.”

A couple of years ago, many companies believed AI agents would finally replace big portions of manual and automated QA work. Currently, most experienced engineering teams are adopting a practical method.

The right pattern is highly hybrid. AI manages assistance and acceleration. Engineers handle testing and decision-making. This division works quite well.

Think of test maintenance as an example. Handling selectors has historically been one of the most frustrating segments of UI automation. Minor DOM changes can break hundreds of tests overnight.

This is typically an ideal AI use case.

A self-healing system can catch similar elements, verify selector confidence, and repair locators automatically without modifying the intent of the test itself.

await page.locator('[data-test="submit-btn"]').click();

await page.locator('button:has-text("Submit")').click();

after a UI update.

Here, AI is correcting a niche maintenance problem with relatively clear boundaries. The system is not deciding whether the business logic is correct. It is simply helping automation survive UI evolution.

The above is an example of right-sized AI in practice.

Mapping AI Types to Real Testing Problems

• Large language models are efficient for creating and explaining things.
• Predictive machine learning models are more suited for catching patterns in historical data.
• Computer vision systems are ideal for visual verification.
• Gen AI models are good for test creation. They help quickly convert requirements into draft automation flows, suggest edge cases, and build framework-specific code.

The value is gained from matching the correct AI functionality to the correct engineering issue.

Take, for example, a QA engineer working with a tool like Playwright might request an LLM to build authentication tests with positive and negative testing scenarios. Rather than starting from scratch, the engineer gets a first draft of implementation that can be refined and tested.

This alone helps save a big chunk of development time.

• flaky test frequency
• defect hotspots
• commit history
• regression failures
• impacted services

This is really valuable in bigger CI/CD environments where running the full regression suite on every commit becomes costly.

Rather than executing 15,000 tests blindly, predictive systems can catch which areas are statistically most likely to fail.

Computer vision AI is another category that has become more valuable, especially for frontend-heavy apps. Usual screenshot comparison tools tend to generate noisy failures for minute visual differences. AI-based visual systems can better differentiate between efficient UI regressions and harmless rendering variations.

Also, the pattern is quite consistent: AI works best when applied to focused, high-volume problems with clear boundaries.

A Practical Decision Framework for AI in Test Automation

One of the most exploited ways to misuse AI is to embrace it without defining where it can and cannot be trusted.

A useful metric many senior QA teams adhere to is simple:

The higher the business risk, the lower the AI autonomy.

Say, for example, utilizing AI to summarize test failures is quite low risk. If the summary is wrong, engineers can still inspect the logs manually.

Using AI to autonomously verify payment calculations in production is quite different. The cost of being incorrect is way more expensive. This highlights why deterministic automation still matters in mission-critical systems.

• payments
• healthcare workflows
• compliance-sensitive operations
• financial reporting
• legal calculations

Then explicit assertions remain essential.

expect(order.total).toBe(149.99);
 expect(tax.amount).toBe(22.50);

instead of allowing an AI system to “infer correctness.”

So right-sized AI is not going to eliminate deterministic verification; it protects it.

The Effective AI Strategy that Works in the Current Setup

• deterministic automation frameworks
• AI-assisted engineering workflows
• predictive optimization
• human review systems

Most modern workflows would look something like this: An LLM helps write the first draft, and Playwright tests from product requirements. Visual AI checks UI rendering modifications. Manual QA engineers review important assertions before release.

This strategy has a higher scaling efficiency than trying full autonomy.

Also, engineers continue trusting the system. And trust is one of the more underappreciated metrics in automation. The moment teams stop trusting test results, the value of automation fails fast.

How AI-Driven Tools Fit into Right-Sized AI

• Fragile traditional automation
• Fully independent AI agents

They demand something more practical. This is where tools like testRigor are useful, as they sit somewhere in the middle. Rather than depending fully on low-level implementation details such as XPath-heavy Selenium scripts, testRigor concentrates on higher-level test intent using plain English prompts. The tool also uses AI-driven functionalities like self-healing behavior and maintenance reduction.

driver.findElement(By.xpath("//button[@id='submit']")).click();

click "Submit"

That abstraction is necessary because most automation instability comes from implementation-level fragility rather than business-flow changes.

This is also a good example of what right-sized AI looks like in practice. The AI is not making release decisions or inventing business logic. Rather, it is helping to bring down maintenance overhead and improve test resilience while engineers still control validation and quality standards.

• UI changes frequently
• Test maintenance consumes a large engineering effort
• QA teams need broader automation coverage
• Non-developers contribute to testing workflows

Conclusion

The conversation surrounding AI in testing has finally started to become more grounded in engineering reality.

The majority of mature QA teams are no longer trying to fully replace their testing team with autonomous AI systems. In place, they are in search of practical ways to bring down iterative work, increase automation stability, and scale quality engineering without losing trust in their test suites.

This is what right-size AI is all about.

Frequently Asked Questions (FAQs)

1. What is right-sized AI in test automation?
   A: Right-sized AI in test automation means using the right level of AI for specific testing problems instead of applying AI just everywhere. It focuses on balancing automation efficiency, reliability, maintainability, and human oversight rather than depending on fully autonomous testing systems.
2. How does right-sized AI reduce flaky tests?
   A: Right-sized AI brings down flaky tests by applying AI selectively to maintenance-heavy areas like locator healing and visual validation while keeping core assertions deterministic. This prevents AI from introducing unnecessary instability into critical workflows.
3. Why is fully autonomous AI testing risky?
   A: Fully autonomous AI testing can become risky because AI-generated tests may miss critical business logic, create flaky automation, or generate false confidence in test coverage.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT