Best Practices on How and When to Use AI in testRigor

Kevin Carroll

AI can significantly speed up test creation and maintenance in testRigor, but not every use of AI serves the same purpose or has the same tradeoffs.

1. Build-time AI — AI that helps create, improve, or repair tests.
2. Runtime AI — AI that is called while a test is actually executing.

The best practices for these two types of AI are different.

The simplest principle to remember is: Use Build-time AI freely to reduce human effort. Use Runtime AI only when you need it.

1. Use Build-time AI as much as possible to speed up test creation and maintenance
2. Review all AI-generated tests before considering them complete
3. Give Build-time AI enough context to understand what you are testing
4. Generate tests in manageable batches
5. Use AI generation and repair asynchronously when possible
6. Make sure AI-generated tests still follow testRigor best practices
7. Use Runtime AI only as a last resort
8. Try other approaches before adding using AI
9. Give Runtime AI clear and unambiguous instructions
10. Prefer normal testRigor logic when it can accomplish the same task reliably

Use Build-time AI as much as possible

Build-time AI is where users should take the greatest advantage of AI.

• generating new test cases;
• generating scenarios from a feature description;
• creating an initial test that you can refine;
• helping fix failing tests;
• maintaining existing tests;
• reducing the amount of test creation that must be done manually.

The purpose is not necessarily for AI to produce a better test than a person could create. The purpose is to save the tester from having to do all of the work manually.

Instead of starting every new test from a blank page, give AI the context it needs and let it create something you can work from.

This changes the tester’s role. Instead of spending most of the time manually creating every test step, the tester can increasingly act as a reviewer of work produced by AI.

AI does the bulk work. The tester verifies that the result is correct.

Review all AI-generated tests

A passing test is not automatically a good test.

Even when an AI-generated test is green, review it.

• it tests the scenario you intended;
• it follows the correct business flow;
• the validations actually verify the important behavior;
• the test is understandable to another person;
• unnecessary or confusing steps have been removed;
• the test follows testRigor best practices.

If an AI-generated test passes but you do not understand what it is doing, the test should not be considered finished.

AI can eliminate much of the manual creation effort, but it does not eliminate the need for human review.

Give Build-time AI enough context

AI does not automatically understand your application, business requirements, or intended testing strategy.

It may be able to interpret what it sees, but it should not be expected to make the correct assumptions about how your application is supposed to work.

Provide useful context through feature descriptions, scenario descriptions, test names, and other available information.

If a particular behavior must work in one specific way, make that requirement explicit.

This is especially important because AI is generally trying to accomplish the requested flow. If one route does not work, AI may attempt another route.

That can be useful in some situations, but it can also hide the exact condition you intended to test.

For example, if a particular button must perform an action and there should be no alternative route, state that requirement in the feature description or add a validation that requires that behavior.

If the required behavior does not occur, the test should fail rather than allowing AI to find another way to reach the end result.

Use AI Context when useful

When working with generated tests, AI Context can help you understand the logic AI is trying to follow.

AI Context is not the same as the generated test steps themselves. It provides additional information about what AI is trying to accomplish.

If the resulting test is going in the wrong direction, correcting the AI Context may help improve the generated test before manually rewriting individual steps.

Manual cleanup should still be used whenever necessary.

Generate tests in manageable batches

AI can generate a large amount of automation very quickly. Humans still need to review that work.

Generating too many tests at once can simply replace a test-creation bottleneck with a test-review bottleneck.

• review the proposed scenarios;
• remove unnecessary tests before additional effort is spent on them;
• correct test names;
• verify that the correct flows are being covered;
• distribute review work among the team.

As a practical guideline, approximately 20 generated test cases per available reviewer should be treated as an upper working limit for a batch. Smaller batches are often preferable.

If several people are available to review generated tests, a larger overall batch may be reasonable because the review work can be divided among them.

Use Build-time AI asynchronously

Some AI operations take time to complete.

There is no benefit in sitting and watching AI generate or repair tests when you could be doing other work.

1. Give AI enough context to begin the task.
2. Start the generation or repair.
3. Continue with other work.
4. Return when the AI work is complete.
5. Review the result.

This allows AI work and human work to happen in parallel.

The productivity benefit of Build-time AI is not only that AI performs work for you. It is also that the work can continue while you focus on something else.

AI-generated tests must still follow testRigor best practices

A test is not good simply because it passes.

AI-generated automation should be held to the same standards as automation created manually.

If AI generates a test that technically works but uses poor automation practices, clean it up.

Best practices matter for long-term stability, readability, and maintainability.

Tests that follow testRigor best practices are also the tests that testRigor can most effectively support and maintain under the testRigor warranty.

AI should accelerate the creation of good automation, not become a reason to accept lower-quality automation.

Use Runtime AI only as a last resort

Runtime AI is AI that becomes part of the execution of the test itself.

In testRigor, it is typically explicitly requested by adding using AI to a command.

click "shopping cart icon" using AI

Runtime AI can be used with commands such as clicks, grabs, validations, and other actions when normal testRigor logic cannot adequately accomplish the task.

However, do not add using AI simply because AI is capable of accomplishing something.

If a normal testRigor command can accomplish the same thing reliably, use the normal command.

Try other approaches before adding using AI

1. normal testRigor commands;
2. relative locations;
3. OCR where appropriate;
4. IF/ELSE conditionals or auto-apply rules when a portion of a test case presents predictability issues;
5. stable identifiers, if and ONLY if they are human readable;
6. AI-based reusable rules.

Runtime AI becomes appropriate when those approaches cannot reliably describe what the test needs to interact with or validate.

Example: an element that is difficult to describe normally

Suppose an application contains a shopping cart icon but does not provide stable visible text that testRigor can use to identify it.

click "shopping cart icon" using AI

AI can interpret what is visible on the screen and identify the object based on the natural-language description.

The problem is not necessarily that the object cannot be understood by a human. The problem is that traditional automation may not have a stable way to refer to it.

Example: understanding visual information

Runtime AI can also be useful when the validation itself requires visual interpretation.

check that page "contains diagram with 61% allocated to bla" using AI

A regular command may not be able to understand the meaning of a diagram or other visual information displayed on the page.

Runtime AI can interpret what it sees and apply the additional logic necessary to evaluate the instruction.

Give Runtime AI clear and unambiguous instructions

When Runtime AI is necessary, describe the task as though you were explaining it to someone who has never seen the screen and cannot make assumptions about what you mean.

The priority should be clarity, not making the instruction as short as possible.

click "the icon" using AI

This provides very little context if several icons are visible.

click "shopping cart icon in the header" using AI

provides AI with more information about what it should find.

If you are unsure whether your instruction is clear, ask yourself:

Could someone who has never seen this application understand what I am referring to from this description?

Prefer normal testRigor logic when it works

Runtime AI adds additional work to every execution.

Instead of testRigor executing its normal logic directly, the task may need to be sent to an LLM along with information such as the current screenshot. The AI must interpret the request, determine what needs to happen, and provide information that testRigor can use to continue the execution.

That process takes time.

Using Runtime AI will generally make a test slower than accomplishing the same action with normal testRigor commands.

Runtime AI also introduces an LLM into every execution of that step. AI is extremely useful, but it is not perfect and may occasionally misunderstand the available context.

For regression testing, the preferred outcome is therefore to use normal testRigor logic wherever possible and reserve Runtime AI for situations where that logic is genuinely insufficient.

More AI does not mean a smarter test

Adding more AI does not automatically make a test better or more intelligent.

AI does not understand your application, requirements, and business context in the same way that you do.

If two approaches accomplish the same task reliably, the approach that does not require Runtime AI will generally be preferable.

The purpose of Runtime AI is to expand what automation can accomplish when ordinary automation cannot do the job well. It is not intended to replace ordinary automation that already works reliably.

Build with AI, run with testRigor

A useful way to think about the overall strategy is:

Use AI heavily while building. Use AI selectively while running.

Build-time AI can generate tests, help repair them, and reduce the amount of manual work required.

Once a test has been reviewed and approved, however, the preferred result is a stable test that primarily runs using normal testRigor logic.

The fact that AI created a test does not mean AI needs to be involved every time that test executes.

This allows teams to take advantage of AI’s ability to create and maintain automation without unnecessarily introducing LLM calls into every regression execution.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT