How Does It All Work?

With all the magical benefits of testRigor, almost no test maintenance, and AI writing tests for you, how does it all work?

What is the struggle?

To understand how we achieved the results it is necessary to understand what test stability actually means. As we all were engineers, we were torn between 100% of our test cases being green 100% of the time (like what would be expected from unit tests) and the fact that our perfectly written end-to-end tests were always failing on any non-trivial change.

However, at the same time if you perform tests manually you could go through the system under test. But why? Why can’t we make end-to-end tests as stable as manual tests?

And there we realized that the way the steps of the tests are described are different from the way we do in our tests. Conceptually different!

When we write unit tests or Selenium tests we use selectors like XPaths, CSS Selectors, IDs, data-test-ids, etc. However, if you think about it, all of these are details of implementation, how we or other engineers wrote the code for this particular change. And, of course, the way it is written is changing all the time since engineers are forced to refactor the code to adapt new changes, new elements in the UI, upgrade the libraries, make pages responsive etc.

How should it work?

In fact, there are 2 levels on which human language is different from what we code:

1. We describe things that are visible to humans from a human’s perspective using visual language and references
2. We don’t even describe steps sometimes. We just say “login” or “add a product to the cart”

And this is exactly how testRigor works. testRigor is a “human emulator” that gets plain-English statements and executes them from an end-user’s perspective.

Specifically, it is done on 2 levels:

1. Parsed English that allows you to describe things in English has a huge number of features, but it has some syntax to it, although we are trying to support any way you could have used it.
2. On top of that, you can write any language, including “add to cart”, and execute it. This is processed using a large language model (LLM) to understand what you want to achieve and the LLM uses our own commands to do the right steps.

How do our basic commands work?

Some of them are not related to the UI like “open new browser” or “call api” will do exactly what you specified. However, if you are using UI-related commands like “click” and “enter””, they will be done the following way:

1. They will talk to the browser/device/OS to get the list of visible elements. If not possible we will use AI to get them from the image of the screen;
2. They will classify those elements into 3 categories:
   1. Clickable elements that we call “buttons” in our documentation (they don’t have to be <button> tags)
   2. Interactable elements like input fields and checkboxes
   3. Just plain non-interactable texts
3. We perform additional iterations that use algorithms and AI to do things like label-to-edit associations
4. We find elements based on the description provided by users the same way a human would find it (keeping the context in mind, for instance, if you say “click” that means you are looking for a “button”)
5. Execute the action (usually through manipulating the mouse and keyboard)

How does any language support work?

We do it step by step. We provide the LLM with a screenshot, the list of steps that it has already done and ask it based on the prompt what should be done now. It’s very straightforward, since we talk to the LLM using our own parsed language that we support. Of course, there could be hallucinations and it can do some of the steps wrong. This is why we allow you to modify the steps that the LLM came up with, so that you can fix the steps. And the steps are still in plain English, the parsed version as described above. You can see how it works here.

This way we avoid any reliance on details of implementation whatsoever, and enable everyone on the team to collaborate:

1. PMs now can review and understand the tests (or even create them before engineers write the code)
2. Manual testers can participate in building and maintaining test automation

Still have questions? You can watch the video here or the whole library of video tutorials here.

Schedule some time to talk to us here.