Test Automation Blueprint

Best practices for using testRigor’s AI

• Always start with generative AI-based test generation and see if you need to correct the test steps (you should expect to tweak a bit).
• Describe your application’s purpose and capabilities in as much detail as possible. The more detailed the description is, the better chance that generative AI will be able to build successful tests.

You can provide this description while creating the test suite, as shown below:

You can add this description later in a test suite by navigating to

Settings -> Test Generation -> AI-based Test Generation.

• Break down in-script prompts (i.e., AI-based rules) into several steps like you would for a description intended for a human. Avoid making assumptions that AI will understand long processes with very few words.
• Avoid using internal jargon, as AI does not understand the abbreviations and terminology you may use internally. Instead, use standardized terminology or words that AI can associate with the content it will see during the scenario’s execution.
• Use generative AI to speed up the test case creation process; however, avoid becoming fully dependent on it. testRigor’s AI is trained to complete the flow using any path it may find. Once the creation process is completed successfully, check whether it is more important to complete the user flow or to validate a specific application functionality.

Note: It is recommended that you disable the AI feature of a rule once it has the correct steps or is close enough to complete them to fulfill the prompt.

Set up test suites for CI/CD

You should decide what structure works best for your organization regarding test suite organization for CI/CD and development. We have introduced ample options for flexibility to allow several approaches since what we consider best practice at testRigor might differ for your organization.

Option 1: Set up one suite per environment

• This option is ideal. It best matches the way testRigor was designed. Create a separate test suite for each test environment of the application.
• Establish a global or parent suite with the test cases, reusable rules, and test data.
• Make the other test suites children of the parent to maintain everything in one place.
• Handle discrepancies in child suites by overwriting rules/test data and creating duplicates in the child suite.
• Use Bash or PowerShell scripts from the CI/CD Integration section to trigger runs as needed in your pipeline.
• Remember that testRigor uses ‘Labels‘ to group cases into subsets. This is necessary to maintain order and structure.

Option 2: Set up one suite per product

• It is also possible to trigger the same cases on branches or different environments of the same product using one test suite.
• From testRigor’s UI, insert the URL of the environment, the branch name, and the commit ID.
• Using the pre-populated Bash or PowerShell scripts, you can trigger runs with the CI/CD tool of your choice.
• You can trigger multiple branches and the baseline simultaneously.
• testRigor’s CI/CD Integration section contains pre-populated scripts to trigger runs of individual test cases and subsets of test cases via labels. There are also scripts to cancel these runs and see reports of runs.
• Handle discrepancies between environments using baseline mutations.

Option 3: Maintain the repository from Git or your CI/CD tool

• Create and maintain test cases, test data, and reusable rules in JSON format or export YAML or txt files from your suite in testRigor and convert them to JSON.
• Using the pre-populated Bash or PowerShell scripts, you can trigger runs with the CI/CD tool of your choice.
• You can trigger multiple branches and the baseline simultaneously.
• testRigor’s CI/CD Integration section contains pre-populated scripts to trigger runs of individual test cases and subsets of test cases via labels. There are also scripts to cancel these runs and see reports of runs.
• Handle discrepancies between environments using baseline mutations.

Involving PMs and developers

testRigor allows cross-team collaboration easily and involves everyone on the team to contribute:

Involving PMs: There are several ways to involve PMs in the creation of test cases.

• testRigor’s natural language commands match common product specifications. These specifications are executable without the need for any coding.
• Empty reusable rules allow PMs to add specifications that go beyond testRigor’s keyword language. Testers then fill in the empty rules with testRigor’s keyword language.
• Similar to empty reusable rules, generative AI or AI-based rules allow PMs to add specifications that go beyond testRigor’s keyword language with the addition of AI to execute them.

Involving Devs: There are multiple tools for involving developers as well.

• Devs can manage test case repositories in Git and then run them using the pre-populated scripts found in the CI/CD section.
• Devs can run test cases in their local environments as well using our command line tool.

Environment-specific testing

Dealing with environments is a two-level process:

• Prepare all test environments: On the first level, you want to separate all different types of environments, such as production, staging, UAT, and dev, into different test suites. Each environment would contain different versions of the system under test and, therefore, would have different tests verifying different functionality.
  In a classical scenario, new functionality goes from the dev environment for the time it is being developed to the staging environment for testing before delivering to production. Therefore, testRigor can promote test cases (with associated rules and, potentially, test data) to the next environment for code promotion.
• Reuse test environment: However, sometimes environments are dynamically created/destroyed for branches, usually for dev environments to test branches. Here, it is possible to reuse the same test suite for development and point it to different test environments.
  You can do so by changing the URL in the Test Suite Details section on the UI.
  

Otherwise, use API or command line, where it is possible to override URL, test data, test cases, rules, and settings via the API itself. This way, it is possible to reuse the same development test suite for multiple (even ephemeral test environments).

Compare test run results on the same or different environments

With testRigor, it is possible to compare the results of a test run between different branches/runs. For that, you can go to the ‘Errors’ section and click ‘Compare to previous run‘.

It is possible to do this on every test suite execution found in the ‘All Runs‘ section in the left hand side pane.

Best practices for naming conventions

• The best practice of naming conventions is to match the language that you use yourself in manual test cases or in functional specifications. This applies to both test cases and reusable rules. Read this article to learn How to Write Maintainable Test Scripts.
• For the test data, it is advised to agree upon naming conventions for the variables, for instance, camelCase.

Folder structures best practices

Create folder structures based on the functionality/module/release.

• Currently, folders are used on the homepage to gather test suites into groups for products or teams. The best way to organize folders depends on the way you organize suites. If you test all environments from a single suite, folders should be organized by teams.
• If you have several suites for different environments of the same product, all the suites of that product should go into a folder, which can then be nested into a team folder.

Effective use of labels and test suites

Test suite runs

• Full runs of test suites can be triggered manually from the testRigor UI via CI/CD tools or API tools like Postman.
• Via testRigor’s UI, full test runs can be scheduled by leveraging our monitoring tool. Suite runs can be triggered as little as once a week and as often as every 4 hours.
• If the time intervals provided by testRigor’s default options do not meet your needs, you can use your CI/CD tool, which usually has more flexible options in this regard. Use these options to trigger runs on the schedule of your preference.

Runs by label

Labels in testRigor have two primary purposes:

1. Easily locate or filter results based on these test labels.
2. Gather test cases to have a test run of subsets instead of full retests. testRigor’s reason for using labels instead of folders for this purpose is that a case can only belong to a single folder, while several labels can be added to the same scenario.

For example, you may have a group of sanity cases, each of which may pertain to different areas of your product. Sometimes, you will want to run those sanity cases as a subset, and other times, you will want to test only specific areas of your product before merging new functionality. Therefore, labels allow you to do that where folders would not.

Apply labels using any of two options:

Option 1: Apply labels manually in bulk

• Filter the test cases that need to be labeled.
• Click Select multiple.
• Select the cases.
• Add/Remove a label in any of the two ways:
  • Add or remove a label by clicking the + button on the right of the Labels field in any one of the selected cases. testRigor will ask you whether the changes should be applied to all selected test cases. Accept the changes.
  • Add or remove a label by clicking Edit Labels. A prompt will pop up, and the user will be able to select existing labels, create new ones, and then add/remove them from all selected test cases.

Option 2: Strategically apply labels automatically via reusable rules

Create reusable rules that should apply only to cases within a specific subset. As the purpose of this method is to be strategic, the steps within this rule can be any of the following:

• a single command or step unique to that subset of cases
• a series of commands or steps unique to that subset of cases
• a commented line

Here is a how-to guide on reusable rules or subroutines in testRigor.

Maintain smoke/end-to-end/regression/functional tests

testRigor minimizes maintenance and is easy to adapt, let’s see how:

Preventive Maintenance

• Create reusable rules for all repetitive processes. If a step changes in that process, the changes made to the rule apply to all the cases where it is used, which is exponentially easier than changing the same step case by case.
• For situations where the changes are known prior to the test run, use Find and Replace in the Test Cases section to make edits in bulk.

• Alternatively, if there is no text or label for the element you need to target and you can only use an ID, class, or another locator, that may change. Then, you can save it in the Test Data section as a more readable variable. If the locator does change, it’s exponentially easier to edit the variable than edit the locator case by case.

Reactive Maintenance

testRigor groups all the cases affected by unique errors and allows these errors to be corrected using the ‘Fix by find and replace‘ functionality in the ‘Errors‘ section.

• Click ‘Fix by find and replace‘ on the unique error that needs to be addressed.
• Edit the portion of the ‘Replace with‘ script to address the cause of the failure.

Test suite reuse: Android, iOS, and Web

• You can inherit a test suite in the Shared suite section on the left-hand side pane. This way, you can reuse the test cases, rules, and data across different test suites for other testing, like Android and iOS.
• Once inherited, you can override test rules, data, and cases to match a slightly different functionality in the test suite. For example, web functionality might have more functions, and the menu can always be visible compared to the Android functionality.
• When this is done, test scripts must account for resolution differences. This is mostly done by using dynamic scrolls and conditional loops in the original test scripts to compensate for discrepancies.
• Create either rules or if/else condition blocks for known UI discrepancies. For example, a navigation pane on a desktop UI is often collapsed into a menu button on mobile. These types of differences must be accounted for in your script.
• Remember that some commands can only be used for certain types of testing. Avoid using desktop web-only commands for mobile web scenarios. testRigor’s documentation mentions when commands or functionality is only available on a specific platform.

Here is an example of a Shared test suite:

When and how to use shared test suites

There are two main use cases for shared test suites:

• Shared test scripts and test data for different types of testing (Android, iOS, Web, Web for mobile)
• Shared reusable rules as a library for multiple projects.

Test data and environments

• Use global test data to store variables for use in individual scenarios. Keep in mind that global variables overwrite built-in variables. Know how to set up global variables in testRigor.
• Use data sets to allow test cases to repeat themselves using different data each time. Keep in mind that data set variables overwrite global variables of the same name.
• Our recommendation is to manage test data of different test environments from their own suites. It is beneficial when the test data is different from one test environment to the next. It is possible to share test cases without sharing test data. If the discrepancies between the parent and child are minimal, it would be best to share test data among suites and create variables with the same name to account for those discrepancies.
• While this is not the most recommended way, if the preference is to test different test environments in the same suite, it is possible to keep one-row data sets for each environment and link them to all cases in bulk.
• Data sets can be shared with a child suite, or they can be overwritten with a data set of the same name inside the child suite. Read this how-to guide to perform data-driven testing in testRigor.

Dealing with passwords and other sensitive data

• The main username and password for a test suite should always be stored in the Authentication -> App-specific Authentication section of Settings. This will enable the built-in ‘login‘ rule to work out of the box. Know How to simulate login action using testRigor.

• For the rest of the testing, it is possible to create hidden test data. When creating test data, you can select type ‘Hidden‘, which should hide the data across the board like the password. This way, it is possible to deal with multiple users for multiple roles. This also applies to test data sets.

• It is also possible to get sensitive data from your vault and pass to testRigor. For this, the variable should be declared ‘Hidden‘, and the data can be passed/overridden by an API call from CI/CD. The process would be like this:
  • Read sensitive data from the vault
  • Construct the updated call to testRigor for overriding the data
  You can also get the secrets directly from vault at runtime like this:

  call api get "https://myvault.com/secret" with headers "Content-Type:application/json" and "Accept:application/json" and get "$.data.password" and save it as "password2"
  enter stored value "password2" into "password"

• Passwords usually should not be generated even if you are testing sign-up scenarios since it will prevent you from logging in to the account in case something needs to be checked manually and generated data is exposed in the UI.

Inheritance and sharing of reusable rules

For environments being tested in different test suites, reusable rules can be shared from the parent suite.

• Navigate to the child suite.
• Go to the Shared Suite section.
• Enable the checkbox for Inherit rules.
• Select the Parent suite from the dropdown and Save.

• If the rules differ slightly, create a rule with the same name in the child suite and modify the step(s) needed to handle the differences.

Access management test suites/teams

Company management must be enabled on the subscription. Once it is enabled, your company admins have several options:

• Enable Single Sign On (SSO): This will allow users with the established email domains to simply login without the need to invite them or have them sign up.
• Invite someone as a Billing Admin: In addition to collaborating with test case building, the billing admin will be able to invite other users, and they will receive the billing emails from testRigor. They will also be able to assign users, servers, and suites to teams.
• Invite someone as a Company Admin: In addition to collaborating with test case building, they will also be able to assign users, servers, and suites to teams.
• Invite someone as a Developer: They can collaborate in the test case building for the suites they are assigned to.
• Invite someone as an External Collaborator. They can collaborate on test case building in the suites they are assigned to.

Note: Anyone not assigned to a team will be able to access all suites.

Rerun only failed test cases

After a test suite run, some test cases may fail because another case was using a resource that the failed case was using at the same time. testRigor has a setting to automatically re-run failed cases up to two times. However, if automatic re-tests are not desired, testRigor also has the functionality that allows the user to re-trigger all the test cases that failed.

To re-run only the failed test cases, do the following:

• Wait until the test run is completed.
• Go to the Test Cases section.
• Click the red ‘Rerun failed‘ message at the top of the screen.
• The button can also be located in the Test Suite Details section.

Test data vs test data sets

• Test Data: Useful when data is shared across test cases/rules or requires modification for different use cases like language/environment/etc.
• Test Data Sets: Useful for the use cases where the same test case must be run on different sets of test data to test different data variations.

Do I need to do any cleanup: delete uploaded files or log out?

• No. You don’t need to perform any file cleanup or log out since the system will start tests on a new VM and browser anyway.
• Do you still need to do the cleanups, though? It might be better to do setup instead of clean up since this is guaranteed to run. However, if the test fails, the cleanup might not run the way you expected.

How to do self-healing?

• Self-healing is an old term used by some old test automation systems to correct the locators for elements. Since testRigor doesn’t use locators, it is more stable than any self-healing system there and is not susceptible to any auto-healing of the wrong element.
• In addition, if you give a function a descriptive name like ‘find a Kindle,’ it should work regardless of how you implemented the search function.

How do I see browser versions?

• Navigate to Settings -> Advanced, Enable ‘Show actual OS and browser versions‘, and Save settings.

• Run test cases.
• Go to the test case execution page.
• See the OS and Browser versions at the beginning of the test cases that appear in vertical fashion.

How to deal with OTP using Email/SMS/2FA?

You can send and test OTPs using testRigor in three ways:

1. Email:
   • Have the test case send the OTP to a testrigor-mail.com email address.
   • Check the email, grab the OTP, and save it as a variable.
   • Go back to the login tab.
   • Insert the OTP variable into the required field.
   Here is a guide on how to do email testing using testRigor?
2. SMS: Know the testRigor commands to test SMS here. You can also go through this detailed how to guide.
3. 2FA: Here is a guide to know how to do SMS, 2FA, and phone call testing using testRigor. Also, you can go through this document to know more about testRigor’s Gmail 2FA support.

How to deal with user certificates for browsers

• Navigate to Settings -> Advanced. You will see an option to Install client certificates, where you can upload the certificates you wish to use.

• In test cases, when the certificate window appears, use testRigor commands type arrow up, type arrow down, and type enter to select certificates.

Note: This will only work on Chrome/Edge on Windows.

Test case import

It is simple to import your TestRail’s existing test cases into testRigor and convert them to automated tests directly. Here is a how to guide to import test cases from TestRail for execution.

• Imported test cases keep testRigor’s keyword steps. If there are steps that are not keyword-based, they are automatically converted into AI-based rules.
• When the phrases are imported as reusable rules, those rules can be reused as functions going forward.

Note: Read testRigor’s best practices for building test cases.