Selenium AI

Pragya Yadav

For decades, Selenium has enjoyed being synonymous with test automation. With the advent of AI-powered intelligent test automation tools, organizations now have some excellent options which solve the issues faced with Selenium. The question arises: Can AI be used with Selenium to enhance its capabilities?

Let’s find out in this article.

Levels of Artificial Intelligence

Artificial Intelligence is dynamic and constantly evolving with computer code, data structures, and algorithms to enhance itself. Here are the four broad levels of AI:

Reactive Machines (Narrow AI): These AI systems can perform specific tasks but cannot learn from new data or adapt to different situations.

Example: Chess-playing programs such as IBM’s Deep Blue can analyze millions of potential moves to choose the best. However, it can’t learn from new opponents, retain past memory, or adapt to other games.

Limited Memory (Weak AI): These AI systems can retain some past experiences and use them to improve performance over time. But still, they focus on specific tasks and don’t possess a general understanding of the real world.

Example: Self-driving cars that can learn from past driving experiences to improve their ability to navigate roads and traffic, but they can’t do tasks unrelated to driving. Other examples are virtual assistants like Siri/ Alexa, purchase recommendations on Spotify/Amazon, and language translation services like Google Translate.

General AI (Strong AI): This AI level involves machines that can understand human emotions, intentions, and beliefs. We are yet to reach here, as this level would apply to AI systems that can engage in natural conversations, understand human emotions, and behave accordingly.

Self-aware AI (Superintelligent AI): This is the hypothetical level of AI where machines possess self-awareness, consciousness, and intelligence, surpassing human capabilities. This level of AI is more speculative and is the subject of philosophical debate.

Concepts to Know in AI

Neural Networks

A neural network is a fundamental concept in artificial intelligence and machine learning. It’s a computational model inspired by the structure and functioning of the human brain, precisely the way neurons process information.

At its core, a neural network comprises interconnected nodes, often called neurons or units, organized in layers. These layers are typically categorized into three types:

Input Layer: This is where the data is initially fed into the neural network. Each neuron in the input layer corresponds to a feature or attribute of the input data.

Hidden Layers: One or more layers between the input and output layers serve as hidden layers. Neurons in hidden layers process and transform the input data through weighted connections and apply activation functions. Hidden layers enable the network to learn complex patterns and representations from the data.

Output Layer: This layer produces the final results or predictions based on the computations performed by the previous layers.

Neural networks use weights and biases associated with the connections between neurons to learn and adapt to the data. During training, the network adjusts these weights and biases iteratively to minimize the difference between its predictions and the actual target values.

Use Cases of Neural Networks

Face Detection: Neural networks are pivotal in advancing face detection technology in AI. They enable machines to detect and locate human faces within images or videos, crucial for various applications such as facial recognition, identity verification, surveillance, photography, and more.

Other use cases are image and speech recognition, natural language processing(NLP), game playing, and autonomous vehicles. For all these to happen, deep learning, a subset of machine learning, utilizes neural networks with many hidden layers (deep neural networks) to learn intricate and hierarchical representations from data.

Reinforcement Learning

Reinforcement Learning (RL) is a machine learning paradigm that trains agents to make sequences of decisions in an environment to maximize a cumulative reward. It can achieve objectives by trying all possible combinations and their outcomes using a set of measurable actions.

Components of Reinforcement Learning:

Agent: A learning entity that interacts with the environment, taking actions and making decisions.

Environment: An external system with which the agent interacts. Based on the agent’s actions, it provides feedback in the form of states, rewards, and potential next states.

State: A representation of the current situation or configuration of the environment that the agent uses to make decisions.

Action: The choices made by the agent that influence the environment’s state transitions and subsequent rewards. The closer a single action takes toward the objective, the more that particular action is rewarded.

Reward: A scalar value that represents the immediate feedback the agent receives from the environment after taking an action. The agent’s goal is to maximize the cumulative reward over time.

Steps in Reinforcement Learning:

1. Agent observes the current state of the environment.
2. Agent selects an action according to its learned policy based on the observed state.
3. Environment responds with a new state and a reward.
4. Agent uses this information to update its policy to improve future actions.

How to use AI with Selenium?

A huge variety of objectives can be achieved through Reinforcement Learning if they fulfill below criteria:

1. The objective must have measurable actions
2. The actions can execute automatically through a system that can access results and update the existing policy.

Example: Consider an e-commerce web application under test(AUT). Let us apply Reinforcement Learning to it and use Selenium to complete the model.

• Environment: Defined by the web elements, links, images, page outlay, etc.
• Actions: Text entered, clicked element, page scrolled, etc.
• Results: Measured through assertions on the elements and the page navigation.

This AI model can be coded using Selenium, where you can simulate the user actions and then assert the results using assertion statements. Since we have a known/measurable set of user actions that can achieve an objective through different combinations/variations, Selenium code for automation testing can be applied.

Examples of Selenium AI

Below are a few products that support AI with Selenium code:

headspin

To perform AI-based element identification in Selenium, headspin has written client-side plugins with access to the driver object. They have created a library that takes an existing Selenium session (a driver object) and uses it for its purposes. The library has access to the Test.ai classification model that already exists as part of the Test.ai + Appium classifier plugin.

For Selenium, they extended the capabilities of the existing Appium classifier plugin to act as a classification server. To use, set up the classifier server and the classifier client, then write Selenium code to find the correct web element using code assertions. The code for this is as below from the headspin website.

Healenium

Healenium, an open-source testing framework extension, enhances the reliability of Selenium-based test scenarios by effectively managing modified web and mobile components. It utilizes a machine-learning algorithm to assess the current status of the page, addressing issues related to NoSuchElement test failures.

This self-healing capability is only supported in Java, and you need to perform the below setup to use it:

1. Start the back end (docker)
2. Dependencies and Page Object Setup (Selenium WebDriver, WebDriverManager, and JUnit5)
3. Identify the page elements, write code assertions, and write test scripts code.
4. Before you run the tests, import the Healenium Maven dependency and reporting plugin.

testRigor

SelfHealingDriver driver = TestRigor.selfHeal(new ChromeDriver(), "a2518f3a-3e8b-484a-befe-23b9160ef166");

There are two stages:

1. After successful test execution, testRigor will store locators and corresponding page information and infer user intent for each locator.
2. Whenever an element is not found using a locator, it will use stored user-level intent and page information. This helps to find the new locator for the intended element. If found, the new locator is used; if not, it will fail the test.

Since testRigor will infer the user-level intent, your test will succeed only when the intended action is possible from the end user’s perspective. If there are element attribute changes and the locator cannot find it – testRigor will fail the test to avoid false positives.

You can save time on two things in test maintenance with the testRigor Selenium plugin:

1. Investigation of why the test failed.
2. Finding what is a new locator to replace the non-working one.

Why Use testRigor Instead?

Even if you surpass Selenium’s existing issues, the Selenium AI integration will increase the overall complexity of the test automation framework. You will also require QA engineers who are well-versed in Selenium, programming languages, and AI/ML. This is a dual problem and is costly and hard to solve.

Better option is available, which will help you skyrocket your automation ROI with the least test creation, execution, and maintenance efforts. Here is a comparison of testRigor vs. Selenium to understand how testRigor is a better choice.

See below a quick list of testRigor’s features that solve the issues Selenium has:

• Solves the major issue of flaky and unstable tests that Selenium poses. Elements are referenced as text visible on UI, not through XPath or CSS locators, simplifying creation, maintenance, and debugging.
• Perform test data management quickly through testRigor’s in-built features without spending hours writing programming code to read, write, and update the data files.
• Any element attributes or application changes are automatically applied to the test scripts using self-healing. This helps save considerable maintenance effort and time compared to Selenium frameworks.
• Unlike Selenium, no external integrations are required for reporting or executing different test types. testrigor provides freedom to perform all prominent testing types, web, mobile, desktop, API, visual, parallel, cross-browser, and cross-platform tests using a single tool.
• Integration support for significant issue management, CI/CD, and infrastructure provider tools ease your testing process. You need not manage dependencies and integrations manually.
• Writing test scripts in Selenium requires coding or language proficiency and is highly time-consuming. testRigor provides the latest in-built AI features to create tests easily through any three choices:
  1. Generative AI: Only provide the test case title; you will have the actual test steps generated by the system within seconds.
  2. Natural Language Processing (NLP): Write test cases using plain English commands, without any special element locators.
  3. Test Recorder: Record your actions, get the test case in the same plain English format, making it easy to add more assertions, edit, and maintain.

The result? See for yourself, how the same test will look in Selenium vs in testRigor:

These capabilities enable everyone on your team to test and contribute. See a complete list of testRigor’s features and use AI to supercharge your testing.

Conclusion

Integrating Selenium and AI is beneficial because it will mitigate the shortcomings of the Selenium framework. However, this amalgamation will increase the complexity manifold since AI is another grand universe.

Also, the reliance on AI models raises concerns about the reliability of predictions and their impact on testing accuracy. Over Reliance on AI can lead to false positives or negatives if the models are not correctly calibrated. Hence, intelligent tools such as testRigor, which have in-built AI capabilities to solve all your testing problems single handedly, are a great choice within budget.