ChatGPT for Test Automation
It is estimated that training the OpenAI ChatGPT model happened in a mere 34 days. Interestingly, the company’s executives predict hitting a revenue of $11 billion in 2025. Every industry and organization uses ChatGPT for some purpose or another. Understanding whether we can use ChatGPT in software testing activities would be interesting.
Let us find out in this article.
What is ChatGPT, and How Does it Work?
Here is a response from the ChatGPT, what it assumes itself to be:
ChatGPT is an AI language model developed by OpenAI, designed to generate text that mimics human-like responses in a conversational format. It’s built on the Generative Pre-trained Transformer (GPT) architecture, enabling it to understand and produce text across a wide range of topics and styles. ChatGPT can engage in dialogues, answer questions, provide explanations, and even create content such as stories, essays, or code, based on the input it receives.
The model is pre-trained on a diverse dataset compiled from the internet, which helps it learn language patterns, context, and information across numerous subjects. However, its responses are generated based on patterns in the data it has been trained on; it doesn’t possess understanding or awareness but rather predicts text that is statistically likely to follow the given prompt.
Hence, ChatGPT is a popular machine-learning model that is trained to chat with people and produce correct-looking results. As you can see above, it produced an impressive explanation and is learning fast.
Does it Mean that ChatGPT can be Trusted?
So, how does ChatGPT work? It is a neural network trained on massive data to answer questions and hold context to a certain degree. In essence, it is a set of functions extrapolating based on “training points“.
This is where ChatGPT’s pitfalls and limitations arise: since it is not nearly as powerful as a human brain, it does not have a high enough degree of complexity to be able to validate and understand either the question or the answer, and therefore, often produces non-sensical results that look legitimate. It becomes hazardous when the most part of the answer looks correct, but the issues are somewhere in the middle of the text. In fact, Stack Overflow banned ChatGPT for “constantly giving wrong answers”.
Using ChatGPT to Create Test Scripts
Similarly to the description above, ChatGTP can produce some results based on its training that can be very helpful in jump-starting your test. Here is an example of a sign-up test in Selenium using Java:
Below is an example of a sign-up test using Selenium with Java. This test assumes you’re testing a web application with a sign-up form that includes fields for a username, email, password, and a sign-up button. Please adjust the element IDs (‘username’, ’email’, ‘password’, ‘signUpButton’) and URL (‘http://example.com/signup’) to match your actual application.
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.*;
public class SignUpTest {
private WebDriver driver;
@BeforeMethod
public void setUp() {
// Set path to your chromedriver.exe
System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
// Initialize the driver
driver = new ChromeDriver();
// Maximize the browser window
driver.manage().window().maximize();
}
@Test
public void testSignUp() {
driver.get("https://www.example.com/signup");
// Fill out the form
driver.findElement(By.id("username")).sendKeys("testUser");
driver.findElement(By.id("email")).sendKeys("[email protected]");
driver.findElement(By.id("password")).sendKeys("securePassword123");
// Click the sign-up button
driver.findElement(By.id("signUpButton")).click();
// Wait for navigation and assert a successful sign-up
// This could be a redirection to a dashboard, login page, or displaying a success message.
// Adjust the condition to fit your application's sign-up flow
new WebDriverWait(driver, 10).until(ExpectedConditions.urlToBe("http://example.com/dashboard"));
// For example, check if the success message is displayed
WebElement successMessage = driver.findElement(By.id("successMessage"));
assertTrue(successMessage.isDisplayed());
}
@After
public void tearDown() {
// Close the browser
driver.quit();
}
}
This script does the following:
- Initializes a new ChromeDriver instance to control Chrome.
- Navigates to the sign-up page of your application.
- Finds the username, email, and password input fields and the sign-up button by their IDs and interacts with them by entering data and clicking the button.
- Waits for a specific condition to be met after the sign-up button is clicked, such as being redirected to a success page or a success message being displayed on the screen. This is where you should customize the script based on how your application confirms a successful sign-up.
- Asserts that the sign-up was successful, for instance, by checking the presence and visibility of a success message. This part of the script must also be tailored to fit the specifics of your application’s UI and flow.
- Finally, it closes the browser window.
In this example, we use the ChromeDriver class from the Selenium library to launch the Chrome browser and interact with the website. The @Before and @After annotations are used to set up and tear down the WebDriver instance before and after each test.
In the testSignUp method, we navigate to the sign-up page, locate the email, password, and sign-up button elements on the page using the findElement method, enter values into the email and password fields, and click the sign-up button.
Finally, we locate the success message element and use the assertTrue method to verify that the text of the success message is “Sign up successful!“.
As you can see, it is helpful, although it has several issues:
- It relies on element ids in the code for element identification; hence, any element attribute change may result in an unstable script and code maintenance.
- It doesn’t generate a unique username/email and uses the hard-coded value every time. Hence, it will fail in the subsequent test runs in a typical test environment.
- It is incomplete, as there is no email confirmation validation or to check that the newly created credentials can be used in the application for login.
What is testRigor?
testRigor is an AI agent that executes plain English commands in the test case precisely like a human would. testRigor is powered by generative AI and is an intelligent end-to-end testing tool. It allows you to perform web, mobile (native, hybrid), desktop, and API testing singlehandedly in plain English.
login create a new lead select "Mr" from "Salutation" generate unique name, then enter into "First Name" and save as "firstName" generate unique name, then enter into "Last Name" and save as "lastName" generate from template "###-###-####", then enter into "Phone" and save as "generatedPhone" enter "testRigor" into "Company" click "Save" check that page contains "created" check that page contains stored value "firstName"
As you can see, the test steps are simple. This empowers the testers or anyone on the team to use testRigor as a test automation tool without dedicating time to programming and debugging exceptions. Just register and start creating the tests in plain English. You can also import your manual test cases and, with some adjustments, use them as testRigor’s test cases. Read here about testRigor as an automation testing tool for manual testers.
Here are a few exciting capabilities of testRigor:
- Email, Phone Call, and SMS Testing: You can use simple English commands to test the email, phone calls, and SMS. These commands help validate 2FA scenarios, with OTPs and authentication codes being sent via email, phone calls, or text.
- File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
- Database Testing: Execute database queries and validate the results fetched.
- Reusable Rules (Subroutines): Create functions for the test steps you use repeatedly. You can use the Reusable Rules to create such functions and call them in test cases by simply writing their names. See the example of Reusable Rules.
- Global Variables and Data Sets: You can import data from external files or create your own global variables and data sets in testRigor for data-driven testing.
- 2FA, QR Code, and Captcha Resolution: testRigor efficiently manages the 2FA, QR Code, and Captcha resolution through simple English commands.
- A Single Tool for All Testing Needs: You can write test cases across platforms: web, mobile (hybrid, native), API, desktop apps, and browsers using the same tool in plain English statements.
- Test AI Features: This is an era of LLMs and using testRigor you can even test LLMs such as chatbots, user sentiment (positive/negative), true or false statements, etc. Read: AI Features Testing and Security Testing LLMs.
testRigor vs. ChatGPT
click "Sign up"
generate unique name, then enter into "Username" and save as "username"
generate unique email, then enter into "Email" and save as "generatedEmail"
generate unique name, then enter into "Name" and save as "generatedName"
enter "PasswordSuperSecure!" into "Password"
save value "PasswordSuperSecure!" as "password"
click "Submit"
check that page contains "Sign up successful!"
check that email to stored value "generatedEmail" was delivered
click "Confirm email"
check that page contains "Email was confirmed"
check that page contains expression "Hello, ${generatedName}"
click "Logout"
login
check that page contains stored value "generatedName"
As you can see, using testRigor, you can outline the steps and follow them the way you described. Its goal is to allow you to create high-quality test cases quickly and precisely execute test steps without writing any code or relying on implementation details.
Additionally, if attribute changes happen, such as a ‘Submit’ button being changed from <button> tag to <a> tag, testRigor, as an AI agent, would understand the button’s role within the application context. Since the elements are identified using the text visible on the screen, even if the attribute changes, the element remains identifiable with the exact text.
Let us hear from the ChatGPT itself what it thinks about testRigor vs. ChatGPT for software testing:
- Use testRigor for automated end-to-end, regression, and UI testing in an efficient, low-maintenance way.
- Use ChatGPT to assist with test strategy, script generation, and debugging issues in automation frameworks.
You can also refer to more testRigor’s top features here. Read: All-Inclusive Guide to Test Case Creation in testRigor.
Conclusion
ChatGPT is a good companion, and its intelligent use can propel the testing activities with great acceleration. However, you can not rely entirely on ChatGPT for script generation or testing. You can use it to suggest applicable test scenarios for an application and use them to create test cases further using intelligent test automation tools such as testRigor.
AI and test automation are helpers of humans, and it is up to us how we utilize the tools available to us currently. Nothing can replace human intelligence, but these tools can surely aid in performing mundane and repetitive tasks so that humans can focus on what we can do better.
Additional Resources
- What is the Role of Quality Assurance in Machine Learning?
- Machine Learning Models Testing Strategies
- Machine Learning to Predict Test Failures
- Top 10 OWASP for LLMs: How to Test?
- How to Test Prompt Injections?
- AI Assistants vs AI Agents: How to Test?
| Achieve More Than 90% Test Automation | |
| Step by Step Walkthroughs and Help | |
| 14 Day Free Trial, Cancel Anytime |
