Assert and Verify Methods in Selenium for Robust Testing

Hari Mahesh

Selenium is one of the widely used automated testing tools for web applications. It supports writing test scripts in different languages, including Java, C#, Python, and Ruby. The primary aspect of writing automation is validation.

Selenium itself doesn’t provide any methods for assertions. However, this is possible by using different libraries based on the programming language you use for Selenium scripting. Popular assertion libraries for various programming languages include JUnit, TestNG, AssertJ for Java, NUnit for .NET, and pytest for Python. These libraries offer a comprehensive suite of assertion methods, allowing testers to make detailed comparisons and validate the accuracy of their test cases effectively. By using methods such as Assert and Verify, in conjunction with Selenium, we can validate any functionalities and create reliable scripts.

So, let’s understand assertions and the different libraries supporting Selenium assertions more deeply.

Understanding Assertions in Testing

Assertions are one of the basic concepts in testing. They verify whether a particular condition is true or not. If the condition is true, test execution continues. If the condition is false, that particular test step will fail. Here is an informative blog to understand Assertion Testing: Key Concepts and Techniques. Asserts help to find if the application behaves as expected and if there is any deviation from the requirement, that is captured and mark the scenario as failed.

Types of Assertions

Assertions are classified into two types based on their behavior. Let us understand more about the types of assertions.

• Hard Assertions: These are the most commonly used assertions. When a hard assertion case fails, the test execution stops at that step and marks the test step failed. Hard executions are mainly used when the subsequent step has a dependency on the success of assertions.
  For example, consider we have an automation script to add an item to the cart. The next step would be to check if the cart is empty, only then we can proceed to the checkout page. To check this we can add a hard assert, which validates that the cart is not empty, assert passes and goes to the checkout page. Otherwise, if the cart is empty, the assert and the test case fails.
• Soft Assertions: Soft assertions are different from hard ones. Here, the test execution will not stop even if the assertion fails. Instead the failure will be recorded, and the test execution will continue. At the end of test, all the failures will be reported. Soft assertions are mainly used to validate the UI elements in the screen which are not critical for functionality.
  For example, we have added an assertion to validate if the cart button is in red color. But in application it has blue color. So, the assertion gets failed. But this won’t break the test case and it continues to execute.

Let’s go through the popular libraries that support assertions for Selenium.

JUnit

JUnit is a widely used open-source testing framework for the Java programming language. It is primarily used to write and run repeatable automated tests to ensure that code behaves as expected. JUnit is an essential part of the development process for Java applications, facilitating unit testing, integration testing, and regression testing.

JUnit integrated with Selenium provides a structured approach to writing and organizing test cases. It also offers features like annotations, assertions, and test suites, enabling effective test management and execution.

JUnit provides different assertion methods for validating the code behavior. Let’s look at them in detail.

assertEquals(expected, actual)

This method checks whether the expected value is equal to the actual value. It is used to verify the correctness of specific values, such as text on a web page, element attributes, or any other data.

• expected: The expected value.
• actual: The actual value obtained from the test.
• message (optional): The message to display if the assertion fails.

String expectedTitle = "Google";
String actualTitle = driver.getTitle();
assertEquals("message",expectedTitle, actualTitle);

assertNotEquals()

This method checks whether the unexpected value is not equal to the actual value. It is used to ensure that certain values do not match, which can be useful for verifying that an incorrect input does not yield an expected output.

• unexpected: The value that should not be equal to the actual value.
• actual: The actual value.
• message (optional): The message to display if the assertion fails.

assertNotEquals("Hello", "World");
assertNotEquals("message", "Hello", "World");

assertTrue()

This method checks whether a specified condition is true. It is commonly used to validate conditions such as the visibility of elements or the truthiness of certain expressions.

• condition: The condition to check.
• message (optional): The message to display if the assertion fails.

assertTrue(5 > 3);
assertTrue("message", 5 > 3);

assertFalse()

This method checks whether a specified condition is false. It is used to confirm that certain conditions are not met.

• condition: The condition to check.
• message (optional): The message to display if the assertion fails.

assertFalse(5 < 3);
assertFalse("message", 5 < 3);

assertNull()

This method checks whether a specified object is null. It is useful for verifying that certain elements or variables have not been initialized or found.

• object: The object to check.
• message (optional): The message to display if the assertion fails.

String str = null;
assertNull(str);
assertNull("message", str);

assertNotNull()

This method checks whether a specified object is not null. It confirms that certain elements or variables are initialized or present.

• object: The object to check.
• message (optional): The message to display if the assertion fails

String str = "JUnit";
assertNotNull(str);
assertNotNull("message", str);

assertArrayEquals()

This method checks whether two arrays are equal. It is used to verify that arrays contain the same elements in the same order.

• expectedArray: The expected array.
• actualArray: The actual array.
• message (optional): The message to display if the assertion fails.

int[] expected = {1, 2, 3};
int[] actual = {1, 2, 3};
assertArrayEquals(expected, actual);
assertArrayEquals("message", expected, actual);

assertSame()

This method checks whether two references point to the exact same object. This is different from assertEquals, which checks for value equality rather than reference equality.

• expected: The expected reference.
• actual: The actual reference.
• message (optional): The message to display if the assertion fails.

String str1 = "JUnit";
String str2 = str1;
assertSame(str1, str2);
assertSame("message", str1, str2);

assertNotSame()

This method verifies that the two specified objects do not refer to the same object in memory. It is used to ensure that two variables refer to different instances, even if those instances might have the same value.

• unexpected: The reference that should not be the same as the actual reference.
• actual: The actual reference.
• message (optional): The message to display if the assertion fails.

String str1 = "JUnit";
String str2 = "Test";
assertNotSame(str1, str2);
assertNotSame("message", str1, str2);

assertIterableEquals()

This method verifies that the two specified Iterable instances are equal, meaning they contain the same elements in the same order. This method is useful for verifying that collections such as lists, sets, or any other iterable structures contain the same elements in the same order.

• expectedIterable: The expected Iterable.
• actualIterable: The actual Iterable to be checked.
• message (optional): The message to display if the assertion fails.

List<String> expectedList = Arrays.asList("JUnit", "Selenium", "TestNG");
List<String> actualList = Arrays.asList("JUnit", "Selenium", "TestNG");
assertIterableEquals("Iterables do not match", expectedList, actualList); // This will pass

assertThrows()

This method verifies that the provided executable throws an exception of the specified type. If the executable does not throw the expected exception, the test will fail.

• expectedType: The type of exception expected.
• executable: The code that is expected to throw the exception.
• message (optional): The message to display if the assertion fails.

assertThrows(NullPointerException.class, () -> {
  String str = null;
  str.length();
});

assertAll()

This method allows you to group multiple assertions into a single test, ensuring that all assertions are executed even if some of them fail. This method is useful for performing several related checks in one test case and getting a comprehensive report on all assertion failures.

• heading (optional): A heading message for the group of assertions.
• executables: The assertions to be grouped.

assertAll("Test Group",
  () -> assertEquals(4, 2 + 2),
  () -> assertTrue("JUnit".startsWith("J")),
  () -> assertNotNull("JUnit")
);

fail()

This method is used to explicitly mark a test as failed. This can be useful in situations where you want to ensure that certain code paths should not be executed, or to signal a failure condition explicitly within the test.

• message (optional): The message to display if the assertion fails.

fail("This test should fail");

TestNG

TestNG is a Java testing framework. It is inspired by JUnit and NUnit but packed with several new functionalities, making it easier to use. It provides different features like assertions, annotations, parallel execution, test configuration, etc.

While JUnit and TestNG share many similarities, there are some differences in how assertions are used in each framework.

Common Assertions in JUnit and TestNG

Both JUnit and TestNG offer similar types of assertions such as:

• Equality checks (assertEquals)
• Inequality checks (assertNotEquals)
• Truth checks (assertTrue and assertFalse)
• Null checks (assertNull and assertNotNull)
• Array equality checks (assertArrayEquals in JUnit, similar in TestNG)

Differences in Assertions between JUnit and TestNG

assertArrayEquals()

JUnit has a dedicated assertArrayEquals method, whereas TestNG uses assertEquals for arrays.

import static org.junit.jupiter.api.Assertions.assertArrayEquals;
assertArrayEquals(expectedArray, actualArray);

import org.testng.Assert;
Assert.assertEquals(actualArray, expectedArray);

assertThrows()

JUnit has a dedicated assertThrows method, while TestNG requires a try-catch block and the use of Assert.fail() to handle expected exceptions.

import static org.junit.jupiter.api.Assertions.assertThrows;

assertThrows(ExpectedException.class, () -> {
  // code that throws the exception
});

import org.testng.Assert;

try {
  // code that throws the exception
  Assert.fail("Expected exception not thrown");
} catch (ExpectedException e) {
  // exception handling code
}

assertAll()

TestNG doesn’t have an assertAll() method, but it’s achieved by using soft assertions.

import static org.junit.jupiter.api.Assertions.assertAll;
assertAll("heading",
  () -> assertEquals(1, 1),
  () -> assertTrue(true)
);

import org.testng.asserts.SoftAssert;

SoftAssert softAssert = new SoftAssert();
softAssert.assertEquals(1, 1);
softAssert.assertTrue(true);
softAssert.assertAll();

Soft Assertions

import org.assertj.core.api.SoftAssertions;

SoftAssertions softly = new SoftAssertions();
softly.assertThat(1).isEqualTo(1);
softly.assertThat(true).isTrue();
softly.assertAll();

TestNG

import org.testng.asserts.SoftAssert;

SoftAssert softAssert = new SoftAssert();
softAssert.assertEquals(1, 1);
softAssert.assertTrue(true);
softAssert.assertAll();

AssertJ

AssertJ is a powerful and fluent assertion library for Java that offers a rich set of assertions to improve the readability and maintainability of your tests. It is especially useful for writing more expressive and flexible assertions compared to the built-in assertions provided by frameworks like JUnit or TestNG.

Key Features of AssertJ

• Fluent API: AssertJ uses a fluent API, which means assertions can be chained together in a readable manner. This makes test code more natural and expressive.
• Rich Set of Assertions: AssertJ provides a wide range of assertions for different types of objects, including collections, maps, optionals, dates, and more.
• Custom Assertions: Users can extend AssertJ with custom assertions to handle specific needs.
• Better Error Messages: AssertJ provides detailed and clear error messages that help diagnose test failures quickly.

Now, let’s look into the assertions provided by assertJ.

assertThat()

The assertThat method in AssertJ is the entry point for all assertions. It allows you to create assertions for various types of objects, making your test code more readable and expressive.

• actual: The actual value to be asserted.

import static org.assertj.core.api.Assertions.assertThat;

assertThat(driver.getTitle()).isEqualTo("Example Domain");

isEqualTo()

This method assert that two objects are equal. This is one of the most commonly used assertions and it checks for equality between the actual value and the expected value.

• expected: The expected value.

String title = driver.getTitle();

assertThat(title).isEqualTo("Example Domain");

isNotEqualTo()

This method assert that two objects are not equal. This assertion checks that the actual value is different from the expected value, which can be useful for ensuring that certain conditions or results do not match specific values.

• unexpected: The value that should not be equal to the actual value.

String title = driver.getTitle();

assertThat(title).isNotEqualTo("Some Other Domain");

isTrue() / isFalse()

The isTrue and isFalse methods assert the truthiness or falsity of boolean expressions. These assertions are useful for validating conditions that should be either true or false in your tests. isTrue is used to verify that a boolean expression is true and isFalse for that a boolean expression is false.

• None for isTrue and isFalse.

int a = 5;
int b = 10;

Assertions.assertThat(a < b).isTrue();
Assertions.assertThat(a > b).isFalse();

isNull() / isNotNull()

The isNull and isNotNull methods are used to assert whether an object is null or not null. These assertions are useful for validating the presence or absence of objects in your tests.

• None for isTrue and isFalse.

String actual = "AssertJ";

Assertions.assertThat(actual).isNull();
Assertions.assertThat(actual).isNotNull();

startsWith() / endsWith()

The startsWith and endsWith methods in AssertJ assert that a string starts or ends with a specific sequence of characters. These assertions are useful for validating the content and structure of strings in your tests. startsWith verifies that a string starts with a specific prefix and startsWith verifies that a string ends with a specific suffix.

• prefix for startsWith and suffix for endsWith.

String text = driver.findElement(By.cssSelector("h1")).getText();

assertThat(text).startsWith("Example").endsWith("Domain");

contains()

This method assert that a string or collection contains a specific sequence of characters or elements. This assertion is useful for validating the presence of specific content within strings or collections.

• substring: The substring that should be contained within the actual string.

String text = driver.findElement(By.cssSelector("h1")).getText();

assertThat(text).contains("Example Domain");

isEmpty() / isNotEmpty()

The isEmpty and isNotEmpty methods assert that a string, collection, map, or array is either empty or not empty. These assertions are useful for validating that objects contain no elements or characters, or that they do contain elements or characters as expected.

• None for isEmpty and isNotEmpty.

String text = driver.findElement(By.id("emptyElement")).getText();

assertThat(text).isEmpty();
assertThat("Hello").isNotEmpty();

hasSize()

This method asserts that a collection, map, or array has a specific size. It is useful for validating that the number of elements in a collection or array or the number of entries in a map matches the expected size.

• expectedSize: The expected size of the collection.

List<WebElement> elements = driver.findElements(By.tagName("p"));

assertThat(elements).hasSize(2);

contains() / doesNotContain()

This method verifies that a collection, map, or array contains or does not contain specific elements. These assertions are useful for validating the presence or absence of elements within collections, maps, or arrays.

• elements: The elements that should or should not be in the collection.

List<String> items = Arrays.asList("item1", "item2", "item3");

assertThat(items).contains("item1", "item2");
assertThat(items).doesNotContain("item4");

containsExactly()

This assertion verifies that a collection or array contains exactly the given elements in the same order. This assertion is useful for validating that the content and order of elements in a collection or array match the expected values precisely.

• elements: The elements that should be in the collection.

List<String> items = Arrays.asList("item1", "item2", "item3");
  
assertThat(items).containsExactly("item1", "item2", "item3");

Custom Assertions

import org.assertj.core.api.AbstractAssert;

public class CustomAssert extends AbstractAssert<CustomAssert, WebElement> {
  public CustomAssert(WebElement actual) {
    super(actual, CustomAssert.class);
  }

  public static CustomAssert assertThat(WebElement actual) {
    return new CustomAssert(actual);
  }

  public CustomAssert hasText(String text) {
    isNotNull();
    String actualText = actual.getText();
  
    if (!actualText.equals(text)) {
      failWithMessage("Expected element's text to be <%s> but was <%s>", text, actualText);
    }

    return this;
  }
}

WebElement element = driver.findElement(By.id("example"));
CustomAssert.assertThat(element).hasText("Expected Text");

Comparing JUnit, TestNG and AssertJ

AssertJ is considered the best for assertions due to its highly readable and expressive fluent API, which makes test code easier to understand. It provides a comprehensive set of assertions for various data types, including collections, maps, and custom objects, allowing for precise and detailed validations. Additionally, AssertJ offers superior error messages, which significantly aid in debugging by clearly indicating why an assertion failed.

Challenges using Assertions

As we know, assertions are the critical component in any automation tool, whether Selenium or any other. With assertions, we can validate a test case. However, even though Selenium has strong assertion libraries like AssertJ, a major drawback occurs with its element locating strategies. Selenium uses the conventional method of element identification, where the element properties like ID, class, CSS selector, or XPath are used.

All these properties depend on the DOM properties of the element. Therefore, if any element’s property or hierarchy in the DOM changes, Selenium won’t be able to locate these elements, making the test case fail even before the assertions are validated. For example, on a sign-in page, if the XPath for the username field changes, Selenium fails to identify it, causing the test case to fail.

These failures are false positives, which means the application did not fail, but an error in the automation script caused the test to fail. These failures make the automation less credible. So, even if we have written strong assertions to validate the application, the test case can fail due to the unstable locator issues with Selenium. That is where modern automation tools such as testRigor make a difference.

Assertions in testRigor

click "cart"
click on the button "Delete" below "Section Name"

That’s it! testRigor supports a variety of assertions, including page validations, element-specific validations, API validations, and others. To learn how to use these assertions, refer to How to perform assertions using testRigor. All these assertions are written in plain English, thereby making the test script creation easier. That means manual testers create automation scripts 4X faster than an automation engineer using any conventional automation tool.

Since there is no dependency on programming languages, anyone, including project managers, stakeholders, or even the testing team, can write automation scripts even if the application is still in development. For the testing team, if the application wireframes are available, they can start creating test cases even before the engineers write the code.

For more clarity, you can go through powerful features of testRigor and its documentation

Conclusion

Assertions have been important for automation testing in Selenium, as they are the pillars of test validations. However, in the current software development, where organizations aim for quick and frequent release, it will be challenging to trust tools such as Selenium, which has a high chance of creating false positive bugs.

In today’s competitive scenario using intelligent tools such as testRigor makes software delivery easy and faster. testRigor makes the use of assertions easy and also speeds up the test execution by a tremendous 100X as compared to Selenium. Here is a list of testRigor’s benefits that can help you decide better.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT