What is Test Coverage?

Hari Mahesh

“Quality is the best business plan” – John Lasseter.

Having a bug-free application is the dream that every developer, tester, and business chases. You may have encountered numerous occasions where unsightly bugs were discovered despite the code being written exactly as per requirement. These issues include performance-related issues or even simply failing to identify real-world use cases. Having ways to ensure every part of the application gets tested before it is made available to the customer is precisely what Test Coverage is for.

Read on for a deeper understanding of this concept.

Test Coverage at a Glance

In software testing, code coverage measures how many lines of code are tested (covered) by automated tests. For example, you have a total of 1000 lines of code, out of which 700 are covered with tests. This is a quantitative test of your code. Since tests are mapped to source code, code coverage can be considered a white-box testing technique.

Test coverage on the other hand, has no single clear definition. Martin Fowler, ThoughtWorks, and Wikipedia say it is the same as code coverage.

The most common definition of test coverage is that it is a qualitative measure of whether all the features in your application are covered with tests. This means that with test coverage, one is not to bother with the internal workings of the application, but rather focus on how well the different components of the application are functioning. In short, it is a measure of how well the testing effort covers the software’s functional requirements, user scenarios, and behaviors.

For example, if your application has a cataloging system, then some of the test scenarios would include ensuring that all items are displayed properly, are searchable, and can be added to the cart. Over here, you will not be worrying about which code method is being summoned to perform each operation, or how many lines of code are being executed for your test case. Thus we see that test coverage is a black-box approach to ensure software quality – since you are only concerned with the output rather than how or what is written in the code.

Test Coverage Example

Consider a simple program that includes three branches: checking if a number is greater than, less than, or equal to zero. To achieve 100% test coverage, you would need to create test cases that check each of these conditions. For example, testing with a positive number ensures the code that handles positive numbers is executed, a negative number tests the code for negative numbers, and zero tests the branch for equal to zero. By covering all these scenarios, you ensure that every part of the code is tested, which increases the reliability and robustness of the software.

Different Types of Test Coverage

Ensuring that your application is tested comprehensively can seem like a difficult task. To provide solid test coverage, you will need to view your application from different angles to cover maximum ground. Below are some points to consider when undertaking this activity.

Product Coverage

This is the top view. Look at your application as a collection of different features or entities. Each of them together forms the final product. This means that you need to write test cases for all these various features while considering:

• Positive test cases
• Negative test cases
• User-centric paths

Product coverage refers to the different aspects of the products that are tested. Product coverage focuses on a broader view that includes functionality, user interactions, interfaces, and system behavior. Product coverage is critical because it ensures that the software product is thoroughly tested from multiple perspectives. Read about Positive and Negative Testing: Key Scenarios, Differences, and Best Practices.

Product coverage includes functional testing to verify that the software performs the intended tasks, usability testing to ensure a good user experience, interface testing to check interactions between different system components, and performance testing to evaluate how the software behaves under various conditions. This holistic approach helps in identifying gaps that might be missed by focusing solely on code coverage, thereby improving the overall quality and reliability of the product.

Example of Product Coverage

Consider an e-commerce website with the features of login, product search, cart, checkout, and payment processing. So, the product coverage covers testing each of these modules thoroughly.

If we look at the checkout module, then it could have test cases like – Am I being redirected to the checkout page? Am I able to make a payment? Am I seeing a success message? If the payment fails, is there a failure message? On failure, does the cart content remain the same? On success, does the cart content update? Your aim should be to test each module of your product.

For example, functional testing covers all the positive and negative scenarios of the login functionality. Usability testing checks how easily the user can navigate among the pages in the application, while interface testing ensures the communication between the front-end and back-end works fine, like in the checkout process. Performance tests would simulate a high-traffic conditions to ensure the payment system can handle multiple transactions simultaneously. So, by covering all these various aspects, product coverage ensures that the user gets a seamless, reliable, and user-friendly experience.

Risk Coverage

Risk coverage focuses on identifying and testing the application’s functionalities that are most likely to fail or that may impact the application if they fail. With this approach, you can prioritize the testing efforts based on the potential risks. You can classify critical and high-risk functionalities in the application and ensure those areas are thoroughly tested. By concentrating on these areas, risk coverage helps to mitigate potential failures that could lead to severe consequences like security breaches, data loss or any other functionality issues. Risk coverage is a proactive strategy that aims to mitigate possible problems, thereby enhancing the stability and reliability of the application.

Example of Risk Coverage

Consider a banking application that includes features such as user authentication, fund transfers, bill payments and account management. When considering the risk coverage, the highest priority will be for user authentication and fund transfer features. Through proper testing, you can ensure that the authentication mechanism is robust against attacks such as SQL injection, brute force and cross-site scripting.

Also, the fund transfer features that include transaction scenarios like transaction failures, duplicate transactions and rollback mechanisms are tested rigorously. By focusing on these high-risk areas, risk coverage ensures the most vulnerable part of the application is well-protected and reliable, thereby minimizing potential risks.

Requirements Coverage

Requirements coverage ensures all the specified requirements of the application are covered by test cases. In this approach, test cases are directly linked with the requirements mentioned in the project documentation. Requirements coverage confirms every functional and non-functional requirement is tested. The primary goal is to verify that the software meets all specified needs and functionalities defined by stakeholders and documented in the requirement specifications. It helps in validating that the final product aligns with the expected outcomes and performs as intended in real-world scenarios.

Example of Requirements Coverage

Consider a project for developing a healthcare management system with specified requirements such as patient registration, appointment scheduling, electronic medical records (EMR) management, and billing. Requirements coverage would involve creating test cases for each of these functionalities. For example:

• For the patient registration requirement, tests would verify that users can input patient details, and that the system correctly stores and retrieves this information.
• For appointment scheduling, tests would ensure users can book, modify, and cancel appointments, and that the system sends appropriate notifications.
• For EMR management, tests would check that patient records can be created, updated, and accessed by authorized personnel only.
• Finally, for billing, tests would verify accurate calculation of charges, generation of invoices, and processing of payments.

Let us say that a new requirement came in asking for a chat application within the website so patients can talk to the health services provider. However, while testing, testers observed that when patients closed all browser tabs, they were logged out of their accounts but were still represented as active on chat. Such issues can lead to potential customer dissatisfaction and confusion.

Compatibility Coverage

Compatibility coverage ensures the application under test is tested across various environments including different devices, operating systems, browsers, etc. The primary goal is to ensure the application performs constantly across all environments. This approach will be helpful for applications deliver same performance in different browsers and devices. With this approach, we can identify and fix environment-specific issues that could affect user experience.

Example of Compatibility Coverage

Consider a web-based email client that needs to function correctly across multiple web browsers and operating systems. Compatibility coverage would involve creating test cases to verify that the email client works as expected on browsers such as Chrome, Firefox, Safari, and Edge, as well as on operating systems like Windows, macOS, Linux, and mobile platforms like iOS and Android.

For instance, a test might involve composing, sending, and receiving emails on each browser and OS combinations. This will ensure the interface displays correctly, emails are sent and received without issues, and all features (like attachments, formatting tools, and spam filtering) function properly.

Branch Coverage

Branch coverage, also known as decision coverage is used during unit testing. It ensures if both nodes of the if-else statements or any other conditional statements are executed by test cases. The main purpose of branch coverage is to ensure that every decision point in the code is tested. This helps to confirm all the possible outcomes of the decisions are tested and reliable. This method also helps to identify logical errors and untested conditions that cannot be covered in any other coverage that we discussed. Branch coverage provides a thorough examination of the code’s logical flow thereby improving the reliability and robustness of the application.

Example of Branch Coverage

Consider an online shopping application with a feature that applies discounts based on user membership status. The code might include an if-else statement to check whether a user is a premium member or a regular member and apply different discount rates accordingly. To achieve full branch coverage, tests would need to cover both the scenario where the user is a premium member (resulting in a higher discount) and the scenario where the user is a regular member (resulting in a lower discount or no discount).

Additionally, there should be tests for situations where the user might not be logged in at all, ensuring the branch that handles unauthenticated users is also executed.

Boundary Values Coverage

Boundary values coverage focuses on testing the edge cases of input ranges. This method is based on the observation that errors often occur at the boundaries of input values rather than within the middle of the range. Boundary values coverage involves creating test cases for the extreme ends of input ranges, such as the minimum, maximum, just inside, just outside, and typical values within the boundary limits. This type of testing is crucial because it helps identify defects that may not be apparent with typical input values, ensuring that the software handles edge cases correctly and robustly.

Example of Boundary Values Coverage

Consider a function in an online form that accepts user ages ranging from 18 to 60. To achieve boundary values coverage, test cases should be created for the boundary values of this range. This would include testing with input values such as 17 (just outside the lower boundary), 18 (at the lower boundary), 19 (just inside the lower boundary), 59 (just inside the upper boundary), 60 (at the upper boundary), and 61 (just outside the upper boundary). By testing these specific values, boundary values coverage ensures that the application correctly handles input at the edges of the accepted range, such as allowing users aged 18 and 60 to submit the form while preventing those who are 17 or 61.

How to Measure Test Coverage?

While measuring test coverage keep below points in mind:

• How much of the functionality and requirements are being tested.
• Test coverage is measured by mapping tests to requirements or use cases.
• It works on functional and higher level, dealing with the application’s behavior and requirements.
• The metrics used are requirement coverage, use case coverage, scenario coverage, feature coverage, branch coverage, etc.
• Tools that you can use are test management tools, RTMs, and other code-based tools.
• Ensure all specified requirements and user scenarios are tested.
• Need to make sure that all aspects of a feature, such as user login, are tested.

Test Coverage Formula by Code

The basic formula for calculating test coverage is:

Test Coverage = (Number of Lines of Code Tested / Total Number of Lines of Code) x 100%

With this formula, you can calculate the test coverage percentage, clearly showing how much of your codebase is covered by tests.

Consider a simple application with a total of 1,000 lines of code. After running your test suite, the coverage tool reports that 850 lines of code were executed during the tests.

Test Coverage = (850/1000) x 100% = 85%

This means that tests cover 85% of the codebase.

Measure Test Coverage with Tools

Here are the key steps and methods used to measure test coverage through a tool:

• Define Coverage Criteria: Determine which type of test coverage you want to measure, such as statement coverage, branch coverage, function coverage, or path coverage.
• Instrument the Code: Use a test coverage tool to instrument the code. This process involves adding additional code or metadata that records which parts of the code are executed during testing. Examples: JaCoCo (Java), Istanbul (JavaScript), Coverage.py (Python), Clover (Java), etc.
• Run Test Suite: Execute your test suite with the instrumented code. This will generate coverage data that shows which parts of the code were executed.
• Generate Coverage Report: Use the test coverage tool to generate a coverage report. This report typically includes metrics and visualizations showing which parts of the code were covered by the tests and which parts were not.
• Analyze Results: Review the coverage report to identify untested parts of the code. This analysis helps in understanding the gaps in testing and planning further tests to improve coverage.

Benefits of Test Coverage

Test coverage provides several key benefits to software development teams:

• Improved Code Quality: The higher test coverage means more lines of code are tested and ensured, thereby leading to improved code quality. Test coverage is directly proportional to code quality. When we add more test cases, more lines of code are tested which makes it easy to identify and fix the defects. An application with higher test coverage means the application is more robust and reliable.
• Early Detection of Bugs: Test coverage helps in identifying the bugs at an early stage of development. While testing a portion of code for test coverage, the defects are identified during the development phase. This helps to save time and effort for debugging the code later in the development cycle.
• Enhanced Confidence in Software: The confidence and reliability of the application will be more, if the test coverage score is high. The score is particularly important when showcasing to stakeholders, as their reliability on the application also grows. As the test covers a large portion of the application, there are very less chances that the applications an fail in various scenarios, thereby providing a positive experience to the stakeholders.
• Better Maintenance and Refactoring: With high test coverage, the developers can ensure the application code is tested and reliable. So for any debugging or refactoring of code, the developers can assure that the existing functionality will not be broken.
• Risk Mitigation: Test coverage helps to mitigate the risks associated with the application. With test coverage, the team can identify which part of the code is untested. Adding test cases to those functionalities reduces the risk of undetected bugs. This proactive way of risk management ensures the application is more reliable. Read: Minimizing Risks: The Impact of Late Bug Detection.

Automation and AI for Test Coverage

Automation is crucial for achieving and maintaining high test coverage in software development. Here are several reasons why automation plays a vital role in enhancing test coverage:

• Efficiency and Speed: Manual testing can be time-consuming and prone to human error, especially as the size and complexity of the codebase grow. Automation enables the rapid execution of test cases, significantly reducing the time required to run comprehensive test suites. Automated tests can be run repeatedly with consistent results, allowing for frequent and thorough testing without the time constraints of manual efforts. Read: Transitioning from Manual to Automated Testing using testRigor.

With tools like testRigor, we can create test scripts in plain English, eliminating the need for coding expertise. This empowers a broader range of team members without any technical background to contribute to automated test creation.

• Consistency and Reliability: Manual testing is prone to human error and inconsistency. Automation ensures that tests are executed in a consistent manner every time. Tools like testRigor are designed for end-to-end testing. testRigor uses a unique approach for element locators, unlike traditional tools that rely on specific element identifiers. You can simply describe elements by the text you see on the screen, leveraging the power of AI to find them automatically. This means your tests adapt to changes in the application’s UI, eliminating the need to update fragile selectors constantly and making the test reliable and consistent.
• Scalability: With manual testing, it’s challenging to execute the test cases in parallel. So, executing in multiple devices or browsers with manual testing seems impossible. With automation testing, we can execute test cases in parallel 24×7, easily, in less time duration. Read: How to execute test cases in parallel in testRigor?
• Early Detection of Defects: With continuous testing, we can run automation suites for every developer code merge, thereby ensuring there is no failures due to the new code. Also we can identify the failures early as soon as the code is merged. testRigor has seamless integrations with many CI/CD tools, infrastructure providers, test management tools. This early feedback loop helps developers identify and address issues before they escalate, reducing the cost and impact of defects.

Here are testRigor’s top capabilities, to gain more clarity.

Read this case study to know how Enumerate went from ZERO to 100% test automation within six months of testRigor’s implementation.

Frequently Asked Questions (FAQs)

1. What is the ideal percentage of test coverage?
2. What are popular test design and test coverage techniques
3. Why is test traceability important?

Conclusion

Test coverage is a vital metric in software testing that ensures a comprehensive assessment of an application’s codebase. High test coverage indicates that the software has been thoroughly tested, which helps in identifying and mitigating potential defects. Test automation with intelligent tools like testRigor is essential for achieving robust test coverage. It ensures that applications are thoroughly tested across various scenarios, enhancing reliability, and quality, and ultimately delivering a better user experience.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT