What is OCR-based Testing? How testRigor Does It

Shilpa Prabhudesai

Traditional automation testing mainly focuses on automating predefined manual tests. Over time, automation testing has evolved so much that automation tools are now using advanced AI techniques to enhance, optimize, and streamline the automation testing process. With the integration of AI, the tools use machine learning (ML) algorithms, natural language processing (NLP), computer vision with optical character recognition (OCR), and other AI techniques to make the testing process smarter, faster, and more efficient.

With revolutionary advancements in business operations, OCR has emerged as one of the most transformative advancements in the technology sector.

• Organizations can use OCR-enabled devices to automate the process of data entry and transcription.
• testRigor handles texts in images using OCR and identifies buttons/texts through ML-based image classification.

This article explores the OCR-based testing concept, why it matters, and how testRigor uses Vision AI and OCR to make it simple and powerful.

What is OCR-based Testing?

OCR stands for Optical Character Recognition, and is a technology that converts images of text (like screenshots, scanned documents, or photos) into machine-readable text.

OCR-based testing is a technique to validate and interact with content that appears as pixels on the screen (for example, text within the image), not just text available in the DOM or API.

• Text inside images, logos, or icons.
• Text rendered in canvas elements or images generated by a third-party library.
• Scanned PDFs, invoices, or reports are shown inside a viewer.
• Desktop apps or legacy systems where there is no DOM access.
• Remote apps are streamed through virtual environments or Citrix-like setups.

1. Taking a screenshot of the screen or a region.
2. Running OCR to extract the text.
3. Using that recognized text to assert, find, or click.

OCR testing lets your automation see the UI closer to how humans does and involves verifying the accuracy of the OCR conversion of visual representations from text to a machine-readable format.

This testing technique ensures that the OCR system can accurately extract text from various document types, recognizing different fonts, languages, and layouts.

As an example, consider a very common situation where an organization has historical data, which is a mix of images and text in physical format.

Now, if you can convert this valuable information into a digital format, it would be highly beneficial as it would be accessible to everyone.

Before OCR, this practice would require a huge effort to enter data and also add alternate text for images. With OCR, the process is streamlined, faster, and more accurate.

• Functional Testing: Validates if the OCR system extracts text from different document types and formats.
• Accuracy Testing: Measures the system’s ability to accurately recognize individual characters and maintain the integrity of the extracted text.
• Performance Testing: Evaluates the system’s speed and efficiency in handling various document complexities and image qualities.

Why OCR-based Testing Matters

OCR-based testing is essential due to the following reasons:

Beyond DOM Locators

Most UI test automation tools are dependent on DOM locators such as XPath, CSS selectors, and IDs. Automation technique using DOM locators is fragile, especially when the frontend framework updates, an element is refactored, or the underlying DOM implementation changes.

With OCR-based testing, these drawbacks no longer exist. This is because OCR testing works on what is visibly rendered to the user. Hence, even when internal implementation changes occur, as long as the user-facing UI does not change, it remains stable.

Handling Non-standard UIs

Modern applications no longer contain only text but contain non-standard components such as canvas-based dashboards and charts, embedded PDF viewers, images containing text for branding pr security reasons, or desktop/native apps with limited locators.

OCR-based testing technique allows testers to interact with these components using visible labels instead of brittle technical hooks.

Accessibility and User-centric Validation

OCR-based testing is user-centric and accessibility-aware. It ensures that important labels, hints, or instructions are actually visible, and dynamic content showing up as text can be verified to be the same that a human can understand.

How does the OCR Test Work?

OCR testing uses OCR software to extract and convert data from physical documents and images into a digital format.

The OCR software analyzes the document image and identifies individual characters based on their shape and features. The extracted data is then output in many formats, such as text, CSV, XML, and JSON.

Here are the steps involved in OCR testing:

1. Scan the Document

To begin with, the physical document is scanned into a digital image using a flatbed scanner, a document scanner, or a mobile device with a camera.

2. Load the Image into the OCR Software

Once the document is scanned, it is loaded into the OCR software. The software then analyzes the image and identifies the individual characters.

3. Extract the Data

The OCR software then extracts the data from the document and converts it into a digital format. The user can specify the format of the output data.

4. Verify the Data

Once the data has been extracted, it is essential to verify it to ensure accuracy. This is either done manually or by using a data validation tool.

OCR testing is utilized to automate various data entry tasks.

How to Create OCR Tests?

OCR tests can be created either using a manual or an automated approach. Both these approaches are described here:

Manual OCR Test

1. Be ready with the image of the text you want to extract.
2. Use an OCR tool to get the text from the image.
3. Compare the extracted text to the expected text.

Here is an example of a manual OCR test:

1. Take a screenshot of the screen.
2. Use an OCR tool to pull the text from the screenshot.
3. Compare the extracted text to the expected text (“Hello, world!”).
4. If the extracted text matches the predicted text => PASS. Otherwise, the test fails.

OCR Test Automation

1. Choose an automation framework that supports OCR, such as testRigor.
2. Record a test that extracts the text from the image and compares it to the expected text.
3. Run the test to verify that the extracted text is correct.

Tips for Creating Better OCR Tests

• Use high-quality images for the best results.
• Ensure that the text on the image is clear and easy to read.
• The OCR tool you use should be compatible with the type of text you are trying to extract.
• Test the OCR tool on different images before using it in production.
• Utilize a range of test cases to encompass various scenarios.

The choice of manual or automated OCR tests depends on the specific needs and requirements.

Typical Use Cases for OCR-based Testing

The following table summarizes some of the concrete scenarios for OCR testing:

Challenges of OCR-based Testing

• Accuracy and Noise: OCR accuracy is affected by low-quality images, small fonts, varied handwriting, or busy backgrounds.
• Performance: OCR processing is resource-intensive and slower than direct code-based text extraction.
• False Positives / Negatives: Issues such as small rendering differences, anti-aliasing, and dynamic content (timestamps, ads, etc.) can confuse comparison logic.
• Complex Integrations: To get a basic OCR-based check working, many tools need to be integrated with OCR engines like Tesseract or cloud OCR APIs. Custom scripting is also required. This makes it a complex system.
• Maintenance: OCR-based testing requires regular refinement and updates to OCR models and test scripts as the application’s UI or content changes.

How testRigor Approaches OCR-based Testing

Most test automation tools tend to struggle with image-based testing. However, with testRigor‘s smart AI engine, OCR-based testing is no longer an issue. This no-code test automation platform enables you to write tests in plain English, eliminating the need for code or UI element locators.

• Vision AI: testRigor uses computer vision models that interpret visual UIs like a human.
• OCR: It uses OCR to read text rendered on the screen.
• ML-based Image Classification: testRigor uses this to identify buttons and other elements in images, not just DOM nodes.

With this, testRigor can perform OCR-based, visual, and functional testing in a single unified manner. Here are some of the ways testRigor makes image testing easy.

1. Vision AI + OCR: Understanding the UI like a Human

testRigor’s Vision AI understands visual elements such as images, icons, buttons, and text messages on the screen, similar to a human tester.

click “Submit”

When such a statement is encountered, testRigor uses its visual understanding and OCR to locate the button labeled “Submit”, even if the underlying HTML or button appearance changes, or the app runs on different browsers or devices.

testRigor uses OCR to handle texts in images and ML-based image classification to identify buttons/texts. Thus, it doesn’t care how the button is implemented. It just looks for a visible “Submit” element on the screen.

2. OCR by Command: Fine-grained Control

testRigor allows you to use OCR at the command level. You can enable OCR for individual steps without turning it on for the whole suite.

click "Book Store Application"
click "silence" using OCR

testRigor searches the page for the images with the word “silence” and clicks on it. The corresponding screenshot is shown below:

• “This specific command requires OCR since the text is inside an image.”
• “The rest of the steps can use regular element recognition.”

• Precision and speed for normal steps
• Deep visual understanding only where needed.

3. Visual Regression + OCR in One Workflow

testRigor also supports visual regression testing through screenshot comparison across runs to highlight visual differences.

• Compare screenshots so that the focus is on meaningful visual changes.
• Allows you to define how much variation is acceptable (tolerances) in percentages.
• Visual AI is used for OCR, screen navigation, self-healing, and recognition of UI elements.

• Read changes in the text via OCR.
• Notice a break in the UI and usability in case a button disappeared or moved.
• Avoids false positives from trivial rendering differences.

For a more detailed explanation of vision AI in testRigor, go through the article, Vision AI and how testRigor uses it.

Handling Advanced Scenarios: 2FA, QR Codes, CAPTCHA, and More

testRigor’s advanced features let you manage 2FA logins, QR codes, and CAPTCHA resolution with simple English commands. You can easily automate these workflows in plain English.

Benefits of OCR-based Testing with testRigor

• Pure Plain-English Tests: In testRigor, tests are written in plain English. There is no need to stitch together scripts and OCR APIs. Even non-technical stakeholders can read and write tests as follows:

  check page contains "Cart" using OCR
  click button "Submit" using OCR

  This radically improves the OCR-based automation.
• Reduced Test Maintenance: Reducing test maintenance by avoiding fragile locators is the core design of testRigor. It achieves this using OCR and Vision AI:
  • Slight changes in UI and internal code refactors have less impact on your tests.
  • As long as user-visible behavior is stable, your tests typically continue to pass.
  • There is no chance of visual/OCR-based assertions breaking every time someone adds a new wrapper div or changes the HTML structure.
• Unified Visual + Functional Testing: Instead of using multiple tools like one for functional testing, one for visual regression, and one for OCR, testRigor allows you to do everything from the same test suite and steps using Vision AI.
  With this, the architecture is simplified, and CI/CD pipelines can easily be managed.
• Works Across Platforms: testRigor supports cross-platform, cross-browser testing, including web (desktop and mobile), native mobile apps, desktop applications, and mainframe applications.
  OCR-based testing in testRigor is beneficial for:
  • Mobile apps that have texts rendered via native components or images.
  • Desktop/mainframe environments that have no DOM-like access.

Best practices for OCR-based Testing in testRigor

Consider the following best practices to get the most out of OCR-based tests in testRigor:

1. Use OCR Only Where Needed: OCR is more resource-intensive than standard element recognition. So use OCR wisely, especially testRigor’s per-command OCR feature.
   • Use standard commands when elements are reliably accessible.
   • Add using OCR only for UI elements that truly require OCR, such as PDFs, images, canvas, etc.
2. Choose Unique and Stable Text: When writing OCR-based assertions:
   • Choose unique phrases such as "Total Amount Due" rather than just "Total".
   • Text such as timestamps or dynamic banners that change frequently should be avoided to improve the clarity and stability of tests.
3. Combine OCR with Context: When checking the context:
   • Validate a phrase that appears in a specific section.
   • Combine OCR checks with navigation and/or previous state. For example, when checking totals, verify the right customer name.
4. Baseline your Visual Tests Wisely: For visual regression:
   • Set a clear baseline version of UI.
   • Set reasonable tolerances using testRigor’s visual testing settings, and ignore regions that are expected to change.
5. Use testRigor’s Ecosystem: Use Live Mode, AI test generation, and reusable rules for testRigor. Use these extra capabilities to:
   • Generate initial tests, including OCR testing steps.
   • Reuse testing flows across multiple OCR-based scenarios.
   • Maintain consistency across complex testing suites.

Conclusion

OCR-based testing bridges the gap between how traditional automation tools interact with them and how humans see them. What appears on the screen can be easily validated using OCR, instead of being limited to DOM locators or API responses.

testRigor takes OCR-based testing to the next level by combining Vision AI, OCR, and ML-based image classification to interpret UIs the way humans do. It offers visual regression, self-healing, OCR, and cross-platform support in one place. One more plus point is that tests can be written in plain English, including OCR steps.

If your application contains PDFs, image-heavy interfaces, canvas-based charts, or complex desktop/mobile UIs, OCR-based testing is the only reliable way to automate the experiences.

With testRigor, you just describe what you see in natural language, add "using OCR" wherever required, and let the tool handle the complexity for you.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT