Mainframe Testing

Pragya Yadav

Today, we are reaping the benefits of Artificial Intelligence, Machine Learning, Generative AI, LLMs (Large Language Models), and advanced models. And in this huge and advanced Universe of software, do mainframe systems still hold any place? We might have an impression that mainframes are outdated and legacy. Who uses them today, anyway? Surprisingly, the truth is far away from this. The reason is simple: Industries like banking, insurance, healthcare, and government still rely on mainframe systems to run mission-critical operations. While newer systems steal the spotlight, mainframes handle billions of transactions every day — reliably and securely.

According to an IBM study, 67 out of the Fortune 100 companies use mainframes as their core systems. Also, 45 of the world’s top 50 banks, 4 of the 5 leading airlines, and 7 of the top 10 global retailers use mainframe systems in their operations.

Now, if the mainframes are being used for such critical tasks, are we testing them enough? Can advanced AI help in testing these mainframe systems effectively, efficiently, and easily? The answer is yes. Let us find out how.

What is a Mainframe?

Mainframe systems can be thought of as the “heavy lifters” of enterprise IT. They can flawlessly handle everything from ATM transactions to airline reservations, payroll processing, and government agencies such as national tax systems. Here are some defining features of mainframes.

1. High Processing Power: Mainframes can process millions of transactions per second, which is essential for businesses with high-volume operations.
2. Extreme Reliability (High Availability): They’re built to run 24/7 with minimal downtime. Some systems have been running continuously for decades.
3. Massive Input/Output (I/O) Capability: Mainframes manage large volumes of simultaneous data inputs and outputs, such as thousands of users accessing data at once.
4. Scalability: You can scale up a mainframe without major rework—just add more processing capacity.
5. Security: Mainframes have advanced security and access control systems, which is why they’re trusted to handle sensitive information (like financial data or personal records).

Components of a Mainframe System

Today, mainframe systems have networks, crypto, storage, and compression cards. They have SAP (System Assist Processors) to have quick data transfer between OS, processors, and I/O devices. Here are a few of the core components of a mainframe system:

How to do Mainframe Testing?

Mainframe testing involves validating the functionality, performance, and security of mainframe applications. This includes testing batch jobs, online transaction processing (OLTP), data integrity, and integration with external systems.

Types of mainframe testing include:

1. Unit Testing: Test individual programs or modules, typically COBOL or PL/I routines.
2. System Testing: Check that the entire mainframe system behaves as expected.
3. Integration Testing: Test interfaces between the mainframe and external systems (like web apps or APIs).
4. Regression Testing: Verify that new changes don’t break existing functionalities.
5. Performance Testing: Measure transaction processing speed, job throughput, and system response times.
6. Security Testing: Test proper authentication, authorization, and encryption.

Challenges in Mainframe Testing

• Legacy Code and Lack of Documentation: Many mainframe applications are decades old. They are written in outdated languages (like COBOL, Assembler, or PL/I) and often lack proper documentation or knowledgeable developers.
• Resource Constraints: Few professionals today are skilled in mainframe technologies. This talent gap makes it difficult to maintain and test mainframe applications effectively.
• Complex Environments: Mainframes often interact with multiple subsystems, such as IMS, DB2, CICS, and VSAM, which need to be tested as part of the complete system.
• Data-Intensive Testing: Mainframe applications process massive volumes of data. Ensuring the quality, consistency, and privacy of test data is a significant challenge.
• Tool Limitations: Traditional testing tools are often not compatible with mainframe environments. Testing is still done manually in many enterprises, leading to errors and inefficiencies.

Mainframe Testing Automation

Using test automation with mainframes may seem like a far-fetched idea due to the legacy nature of mainframes. But it’s very achievable and increasingly essential for organizations aiming to modernize and adopt DevOps or CI/CD practices. Here are the steps to execute test automation on mainframe apps.

Step 1: Understand the Mainframe Testing Scope

Mainframe systems are used for batch processing, online transactions, and data-intensive operations. Common areas to test include:

• Batch Jobs (e.g., JCL scripts)
• CICS Transactions
• IMS or DB2 Database Interactions
• VSAM or Flat File Validations
• Integration with APIs, web apps, or third-party systems

Step 2: Choose the Right Test Automation Tools

Not all testing tools work natively with mainframes. Look for tools that support terminal emulation, 3270 interfaces, database testing, or API testing, depending on your architecture. If your project/organization has applications which run on web, mobile, desktop, APIs, mainframes, then it is a good idea to use a tool which can singlehandedly execute test cases on all these platforms. One example is the AI agent for test automation: testRigor. We will learn more about its advanced capabilities in the next section.

Step 3: Automate Batch Job Testing

Many mainframe operations run through JCL-based batch jobs. Automating their testing involves:

• Submitting jobs automatically via script or API.
• Monitoring job execution using job logs or exit codes.
• Validating job output (log files, datasets, DB2 entries).

Step 4: Automate CICS/IMS Screen Testing

Mainframe applications often involve 3270 green screen interfaces, which can be automated using terminal emulator scripting or tools that support 3270 protocols.

Ways to automate:

• Use testRigor to simulate user actions on the screen.
• Build automated test scripts that can:
  • Navigate through panels
  • Enter data and retrieve results
  • Validate screen content against expected output

Here is an example from testRigor. This intelligent tool can identify fields, text, data, etc., on the UI and navigate through screens using just English commands. We will discuss this in detail in the next section.

Step 5: Automate Database Testing (DB2, IMS)

If your app relies on mainframe databases:

• Use SQL scripts to query DB2 or IMS databases before and after tests.
• Validate:
  • Record insertion
  • Data updates
  • Referential integrity
• Automate this via tools like testRigor, IBM Data Studio, or custom scripts (Python, JDBC, REXX).

Step 6: Integrate with DevOps and CI/CD

Mainframe systems don’t have to be left out of your DevOps pipeline.

Integration Points:

• Jenkins or GitLab CI/CD can trigger mainframe test jobs.
• Use z/OSMF REST APIs, Zowe CLI, or Endevor to interact with mainframe code during pipeline runs.
• Include:
  • Static code analysis (e.g., with SonarQube for COBOL)
  • Automated unit and integration testing
  • Regression test execution
  • Report publishing

Step 7: Manage Test Data Effectively

Test automation is only effective if the test data is controlled and reusable. Use tools like:

• IBM Optim – For data masking and subsetting
• Compuware File-AID – For test data creation and validation
• Mock or stub external dependencies where needed

Best practices:

• Create data sets tailored for different test scenarios
• Mask production data to ensure privacy
• Reuse consistent test data for regression testing

Step 8: Run Regression and Smoke Tests Automatically

Once your test cases are automated, organize them into suites:

• Smoke Tests – Quickly validate core functionalities after a deployment.
• Regression Tests – Ensure recent changes haven’t broken anything.

Trigger these suites:

• After code check-in
• On a nightly schedule
• Before a release to production

Step 9: Analyze Results and Feed Back

Use dashboards or test management systems to capture:

• Pass/fail rates
• Defects discovered
• Test execution trends
• Coverage levels

Tools like testRigor, JIRA, qTest, TestRail, or custom dashboards help tie test results into your agile tracking process.

Mainframe Testing in Plain English Using testRigor

Let us take the example of a Mainframe System (banking app). We will create end-to-end test cases in testRigor using plain English test steps.

Step 1: Log in to your testRigor app with your credentials.

Step 2: Set up the test suite for the website testing by providing the information below:

• Test Suite Name: Provide a relevant and self-explanatory name.
• Type of testing: App shows following options: Desktop Web Testing, Mobile Web Testing, Native and Hybrid Mobile, Mainframe. Select Mainframe.
• Address: Provide the mainframe app’s address, that you want to test.
• Username and Password: You can provide the app’s user login credentials here and need not write them separately in the test steps then. Here, we are leaving it blank.
  Note: These credentials will work only if the mainframe uses standard input fields for the login form, such as “username”, “email”, “password” etc. In our example below, the app uses “userId”, so we should provide the value or use the testRigor variables username and password if we are providing this information while creating the test suite.
• OS: We are choosing Ubuntu to run the test cases.

Step 3: Click Create Test Suite.

Step 4: To create a new test case, click Add Test Case.

Step 5: Provide the test case Description and start adding the test steps.

Test Case: Search for an Account

Step 1: We will add test steps on the test case editor screen one by one.

testRigor automatically connects to the Address you provided during the Test Suite creation. There is no need to use any separate function for it. Here is the mainframe app’s login screen, which we intend to test.

Let us enter the Userid and Password on the screen and then enter ‘Enter’.

enter "IBMUSER" into "UserId"
enter "*****" into"Password"
type "Enter"

Step 2: Once we type ‘Enter’, the next screen is displayed as below:

Now, we want to assert and check that the expected screen is displayed and user is able to login successfully. We will check that the screen has “SIMPLATFORM MAIN MENU” and “BANKTEST”. Just use plain English commands to check the presence of the text.

check that page contains "SIMPLATFORM MAIN MENU"
check that page contains "BANKTEST"

testRigor highlights the text as a result of the assertions on the screen captured during the test execution.

Step 3: The user is logged into the application. The next step is to enter F1, as shown in the above screen. Then, we will use the clear command, which in testRigor is mapped to Ctrl + C. As you might know, Clear is a standard command for mainframe applications.

type "F1"
type "Ctrl + C"

Step 4: Now, we want to enter “bank” into the first input on the screen.

enter "bank" into first input
type "Enter"

Step 6: Assert that the screen is displayed correctly and contains the ‘Browse’ option to search the accounts.

check that page contains "BROWSE   Browse Accounts  PF1"

testRigor is able to find the above text on mainframe screen, and hence moves to the next step of search.

Step 7: First enter F1, and then we want to enter the account number in the ‘Account Number‘ field and then type ‘enter‘.

type "F1"
enter "123456789" into "Account Number"
type "Enter"

Step 8: The mainframe application is able to find the account details, as shown below, and displays the message ‘Account Found‘.

Now, we will perform the checks to ensure that the Account Balance is correct and that the message “Account Found” is displayed on the screen.

check that "Balance" has value "56.72"
check that page contains "Account found"

testRigor highlights the assertion to make it clear that the text was found correctly, as shown below:

Complete Test Case

Here is how the complete test case will look in the testRigor app. The test steps are simple in plain English, enabling everyone in your team to write and execute them.

Click Add and Run.

Execution Results

Once the test is executed, you can view the execution details, such as execution status, time spent in execution, screenshots, error messages, logs, etc. In case of any failure, there are logs and error text that are available easily in a few clicks.

You can also download the complete execution with steps and screenshots in PDF or Word format through the View execution option.

testRigor’s Capabilities

• Quick Test Creation: Create tests using testRigor’s generative AI feature; just provide the test case title/description, and testRigor’s generative AI engine will automatically generate most of the test steps. Tweak a bit, and the plain English (or any other natural language) automated test cases will be ready to run.
• Eliminate Test Maintenance: There is no maintenance nightmare because there is no reliance on implementation details. This lack of XPath and CSS dependency ensures ultra-stable tests that are easy to maintain.
• Import Existing Manual Test Cases: You can import and refine your manual test cases with reusable steps (subroutines). Import your existing manual test cases from test management tools such as TestRail, PractiTest, Zephyr, etc. Read: Import test cases from TestRail for execution.
• Codeless Testing: testRigor eliminates the need for programming language knowledge by converting English test scripts into actual code internally using advanced Natural Language Processing (NLP). You can also use our test recorder to record UI actions and create the test cases easily in plain English, meaning learning to code is unnecessary.
• Everyone in Team Tests: Product managers can review test cases; testers and your business analysts, sales, and marketing teams can write and execute test cases using testRigor.
• Shift Left Testing: Leverage the power and advantages of shift left testing with testRigor. Create test cases early, even before engineers start working on code using Specification Driven Development (SDD).
• Single Tool for Every Testing Need: testRigor enables you to test web, mobile (hybrid, native), API, and desktop apps with minimum effort and maintenance.

Top 5 Mainframe Testing Tools

testRigor

AI-powered, intelligent no-code test automation that spans mainframe, web, mobile (hybrid/native), desktop, database, APIs, and modern systems. testRigor is changing the mainframe testing scenario by enabling testers to create automated test cases using just plain English through generative AI. Build true end-to-end test flows — all without writing a single line of code.

Key Features:

• Test legacy terminal apps (3270/5250) in plain English
• Natural language test creation (no scripting)
• End-to-end test coverage: mainframe + web + mobile (hybrid/native) + desktop + APIs
• Low maintenance, with AI-powered self-healing
• Seamless CI/CD integration

Cons:

• testRigor’s AI is learning in terms of deep mainframe protocol coverage

Best For: Organizations aiming to modernize QA by involving non-technical users and automating legacy systems in an Agile or DevOps pipeline.

IBM Application Performance Analyzer for zOS

Get deep insight into application performance on the mainframe. This IBM tool is designed to help developers and testers analyze and optimize the performance of applications running on z/OS. It provides detailed execution profiles, helping teams identify performance bottlenecks and resource usage patterns.

Key Features:

• Analyzes COBOL, PL/I, C/C++, and assembler applications
• Monitors CPU, I/O, and memory usage
• Useful for both batch and online applications
• Non-intrusive: no source code modification required

Cons:

• Focused on performance, not functional testing
• Complex to set up and interpret without IBM system experience
• Primarily for experienced developers and performance engineers

Best For: Performance engineers and developers looking to fine-tune resource-heavy mainframe applications.

Rocket Software – BlueZone Terminal Emulator Testing

Lightweight and secure automation for terminal emulators. Rocket’s BlueZone is a secure terminal emulator that supports automation via scripting and macros. While not a full testing framework, it’s often used to validate user journeys within green screen environments.

Key Features:

• Automates 3270, 5250, and VT terminal sessions
• Scripting support (VBScript, JavaScript)
• Secure access via SSH, TLS, SSL
• Easy to deploy and low memory footprint

Cons:

• Not a dedicated testing framework (no test case management, CI/CD)
• Requires scripting knowledge
• Limited integration with broader test automation tools

Best For: QA teams who need a automation solution for terminal interactions, especially when paired with other tools.

IBM REXX

A scripting language for flexible testing and automation. REXX is a powerful scripting language commonly used on IBM mainframes for test automation, data manipulation, and task scheduling. Though it requires coding expertise.

Key Features:

• Great for custom mainframe test logic
• Embedded in TSO, ISPF, and other IBM environments
• Useful for writing utilities, test data generators, and test scripts

Cons:

• Requires deep coding knowledge
• No UI, test management, or built-in reporting
• Not ideal for collaborative or large-scale test automation

Best For: Advanced users who need custom, code-heavy test scripts and prefer scripting over GUI-based automation.

BMC AMI DevX

Integrated DevOps tooling for continuous mainframe delivery. BMC AMI DevX is a full suite of tools designed to bring Agile and DevOps to the mainframe.

Key Features:

• Automates unit and functional testing for COBOL, PL/I, and more
• Enables shift-left testing and continuous delivery
• Integrates with Git, Jenkins, SonarQube, and other DevOps tools
• Includes Topaz Total Test, Xpediter (debugger), and File-AID (data management)

Cons:

• High complexity and cost – best suited for large enterprises
• Requires training and adoption across multiple roles
• Focused mainly on COBOL/PL/I; not ideal for hybrid stack teams

Best For: Enterprises seeking end-to-end DevOps automation for mainframe apps.

Mainframe Testing Tools: Summary Table

Best Practices for Mainframe Testing

Conclusion

It is easy to assume that mainframe systems are relics of the past. However, in reality, these systems are handling billions of secure, real-time transactions with unmatched reliability and performance. And their critical role demands an equally advanced and efficient testing approach. Traditional or manual testing methods are no longer sustainable in the face of rapid release cycles, complex integrations, and growing expectations for flawless user experiences.

With modern tools like testRigor, organizations can bridge the gap between legacy systems and modern development practices. Scriptless, AI-powered automation enables teams regardless of their technical background, to write robust test cases in plain English. This drastically reduces testing time, eliminates maintenance overhead, and empowers continuous testing as part of a larger DevOps or CI/CD strategy for mainframes.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT