Magento Testing

Magento (both a free, open-source version and a commerce version called Adobe Commerce) is a popular choice for building e-commerce websites. It is a powerful platform with its own testing frameworks to help you test your website. However, there are certain limitations when it comes to testing that you ought to know.

• Unit Testing
• Integration Testing
• Magento Functional Testing Framework (MFTF)
• End-to-End Testing

Magento Unit Testing

As the testing pyramid suggests, unit tests should be at the very beginning, encompassing the base of all tests. Magento supports unit testing of all algorithms and methods. It utilizes the PHPUnit framework. It supports PHP as well as JavaScript unit testing. Since Magento has some predefined features, it also comes with some predefined unit test cases. These unit tests are written by following the Test Data Driven (TDD) approach.

Ways to Execute Magento PHP Unit Tests

Magento unit tests can be executed in the following ways.

• Command Line Interface (CLI)
  It is possible to run all or a select few tests using the CLI. Their specific directory paths need to be mentioned for the respective cases to be executed. For further reading, refer to this link.
• PhpStorm IDE
  It is also possible to run unit tests through IDEs like PhpStorm. Just like the CLI execution, the IDE execution also supports running either all or a select group of unit tests. However, the IDE needs to be configured to run the correct set of cases. Further details on the configuration and commands can be found here.

Ways to Execute Magento JavaScript Unit Tests

Magento provides sophisticated frontend capabilities for frontend unit testing. The Commerce version of Magento supports testing JavaScript code using Jasmine and a custom version of Grunt CLI task runner. Further details on the configuration and commands can be found here.

Magento Integration Testing

The next layer after unit tests is typically integration. As you know, the goal of these tests is to make sure various components interact with each other as expected. Some examples include integration with a third-party system or an API. Keep in mind that integration tests in Magento require a runtime environment.

1. HTTP tests to test APIs
2. Third party integration tests
3. Browser tests (although you might want to prefer them to be a part of end-to-end tests)

All of the details on setting up the integration testing with Magento can be found here.

<?php

namespace IntegerNet\Categorydisplay\Test\Integration\Controller;

class NewProductTest extends \Magento\TestFramework\TestCase\AbstractController
{
    public static function loadFixture()
    {
        include __DIR__ . '/../_files/categories.php';
        include __DIR__ . '/../_files/products.php';
    }

    /**
     * @magentoDbIsolation enabled
     * @magentoAppIsolation enabled
     * @magentoDataFixture loadFixture
     */
    public function testDisplaysCorrectProduct()
    {
        $this->dispatch('catalog/category/view/id/xyz');

        $this->assertContains('OurProduct 1', $this->getResponse()->getBody());
        $this->assertNotContains('Our Product 2', $this->getResponse()->getBody());
    }
}

Magento End-to-end Testing

End-to-end tests sit on top of the testing pyramid. They are typically the slowest (compared to other types of tests) but also the most representative. Since E2E tests represent the entire user flow, having a tight coverage is the only way to ensure a great user experience.

1. Magento Functional Testing Framework (MFTF)
2. Selenium, or any other framework based on it
3. testRigor

We won't get into Selenium-based testing here since it might not be the most optimal way to use for end-to-end testing due to its limitations.

Magento Functional Testing Framework (MFTF)

MFTF aims to simplify the testing of complex use cases that are likely to be encountered in e-commerce websites. It tends to target the pages that the end users and admins of the system will interact with. As your website grows, it becomes essential to have automation testing strategies in place to validate the integrity of the ever-growing e-commerce website constantly. This framework is quite stable as it is based on technologies like Selenium, Allure, and Codeception. Since it is meant for Magento websites, it is highly customizable and scalable. It is open source which makes it a popular choice for Magento applications testing. As Magento provides some common functionalities that are used in e-commerce websites, it also has a directory where functional test cases for these features are stored.

It is necessary to set up the environment for functional testing by downloading the required versions of PHP, Java, Composer, and web driver. A detailed overview of this setup can be found here.

All test cases are written in XML. The XML files are built to generate PHP class files which are then executed. It is possible to add assertions, data sets, and club repetitive actions into action groups using MFTF. Details on how to convert manual test cases into MFTF cases can be found here.

Reports of these functional test cases are generated using Allure.

testRigor End-to-end Testing

testRigor specializes in end-to-end testing. Among the key features are ease of test creation, unrivaled test stability, and low maintenance - a rare combination for E2E tests.

1. Support email testing, as well as testing for attachments as part of the tests
2. Support visual testing if needed (ex: you can compare an entire screen to a previous version)
3. Test two-factor authentication logins on Gmail, SMS, etc

End-to-end tests in testRigor represent how a real user would interact with the application. What this means is that you don't need to use CSS Selectors or dig into the DOM structure to define an element. Commands are in simple English terms, such as click "Add to Cart", or check that checkbox "Remember me" is checked.

click "Sign in"
enter "Joe" into "Username"
enter "JoePass" into "Password"
click "Verify me"
check that sms to "+12345678902" is delivered and matches regex "Code\:\d\d\d\d" and save it as "sms"
extract value by regex "(?<=Code\:)[0-9]{4}" from "sms" and save it as "confirmationCode"
enter saved value "confirmationCode" into "code"
click "Continue to Login"
check that page contains text "Welcome, Joe!"

enter "Jack" into table "Client Info" at row "11" and column "First Names"

The best part is this test will work regardless of being rendered as div-based or as an HTML table.

As you can see, it's very straightforward to create end-to-end tests covering the entire user flows while keeping abstraction from the details of implementation.

How to do End-to-end Testing with testRigor

Let us take the example of an e-commerce website that sells plants and other gardening needs. We will create end-to-end test cases in testRigor using plain English test steps.

Step 1: Log in to your testRigor app with your credentials.

Step 2: Set up the test suite for the website testing by providing the information below:

• Test Suite Name: Provide a relevant and self-explanatory name.
• Type of testing: Select from the following options: Desktop Web Testing, Mobile Web Testing, Native and Hybrid Mobile, based on your test requirements.
• URL to run test on: Provide the application URL that you want to test.
• Testing credentials for your web/mobile app to test functionality which requires user to login: You can provide the app’s user login credentials here and need not write them separately in the test steps then. The login functionality will be taken care of automatically using the keyword login.
• OS and Browser: Choose the OS Browser combination on which you want to run the test cases.
• Number of test cases to generate using AI: If you wish, you can choose to generate test cases based on the App Description text, which works on generative AI.

Step 3: Click Create Test Suite.

On the next screen, you can let AI generate the test case based on the App Description you provided during the Test Suite creation. However, for now, select do not generate any test, since we will write the test steps ourselves.

Step 4: To create a new custom test case yourself, click Add Custom Test Case.

Step 5: Provide the test case Description and start adding the test steps.

For the application under test, i.e., e-commerce website, we will perform below test steps:

• Search for a product
• Add it to the cart
• Verify that the product is present in the cart

Test Case: Search and Add to Cart

Step 1: We will add test steps on the test case editor screen one by one.

testRigor automatically navigates to the website URL you provided during the Test Suite creation. There is no need to use any separate function for it. Here is the website homepage, which we intend to test.

First, we want to search for a product in the search box. Unlike traditional testing tools, you can identify the UI element using the text you see on the screen. You need not use any CSS/XPath identifiers.

click "What are you looking for?"

Step 2: Once the cursor is in the search box, we will type the product name (lily), and press enter to start the search.

type "lily"
enter enter

Search lists all products with the “lily” keyword on the webpage.

Step 3: The lily plant we are searching for needs the screen to be scrolled; for that testRigor provides a command. Scroll down until the product is present on the screen:

scroll down until page contains "Zephyranthes Lily, Rain Lily (Red)"

When the product is found on the screen, testRigor stops scrolling.

Step 4: Click on the product name to view the details:

click "Zephyranthes Lily, Rain Lily (Red)"

After the click, the product details are displayed on the screen as below, with the default Quantity as 1.

Step 5: Lets say, we want to change the Quantity to 3, so here we use the testRigor command to select from a list.

select "3" from "Quantity"

click "Add to cart"

The product is successfully added to the cart, and the “Added to your cart:” message is displayed on webpage.

Step 6: To assert that the message is successfully displayed, use a simple assertion command as below:

check that page contains "Added to your cart:"

Step 7: After this check, we will view the contents of the cart by clicking View cart as below:

click "View cart"

Step 8: Now we will again check that the product is present in the cart, under heading “Your cart” using the below assertion. With testRigor, it is really easy to specify the location of an element on the screen.

check that page contains "Zephyranthes Lily, Rain Lily (Red)" under "Your cart"

Complete Test Case

Here is how the complete test case will look in the testRigor app. The test steps are simple in plain English, enabling everyone in your team to write and execute them.

Click Add and Run.

Execution Results

Once the test is executed, you can view the execution details, such as execution status, time spent in execution, screenshots, error messages, logs, video recordings of the test execution, etc. In case of any failure, there are logs and error text that are available easily in a few clicks.

You can also download the complete execution with steps and screenshots in PDF or Word format through the View Execution option.

testRigor’s Capabilities

Apart from the simplistic test case design and execution, there are some advanced features that help you test your application using simple English commands.

• Reusable Rules (Subroutines): You can easily create functions for the test steps that you use repeatedly. You can use the Reusable Rules to create such functions and call them in test cases by simply writing their names. See the example of Reusable Rules.
• Global Variables and Data Sets: You can import data from external files or create your own global variables and data sets in testRigor to use them in data-driven testing.
• 2FA, QR Code, and Captcha Resolution: testRigor easily manages the 2FA, QR Code, and Captcha resolution through its simple English commands.
• Email, Phone Call, and SMS Testing: Use simple English commands to test the email, phone calls, and SMS. These commands are useful for validating 2FA scenarios, with OTPs and authentication codes being sent to email, phone calls, or via phone text.
• File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
• Database Testing: Execute database queries and validate the results fetched.

testRigor enables you to test web, mobile (hybrid, native), API, and desktop apps with minimum effort and maintenance.

Additional Resources

• Access testRigor documentation to know about more useful capabilities
• Top testRigor’s features
• How to perform end-to-end testing