Why Playwright sucks for end-to-end tests in 2024?

Anushree Chatterjee

End-to-end testing is an integral part of determining whether the application under test is ready to face the real world. With this type of testing, you verify whether different parts of the application are working together to produce the expected results from an end-user’s perspective.

One of the tools that can assist you with this form of testing is Playwright.

What is Playwright?

Playwright is a node.js-based framework maintained by Microsoft that was created to meet end-to-end testing needs. It supports modern rendering engines, including Chromium, WebKit, and Firefox. You can test on Windows, Linux, and macOS, locally or on CI, headless or headed, with native mobile emulation of Google Chrome for Android and Mobile Safari.

Writing tests using Playwright

With Playwright, you can write actions in the form of steps and then assert the state against expectations. Both actions and assertions require coding skills to create test scripts.

Actions include navigations and interactions. Playwright includes test assertions in the form of ‘expect’ function. To make an assertion, you need to call expect(value) and choose a matcher that reflects the expectation. There are many generic matchers: toEqual, toContain, and toBeTruthy, that can be used to assert any conditions.

Now, you need to identify the locators of the element or page to process actions or assertions, which itself seems like a task. For example, ‘expect(locator).toContainText()’ checks whether the element contains the text mentioned. To identify the locators, you can use below:

• page.getByRole() to locate by accessibility attributes
• page.getByText() to locate by text
• page.getByLabel() to locate by a label’s text
• page.getByPlaceholder() to locate an input by placeholder
• page.getByAltText() to locate an element by its text alternative
• page.getByTitle() to locate by title attribute
• page.getByTestId() to locate based on data-testid attribute
• page.locator() to create CSS or XPath locators

await expect(page.getByText('Welcome, John')).toBeVisible();

Do you think it is too wordy and confusing for a beginner automation engineer or manual tester? Finding the right locator using the old ways (id, data-testid, CSS/Xpath) can be cumbersome for modern dynamic websites.

Here is an example that shows the code complexity of the Amazon.com website. We can see 10+ levels of nested div elements. Just imagine using CSS/XPath locators to identify an element. Also, if any of the nested elements change, test maintenance will be a nightmare.

Limitations of Playwright

const { test, expect } = require('@playwright/test');

test('Amazon search for headphones on PC page', async ({ page }) => {
  await page.goto('https://www.amazon.com/');
  await page.click('.nav-logo-link');

  // Check that page contains "Amazon Devices" below "Get to Know Us"
  const getToKnowUsText = await page.$eval('a[href="/amazon-devices/b/?ie=UTF8&node=2102313011"]', (el) => el.textContent);
  expect(getToKnowUsText).toContain('Amazon Devices');
  await page.click('a[href="/amazon-devices/b/?ie=UTF8&node=2102313011"]');

  // Check that page contains "Echo & Alexa"
  const echoAlexaText = await page.$eval('a[href="/Echo-Devices/b/?ie=UTF8&node=9818047011"]', (el) => el.textContent);
  expect(echoAlexaText).toContain('Echo & Alexa');
  await page.click('a[href="/Echo-Devices/b/?ie=UTF8&node=9818047011"]');

  // Check that page contains "Alexa App" below "Alexa Resources"
  const alexaResourcesText = await page.$eval('.s-navigation-indent-1 a[href="/gp/help/customer/display.html/?nodeId=GJDWQ6WJ6VYJX9XN"]', (el) => el.textContent);
  expect(alexaResourcesText).toContain('Alexa Resources');
  const alexaAppText = await page.$eval('.s-navigation-indent-2 a[href="/gp/help/customer/display.html/?nodeId=GDKGDTXNTH7A5BQA"]', (el) => el.textContent);
  expect(alexaAppText).toContain('Alexa App');
  await page.click('.s-navigation-indent-2 a[href="/gp/help/customer/display.html/?nodeId=GDKGDTXNTH7A5BQA"]');

  // Check that page contains "Search Amazon"
  const searchAmazonText = await page.$eval('label[for="GLUXSearchInput"]', (el) => el.textContent);
  expect(searchAmazonText).toContain('Search Amazon');
  await page.fill('#GLUXSearchInput', 'headphones');
  await page.press('#GLUXSearchInput', 'Enter');
  await page.waitForNavigation();
  await page.click('a[href*="headphones"]');
  await page.evaluate(() => window.scrollBy(0, window.innerHeight / 2));

  // Grab value from "Sony ZX Series Wired On-Ear Headphones, Black MDR-ZX110" and save it as "headphone1"
  const headphone1Text = await page.$eval('a[href*="MDR-ZX110"] span', (el) => el.textContent);
  const headphone1 = headphone1Text.trim();

  // Grab value from "Beats Solo3 Wireless On-Ear Headphones"
  const headphone2 = await page.innerText('text=Beats Solo3 Wireless On-Ear Headphones - Apple W1 Headphone Chip, Class 1 Bluetooth, 40 Hours of Listening Time, Built-in Microphone - Black (Latest Model)');
  await page.evaluate(() => window.scrollBy(0, -window.innerHeight/2));
  await page.fill('input[type="search"]', 'pc');
  await Promise.all([
  page.waitForNavigation(),
  page.click('text=pc')
]);

// Check that page does not contain "headphone1"
const hasHeadphone1 = await page.isVisible(`text=${headphone1}`);
expect(hasHeadphone1).toBeFalsy();

// Check that page does not contain "headphone2"
const hasHeadphone2 = await page.isVisible(`text=${headphone2}`);
expect(hasHeadphone2).toBeFalsy();

Maintenance Nightmare

As we can see in the above Playwright code snippet, there are so many lines of code. For someone looking to automate a trivial use case like ‘checkout cart items’, this is quite intense. Now, imagine the plight of a manual tester or a business stakeholder who wants to collaborate in the testing process. Such frameworks do not allow those without coding expertise to partake in the test automation process. You have to rely on specialists for the job.

As with any coding-based test automation tool/framework (Read: Why Selenium sucks for E2E tests), the fragility of test scripts always exists. Any change in the element attribute could cause the locator to become invalid. Subsequently, the test script will break and provide inaccurate test results.

/html/body/div[4]/div[2]/div/div[1]/div/div[2]/div/div[1]/div/div[1]/div[2]/div/div[2]/a

//*[@id="zg_left_col1"]/div[1]/div[2]/div/div[2]/a

Maintaining such locators and test scripts will rob you of your valuable time, effort, and cost. It will definitely affect the product’s quality and delivery. Who wants innumerable bugs in production anyways?

You face below issues due to Playwright’s maintenance requirements:

• Time spent on maintenance: When the test suite size in Playwright is huge, many teams spend up to 50% of their time on test maintenance rather than on more productive tasks. You could have used this time to create more robust new tests. However, this precious time is being spent on maintenance, which could have been clearly avoided by using better-performing automation testing tools.
• Third-party app updates: During cross-system testing, where you don’t control the HTML of the system being tested, no amount of BDD or Shift-left practices will reduce the maintenance burden. You will be required to keep up with changes in third-party apps such as ERP or CRM systems (like Salesforce), where you have no control over updates. With Playwright, this gets really difficult to handle. Learn more about how to perform Salesforce Testing.
• Flakiness impacts CI/CD results: Flaky tests in Playwright can have a major impact on a CI pipeline due to their inconsistent behavior. Your same code commit will provide unpredictable test results on different test runs. Therefore, to achieve a successful deployment, the pipeline often needs to be rerun multiple times after a failure. This results in delivery delays, as deployments seem to be affected by random, unreliable test outcomes. This could prove hazardous in an Agile environment, where speed and quality are both required.

Limited Test Case Creation Speed

One of its primary limitations is its reliance on programming languages, which means that testers without prior experience in coding might face a steep learning curve. Your manual testers can not use their testing expertise with Playwright’s programming knowledge requirement, and test case creation is slow.

Debugging Errors and Exceptions

Playwright recommends code editor VS Code Extension to debug your tests right in VS Code, see error messages, set breakpoints, and step through your tests. There are other ways also available including Playwright Inspector (built-in), Playwright Trace Viewer, API logs, etc.

However, the better approach is to use a test automation tool where you do not get into such frequent errors and exceptions. You need to learn async/await patterns, handle timeouts, and understand advanced Playwright features like browser contexts or multiple-page handling to get going with Playwright test automation, which is definitely difficult.

A very common exception in Playwright is TimeoutException, which may cause a waste of your time and effort. Read: How to Handle TimeoutException in Playwright?

Integration Issues

Playwright’s ecosystem, while growing, is less mature than some older testing frameworks (like Selenium). Integrations with third-party tools like test management systems or reporting platforms may require more custom work, dependency setup, or external plugins.

Complex CI/CD Setup

Playwright requires specific dependencies to run on certain CI/CD systems (e.g., for headless browser support). Configuring those dependencies, particularly on certain cloud providers or CI environments, can sometimes be challenging and time-consuming.

Limited Capabilities

• Playwright’s support for mobile testing is limited. It only provides emulated mobile devices rather than real devices, which may not accurately represent actual user experience.
• Another limitation of Playwright is its lack of native visual testing or reporting support, as it requires third-party integrations to conduct these tasks effectively. This could potentially increase the complexity of the testing process and necessitate additional maintenance efforts.
• While extensive, Playwright’s documentation may not always be up-to-date or cover all use cases, making it difficult for developers to find solutions for specific issues. Since Playwright is a relatively new tool, it may not have the same level of community support or resources as more established testing frameworks.

How Should End-to-End Testing Work?

What is the purpose of end-to-end tests? They are meant to validate that your app’s functionality works as intended from an end-user’s perspective, following the actual user flow in real-world scenarios.

This means you should reference elements in the way the user experiences them, not how a developer views the application. For users, the only things that matter are identifying the right input fields or finding the correct button to click. Therefore, it’s essential to have a simple, reliable way to interact with forms, tables, files, etc., that mimics how a user would interact with a browser or device.

Example: Form Interaction

Let us take an example of a website form to help you understand.

As an end user, you will navigate to each field and fill in the data such as ‘City’; you are not concerned about the id/CSS/XPath, etc. of the ‘City’ form element. Here is the HTML of the form:

In this HTML, the ‘City’ element’s ‘id’ and ‘name’ are clearly mentioned. But this won’t be the case with every application.

Also, if there is a change in the UI framework (as in React), you might lose all these IDs. Another case is that you migrate to a rigid back-end framework or upgrade to a newer version (ASP.NET), and you might be forced to change these element names as well. Ironically, this is precisely when you need your end-to-end tests to work, i.e., right after migrating to a new framework. You need E2E tests to rely on and ensure everything functions as expected.

Therefore, your end-to-end testing tool shouldn’t depend on the internal code of your application. Instead, it should mimic the perspective and actions of the end user. Take the “City” input field in the screenshot, for example. Its internal structure could change, but there will likely always be a placeholder or label that the user identifies as “City.” However, not every site has a well-structured HTML setup, like Amazon, with its ‘label for’ attributes. You can’t always rely on the underlying code structure.

What’s really needed is a way to identify and describe elements from the end user’s point of view based on what they see: such as a label or placeholder..

enter "San Francisco" into "City"

Isn’t it really simple and intuitive? Let us see another example.

Example: Table Handling

Here we are looking at a Salesforce table:

Now, from a user’s perspective, what matters is that the row containing Company as “ProperUniqueCompany” displays a specific ‘Lead Status’. Similarly, the down arrow icon in the last column of that row should be clickable. They are not worried about the implementation of these fields they just want to access and take some actions.

validate that table at row containing "ProperUniqueCompany" and column "Lead Status" contains "Open - Not Contacted"

click on the table at the row containing "ProperUniqueCompany" and the last column

This is how humans think and take action, and it should be tested in the same manner. This command will work no matter how the table is rendered – whether it’s HTML <table> (like in Salesforce example) or using <div> rendering (like in Amazon example).

testRigor for End-to-End Testing

The English commands that you have seen above are actual testRigor commands. We have seen that there are many reasons why Playwright is not suitable for E2E testing today. The tool that you use for E2E testing should be as simple as possible and help minimize the number of bugs in production.

It is possible with the available advanced AI technologies and intelligent AI agents. One such tool that you can rely on is testRigor. It is a codeless automation tool powered by generative AI and ML technologies that help generate/create easy and automatic test scripts and data in plain English. With testRigor, test script maintenance requires almost zero effort. Read here how you can decrease 99.5% of your test maintenance time.

You can build 15X more automation tests than Selenium/Playwright. Your manual testers can build 4X more tests than Selenium QA automation engineers. Isn’t it fascinating?

Let’s take a look at its capabilities and see how easy it is to use.

You have the following advantages with testRigor:

• No programming required: It is a codeless tool, meaning tests are written using plain English commands. This opens the ability for the entire team to author tests (including manual testers) and significantly increases the speed of creating tests.
  Its record-and-playback tool can further speed up test creation. Since the recorded tests will be in the same plain English format, it becomes viable for anyone to edit and maintain when desired.
• Stable locators: There is no hassle mentioning XPaths or any technical parameters for locating elements on the screen. All you need to do is mention relative positions or how you see an element on screen. You can click on a button below the title by simply writing click “button” below “Title”.
• Cross-browser and cross-platform support: The tool allows you to perform cross-browser and cross-platform tests; parallel execution feature allows you to get the test results in minutes.
• No installation needed: Being a cloud-based tool, you just need to register and get started with test automation in no time.
• Supported integrations: This can be easily integrated with most CI/CD tools, test management tools, and issue-tracking tools.
• Advanced reporting and logging: It offers good reporting capabilities and captures screenshots at every step. It provides clear error messages in English. You can also view the video recordings or error logs of the execution for clarity.
• A single tool for all testing needs: You can write test cases across platforms: web, mobile (hybrid, native), API, desktop apps, and browsers using the same tool in plain English statements.
• Test AI features: This is an era of LLMs, and using testRigor, you can even test LLMs such as chatbots, user sentiment (positive/negative), true or false statements, etc. Read: AI Features Testing.

Writing tests with testRigor

Writing tests with testRigor is as simple as you are describing the steps to access the application to a friend in English. The testRigor documentation has all the commands in detail, and you can even define your own preferred commands (subroutines) in plain English.

click "amazon"
check that page contains "Amazon Devices" below "Get to Know Us"
click "Amazon Devices"
check that page contains "Echo & Alexa"
click "Echo & Alexa"
check that page contains "Alexa App" below "Alexa Resources"
click "Alexa App"
check that page contains "Search Amazon"
enter "headphones" in "Search Amazon"
click "headphones"
scroll down by 1/2 of the screen
grab value from "Sony ZX Series Wired On-Ear Headphones, Black MDR-ZX110" and save it as "headphone_1"
grab value from "Beats Solo3 Wireless On-Ear Headphones - Apple W1 Headphone Chip, Class 1 Bluetooth, 40 Hours of Listening Time, Built-in Microphone - Black (Latest Model)" and save it as "headphone_2"
scroll up
enter "pc" into "headphones"
click on "pc"
check that page does not contain "headphone_1"
check that page does not contain "headphone_2"

Read How to do End-to-end Testing with testRigor for more details.

Curious to know what more testRigor can do for you? Take a look at their feature list today.

Conclusion

“The most precious resource we all have is time” – Steve Jobs.

No one in the software industry wants to waste time in solving errors, exceptions, and tiring test maintenance. If tools are available, why not utilize them for everyone’s betterment? Playwright’s reliance on coding skills, complexity in locator management, and high maintenance demands create significant barriers for teams, especially those with limited technical expertise.

Additionally, its limitations in handling mobile testing, integration challenges, and flakiness in continuous integration/delivery pipelines make it difficult to scale efficiently. For modern applications that demand speed, simplicity, and reliability, intelligent codeless tools like testRigor are a more user-friendly and efficient test automation alternative.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT