testRigor vs Playwright vs Selenium: How Plain-English UI Tests Can Cut LLM Token Usage

Gabriel Ernandes

In 2025, the enterprise LLM market was valued at USD 8.65 billion and is expected to grow to USD 33.72 billion by 2035.

In test automation, when an LLM generates an automated UI test, the representation (code, keyword-driven, natural language) you ask it to produce matters.

A testRigor’s four-line plain-English test can express the same user journey that takes considerably more JavaScript or Java with imports, framework syntax, element-location logic, assertions, control flow, and cleanup. And that is how LLM token usage and eventually the cost is impacted by the test representation you are choosing.

Let us understand this in detail in this article.

Why Tokens Matter for LLM-Generated UI Tests

As we know, an LLM does not process text one character or one word at a time.

It divides text into tokens.

A token can be a complete word, part of a word, punctuation, a character, or another frequently occurring sequence. Spaces and capitalization can affect tokenization as well. OpenAI notes that token counts depend on the model, encoding, and language.

Its common English planning rule is about four characters or roughly three-quarters of a word per token.

That becomes particularly relevant when our output is code. The following are the artifacts that a model has to generate for a framework-based UI test:

imports
test wrapper
browser/page API calls
selectors or locators
method calls
arguments
quotes and punctuation
assertions
exception/control-flow logic
cleanup

A plain-English test can remove several implementation details and turn them into simple, human-readable commands.

• Generated Tokens Can Be Billable Output Tokens: OpenAI distinguishes input tokens from output tokens. Also, the reasoning tokens may count toward output usage even though they are not visible in the answer.
• Generated Tests Come Back Into Context Later: An AI agent might generate a test, diagnose a failure, read the existing test, modify it, perform a code review, and repair it again. It is therefore reasonable to infer that a smaller persistent test representation can reduce not only initial output but also some later input context, whenever that artifact is sent back to the model.
  The actual savings depend on the agent architecture, caching, context management, and whether the system sends the entire test on every iteration. OpenAI bills and reports input and output tokens separately, and complete API requests can include additional formatting, tools, schemas, images, and other content.
• Context Itself is Finite: Smaller artifacts leave more room in the model’s context window for requirements, DOM snapshots, logs, application code, historical failures, business rules, and other information. OpenAI recommends counting representative workloads rather than assuming visible response length alone represents total cost.

This is why “How many tokens does the test take?” is more interesting than it might initially sound.

It is not just an API-billing question. It is also a question about how much representational overhead your automation approach asks an AI system to carry around.

There is one important caveat we should consider: Tokens are not a measure of test quality.

For example, a 20-token test can be poorly specified; a 300-token test can be excellent, or vice versa. Token counts also don’t measure runtime speed, browser execution costs, coverage, defect-detection capability, or total cost of ownership.

Token count is one efficiency dimension, an increasingly relevant one when AI generates and maintains automation. But it should not be confused with the whole engineering decision.

The Test Examples for Token Usage

Let us take an example of a ‘login’ test for testRigor, Playwright, and Selenium and look into the token usage and associated cost.

Methodology and Assumptions

• Fixed Test Scenario: Log in using stored credentials, click Log in, and verify that the Dashboard appears.
• Consistent Samples: Used the same tests without modifying them after comparison.
• Measured Length: Counted characters, physical lines, and nonblank lines, including spaces, indentation, and punctuation.
• Estimated Tokens: Used the formula Estimated tokens = round(characters ÷ 4). This is a planning estimate, not an exact tokenizer count.

testRigor Login Test

enter stored value "user" into "name"
enter stored value "pass" into "pwd"
click "Log in"
check that page contains "Dashboard"

Size: 126 characters, 4 physical lines, approximately 30 normalized GPT-style tokens.

Let us take these four lines as a benchmark, noting that production tests may need additional context, targets, setup, reusable rules, or other statements depending on the application.

Playwright Login Test

javascript

await page.goto('https://example.com/login');
await page.getByLabel('Email').fill('[email protected]');
await page.getByLabel('Password').fill('myPassword123');
await page.getByRole('button', { name: 'Log in' }).click();
await expect(page.getByText('Dashboard')).toBeVisible();

Size: 276 characters, 5 lines, approximately 69 normalized tokens.

This fragment is quite readable, and its use of getByLabel() and getByRole() is well aligned with Playwright’s own recommendation to favor user-facing attributes and explicit test contracts.

But it is a fragment, not a complete standalone Playwright test file: the import and test wrapper have been omitted.

That brings us to the more comparable complete Playwright JavaScript version, under the requested assumption that there are no usable IDs, labels, roles, or other strong input hooks:

javascript

const { test, expect } = require('@playwright/test');

test('login', async ({ page }) => {
  await page.goto('https://example.com/login');

  await page.locator('input').nth(0).fill('[email protected]');
  await page.locator('input').nth(1).fill('myPassword123');
  await page.locator('button').filter({ hasText: 'Log in' }).click();

  await expect(page.getByText('Dashboard')).toBeVisible();
});

Size: 396 characters, 11 physical lines, 8 nonblank lines, approximately 99 normalized tokens.

One terminology nuance is worth mentioning: in real HTML, elements can have implicit accessibility roles even when developers do not write an explicit ‘role’ attribute. So “without roles” here really means “without usable semantic role/label hooks for this benchmark.” The test intentionally falls back to generic and positional element selection.

That fallback has an engineering consequence. locator('input').nth(0) and .nth(1) encode DOM order into the test. Add another input above the email field and the test can now address the wrong element.

Playwright’s documentation explicitly recommends user-facing attributes for resilient tests and warns that CSS/XPath selectors tied to DOM implementation can become non-resilient when the DOM changes.

Selenium Java Test

java

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

public class LoginTest {
    public static void main(String[] args) {
        WebDriver driver = new ChromeDriver();

        try {
          driver.get("https://example.com/login");

          driver.findElement(By.name("email")).sendKeys("[email protected]");
          driver.findElement(By.name("password")).sendKeys("myPassword123");
          driver.findElement(By.xpath("//button[contains(text(),'Log in')]")).click();

          WebElement dashboard =
            driver.findElement(By.xpath("//*[contains(text(),'Dashboard')]"));

          if (!dashboard.isDisplayed()) {
            throw new AssertionError("Dashboard is not visible");
          }
        } finally {
          driver.quit();
        }
    }
}

Size: 906 characters, 27 physical lines, 22 nonblank lines, approximately 227 normalized tokens.

This time the comparison includes the Java imports, class, main() method, WebDriver construction, try/finally, explicit assertion logic, and browser cleanup.

Selenium’s own “first script” documentation likewise demonstrates that Java WebDriver programs require framework objects, browser-driver creation, element lookup, and program structure around the actual interaction.

The Selenium sample uses By.name() for the inputs and XPath for the button and result. Selenium’s locator guidance says predictable unique IDs are preferable when available. When they are not, it recommends a well-written CSS selector and notes that XPath is an option. While it is flexible, it has more complicated syntax and can be harder to debug.

There is another subtle point and another reason not to oversell the token comparison.

The sample above performs an immediate lookup of the ‘Dashboard’ after clicking. A production application that renders asynchronously might require additional synchronization or waiting logic. Adding that would make the test more robust but would also make the Java artifact larger.

In other words, this particular Selenium measurement may actually understate the representation size of a production-hardened version.

The comparison is easy to see visually. However, the interesting comparison is not merely testRigor’s 30 versus Selenium’s 227. It is what those tokens are doing.

In the testRigor version, most of the artifact expresses business intent: use these credentials, click Log in, verify Dashboard.

In the framework versions, some output expresses business intent, while other output describes implementation mechanics: objects, methods, locator APIs, JavaScript callbacks, Java classes, exception handling, DOM queries, and browser lifecycle management.

That is the architectural reason the token difference can grow as AI generates more tests.

Token Comparison: testRigor vs Playwright vs Selenium

For the representative login flow analyzed here, the testRigor-style version is 126 characters across 4 lines. The complete Playwright JavaScript version is 396 characters across 11 lines, while the complete Selenium Java program is 906 characters across 27 lines.

Those character and line counts are exact for the frozen examples below.

Note: For token planning, this article uses a modern GPT-style tokenization assumption and OpenAI’s published rule of thumb of roughly four characters per token for common English text.

That produces normalized estimates of 30 tokens for the testRigor-style test, 99 for complete Playwright, and 227 for Selenium Java. Under that normalization, the testRigor artifact uses about 74% fewer visible-output tokens than complete Playwright and 89% fewer than Selenium Java. The shorter Playwright fragment, not a complete test file, lands at about 69 estimated tokens.

OpenAI explicitly cautions that actual tokenization varies by model, encoding, language, spaces, punctuation, and request structure, so these are planning estimates rather than billing-exact tiktoken measurements.

*Normalized Estimate = characters ÷ 4, rounded to the nearest whole token.

This is deliberately a consistent planning metric, not a claim that every production LLM will tokenize code at exactly four characters per token.

Why not call these “exact tokens”?

Because that would create false precision.

OpenAI explicitly states that the same text can produce different counts depending on model and encoding, and a raw plain-text count does not necessarily equal the complete token count of a structured API request. For billing-critical benchmarking, the right procedure is to freeze a particular production model and run every artifact through that model’s tokenizer or token-counting endpoint.

The strongest publishable conclusion is not “testRigor always costs 88.5% less.”

It is that, for this representative test, the plain-English representation is dramatically smaller than the equivalent generated framework code. Exact savings will vary with test complexity, language, coding style, model, tokenizer, generated helper functions, and how much defensive code the LLM adds.

• testRigor creates end-to-end tests from the end user’s perspective using plain English, with support for natural language commands and stored-value operations.
• Playwright uses JavaScript/TypeScript APIs and locators. Its documentation recommends user-facing attributes, such as roles, where possible, as DOM-oriented selectors can make tests less resilient.
• Selenium similarly requires WebDriver code and element-locator strategies. Its documentation recommends predictable IDs when available, followed by well-written CSS selectors.

Test Generation Cost

Let’s turn the token delta into dollars. But without making a tiny login test sound like a multimillion-dollar optimization.

As of October 7, 2026, OpenAI’s standard short-context output pricing includes $0.50 per million tokens for GPT-6 Luna, $10 per million for GPT-6.1 Sol, and $50 per million for GPT-6 Astra. That’s $0.0005, $0.01, and $0.05 per 1,000 output tokens, respectively.

Prices vary by model and processing mode, and long-context requests can have different rates.

For a middle-of-the-road sensitivity baseline, use $10 per million output tokens, or:

$10 / 1,000,000 = $0.01 / 1,000 tokens

At that assumed output price, generating the representative artifact 100,000 times produces an illustrative visible-output difference of $69 between testRigor-style English and complete Playwright, and $197 between testRigor-style English and Selenium Java.

The per-test number is tiny. That is the honest answer.

The more interesting economics begin when generation becomes iterative. Imagine an AI system that writes a test, revises it after execution, repairs selectors after a UI change, reviews it, and adds an assertion. Additionally, it later consumes that test as context while generating related tests.

Because API accounting includes input as well as output tokens, a smaller persistent artifact can potentially create downstream savings too. But that would require workload-specific measurement rather than simply multiplying the initial output difference.

These three scenarios correspond to current standard short-context output rates of $0.50/M, $10/M, and $50/M tokens on representative OpenAI model tiers as of October 7, 2026. They are illustrative LLM-generation costs only.

They do NOT include input prompts, reasoning tokens, retries, logs, DOM/context ingestion, tool calls, execution infrastructure, browser grids, product licenses, engineering labor, or test maintenance.

Engineering Tradeoffs of Tokens in AI-Generated Test Automation

So, now we are into a bigger discussion: tokens are useful, but what about the engineering trade-offs?

Maintainability

There is a meaningful distinction between maintaining intent and maintaining implementation.

• testRigor: Uses plain-English tests focused on the user’s intent, reducing the amount of source code that humans or LLMs need to read and modify.
• Playwright: Reduces automation boilerplate with auto-waiting, retryability, and readable locators such as getByRole() and getByLabel(), especially when accessible application hooks are available.
  Playwright’s limitation: When semantic hooks are missing, tests may rely on nth(), CSS, XPath, or other DOM-dependent logic, which can make them less resilient.
• Selenium: Offers flexible element-location strategies, but test authors or AI-generated tests must choose appropriate locators, balancing maintainability and reliability across IDs, CSS, and XPath.

So token size and maintainability can be related, but they are not synonymous.

A short test is useful only when the automation engine can reliably turn that short intent into the intended interaction.

Brittleness

Look closely at these two lines:

javascript

await page.locator('input').nth(0).fill('[email protected]');
await page.locator('input').nth(1).fill('myPassword123');

They are compact JavaScript, but they encode an assumption: email is the first input and password is the second.

Now imagine the application team inserts:

html

<input type="search">

above the login form.

The selectors can now point to different elements even though the user-visible business behavior—“enter username, enter password”, has not fundamentally changed.

• Playwright: DOM changes can cause selectors to target different elements even when the underlying business behavior remains unchanged. User-facing attributes and explicit test contracts help improve resilience.
• Selenium: Faces similar risks when locators depend on changing implementation details. Its documentation recommends stable IDs when available and CSS selectors when IDs are not available.
• testRigor’s Plain-English Automation: Reduces visible locator complexity with AI context and Vision AI, helping reduce maintenance costs with self-healing capabilities.

So, evaluate automation reliability and maintenance effort against your own test suite rather than judging performance solely by test syntax or code length.

Accessibility

Accessibility still matters.

Playwright’s user-facing locators encourage semantic markup, while Selenium can also access accessibility information.

Plain English doesn’t guarantee accessibility: Readable tests do not automatically make the underlying application accessible. testRigor can reduce reliance on manually written locator code, but accessible, semantic markup remains important. Plain-English tests let AI focus more on user intent and less on automation code.

However, with testRigor, you can perform accessibility testing in natural language with ease.

Potential token savings: For this example, Playwright’s representation is approximately 3.8× longer, and Selenium Java’s is 8.7× longer than testRigor’s, based on the normalized token estimates.

Conclusion

AI-generated testing has made test representation size an economic variable. In this representative login test, testRigor-style plain English expresses the workflow in 126 characters, compared with 396 for complete Playwright and 906 for Selenium Java. That means less visible material for an LLM to generate and potentially less material to carry through later AI-assisted maintenance cycles.

The exact dollar impact depends on scale and model choice, but the representational advantage is already visible before the first API bill arrives.

The practical takeaway is straightforward: when AI is generating tests at scale, every piece of framework syntax the model does not have to emit is output you do not have to generate, pay for, review, feed back into subsequent prompts, or maintain.

The dollar difference for one tiny test is microscopic.

At thousands or hundreds of thousands of generation and regeneration operations, the cumulative effect becomes much easier to see.

Frequently Asked Questions (FAQs)

• What is the biggest token-saving result in this comparison?
  For the exact login artifacts measured here, the testRigor-style test is 126 characters and four lines, versus 396 characters and 11 lines for complete Playwright and 906 characters and 27 lines for Selenium Java. Under the standardized four-characters-per-token estimate, that works out to approximately 30, 99, and 227 tokens respectively, about 74% fewer estimated visible-output tokens than complete Playwright and 89% fewer than Selenium Java. Those percentages describe this benchmark, not every possible test.
• Does testRigor always use fewer tokens than Playwright or Selenium?
  Not necessarily. Token counts depend on what you test, how much setup you need, how reusable functions are structured, the generated coding style, and the target tokenizer. OpenAI specifically notes that tokenization varies by model and encoding. The sensible claim is that higher-level plain-English tests can materially reduce generated representation size, as they do in this example.
• Why include the Playwright fragment as well as the complete Playwright test?
  Because it illustrates an easy benchmarking mistake: five lines of interaction code may look like the whole implementation, but a runnable Playwright test also needs its test-framework wrapper and import. Comparing the fragment with a four-line plain-English test is useful, but comparing complete artifacts is fairer. Playwright’s official documentation is built around framework tests and its locator APIs.
• Are fewer tokens actually a meaningful cost advantage?
  For a single login test, the absolute API-dollar difference is tiny. The effect becomes more relevant in AI-heavy workflows that repeatedly create, modify, inspect, repair, and regenerate large test suites. Because LLM APIs account for both input and output tokens, shorter persistent artifacts may also reduce some later context consumption when tests are fed back to the model. That second-order saving is workload-dependent and should be measured rather than assumed.
• Should teams choose a UI testing platform based on token counts alone?
  No. Token efficiency should sit alongside reliability, maintainability, debugging experience, ecosystem fit, accessibility practices, test coverage, execution performance, team skills, and total cost of ownership. Playwright deliberately encourages resilient, user-facing locators, while Selenium offers flexible WebDriver-level locator strategies. Token savings are an increasingly relevant AI-era metric, but they are only one metric.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT