Cucumber is Dead, and AI is Replacing It

Pragya Yadav

In February 2023, SmartBear laid off the last remaining person and co-founder of the Cucumber open-source project. The former employee had been involved in determining Cucumber’s future direction within SmartBear. While he had anticipated an end-game, the sudden termination came as a surprise.

Cucumber isn’t the only open-source framework that saw challenges. TestProject faced end-of-life because Tricentis chose to concentrate on its SaaS-based, AI-powered test automation tool, Tricentis Testim. New TestProject accounts were disabled starting November 16, 2022, and all existing accounts were deactivated by the end of March 2023.

Vision Behind Cucumber

Cucumber is primarily a collaboration tool rather than just a testing tool. By using Gherkin, a plain-text language that is readable by non-technical team members. Cucumber allows all team members to contribute to the definition of application behavior. Its focus is on defining requirements collaboratively before implementation. Read: What is a BDD Framework?

Distinct Cucumber Features

• Bridgesthe Gaps Between Stakeholders: Cucumber was designed to improve communication between developers, testers, business analysts, and other stakeholders through the common language Gherkin. Read more, Mastering Gherkin for Software Testing: A Step-by-Step Guide.
• Supports Behavior-Driven Development (BDD): Cucumber promotes the practice of BDD, where the focus is on specifying the desired behavior of the application rather than just its implementation. This ensures that development is driven by business goals and user requirements.
• It is a Living Documentation: One of the core ideas behind Cucumber is to create living documentation that evolves with the codebase. The feature files written in Gherkin serve as up-to-date documentation of the system’s behavior.
• Test Automation: Cucumber aims to integrate test automation into the development process seamlessly. By writing tests in a language that describes the application’s behavior, tests can serve both as documentation and as automated checks.
• Builds Collaboration: By involving non-technical stakeholders in the writing of feature files, Cucumber encourages collaboration. All perspectives are considered during the development process, which helps identify and address potential issues early on.
• Agile and Continuous Integration: Cucumber fits well within Agile methodologies and continuous integration practices. Its approach supports iterative development and frequent feedback, which are key principles of Agile. Automated tests written in Cucumber can be integrated into CI pipelines, providing continuous validation of the application’s behavior. Here is a case study: Mastering Agile with BDD.

Its misuse occurs when Cucumber is treated solely as a testing tool. When we miss its collaborative and documentation benefits.

What is the Problem with Cucumber?

Yes, Cucumber is not dead literally, but is not favored by most of us either. These events indicate a shift in the test automation industry. Now, companies are increasingly focusing on consolidating their product offerings and prioritizing tools that offer more advanced features, such as SaaS-based, AI-powered solutions. This trend suggests that the market for open-source test automation tools may be facing challenges in terms of long-term viability. Read: What is Cucumber Testing?

Challenges with Cucumber

As Cucumber’s founder Aslak Hellesøy said: “Let’s face it — most people who use Cucumber don’t use it for BDD. They use it for testing, writing their tests afterwards. If that’s your workflow, Cucumber isn’t significantly better than many other testing tools.”

Cucumber is an excellent framework for BDD; however, the problem starts when it is also used for test automation. Here are the challenges associated with it when used for test automation:

• Maintenance Overhead: As the number of features and scenarios grows, maintaining the test suite can become challenging. Keeping the feature files, step definitions, and test data in sync requires constant effort.
• Learning Curve: For teams new to BDD and Cucumber, there can be a steep learning curve. Writing good Gherkin syntax and effectively using Cucumber’s features requires practice and experience.
• Performance Issues: Large test suites with many scenarios can lead to long execution times. When you run numerous scenarios in a single test suite, it may slow down the feedback loop. This, in turn, impacts the efficiency of the development process.
• Readability vs. Complexity: While Gherkin aims to make test cases readable by non-technical stakeholders, achieving this balance can be tricky. Complex scenarios can lead to verbose/less readable feature files and can defeat the purpose of using Gherkin altogether.
• Test Data Management: Managing test data for different scenarios can be cumbersome. Test data should remain relevant and up-to-date to maintain the reliability of tests. Read: Optimizing Software Testing with Effective Test Data Management Tools.
• Parallel Execution: If you are running tests in parallel, it can be challenging due to shared resources and dependencies. Tests that do not interfere with each other require careful planning and setup. Read: Parallel Testing: A Quick Guide to Speed in Testing.
• Integration with Tools: Cucumber’s integration with other tools (e.g., CI/CD pipelines, test reporting tools) can sometimes be non-trivial. It may require additional configuration and scripting. It is more evident because of the differences in syntax, setup, reports, etc.
• Overhead for Simple Tests: For simple test cases, the overhead of writing Gherkin scenarios and step definitions may not be justified. In such cases, traditional testing frameworks may be more efficient.
• Maintenance of Step Definitions: Over time, step definitions can become bloated, and duplicate steps might appear. You might face maintenance difficulties due to this. Organizing and refactoring step definitions is necessary to keep the test suite clean. Read How to Write Maintainable Test Scripts.
• Dependency on Development Workflow: Successful implementation of BDD and Cucumber requires an alignment with the development workflow. If not adopted fully by the team, the benefits of using Cucumber may not be visible.

Intelligent Alternative to Cucumber

To be honest, these recent changes in the testing arena are not necessarily negative for companies. Today, you have intelligent generative AI-powered tools such as testRigor where you can use your specifications as automated test cases directly. Yes, you will need to make a few changes in the specifications according to testRigor’s language, but everything will be in plain English. Know about Test Automation with AI.

Your Business Analysts, Product Owners, Developers, QA team, and all other stakeholders can use the same specifications and then execute them as automated test cases. This supports team collaboration as Cucumber does, and you do not need separate teams to write the programming code for implementation.

When Cucumber is used in conjunction with test automation, the extra layer of coding becomes an overhead for the QA team.

enter "Kindle" into "Search"
enter enter
click "Kindle Scribe"
click "Add To Cart"
check that page contains "Item Added To Cart"

Using testRigor, manual QA professionals can create end-to-end automated tests, regardless of their length or complexity, while spending minimal time on maintenance. This eliminates the common disconnect between manual and automation QA, accelerates the test creation process, and scales progressively until the desired test automation percentage is achieved.

Use this ROI calculator to see why AI-based no-code test automation outperforms other methods.

If you’re considering a transition away from Cucumber, it’s never been easier. Since testRigor is a specification-driven tool, you can effortlessly import your existing Gherkin feature files and transform them into automated tests, removing the need to write Selenium tests or any code at all.

testRigor Features

You can use testRigor and get quick and visible benefits with its generative AI-powered capabilities. Here are some notable features:

• Global Variables and Data Sets: You can import data from external files or create your own global variables and data sets in testRigor to use them in data-driven testing.
• 2FA, QR Code, and Captcha Resolution: testRigor easily manages the 2FA, QR Code, and Captcha resolution through its simple English commands.
• Email, Phone Call, and SMS Testing: Use simple English commands to test the email, phone calls, and SMS. These commands help validate 2FA scenarios, with OTPs and authentication codes being sent via email, phone calls, or phone text.
• File Upload/ Download Testing: Execute the test steps involving file download or file upload without the requirement of any third-party software. You can also validate the contents of the files using testRigor’s simple English commands.
• Database Testing: Execute database queries and validate the results fetched.
• Cross-browser and Cross-platform Support: testRigor supports seamless cross-browser and cross-platform testing on real devices, ensuring your application functions correctly on various browsers and operating systems.
• AI Feature Testing: testRigor is an AI agent and it lets you test your LLMs, graphs, chatbots, etc as well. Read more about how to perform AI Features Testing.

Read more about other top features and testRigor’s documentation.

Conclusion

The vision behind Cucumber is to enhance the development process by making it more collaborative and transparent. It is focused on delivering value to users through well-defined and tested behaviors. However, it might not suit everyone, especially manual testers or people who are not programming experts.

Team members with extensive testing and domain knowledge can easily generate/create plain English test cases using testRigor and execute them. This provides greater test coverage with minimum time, effort, and cost. This is why testRigor is a Test Automation Tool For Manual Testers, which lets everyone on the team contribute and use their distinct expertise.

As you consider the benefits of AI-based test automation tools, don’t hesitate to schedule a demo with us to gain deeper insights and discover how these solutions can positively impact your organization.

“We spent so much time on maintenance when using Selenium, and we spend nearly zero time with maintenance using testRigor.”

Keith Powe VP Of Engineering - IDT