 |
Cypress aspired to be an end-to-end testing framework. But it is not living up to the expectation. Moreover, all of its features are a subset of Selenium. And, at least on paper, it is really worse than Selenium.
Let’s go through the properties of a good end-to-end framework and see how Cypress manages them.
Lack of Browser Support
It lacks support for Internet Explorer, Safari, and virtually any mobile browser: like Chrome on Android, or Safari on iOS. That’s just basic stuff that even Selenium can do.
Today, and for quite some time now, more than half of all web traffic comes from mobile.
Non-developers Can’t Use It
Modern frameworks like testRigor allow wide audiences to be able to not only record steps, but also to convert these steps into a test – and understand what is going on there with just an understanding of English. Not so much Cypress. It is only available for developers, and if you are not one, then you won’t be able to understand the code behind it nor the test failures.
Your product manager has no idea what you are testing and why. And she/he has no way of understanding what happened in case of a failure. And, of course, there’s no chance for her to create a test for you before a feature is developed – unlike what you can do in testRigor.
Moreover, your product manager can’t participate in creating the executable specifications for you or your team and/or review the ones that had been create by you/QA. No chance of an effective ATDD like this one.
It is Not Stable Enough
The code in Cypress is not actually testing end-to-end scenarios. It is testing the implementation of your pages since it hooks up to the details of the implementation of a page like CSS selectors. Therefore, as soon as the implementation changes – tests break with a false-positive and require a time investment to update the test to match the new implementation of the page. This is the exact opposite of what an end-to-end test is supposed to do. If the page changes, but from an end-user’s perspective everything is still the same – tests should confirm it. Moreover, true end-to-end tests should allow for framework migration (think Angular to React) and validate that everything still works fine from end-user’s perspective.
Failure to Test Forms
Imagine that you started to render your controls manually and had good-old names on the form inputs. That is wonderful. But then you migrated to fancy Telerik controls, which would, in particular, migrate all of your selects to div-based rendered dropdowns. That’s it. Say goodbye to all of your test code. You need to rewrite everything from scratch. Even though from the user’s perspective everything is still the same.
Failure to Test Tables
Once again, a very common scenario is that you start from plain and simple bootstrap to render your tables. But then you’ll figure out that you don’t want to manage all of the boiler-plate code related to getting the data, rendering the table, being reactive, sorting, filtering etc., and migrate to a nicer library like AG Grid. Unfortunately for you, AG Grid is a modern UI framework, and it renders tables using divs for various reasons including being responsive. Boom! All of your tests are gone yet again! Compare that to doing nothing and validating that everything still works fine.
Hard to Deal with Test Issues
Imagine you really invested in Cypress and wrote 1,000 “end-to-end” tests. And then the design of the home screen had changed – affecting, let’s say, 800 tests.
This is a painful situation since you are in the dark about what is going on. A lot of the tests failed, but you don’t know how many unique issues are there and it is almost humanly impossible to go through 800 individual broken tests. Modern frameworks like testRigor will help you to work with unique issues as opposed to just a huge list of broken tests.
It Can’t Do End-to-end
Even in a simple scenario where a test should span your system and integration with, say, Microsoft Dynamics CRM, Cypress can’t pull it off. Microsoft developers change the rendering of their pages and elements significantly every two weeks, which would keep failing most of the tests every couple of weeks. Not exactly what you’d want to have in your CI.
On top of it, the notion of “end-to-end” also includes creating tests, for instance:
- Spanning both web and mobile
- Testing emails for things like signup flow
- Testing 2FA with SMS or Authenticator
- Testing audio and video
- Many other features that you’ll find in a modern application
None of which is possible with Cypress.
It Impedes Your Ability To Move Fast
Your goal as a team or a company is to move fast without breaking things. The goal of test automation is to help you to move fast without breaking things.
However, as we see from the above, Cypress will impede your progress and will add unnecessary overhead maintaining/rewriting your tests instead of helping you to validate functionality and move faster.
Low ROI
Think about it, Cypress requires investing a lot of expensive engineering time on creating and maintaining UI-level tests. Plus you need to also involve Appium or something like this to complement it with testing your mobile app.
Compare that with your manual testers being able to do the same thing faster. And spend almost no time maintaining the tests.
Plus, since it is code, it adds additional overhead on communications since only Engineers will be able to understand code failures.
We have customers that saved 60%+ of the budget migrating from Cypress to testRigor.
Conclusion
Cypress might be OK as a unit testing tool, but it is not applicable for end-to-end testing. And never was.
You can’t afford it when you are a fast-moving team or a startup.
Unlike more modern AI-based alternatives, Cypress doesn’t scale well with the number of test cases you need to cover.
Would you like to learn more about how modern frameworks will allow you to move 10X faster in building tests and eliminate painful test maintenance? Schedule a demo with us. Or go ahead and get your free account.
