Why Did We Kill Our Test Environment?
 |
Yes, we did indeed completely remove the test environment from our process and de-provisioned it.
The story is as follows:
The Problem
We started like everyone else with people writing and committing code to the code base and then eventually we would deploy it to the test environment and after validation, it would go to production.
And yes, for the record, we do use testRigor to test testRigor. Well, that wasn’t enough. Because testRigor tests only deviations from the previous build. And if some untested code gets into the test, testRigor more often than not will just update the baseline thinking that this is how things should work.
Well, it wasn’t always working as it should have. And even more, deployment to test things before it gets to production proved to be a painful experience. It was sporadic and time-consuming.
De-facto what was happening is if there was a blocker-level issue we would deploy it to production ASAP anyway. Yes, without testing. And guess what – it did break the production.
Moreover, the quality overall wasn’t that great, since everyone was relaxed thinking that it WILL be tested in a test environment before production.
And, on top of it, reviewing new changes on tests proved to be tedious because they had a tendency to come in batches with several people finishing several large features at the same time.
Needless to say, we were struggling with quality (sic!). And we can’t afford it obviously since we ARE a quality testing tool!
The Solution
After some time of painful struggle, it was decided to get rid of the test environment altogether. We embraced full CD and set up a master build after the merge to go directly to production.
Boom!
What was the effect of this?
Controversially – we got to WAAAY better quality in production!
But why??
Well, I believe several things:
- We started to test it in production. Nothing is closer to production than production.
- Most importantly, developers now got a sense of ownership. We implemented a process where you need to deploy AND test your changes on your (temporary) dev environment. And that was super effective to drive quality up.
Check out what we envision:
the system is not public yet, however, if you want it – just shoot an email to [email protected]
