Two months ago, the M2Q Agile team found itself in a situation where most of the automated tests were put on hold because they were taking too much time to execute. However, these were necessary to be performed with each build.
After a very intensive retrospective, we decided to devote a short sprint to speeding up our tests, because the faster the tests are, the more often they will be run.
The tests that are not performed do not catch problems, and the less often they are performed, the more confusing it is to understand what is wrong. Fast tests are often run, and that helps us work in smaller, safer steps.
The first thing we did was take a look at what the tests actually do. A good rule of thumb is that each test should do exactly one thing unless otherwise justified. This atomic nature is key to creating unit tests, but it is equally important for integration or end-to-end testing. It is common for lower-performance tests to be overloaded because too many things are running simultaneously. By reducing the number of steps, which each test must perform, the total time for each test decreases. You may find that running multiple tests with one goal is faster than running one overloaded test.
Once we identified less-performing tests and there was no obvious bottleneck, we ran the test with a monitoring tool to see why that test took so long to execute. With that information, we were able to identify bottlenecks and consider solutions to increase performance.
This is how you can reduce the Time To Market:
After reviewing and implementing the above tips, our tests were no longer a bottleneck to the project. We could go back to testing faster.
But as a fellow tester once said, being the bottleneck is not necessarily a bad thing. E.g.. : Security is always a bottleneck at airports. Sometimes you have to queue for a long time to get past the security checkpoint. But there is no one speaking out to say “Let’s do away with security because I want to save 15 minutes”?
People who judge the speed of testing may not have thought about the value of the process. Testing is like insurance, in the sense that you don’t take insurance because you hope to make a profit. You take out insurance as protection against loss. Testing has a price in return for that investment and time.
We get a chance to discover important and elusive problems, and therein lies the value of testing.