Two months ago, the M2Q Agile team found itself in a situation where most of the automated tests were put on hold because they were taking too much time to execute. However, these were necessary to be performed with each build.

After a very intensive retrospective, we decided to devote a short sprint to speeding up our tests, because the faster the tests are, the more often they will be run.

The tests that are not performed do not catch problems, and the less often they are performed, the more confusing it is to understand what is wrong. Fast tests are often run, and that helps us work in smaller, safer steps.

The first thing we did was take a look at what the tests actually do. A good rule of thumb is that each test should do exactly one thing unless otherwise justified. This atomic nature is key to creating unit tests, but it is equally important for integration or end-to-end testing. It is common for lower-performance tests to be overloaded because too many things are running simultaneously. By reducing the number of steps, which each test must perform, the total time for each test decreases. You may find that running multiple tests with one goal is faster than running one overloaded test.

Once we identified less-performing tests and there was no obvious bottleneck, we ran the test with a monitoring tool to see why that test took so long to execute. With that information, we were able to identify bottlenecks and consider solutions to increase performance.

This is how you can reduce the Time To Market:

• Perform low value(priority) tests less frequently. The value is calculated by relating the probability of failure to the priority of a bug found by an error. If a test never fails or only finds low priority bugs (or is redundant with another test), we run it less frequently, perhaps overnight or weekly.
• Remove “sleep-time” in your testware. They waste time, and they are an indication of a poor test. (tests are of poor quality if they show a seemingly random result despite testing code that has not changed). The tester doesn’t really know what to wait for and ends up waiting a long time. If the test is important, there are ways to check for the required data more often(polling) so that the test can be resumed earlier.
• Move testing lower in the pyramid: trade integration for speed. Can a system test become an integration test? Can the integration test be replaced by some good unit tests? The lower the level of the test, the faster it usually is.
• Do the setup of the application and all supporting service only once, not with each test case.
• Whenever possible, use data that can remain in memory(databases) and small data sets.

After reviewing and implementing the above tips, our tests were no longer a bottleneck to the project. We could go back to testing faster.

But as a fellow tester once said, being the bottleneck is not necessarily a bad thing. E.g.. : Security is always a bottleneck at airports. Sometimes you have to queue for a long time to get past the security checkpoint. But there is no one speaking out to say “Let’s do away with security because I want to save 15 minutes”?

People who judge the speed of testing may not have thought about the value of the process. Testing is like insurance, in the sense that you don’t take insurance because you hope to make a profit. You take out insurance as protection against loss. Testing has a price in return for that investment and time.

We get a chance to discover important and elusive problems, and therein lies the value of testing.

Author: Oana