It’s one of the most frequently asked—yet also one of the most difficult — questions in software development: When do you stop testing? Ask a developer, and the answer is often: “When the code is done.” Ask a project manager (PM) and you’ll hear: “When the deadline is reached or the budget is exhausted.” But ask an experienced test manager, and the answer is invariably: “When the risk of going live is acceptable.”
There is no black-and-white answer. In theory, software is never “100% finished,” and it’s impossible to test every conceivable combination of actions and systems. In this comprehensive guide, we dive deep into the world of testing strategies, exit criteria, and risk management. How do you determine the tipping point at which software is ready for production? What are the best practices, and what should you do when commercial pressure mounts?
The key question behind “When do you stop testing?” is really: When is my software ready to go into production? As a test manager, QA engineer, or tester, it is your job to demonstrate, in an objective and transparent manner, the current state of the software at a specific point in time. You stop testing when you can prove that the application has been thoroughly tested and that the remaining risks are minimal and acceptable.
In practice, unfortunately, this process doesn’t always go according to plan. Many test managers have encountered situations where a project manager simply says , “We’re stopping now because we’re going live.” Without any valid, quality-related reason for doing so. Decisions like this significantly increase the risk of critical errors in production. To prevent this, you need to establish clear agreements at the start of every project: the so-called exit criteria.
At the start of a project, the QA team develops a test plan or test strategy. This document addresses the following key questions:
But the most important question—one that is often overlooked—is: When do we stop?
You often hear people say , “We’ll stop when the money runs out” or “when the deadline is reached.” While budget constraints and deadlines are a harsh reality of the business world, they are not valid qualitative exit criteria. If the budget is gone, but the core functionality still crashes with every click, then the software is simply not ready for the end user. Good exit criteria must be based on measurable quality.
Strategy 1: Defect Status (Bug Triage)
One of the most commonly used and effective exit criteria is based on the defect list (the bug log). During testing, you log all findings in a defect management system (such as Jira or Azure DevOps) and assign priorities: Critical, High, Medium, and Low.
For example, a sound exit criterion is:
“We will stop testing and move to production as soon as all ‘Critical’ and ‘High’ defects have been fully resolved and a business-approved workaround is in place for any remaining ‘Medium’ defects.”
Strategy 2: Code Coverage (White-box Testing)
A more technical approach involves measuring code coverage through white-box testing. This involves running software in the background that tracks what percentage of the written code is actually covered by the tests (code coverage or test coverage).
Can you agree to stop once you reach 100% code coverage? That depends heavily on the type of application. For critical medical software or aviation systems, extremely high coverage is required. For an average business application, 100% coverage is often unaffordable and unnecessary; in such cases, the focus is more on a reasonable standard of, say, 80% to 85%, combined with functional testing.
Strategy 3: Requirements Coverage
You can also link exit criteria to the functional specifications. If you have a list of 50 requirements (needs and wishes), and there are 200 test scenarios associated with them, then the criterion could be:
“We won’t stop until 100% of the critical test scenarios have been run and successfully completed (green).”
The trick isn’t in coming up with criteria, but in choosing the right combination that fits the context of your project.
Sometimes the deadline is so strict that the business has no choice. Consider, for example, the historic transition to the euro on January 1, 2002 (or the preparations in the banking sector in the years leading up to it). That was an absolute deadline. The law mandated that systems had to be ready by that date. Whether your testing phase had been perfectly completed or not: going live was mandatory.
As a test team, how do you handle such extreme pressure?
The paradox of software testing is that you never really stop. Once an application goes live, preparations for the next release often begin immediately. This creates a unique set of dynamics:
Regression testing
When releasing a second or third version of software, you want to be sure that the new features haven’t broken the existing, properly functioning parts. We verify this through regression testing. This can be done manually, but ideally it is (partially) automated to maintain speed.
Post-Go-Live Testing
Were you unable to test everything before going live due to a tight deadline? If so, it’s an absolute best practice to continue testing in a stable environment immediately after the go-live. At that point, you focus on the least critical functionalities that were previously left out. If you don’t do this, the end user will eventually find the errors in production. And that is a scenario you naturally want to avoid; after all, fixing a bug in production is many times more expensive than in the testing phase.
It’s every software tester’s ultimate dream: an unlimited budget and no time pressure. How would a test manager approach it in that case?
Surprisingly, this doesn’t change the fundamental testing strategy. Even with billions in the bank, the basic structure remains intact. The big advantage, however, is that the pressure is off. You get the space to tackle everything thoroughly. You can dive deep into non-critical functionalities, run extensive performance tests, conduct security audits, and perform usability studies with real end users. In the real world, however, budget is almost always the limiting factor, which means that prioritization remains a tester’s most important skill.
Exit criteria in practice: The example of an online store
To make exit criteria more concrete, let’s look at a familiar example: a large, well-known online store. What are the top priorities here, and how do you determine when to stop testing?
An online store is all about one main goal: generating revenue and selling products. Its core functionality consists of a clear funnel:
These are the five critical pillars. When establishing the exit criteria for launching an online store, you should focus entirely on this process. Does the integration with the bank (iDEAL, Bancontact, credit card) work flawlessly? Can orders be processed correctly? If that works 100%, it’s acceptable to go live.
Minor issues, such as a button that’s just three pixels to the left, a color discrepancy in the footer, or a typo in the terms and conditions, are annoying, but they don’t prevent the online store from generating revenue. These are typical issues that you leave on the defect list as ‘Low’ or ‘Medium’ and that are only resolved after the launch (or in a subsequent sprint).
Another highly scientific way to determine when to stop testing is to look at the trend line of the number of defects found: the defect curve.
When a testing team starts working on a brand-new release, you almost always see a sharp spike in the reports. A large number of bugs are found in a short period of time. As the developers fix these bugs and the testing team performs retesting, the curve flattens out. Fewer and fewer new defects are discovered.
As a test manager, you might decide: “We’ll stop testing as soon as the number of new defects found per test day drops below a certain level.” That’s when the law of diminishing returns kicks in. The time, energy, and money you have to invest to find that one last small bug is simply greater than the impact that bug might have in production.
Pausing the test phase: Pulling the emergency brake
The defect curve can also be a warning sign. Sometimes you don’t see the curve dropping, but rather rising exponentially. You log one bug, the developer tries to fix it, but accidentally introduces four or five new bugs. This often happens when the code is too complex (spaghetti code), the developers are under too much time pressure, or when they don’t fully understand the functional context.
As a test manager, you need to have the courage to say, “We’re putting the testing phase on hold.” Continuously testing against a wall of poor quality is a waste of time. It demotivates the test team and costs an unnecessary amount of money in administrative expenses. At that point, you go back to the drawing board with the development team to first get the software’s basic stability in order.
When we talk about bugs introduced by developers, we also need to take a critical look at the phase that comes before it: the analysis. After all, when is a functional or technical analysis complete?
Software bugs often arise because human interaction breaks down early on. An analyst listens to the stakeholders’ requirements, documents them, and both development and QA have to work with that. If there are already gaps or incorrect assumptions in that documentation, the developer builds a “bug” that is actually exactly according to the specifications.
This is where the “Shift Left” principle comes into play. This means that the testing team is involved in the project at a very early stage, even before a single line of code has been written. By critically validating the analyses for logical inconsistencies and ambiguities as a tester, you catch the errors before they are programmed. This saves a tremendous amount of time in the later testing phase and ensures that you meet your exit criteria much faster.
What if, as a tester, you join a project where there is no documentation, no analysis, and no test cases ready to go? You can still test. We call this ad-hoc testing or exploratory testing. You simply ask an expert or end user to use the system intuitively and see where it breaks.
Although this is an excellent way to quickly get a sense of an application’s stability (for example, through a brief “smoke test” or a half-hour “free-wheeling” session with a new daily build), ad-hoc testing should never form the basis of your exit criteria. It is not documented, not repeatable, and does not provide an objective guarantee of quality.
Do teams today want to stop testing sooner than they used to?
With the rise of Agile, Scrum, CI/CD (Continuous Integration/Continuous Delivery), and shorter release cycles (sometimes going live every two days or even multiple times a day), it seems as though there is less time for testing. But do teams actually want to stop testing sooner?
No, the drive for quality remains unchanged. However, technology has changed the context. Thanks to modern tools, Artificial Intelligence (AI), and extensive test automation, testing speeds are now many times faster than they used to be. But the fundamental principles of QA remain the same: you still need to sit down with stakeholders, assess the risks, and define clear exit criteria.
So when should you stop testing? The answer lies in a combination of clear agreements and risk management. Use the following checklist to determine whether the testing phase can be successfully completed:
Testing is not an end in itself; it is a means to facilitate decision-making. And when you can demonstrate with data and facts that the risk to the business is minimal, that is the moment to confidently conclude the testing phase.
Talk to us about the right approach for your business during a free online consultation. Register your consultation with the CEO here.
