You can’t miss it these days. AI pops up in every LinkedIn ad, is the buzzword at every conference, and is used by just about everyone—for a mix of the most sensible and the most nonsensical things. Even within software testing, the question is no longer whether AI has an impact, but rather how significant that impact is, and what concrete changes it brings for testers, QA teams, and the quality of the software that is ultimately delivered.
To answer that question, two test consultants from M2Q sat down for a conversation on our podcast. No scripted talk—just an honest, off-the-cuff chat about what AI is already doing well in a testing process today, where its limits lie, and why testers are far from becoming obsolete. In this article, we’ll summarize the key insights from that conversation.
🎧 Listen to the full episode: “The Impact of AI on Testing and Software Quality”
Prefer to read? You’ll find the key points from the conversation below.
Not automatically, no. For developers, AI is first and foremost a catalyst: whereas it used to take a long time to translate an algorithm into code, AI can do that in a matter of seconds today. That means faster programming and faster delivery, but faster isn’t the same as better.
The challenge lies not in generating the code itself, but in what happens afterward. Once AI-generated code is integrated into an existing codebase, it must be able to communicate with the existing code. Without a thorough analysis beforehand, that’s exactly where the risk lies: sections of code that suddenly become inaccessible, or unexpected errors that only become apparent during integration. So testing isn’t going away; it’s simply evolving alongside a new way of coding in which AI and the developer produce the code together, rather than the developer alone.
An interesting side note from the conversation: AI-generated code may be slightly more structured and better documented than code written by an individual programmer, simply because an AI model consistently applies the same conventions. Whether that’s actually true remains to be seen, but it could certainly improve readability for testers, especially in the context of automation.
AI already provides clear added value in two areas of the testing process: in the generation of test scenarios and in the automation of their execution.
In practical terms, this means that when creating test scenarios, instead of manually coming up with all the ways a user might make a mistake, an AI tool can suggest a large portion of those scenarios based on a user story and acceptance criteria. In practice, about 80 to 85% of these AI-generated scenarios prove to be usable, representing a significant time savings compared to the traditional, entirely manual process. The remainder—and especially the scenarios based on how a real user behaves rather than what the documentation describes—remains the work of humans. No AI tool today can simulate the intuition of an end user.
In addition, AI also accelerates the automation process itself. Whereas test scripts used to have to be recorded or coded manually, an AI agent can now generate code on its own to execute a scenario in a specific environment. This reduces a turnaround time that used to take weeks to just a matter of days.
An important difference here is that a generic AI chat tool produces generic results. An AI agent specifically trained for testing—for example, integrated with Jira and trained to write test cases directly from a ticket—delivers noticeably better and more consistent results than an AI tool that “just happens” to be used for testing as well. For this reason, many test management tools are now developing their own specialized AI agents.
How well AI performs depends largely on how well you ask. That sounds obvious, but in practice, it’s exactly where a lot of AI-generated output falls short: people do write what they want, but they rarely specify what they definitely don’t want, or the boundaries within which the AI must operate.
As a result, the tester’s role is shifting: from someone who performs every step themselves to someone who directs and evaluates AI—from a trumpet player to an orchestra conductor. This shift requires a new skill: knowing exactly what question to ask an AI tool, how to provide context, limitations, and the desired outcome, and how to then critically evaluate the results.
That decision is also a matter of time. For a simple, well-documented story, prompting quickly yields usable output. But with a story that’s already unclear, the AI doesn’t recognize that ambiguity either, and you sometimes end up going back and forth with the AI tool—which takes longer than just writing it yourself. So the question a test team must ask itself each time is: Does the time spent on prompting outweigh the time it saves, and how often does that scenario recur?
AI generates test scenarios based on available documentation, which immediately highlights the biggest future challenge: just how up-to-date, uniform, and consistent is that documentation, really? This isn’t a new problem. Testers used to run into conflicting documentation as well—where page 5 claimed something different from page 100—and would consult the analyst to resolve it. The difference is that the quality of that documentation now directly determines the quality of what the AI generates from it.
Organizations that want to use AI systematically for testing would therefore be wise to first invest in the quality of their own documentation before expecting AI to generate useful, reliable scenarios from it.
AI doesn’t replace the tester, but it does shift where the tester spends their time. The same argument was made about five to ten years ago regarding test automation: back then, people thought manual testing would disappear entirely. In practice, manual testing still exists today—only in combination with automation. AI will likely follow the same path.
What is changing, however, is the speed: whereas a thorough scenario analysis used to take several weeks, AI can reduce that work to a matter of days. But even after that initial generation, a tester is still needed to verify that the AI’s output is correct, to assess whether scenarios are relevant, and to implement what is actually necessary. It is precisely this monitoring and evaluation role that a tester’s responsibilities are shifting toward—not away from testing, but toward guiding and validating what the AI proposes.
You can never be certain that what AI generates is actually correct, and in testing, that’s a risk you can’t just brush aside. Just as a developer shouldn’t test their own code, the same question applies to AI-generated test results: how do you verify that what the AI claims to have tested has, in fact, been thoroughly tested?
An example from the discussion illustrates that risk: during an automated security and hacking test, an AI tool was given clear boundaries regarding what could and could not be tested, and yet the tool still found a way to move from the test environment to the production environment. Security and boundaries are therefore not a side issue, but a prerequisite as soon as AI is deployed more autonomously.
Nevertheless, some organizations will choose to consciously accept that risk, for example, because the impact of an error is estimated to be low. That can be a valid choice, as long as it is made explicitly and supported by the appropriate stakeholders, rather than arising tacitly. Anyone who wants to consciously approach this process will immediately recognize the connection to risk-based testing: assessing risks, prioritizing them, and consciously deciding what to trust and what not to trust.
At M2Q, it makes little difference whether an application was built entirely, partially, or not at all by AI. What matters is whether it does what the customer asked for—functionally, securely, and reliably. That functional focus doesn’t change because of AI.
However, M2Q does use AI as a tool to generate usable test scenarios more quickly and, in combination with risk-based testing, to prioritize them based on the actual risk. But the validation itself—verifying that what is delivered is actually correct and secure—remains a human task. A tester is in the driver’s seat, not the AI tool. This is especially true for sensitive scenarios such as payments or personal data, where the question “Has this been tested for security?” should never simply be left to a tool.
Will AI replace the software tester?
No. AI is changing how a tester spends their time—from manually devising each scenario to directing, evaluating, and validating what the AI proposes—but the tester’s role remains essential.
Can AI write test cases for me?
Yes, based on a user story and acceptance criteria, AI can generate test scenarios, 80 to 85 percent of which are usable on average. The specific test steps and scenarios based on the behavior of a real end user are generally still done by humans.
Is a generic AI chatbot sufficient for test automation?
A generic AI tool can be helpful, but an AI agent specifically trained for testing—for example, one integrated with your test management tool or Jira—typically delivers significantly better and more consistent results.
How important is documentation when you want to use AI for testing?
Crucial. AI generates scenarios based on available documentation. If that documentation is outdated, contradictory, or incomplete, the AI simply carries those shortcomings over into the generated test scenarios.
What is the biggest risk posed by AI in software testing?
You can never be completely certain that what AI generates or “tests” is actually correct and thorough. Especially in sensitive scenarios, such as security or payments, human validation remains essential.
How does M2Q handle AI in testing processes?
M2Q uses AI as a tool to develop usable test scenarios more quickly, combined with risk-based testing to prioritize them. The final validation and verification are always performed by a test consultant.
AI, in the words of our own test consultants, is still very much a work in progress, yet it is already a valuable tool in software testing. It speeds up the generation of test scenarios, accelerates automation, and changes how a tester spends their time. But it does not replace human judgment, a critical eye, or oversight of what is ultimately delivered.
Curious about the full conversation, with even more real-world examples? Listen to the podcast episode “The Impact of AI on Testing and Software Quality.” Want to know how AI and risk-based testing can specifically contribute to the quality of your software projects? Contact M2Q for a no-obligation consultation.
