Automate testing without programming. So how do you tackle this without getting lost in the forest of tools and vendors? All state that their tool is the best, can provide reference cases demonstrating this, etc….

In this series on Test Automation Tools, Dirk Sundermann, QA Officer @M2Q.be, gives you a structured way to tackle this. The podcast is followed by 4 tool specific podcasts, namely Katalon, TestComplete, Tosca and
UFT
.

>>> Listen to our podcast on Test Automation without Programming Skills. <<<

1. Where does low code test automation fit into the landscape of testing?

Low Code Test Automation (LCTA) is currently a full-fledged alternative to the more mainstream way of automating testing. LCTA is usually introduced and used in 2 ways viz.

• As a test automation solution
• As an additional test automation solution to increase test automation level and coverage.

But, stepping back for a moment, testing is only 1 part of ensuring quality. There are a lot of methodologies and techniques for improving quality but that is not the subject of this blog.

Test automation offers a host of benefits whether done the regular way or through LCTA. The most important are the following (if done correctly):

• Reducing lead time to run tests
• Greater accuracy in repeating tests and repetitive tests (e.g., unit testing)
• Improving test coverage
• Test automation is a necessity for performance testing
• Continuous testing is only possible through test automation and thus enables shift-left of testing
• May lead to shorter time to market
  NOTE – cost/benefit analysis of what to automate

LCTA offers the ability to automate testing without or with minimal code. Today there is a wide variety of tools and offers viz. solving:

• 1 specific problem
• A whole range of test problems and challenges

In the world of LCTA, there are 3 types of suppliers viz.

• Software vendors – provide a wide variety of tools and services and are not specific test companies such as IBM, HP, Broadcom, etc….
• Test Companies that focus on testing such as Eggplant, Leapwork, Tricentis, etc., among others….
• Niche players that focus on solving a specific problem such as Froglogic and Browserstack, among others
• Froglogic Squish – testing graphical user interfaces (GUIs) and human-machine interfaces (HMIs)
• Browserstack – cross-browser and device testing (on physical devices hosted by Browserstack)
  Beware – there is a consolidation going on that may cause companies to disappear (not necessarily the products). Companies want to increase their portfolio in this way.

2. Why you should do low code test automation:

LCTA is an alternative to the standard way of test automation via 1 of the used test automation frameworks (TAF). A TAF requires technical knowledge both in the technology of the framework and its use. Some examples of frameworks are:

• Selenium
• Cypress
• Playwright
• Appium
• etc…

The big challenge today (24/08/2023) is to find people with technical knowledge (e.g. Java) and knowledge of 1 or more frameworks. The reason is that these technical profiles require similar knowledge as developers.

Additional challenges include:

• High initial cost
• Long-term investment nl you realize added value only over time
  The challenge will also become greater in the coming years. There is going to be more and more demand for test automation without the number of people increasing at the same rate. Just a little more info on this – the test automation market is about US$22 billion by 2022. According to Global Market Insight – Automated Testing, this market is going to increase by 15% between 2023 and 2032.

The growth of this market is driven by

• DevOps and Agile strategies – both work sub-optimally without automation
• AI & ML – without, among other things, test automation, this is difficult to test
• Mobile applications – there are many mobile devices with different OS which can be best and fastest tested via automation
• Further growth of digitization – more and more manual work and paper documents are being digitized which further increases the demand for test automation.

3. Approach – What are the main steps to take while selecting a tool to select?

Simply buying a tool and implementing it certainly does not guarantee success both in selecting the correct tool and implementing and using it. It is therefore important to take a structured approach to selecting the LCTA tool for your company. Just because a certain tool works in one company does not mean it will work in another.

Below is a list of recommendations to address selection. They are in chronological order but some can be done simultaneously.

• Define the scope of LCTA e.g. web, mobile, API, …
• Determine the selection criteria. These are crucial and are determined by scope. Apply these strictly as well, and if you make changes during the evaluation, apply them to all vendors. Some points of interest in creating selection criteria are:
  • Not just GUI – if your scope goes beyond GUI, focus on that too
  • Needs of today and tomorrow – if you have plans to introduce e.g. mobile soon then you should include it in the selection criteria.
  • Functional and technical needs
  • Use use cases to evaluate the tool – use real life scenarios to evaluate tools
  • Integration with other tools such as Jira and Azure DevOps because a tool is rarely used as a standalone
  • Security is an important aspect of tool selection but this from different perspectives
    • Security problem in the tool – check vendor response time
    • Performing security testing such as pen tests, among others
    • Turn off (bypass) security check e.g. captcha, 2-factor auth
  • Plus several others
    • Roadmap – past (number of major/minor releases per year) and future (usually only via an NDA). It is also important to know how quickly you move to a new version e.g. you may be 1 major release behind
    • Life cycle support e.g. promote testing from acceptance to pre-production
    • Reuse of tests – how easy is it to reuse a test or piece of test
    • Under shelf life testing – how easy is it to maintain testing
    • Knowledge in Belgium – through the supplier or a partner. You don’t want to make people travel to Belgium every time.
    • Pricing model
    • PoC & trials – the opportunity or proof-of-concepts & trials during selection. The important thing is that you can use this yourself and validate how it works and not just rely on the vendors.
    • Hosting flexibility – what options are available
• Gathering information on different suppliers through different channels
  • Pre-study & market analysis – a lot of information exists about the different vendors and their tools
  • Consult independent experts e.g. Gartner, Forrester
  • Experts by experience such as businesses and people – look into professional circle of friends
  • Plus
    • Customer satisfaction reports
    • Customer visits
• LCTA vendor selection – make a selection of the tools you want to evaluate
  • Reason is that you can’t evaluate all the tools
• Evaluation and selection
  • Decide how you will involve vendors during evaluation and selection
  • When you ask questions, also pay attention to how quickly they answer. If it is slow during the PoC, it will not improve 1-time you use the tool in production.

4. Implementation – What are the initial steps you need to take to make low code test automation a success?

1-After you have made a selection and it has been approved (e.g. via an Architecture Board), you still need to implement it.

The following steps guarantee implementation success

• After selection is made, it is important to roll out the tool to realize the benefits
• Set-up – involve the chosen vendor or 1 of the partners in the set-up.
  • In the early years of LCTA, many projects failed because the set-up was wrong
• Best to have a gradual roll-out and not a big bang roll-out. There are a number of reasons to do a graduated rollout.
  • Set up the tool correctly for your work environment
  • Get to know the tool and gain experience – including what doesn’t work well
  • Build the missing module e.g. link with a test management tool such as TestRail
  • Update initial best practices and guidelines with experience gained.
• Gradual rollout – selection of teams and/or areas
  • 1, 2, 3, … teams is not the main criteria for initial rollout
  • More important – focus on different types of tests nl a diversification of different tests
    • Greater variation yields greater knowledge of the tool
  • Further rollout can be done through 1 or more teams
  • During rollout, keep best practices and guidelines up-to-date
    • Setting up a Community of Practice (CoP) or something similar (E.g. Special Interest Groups, Guilds) is a good idea

5. Any closing remarks?

• Define the scope – also take your time to determine when it is a success or not
  • The correct scope also determines the selection of the tool
    • Niche solution vs. broader solution
  • The scope also determines the selection criteria e.g. if you don’t do API testing and have no plans, you don’t need to select on that
• Selection criteria – as complete as possible
  • Involve others to establish these selection criteria
  • Look beyond functional and non-functional needs
    • Integration
    • Security
    • Cost of license and maintenance
    • Training
    • Etc…
  • Gradual rollout – learning from the tool in your environment
    • Do this and the full rollout through a project
    • Change – keep that in mind because changes are going to happen during the rollout.
    • Ensure that a CoP exists and that guidelines/best practices are up to date and maintained.

Listen to our podcast on Test Automation without Programming Skills.