What levels of automation do we need to support Continuous Integration/Continuous Deployment (CI/CD) pipelines?
How can we get a return on investment (ROI) from our automation testing?
How do you implement automation testing in agile projects?
Why are our automation tests flaky and keep failing?
How should I choose an automation testing tool for my project?
These are some common questions I get asked by organisations I work with.
In this article, I will share some key things that needs to be considered when creating a test automation strategy to support CI/CD pipelines and deliver a quick ROI from automation testing in Agile projects.
Why use automation testing?
In Agile software development small teams work on product increments and deploy to production as often as possible. In continuous deployment test automation becomes even more important due to its ability to provide quick feedback to the development team on the health of the application. In order to get quick feedback automated tests need to be executed as often as possible, ideally after every code change that is pushed to the repository. Tests need to be fast, delivering consistent and reliable results.
The major challenge for QAs on any agile project is the amount of Regression testing that needs to be performed before each release; this can be addressed using automated tests.
Prepare a test automation strategy
The test automation strategy should be result-oriented and must fit well with the team and organisation as a whole. There is no one-size-fits-all when it comes to this strategy, so you need to tailor it to each particular project/application, ensuring that it serves the intent and helps to attain the team’s objectives.
It’s a good idea to consider the team size and environments available for testing while preparing a test automation strategy.
It’s important to define objectives for your test automation strategy. Well defined objectives will help you set up parameters to measure its success. The objectives could be different – get fast feedback for the developers, reduce testing efforts in later sprints, enhance confidence in the code through test coverage (Unit, Integration, User Interface (UI), Performance, etc.), faster time to market, cost-effective testing, reusable tests, support continuous integration and much more.
Select suitable automation tools
There are many commercial and free open-source test automation tools available on the market. Choice of the tool will depend on your testing objectives and team requirements.
Here are some key parameters to consider when shortlisting a tool:
- Compatibility of the tool with the project tech stack (e.g. support for containerisation)
- Internal and external support for the tool (Example implementation, blog posts, etc.)
- Users of the tool (QA, dev, PM, etc.)
- Easy interface to create and maintain tests
- Support for different testing techniques (BDD, TDD, etc.)
- Support for different testing levels (Unit, Integration, UI)
- Support for running tests in headless mode (To speed up execution of UI automation tests)
- Cost of licensing, maintenance and training (open source vs paid)
Create automation sub-tasks
Create test automation sub-tasks under the User Stories so that the automation test scripts can be implemented at the end of the current iteration or at the beginning of the next. In addition, this will help in tracking the progress of automation tests and the test coverage.
Build automation tests
Here are a few things to be considered when it comes to creating automation test scripts:
Reusable and maintainable test scripts
Reusability is one of the key reasons for considering test automation. It is important to consider this aspect in your automation test strategy. Reuse of test scripts will make the process cost-effective and speeds up the automation process. It is important to create a reusable framework that can be optimised for testing across other device platforms or for other projects. Due consideration should be given to concerns like maintainability and execution time.
Automation Test pyramid
As part of the test automation strategy, we need to implement the automation test pyramid to increase the test levels.
Automated Unit Testing
- Unit tests are written by developers for any new feature that is developed. Test automation starts at the unit level. These tests will test each testable unit separately.
- Unit tests can be run on the developer’s local environment as well as the CI/CD environment.
- Unit testing provides the greatest ROI for the team as they are very quick to run, easy to maintain and modify, and give quick feedback to the developer about the quality of the code.
Automated API / integration testing
- Integration testing forms the next level up from unit tests to test the functionality of each service (API) independently. These tests are executed at the API layer independently of the User Interface (UI).
- Mocks and stubs will be used to factor out the dependence on other services or 3rd party systems.
- These tests are executed once Unit tests are run and passed.
- Integration tests can be created by both Developers and Automation Testers.
- These tests are fast to execute as they do not depend on UI and can be run in local environments and in the CI/CD environment.
User Interface (UI) automation testing
- The UI automation tests cover typical user flows, journeys, and end-to-end scenarios.
- These automation tests simulate a user’s interaction with the application through the UI which provides good and meaningful scenarios, however it is prone to issues such as: a dependency on html locators, slow page load-times, limited testing, etc. So, it is a good strategy to keep the number of these tests to a minimum.
It’s a good idea to employ the same version control (Github, Bitbucket, etc.) tools as the developers, to stay in line with the project tech stack. All unit tests and integration tests can live in the main repository with the application. UI automation tests may be better held in a separate remote repository as these tests depend on third party tools.
Run automation tests in CI/CD pipelines
A CI/CD pipeline helps to automate steps in the software delivery process, such as initiating code builds, running automated tests, and deploying to different environments such as Integration (System test), Staging (Pre-production) and Production.
A pipeline procedure is triggered when code is committed to a repository hosted somewhere like GitHub or Bit bucket. This pipeline can build the application, execute unit tests and report the results.
Once unit tests have run successfully it is deployed to an integration (system test) environment where automated integration (API) tests and UI smoke tests can be executed. Once all tests have passed then a build will be triggered to deploy it to the staging (pre-production) environment where a full set of regression tests are run.
Even with good test scripts it is difficult to find bugs through automation if there is no adequate reporting available. Clear and comprehensive reporting helps us to reach a conclusion after script execution has completed. The reporting format might be different for each tool but should be easily understood. It is also helps to include screenshots in test reports. There are many third party reporting tools such as Allure Test Report, etc. are available in the market.
The main challenge for the QAs on any agile project is the amount of regression testing that needs to be performed before each release. This can be addressed using automation testing.
Your test automation strategy should be integrated into your Software Development Life Cycle (SDLC) and should include different levels of testing such as unit, integration and UI.
Think about the test automation pyramid while creating your test scripts. Creating more UI tests will make your automation slow, it will become out dated quickly, and be difficult to maintain in a long run.
Test automation is a long term investment. You can get a quick ROI when you run your test scripts in CI/CD pipelines as part of the release process. You can see the benefits of automation testing when you select the tools that will fit into the project tech stack and the way the team works. One should think about reusability and maintainability while developing test scripts and should have clear and comprehensive reports.