Venu Botla

Don’t gamble with quality

How understanding stakeholders optimises your testing strategy and the software you end up with.

In my experience working as a Senior QA consultant at Triad, I have seen many organisations consider Quality Assurance (QA) only as an additional cost in the project and ignore the intrinsic value that it brings to software quality.

The main reason seems to be that the project managers and key stakeholders are not be aware of the benefits of QA and the value it adds to the development team.

In this article, I will explain why it is important to know who your stakeholders are and how to optimise your testing strategy to include the interests of these stakeholders.

Because better engagement leads to a better testing strategy that results in better software.

Quality Assurance in Agile projects

In Agile Software development, it is important to create a test strategy that is results-oriented and fits well with the team and organisation as a whole. In Agile, Quality Assurance is about integrating testing into every stage of the software development life cycle, hence the testing strategy should focus on achieving this.

As with other aspects of software development, to create an effective test strategy you need to know who your stakeholders are. You need to understand their requirements, interests, concerns and goals at the outset, and be prepared to accommodate change. In turn, this will make buy-in from the wider team easier to secure, and they will be interested in the testing outcomes.

Who are the tester’s stakeholders?

Most commonly they are those people who have an interest in the outcome of tests and the information that QAs provide: developers, designers, product owners, project managers, users and sponsors.

Developers

Developers are the people who build the product. These stakeholders need to know where their product fails so they can fix defects [close to their source]. For them, it is important to identify issues early in the development process, because the cost of remediation increases over time, and the later defects are found the less flexibility a project has to respond to them.

The test strategy should contain good development practices like feature branching, pull requests, peer reviews and quality measures such as unit test coverage, integration test coverage, automated smoke/regression tests, performance tests, security tests and code quality checks. These measures provide a safety net giving developers greater confidence to make changes in the application.

Designers

Designers are responsible for designing the User Interface (UI) and User experience (UX) of a product. These stakeholders also need to know where their design fails so they can fix the issues. It is important to identify any design issues as early as possible, typically before the development starts.

The test strategy should include testing techniques such as design reviews, usability testing, accessibility testing, etc. to address the interests of these stakeholders.

Product owners and Business Analysts (BAs)

Product owners and BAs may work together to develop user stories and discuss priorities. They are interested in understanding whether a feature meets their needs and expectations. The QA team should work collaboratively with the POs and BAs to understand the requirements (user stories) and effectively communicate them to the wider team.

A well-defined user story should include background, acceptance criteria and non-functional requirements for that feature. The test strategy should contain activities to review that the user stories are testable and to create acceptance tests to verify that the implemented feature meets the acceptance criteria.

Project Management

These stakeholders need to know the status of deliverables (i.e. application features): are they available, acceptable, and reasonably free of defects? If there is a problem, the manager may need to re-plan or at least manage expectations.

It is useful to include the test deliverables (i.e. test scenarios, test cases, requirement traceability matrix (RTM)), schedules and quality measures in your QA strategy to prove the quality of the application being developed.

Sponsors

These stakeholders need evidence that the system can support their business goals and that the risk of failure is acceptable. It is important to understand the business goals and product risks and include them in the test strategy.

Demonstrate an understanding of the business goals and risks in your testing strategy, it will also help you in defining the right test approach.

Users

These stakeholders need evidence that the system will work for them. As a QA, you should always think as an end-user.

Your test strategy should address real-time use case scenarios such as testing in different browsers, operating systems, devices, usability testing and accessibility testing.

Conclusion

As QAs we need to embrace that we are accountable to stakeholders; you need to engage, consult, and communicate with a range of stakeholders and reflect their needs into your testing strategy. Testing approaches should focus towards meeting their needs at all times.

However, even once the testing strategy is agreed and being put into practice, for QA’s, it is important to communicate effectively with those stakeholders. Sometimes, this will take the form of documentation, shared wikis, Kanban or Scrum boards. If your test strategy addresses the concerns of the stakeholders, then they will feel involved and will support the QA team during critical times.

The best test strategies are the result of exploration, thinking and collaboration.

How to setup OWASP ZAP to scan your web application for security vulnerabilities

Recently, I had an opportunity to work alongside my excellent team mates from Triad and the Department for Transport (DfT) as a QA practice lead, developing the new Manage Motor Fuel Greenhouse Gas Emissions service for GOV.UK.

For this project, we wanted to strengthen our in-house penetration testing (pen test) capability to enable us to prove the security of our web application from the outset, rather than having to wait for the results of our independent pen test towards the end development. Being relatively new to penetration testing, we wanted to choose a tool that was easy to setup and could find as many vulnerabilities as possible. Having considered several free and paid tools, we chose OWASP Zed Attack Proxy (ZAP) due to reasons given above and expanded on below.

In this article, I will demonstrate how to setup and use OWASP ZAP to test the security of a typical web application.

Before I continue, I feel obligated to warn you that you should use this tool only with an application you’re hosting yourself, or one you’ve been given explicit permission to test, as ZAP attempts to modify data and insert malicious scripts in the web application.

What is OWASP?

The Open Web Application Security Project (OWASP) is an open, online community that creates methodologies, tools, technologies and guidance on how to deliver secure web applications. It is an international collaborative initiative comprised of both individuals and corporations. The project aims to standardise security approaches in web development and spread associated knowledge.

What is OWASP ZAP?

OWASP ZAP (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers. It can help to find security vulnerabilities in web applications. It’s also a great tool for experienced pen testers and beginners.

ZAP can scan through the web application and detect issues related to:

SQL injection
Broken Authentication
Sensitive data exposure
Broken Access control
Security misconfiguration
Cross Site Scripting (XSS)
Insecure Deserialization
Components with known vulnerabilities
Missing security headers

Why we chose OWASP ZAP?

As it is designed to be used by people with a wide range of pen testing experience, it was ideal for our team who were new to penetration testing.

ZAP is a free open-source tool which is easy to setup and use. As it is used by the wider community, there is a lot of help available online through the ZAP blog and other articles to help you setup and use the tool.

ZAP is cross platform i.e. you can install it in Windows, Linux or Mac OS.

ZAP can be run in a Docker container, which suited our project tech stack. Also, its functionality is scalable with many diverse extensions published on GitHub.

ZAP Jenkins plugin can be setup to run the scans as part of CI / CD pipelines.

How it works

ZAP is what is known as a “man-in-the-middle proxy.” It stands between the browser and the web application. While you navigate through all the features of the website, it captures all actions. Then it attacks the website with known techniques to find security vulnerabilities.

As ZAP spiders the web application, it constructs a map of the web applications’ pages and the resources used to render those pages. Then it records the requests and responses sent to each page and creates alerts if there is something potentially wrong with a request or response.

Setting up ZAP

To begin with, you need to download and install OWASP ZAP scanner and set it up correctly. ZAP is platform agnostic so you can install it on Windows, Linux or Mac OS. You need Java 8+ installed on your Windows or Linux system.

For the purposes of this article, I’m going to concentrate on its use on Windows.

Starting ZAP

Once setup you can start ZAP by clicking the ZAP icon on your Windows desktop or from the start menu.

When the app launches, it asks you whether you want to save the session or not. If you want to use the current run configuration or test results later, you should save the session for later. For now let’s select “No, I do not want to persist this session at this moment in time”.

Once you click the “Start” button, the ZAP UI will be launched.

Spidering the web application

Spidering a web application means crawling all the links and getting the structure of the application. ZAP provides two spiders for crawling web applications;

Traditional ZAP spider: The traditional ZAP spider discovers links by examining the HTML in responses from the web application. This spider is fast, but it is not always effective when exploring an AJAX web application.

AJAX spider: This is more likely to be effective for AJAX applications. This spider explores the web application by invoking browsers which then follow the links that have been generated. The AJAX spider is slower than the traditional spider.

Automated scan

This option allows you to launch an automated scan against an application just by entering the URL. If you are new to ZAP, it is best to start with Automated Scan mode.

To run a Quick Start Automated Scan:

Start Zap and click the large ‘Automated Scan’ button in the ‘Quick Start’ tab.
Enter the full URL of the web application you want to attack in the ‘URL to attack’ text box.
Click the ‘Attack’ button.

Once you click the ‘Attack’ button, ZAP will start crawling the web application with its spider and passively scan each page it finds. Then ZAP will use the active scanner to attack all of the discovered pages, functionality and parameters.

Exploring the web application manually

Spiders are a great way to explore the basic site, but they should be combined with manual exploration to be more effective. This functionality is very useful when your web application needs a login or contains things like registration forms, etc.

You can launch browsers that are pre-configured to proxy through ZAP via the Quick Start tab. Browsers launched in this way will also ignore any certificate validation warnings that would otherwise be reported.

To Manually Explore the web application:

Start ZAP and click on the large ‘Manual Explore’ button in the Quick Start tab.
Enter the full URL of the web application to be explored in the ‘URL to explore’ text box.
Select the browser you would like to use and click the ‘Launch Browser’ button.

This will launch the selected browser with a new profile. Now explore all of the targeted web applications through this browser. ZAP passively scans all the requests and responses made during your exploration for vulnerabilities, continues to build the site tree, and records alerts for potential vulnerabilities found during the exploration.

What is passive scanning?

Passive scans only scan the web application responses without altering them. It does not attack or insert malicious scripts to the web application, so this is a safe scan; you can use it if you are new to security testing. Passive scanning is good at finding some vulnerabilities and as a way to get a feel for the basic security of a web application.

What is active scanning?

Active scan attacks the web application using known techniques to find vulnerabilities. This is a real attack that attempts to modify data and insert malicious scripts in the web application.

Active scans put the application at risk, so do not use active scanning against web applications you do not have permission to test.

Running automated scan againt the web application

Inspecting the test results

Once the scan is completed, ZAP generates a list of issues that are found during the scan. These issues can be seen on the Alerts tab that is located in the bottom pane. All the issues are marked with colour coded flags. You can also generate an HTML scan report through the ‘Report’ menu option on the top of the screen.

Summary

ZAP is a free open source platform agnostic security testing tool that scans through your web application to identity any security vulnerabilities as possible. It is a great tool for experienced pen testers, as well as beginners.

ZAP spiders the web application under test and scan for any known vulnerabilities.

For beginners it is easy to start with Automated Scan that will crawl the given URL with spider and passively scan each page it finds. You can do a more in-depth scanning by exploring the web application manually.

ZAP generates the scan report in the form of Alerts that are marked with colour coded flags. You can even download HTML reports from the “Report” menu option.

ZAP can also be integrated into CI/CD pipeline using ZAP Jenkins plugin.

Test Automation VS Process Automation

Often when we talk to Test Automation Engineers about Robotic Process Automation (RPA) the first thing we hear is “It is same as test automation with Selenium or similar tools”. I should confess that this is what I thought of it when I heard about RPA for the first time. However, my view has changed since I started learning RPA with UiPath. In this article, I will present the differences I have noticed between the test automation and robotic process alongside some views on how to maximise the benefit from different aspects.

What is Robotic Process Automation (RPA)?

Robotic Process Automation also known as Process Automation or Intelligent Automation helps in automating repetitive tasks without using human power. RPA is a kind of technology that consists of software bot, or a “robot” to emulate and integrate the actions of a human interacting within digital systems to execute a business process. The RPA bots can do a lot of things such as entering data, logging into applications, accomplishing tasks, reading and sorting emails, log out, etc.

RPA is primarily used in applications that require repetitive actions. Processes that require complex calculations, accuracy and precision can be assigned to RPA robots to get the task done efficiently. RPA robots utilize the user interface to capture data and manipulate applications just like humans do. RPA tools like UiPath incorporate elements of Artificial Intelligence (AI) and can be used to automate any structured data.

In summary, RPA technology is used to reduce or eliminate the need for humans to perform high-volume IT support, workflow, and back-office processes, such as those found in finance, accounting, supply chain management and customer service.

What can RPA tools do?

RPA tools are used by business users to automate any business process or functionality. They can be used to automate the process in web, desktop, mobile applications, mainframe and SAP based applications and also the applications that are running on virtual machines (Citrix based) etc. Unlike test automation tools, RPA is normally used only in production environments.

In my opinion, RPA tools can also be used for test automation, however, this would be an expensive option. RPA tools are more powerful than the conventional test automation tools. It is easy to setup and use RPA tools with no coding experience required, hence, they suited and mostly used by business users. Most of these tools have ready to use features like “Robotic Enterprise Framework” (RE Framework) in UiPath that speeds up process design.

List of Tools

What is Test automation?

Test automation is used as part of the development lifecycle to help the development team get quick feedback about the quality of their software. It is mostly used by the QA team to perform Regression testing. Automation testing tools like Selenium webdriver are used to test the functionality of a web application through a browser interface during the development phase of an application. Test automation is popular in Agile software development to increase team velocity.

There are different approaches for test automation such as Unit testing, Integration testing (API testing) and Graphical user interface testing. Each have separate tools available to support these test approaches.

In order to get quick feedback, automated test scripts need to be executed as often as possible, ideally after every code change that is pushed to the repository. You can read more about test automation in my blog post “How to optimise test automation on Agile projects”.

List of test automation tools

Here are the few differences between RPA and test automation;

Test automation is automating the software product and RPA is automating the business process and repetitive tasks.
Most of the test automation tools help to automate current web pages, unlike RPA tools that can automate all the backend processes.
The automation tests run in test environments such as Integration, Staging, etc. whereas RPA tools are used in production environment.
Test automation tools are used by QA team to get a quick feedback about the quality of the software. RPA tools are used by business users to speed up the processes.
Basic or no coding is required for RPA as it is Wizard-driven process, whereas automation tools like Selenium required proficient coding knowledge.
High domain knowledge is required for RPA compared to test automation.
Test automation is part of a software development lifecycle, whereas RPA has its own development lifecycle (Analysis, development, testing and support & maintenance).
There are a number of opensource test automation tools available in the market. When it comes to RPA tools, UiPath community version is free but the rest are paid-for tools.
Test automation tools have limitations when it comes to the type of application they can support, on the other hand RPA tools can automate any application including web, desktop, Citrix, etc.

Conclusion

Hopefully this article has explained how test automation and process automation tools both have a role in delivering automation solutions. The way I see it, the main difference is the environment in which they are used and the type of applications they can automate.

Test automation is used by QA team to verify the application under development as part of development life cycle, on the other hand RPA is used by business users to automate the repetitive tasks.

RPA uses bots (robots) to emulate the human actions to complete a process. Test automation is applied to verify the quality of the product whereas RPA is applied to the process associated with the business environment. RPA tools can be used for test automation, but it will be an expensive option.

Once this distinction is understood better, I believe that the opportunities for RPA in particular become easier to identify and more numerous across a typical business. The analysis above is based on my experience with Test automation and RPA.

Why are Agile ceremonies important for Quality Assurance?

As Agile becomes the default approach to software development, it’s easy to start going through the motions without paying too much attention to the underlying rationale and beginning to let some of the steps slip. In my experience as a Test Automation lead working with a variety of Triad clients helping them produce higher quality software efficiently, Agile meetings or “ceremonies” are a key part of successful Agile development.

In this article, I’m going to run through the rationale for them and then focus in on the different aspects that maximise the Quality Assurance aspect of a development project, although doing these well will undoubtedly benefit the entire project.

Perhaps Agile ceremonies’ loftiest and more critical role is that they should bring transparency to the team and set a common goal and vision. These ceremonies help the team solicit feedback, assess their progress, and align what they are doing with the needs of the client. These ceremonies should ensure that everyone in the team are in-sync, making it important for the whole team including Business Analysts/product owners, the development team (developers and testers), delivery managers/scrum masters and stakeholders are involved. Agile ceremonies play a key role in improving the product and process quality – everyone should believe and commit to that and deliver their own participation.

The Agile ceremonies that are particularly important for Quality Assurance:

Backlog refinement meeting

Backlog refinement is the act of adding detail, estimating, and prioritising the items in the Product backlog. This is an ongoing process in which the Product owner, stakeholders and the development team collaborate to steer the product in a way that delivers the most value as early as possible.

During the product backlog refinement, items are reviewed, revised and acceptance criteria are updated depending on the common understanding of the requirements. It is advised to conduct the refinement meetings on a regular basis, ideally every two weeks. This helps to improve the quality of the requirements and to have a prioritised product backlog ready for development.

In Agile Scrum the backlog refinement needs to be done before the team goes into the Sprint planning meeting. This gives an opportunity to verify that the user stories meet the ‘definition of ready’.

A ‘definition of ready’ is the set of criteria agreed amongst the team in order for the story to be considered ready for development. A definition of ready could stipulate for example, that the story should contain a proper description and a summary, that acceptance criteria are complete, relevant screenshots or wireframes have been attached, and business rules have been fully documented.

The team can also do Bug triage as part of this meeting and add any bugs into the backlog depending upon the priority. Bug triage is a process where each bug is prioritised based on its severity, frequency, risk, etc.

In my experience, most large-scale issues that arise in software development projects start due to poor or gaps in requirements, refinement meetings are used to improve the quality of the requirements.

Sprint planning meeting

The product owner, stakeholders and development team meet and review the prioritised backlog. The team discuss each user story to get a clear understanding about how to implement the user story.

The Sprint planning meeting happens before every Sprint and the goal of Sprint planning is to answer the questions “What are we going to work on, and how we are going to do it?” The team creates a Sprint backlog with the list of items from the prioritised product backlog that they plan to work on during the specific Sprint.

The development team will add more technical detail about the implementation of the user stories and create sub-tasks such as implementing backend changes, front-end changes, creating automation test scripts, etc. under each user story so that everyone in the team know what they need to do in order to complete that user story.

The planning meeting helps the team to understand the requirements and to know what they need to do in order to implement the requirement. The most successful projects take prioritisation, with all views considered, extremely seriously. When it comes to the QA phase of a project if just the technical agenda has been reflected cracks and change control issues often arise. So, again active engagement throughout the project is key to ensuring a high-quality output and smooth QA process.

Daily stand-up meeting

A time boxed meeting (ideally 10 to 15 minutes) happens every day where the product owner, stakeholder and the development team meet to give an update of their work. Each person provides highlights of the previous day, planned work for the day, and any blockers that may hinder them, their team or the delivery.

It’s called a stand-up because the team stands the whole time, encouraging the meeting to be speedy as it is time-boxed to 15 minutes. The stand-up meetings are typically held in the same location and at the same time each day.

Daily stand-up meeting helps to get transference in the team and to see how the team is progressing to meet the sprint goal.

Sprint review / show and tell / demos

It is advised to have regular Sprint reviews, often called show & tell sessions. These should be held at the end of every sprint (typically every two weeks) to showcase what has been achieved so far, what is likely to be the goal for the next phase, and to highlight any risks/issues. This meeting is focussed on the product, rather than the process and the Product owner, stakeholders and development team should all participate.

These show and tells, give an opportunity for the development team to demonstrate the work they have completed so far and to get feedback from stakeholders and end users. The feedback from this meeting can be useful for the team to reprioritise the product backlog, add new requirements into the product backlog, and to alter and align the teams working process to fulfil the client needs.

Retrospective meeting

This is the meeting where the development team and product owner get together to discuss how the last phase of development was handled. This focussed on the process, rather than the product. The retrospective will be used to gather thoughts on how the sprint went, and then discuss actions to improve the team’s welfare/ways of working/process etc.

The team should identify what they felt went well (celebrating success), what didn’t go so well (what are the opportunities), and what they can do to improve the process next time around (Even Better If).

The retrospective is an important mechanism that allows a team to continuously evolve and improve throughout the life of a project. During this meeting efforts are made by the team to find effective solutions to problems and to develop action plans.

Other stakeholders ‘may’ be invited to the retrospective, but it is unconventional to have ‘management’ attend. This is because it is a meeting for the team to discuss issues in a safe environment without being led by management or feel held back from expressing concerns.

Conclusion

Agile ceremonies are an important part of Agile development. In Agile, requirements and solutions evolve through collaboration between self-organised cross functional teams and stakeholders. This makes it important to have a regular interaction between QAs, developers, product owner, stakeholders and the end users through daily stand-ups, planning meetings, backlog refinement meetings, regular demos and retrospectives.

These ceremonies bring a lot of clarity into the requirements and a better understanding about the work that is expected from the team. Conducting these ceremonies correctly will help the team to improve the quality of the process and the product, constantly.

In my experience, most large scale issues that arise in projects start when these meetings are not conducted effectively and when the participants in these meetings have skipped or not fully engaged. As such, it is worth reflecting on how well your Agile ceremonies are running and ensuring everyone understands the different purposes they each perform plus feel equipped to participate fully.

How to integrate Quality Assurance into Agile projects

The biggest challenge in Agile testing is to know how to integrate testing into the Agile process. In this article I will present how I have successfully integrated testing into the Agile artefacts.

What is Agile software development?

Agile software development refers to software development methodologies based around the idea of iterative development, where requirements and solutions evolve through collaboration between self-organising cross functional teams and stakeholders. The focus of the Agile manifesto is “Continuously delivering working software while allowing for and supporting changing requirements”.

More and more organisations are adopting Agile processes due to the benefits of faster application development cycles and quicker turnaround. However, shorter and faster development cycles are generally questioned for quality, hence it is important to integrate testing into the process rather than as an activity that occurs at the end of the development phase. That is where Quality Assurance (QA) comes in.

Quality throughout the development cycle

QA activities need to be integrated into the entire development cycle in order to assure the quality occurring in the software. Quality must be on the minds of all and built into the team’s day-to-day activities rather than occurring at the end. It should be built into project processes from initial requirements gathering, through to development and testing activities.

Quality in User Stories

A user story is a description of a software feature from an end user perspective. It describes the type of user, what they want and why. A user story helps to create a simplified description of the requirement.

User stories should take account of INVEST principles (but not necessarily be rigidly held to them as not all stories will satisfy all criteria, but they should satisfy most).

Importantly, A user story may be refined many times before being implemented. A good user story should contain details such as a description, Acceptance criteria, background material and relevant screenshots or wireframes.

Acceptance criteria are the conditions that have to be fulfilled in order to mark the story as “Done”, which is exposed to users, stakeholders and the team. They contain different scenarios explaining the workflow of the feature and can be written using BDD Gherkin syntax following “Given, When, Then” format. The acceptance criteria should cover the positive (happy path) and negative scenarios. It is also advised to include any specific non-functional requirements that are associated with the user story.

Definition of ready

A Definition of ready is the set of criteria agreed amongst the team in order for the story to be considered ready for development. A definition of ready could stipulate for example that the story should contain a proper description, acceptance criteria are complete, relevant screenshots or wireframes have been attached, and business rules have been fully documented.

Definition of done

Definition of Done (DoD) is a shared understanding of what it means for a user story to be complete. Each Agile team will have its own DoD checklist of features and activities. Basically, a team agrees on a list of criteria which must be met before product increment can be considered “done”.

The idea of DoD is that it ensures everyone on the team knows exactly what is expected of everything the team delivers. It ensures transparency and quality fit for the purpose of the product and organisation.

Here is the checklist for definition of done:

Coding is completed for the presumed functionality
Peer code review performed
Project builds without errors
Unit tests written and passing
Integration tests written and passing
Project deployed to a test environment
Testing performed and issues resolved
Accessibility tests passed
Feature meets the acceptance criteria
Project documentation updated
Relevant automation test scripts created, peer reviewed and checked into version control
Tested on devices/browsers listed in the project assumptions

Each team should collaborate and come up with the definition of done that suits its unique environment.

Documentation

It is always useful to maintain project documents such as User research out comes, Technical implementations, Requirements, Service standards, etc. in a central repository such as Confluence or SharePoint. This documentation can be used for knowledge sharing and can create a living document for the team.

Rest APIs can be designed and documented with tools such as Swagger. Swagger files can also be used to validate the API endpoints.

Source code should be self-explanatory. Whilst comments serve to document the code, it is advised to use comments only where it adds value and to avoid liberally commenting code that is already readable.

Manual and Automation test scripts can be written using BDD Gherkin syntax following “Given, When, Then” format which can be used as a living document for the application.

Test Environment

It is advised to have the test environments setup and deployments tested even before the developers start writing code. This ensures that there is an environment available to test the code once the code is developed. You can have different environments such as local, Integration (QA or System test environment), Staging (pre-production) and Production.

Local environments are used by the developers and testers to run the application in their local machines for development and testing.

The Integration environment is where the code is deployed through Continuous Integration / Continuous Deployment (CI/CD) pipelines for System testing. This environment is used by the QA team to perform manual testing and to run automated Integration (API) tests and Smoke tests ideally every time a code change is pushed to the remote repository.

A staging environment is similar to a production environment setup used by QA teams for manual and automated Regression testing, Performance/Load testing and Security testing.

Ticket workflow in a Sprint or Kanban

Ticket workflow is an assortment of statuses, which are used to track the progress of a ticket. The below table illustrates the ticket workflow that can be used in Agile Scrum model or in Kanban model. This workflow board can be created in any project management tool like Jira or Azure DevOps.

To Do

In Dev

Blocked

Ready for code review

In code review

Awaiting QA

In QA

Done

To Do: This is the first state, indicating it is ready to be picked up by a developer. The user story in this state should meet the definition of ready.

In Dev: This state indicates that the feature is under development.

Blocked: This state indicates that the respective story is being blocked and cannot proceed, for example if further information is required to progress or a dependency exits.

Ready for code review: Once the feature is developed, the developer moves the ticket to this state indicating that the ticket is ready for review by a peer.

In code review: The ticket which is currently under peer review is moved to this state with the assignee specified. If there are any comments, then relevant comments are posted against the ticket and assigned back to the appropriate developer.

Once the review comments are fixed, the ticket is moved to “Ready for code review“, then moved to “In code review” by a peer. Finally if it has passed the review, then the code is approved by the peer and merged, ready for QA.

Awaiting QA: Tickets in this state are ready for testing.

In QA: Tickets in this state will be picked by the QA team and undergo thorough manual testing. If the acceptance criteria are not met, then it has failed QA and is moved back to “To Do” and assigned back to the developer with relevant comments, steps to reproduce the issue, relevant screenshots and expected behaviour.

After fixing any defects, the ticket undergoes all the above steps and is retested.

You can follow the Exploratory Testing technique for manual testing.

Done: After the ticket has passed QA, it is moved to “Done” with any relevant comments, and closed. In principal, a feature should be moved to “Done” by, or in collaboration with the Product Owner to indicate the Product Owner has accepted this feature as complete (as opposed to the development team just saying it is complete). Depending on the project team structure, an additional status ‘Ready for PO review’ could be introduced.

Exploratory Testing

Exploratory testing is about discovery, investigation and learning. As the name suggests, it’s about testers exploring the application to identify potential edge cases.

Exploratory testing is the simultaneous process of test design and test execution. It is also complementary to test automation; that is while automated checks are checking for regression issues, exploratory testing focuses on new features which have been developed.

Test Automation

The major challenge for QAs on any Agile projects is the amount of regression testing that needs to be performed before each release; this can also be addressed using automation tests.

In order to get quick feedback, automated tests need to be executed as often as possible, ideally after every code change that is pushed to the repository. Tests need to be fast, delivering consistent and reliable results.

It is important to prepare a test automation strategy that is result-oriented and fits well with the team and organisation as a whole. You can read more about creating an affective Automation test strategy in my blog post “How to get ROI from Test Automation in Agile projects”.

Communication

No matter how good the process is, if there is a lack of communication amongst the team members then it is difficult to produce a high quality product. Co-locating the team in a shared workspace will improve face-to-face communication and helps to build trust among the team members. Wherever co-locating is not possible you can use team collaboration tools such as Skype, Slack, Microsoft Teams or similar.

In Agile where requirements and solutions evolve through collaboration between self-organising cross functional teams and stakeholders. This makes it important to have a regular interaction between QAs, developers, the BA/Product Owner and stakeholders through daily stand-ups, planning meetings, backlog refinement meetings, regular demos and retrospectives.

Conclusion

Your team can enjoy the benefits of the Agile development process when you produce a high-quality product frequently and repeatedly. To achieve this, the QA process needs to be integrated into the development cycle and should be in the minds of all team members. It should be part of the team’s day-to-day work rather than something that occurs at the end of development. It is also equally important to have effective communication amongst the team to improve the quality of the product.

10 Best tools for REST API Testing

What is API/Web Service Testing?

Application Programming Interface (API) is a set of rules and mechanisms by which one application or component interacts with another. API can return data in a convenient format such as JSON, XML, etc. APIs for web applications are often referred as Web services.

Testing these APIs directly as part of integration testing to determine if they meet the expectations for functionality, reliability, performance, and security is known as API Testing.

API testing is performed at the business layer and can validate application logic. This is different from browser based Graphical User Interface (GUI) testing. API testing doesn’t concentrate on the look and feel of an application.

Why is API testing important?

From past few years the software development process has evolved using micro services and CI/CD pipelines to support frequent release of the working software to the end users. Hence increasing the use of Integration testing (API testing). According to Google trends, the interest in API testing has increased considerably from last five years.

Automated API testing is one of the crucial elements for implementing testing in Continuous Integration/Continuous Deployment (CI/CD) environment.

How to test REST API?

API testing is sending different requests to the API server and verifying the responses from the server. API testing requires a tool/framework that can send requests to the API and display the response from server.

API testing tools

There are number of open-source and commercial tools available for API testing. The below mind map shows the 10 best tools for REST API testing.

Having the right process, tools and solutions for API automation testing are more critical than ever. Finding the perfect tool is a tough job. It is important to carefully consider the project requirements, pros and cons of each tool. It is a good idea to try and create a POC with more than one tool from your shortlist. This gives you a better knowledge of the tool and it’s suitability for your project.

RESTful Web Service Testing for beginners

What is API/Web Service Testing?

Testing these APIs directly as part of integration testing to determine if they meet the expectations for functionality, reliability, performance, and security is known as API Testing. API Testing can be performed manually and also with automation tools. Automated API testing is crucial for implementing testing in Continuous Integration/Continuous Deployment (CI/CD) environment.

API testing is performed at the business layer and can validate application logic. This is different than browser based Graphical User Interface (GUI) testing.

What are RESTful services?

Representational Stage Transfer (REST) is an approach and an architectural style that defines a set of constraints to be used for creating Web services. The word “Representational” means that a representation of the resources is transferred i.e. when a client request a resource, the server do not sent the actual resource, but rather a representation of that resource in a format that is readable by the client. The resources can be a list of users, pictures, documents, videos, posts, web pages, etc. that can be transferred through web. REST relies on a stateless communications protocol, most commonly, Hypertext Transfer Protocol (HTTP). REST can structure data in HTML, XML or YAML formats; however JSON is the most widely used data structure in RESTful services.

The web services that implement REST architecture are typically referred as RESTful services.

What does RESTful web services do?

RESTful services performs 4 basic operations through HTTP protocol:

Retrieving data – Retrieve data from the given URI
Create new data – Create a new entry in the database
Update data – Update existing entities in the server
Delete data – Delete a given resource at the specified URI

These operations are also called as CRUD (Create, Read, Update and Delete).

HTTP requests

In REST architecture, the client makes HTTP request to the server in order to retrieve or modify data. A HTTP request generally consists of:

A HTTP request method, which defines what kind of operation to perform.
A HTTP header, that allows the client to pass additional information about the request.
A request target. This is usually a Uniform Resource Locator (URL) of the resource.
An optional request body containing data to be sent to the server. The request body is a structured data such as XML, YAML or JSON. GET and DELETE request methods usually don’t need a message body.

HTTP request methods

The request method indicates the operation to be performed on the resource identified by the given request URI. The method should always be given in uppercase. The HTTP request methods are generally called HTTP verbs.

GET – GET method is used to retrieve data from server at the specified resource/URI. GET should only extract data and should have no other effect on the data.

POST – POST method is used to create new entity. It can also be used to send data to the web server to create or update a resource (e.g. customer information, file upload, etc. using HTML forms). The POST request requires a URI, header and a message body.

PUT – PUT method is used to send data to the API to create or update resource. The difference is that PUT requests are idempotent, meaning calling the same PUT request multiple times will always produce the same result. The PUT request requires a URI, header and the message body.

DELETE – DELETE method is used to delete the resource at the specified URI.

HTTP Response

When a client sends a valid HTTP request to the server, the server sends back the HTTP response relates to that request. The aim of the response is to inform the client that its request has been processed; or else inform the client that an error occurred in processing its request.

An HTTP response contains:

Status code
HTTP headers
A response body

Status code

The HTTP response from the server contain status code to inform the client about the status of its request. There are many status codes available which are grouped in five categories:

Category	Description	Example
1xx: informational	Communicates transfer protocol-level information	100 continue
2xx: Success	Indicates that the client’s request was processed successfully	200 OK 201 Created 204 No Content
3xx: Redirection	Indicates that the client must take some additional action in order to complete their request	300 Multiple choice 302 Found
4xx: Client Error	Indicates that there is an error in the request from the client	400 Bad Request 401 Unauthorized 403 Forbidden
5xx: Server Error	Indicates that there is a problem at the server side	500 Internal Server Error 501 Not Implemented 502 Bad Gateway

HTTP Headers

The response from the server contain different types of HTTP headers. The headers contains information that a client can use to find out more about the response, and about the server that sent it. These headers also contains information about the response body such as ‘Content-type’, ‘Content-length’, ‘Date’, etc.

Response body

For a response to a successful request, the response body contains the resource requested by the client (like content of an HTML form or a document, etc.) or some information about the status of the request. Not all responses have a body. The ‘content type’ and ‘content length’ of the body are specified in the HTTP header of the response.

For a response to an unsuccessful request, the response body might provide further information about the reasons for the error.

How to test REST API?

API testing is sending different requests to the API server and verifying the responses from the server. This is different from browser based User Interface (UI) testing. API testing requires a tool/framework that can send requests to the API and display the response from server.

There are number of open-source and commercial tools available for API testing. Postman, SoapUI, Katalon Studio and Advanced REST client are the popular open-source tools for API testing. Swagger is an open-source tool for API documentation and can also be used to validate the API endpoints.

Testing REST API with Postman

To get started, download and install latest version of Postman.

Create and send HTTP Request

You can launch the Postman from the start menu or the desktop shortcut. Once the Postman is launched you can start creating the request by clicking on the new tab with + icon.

Select GET request from the request method dropdown
Enter the request-URL (http://httpbin.org/anything) in the URL fields
Click “Send” button

Verify HTTP Response

Once a successful GET request is sent to http://httpbin.org/anything, the server has sent back the HTTP response. The response contains Status-code, Header and Response body.

The status code is set to 200 OK, which means that the request was processed successfully.

The response headers contains information about the response body such as:

Content-Type: application/json – indicates that the data in the response body is in JSON format.
Date – date and time the message was sent.
Content-Length: 279 – indicates the size of the data in the response body

The response headers also contains other information that is required by the client to process and display the response to the user.

The response body contains the resource requested by the client from the given URL. In this case the response body is some data presented in JSON format.

Read more about using Postman for API testing here.

API Testing Mind map

Happy testing!

Using the Agile Testing Quadrants to create your Test Strategy

What is the Agile Methodology?

To achieve this it’s important to create a Test strategy that is result-oriented and that fit well with the team and organisation as a whole. There is no one-size-fit-all when it comes to the strategy, so you need to tailor it to each particular project/application.

In this article I intend to introduce the Agile Test Quadrants that are defined by Brain Marick, which can be used as a starting point for your Agile test strategy.

What are Agile Testing Quadrants?

Quadrant 1 – Technology-facing tests that support the team

Quadrant 1 consists of test cases that are technology driven and are performed to support the development team. Q1 is associated with Automation testing with focus on code quality, and covers tests such as Unit tests, Integration/API tests and Component tests. These tests are fast to execute, easy to maintain and modify, and gives rapid feedback to the development team about the quality of the code. The tests in this Quadrant plays an important role when you want to develop an application in a Continuous Integration and Continuous Deployment (CI/CD) environment. Some example frameworks and tools used in this quadrant are Junit, Nunit, Xunit, RestSharp, RestAssured, Jenkins, Visual Studio, Eclipse, etc.

Quadrant 2 – Business-facing tests that support the team

Quadrant 2 consists of tests that are business driven and are performed to support the development team. Q2 is associated with Automation & Manual testing with focus on requirements, and covers tests such as Functional, Examples, Story tests, Prototypes and Simulations. Skilled testers can collaborate with the stakeholders and clients to understand the business requirements and use that knowledge to validate the product. Tools and frameworks such as BDD Cucumber, Specflow, Selenium, Protractor, etc. can be used.

Quadrant 3 – Business-facing tests that critique the product

Quadrant 3 consists of business facing, system or user acceptance level tests such as Exploratory testing, Scenario based testing, Usability Testing, User Acceptance testing, demos and Alpha/Beta Testing. Mostly manual testing methods are used to evaluate the application based on the user requirements. The skilled testers can pair with the customers to perform User Acceptance testing.

Quadrant 4 – Technology-facing tests that critique the product

Quadrant 4 consists of technology-driven test cases that critique the product. This quadrant focuses on the non-functional requirements such as Performance, Load, Stress, Scalability, Security, Reliability, Maintainability, Compatibility, etc. Automation tools such as Jmeter, Taurus, Blazemeter, BrowserStack, OWASP ZAP, etc. can be used.

These quadrants helps the teams to plan their testing. There are no hard and fast rules about what goes in what quadrant and no mandate to follow the quadrants in a sequential manner as explained above. A tester may start from any of the quadrants as per their requirements, priority, risks and their own choice. You can always customise the quadrants according to your project and team requirements and can act as a guide to build your Agile Test Strategy.

A practical guide for Web Accessibility Testing

What is web accessibility?

Web accessibility means that websites, tools and technologies are designed and developed so that people with disabilities can use them. Web accessibility encompasses all disabilities that affect access to the web, including vision, hearing, cognitive, learning, neurological, physical/motor and speech, as well as neurodiverse conditions such as Dyslexia, Autism, Developmental Coordination Disorder (DCD) and more…

Why accessibility matters?

The web has become an essential part of society, but 1 in 5 people have some form of disability that makes it more difficult for them to work in a digital world than the rest. It has, therefore, become increasingly important to make your web applications accessible by all, including people with disabilities.

Web accessibility also benefits people without disabilities; such as people using different devices, people with changing abilities due to ageing, temporary disabilities, situational limitations such as in bright sunlight or in an environment where they cannot listen to audio, and people with a slow internet connection or limited bandwidth.

Web content Accessibility Guidelines (WCAG) 2.1 design principles

The WCAG 2.1 are an internationally recognised set of recommendations for improving web accessibility. They explain about how to make digital services, websites and apps accessible to everyone, including users with impairments to their:

Vision – like severely sight impaired (blind), slight impaired (partially sighted) or colour blind people.
Hearing – like people who are deaf or hard of hearing
Mobility – like those who find it difficult to use a mouse or keyboard
Thinking and understanding – like people with dyslexia, autism or learning difficulties

WCAG 2.1 design principles

WCAG 2.1 is based on 4 design principles:

Perceivable – the information and user interface components must be presented to the user in a way that they can recognise and use the service with the senses that are available to them
Operable – the users can find and use the content, regardless of how they choose to access it (e.g. using keyboard or voice commands)
Understandable – the users can understand the content and how the service works
Robust – the content can be interpreted reliably by a wide variety of user agents (e.g. assistive technologies, different browsers etc.)

Web Accessibility testing techniques

Here are the accessibility testing techniques for different types of disabilities.

Vision impairment

There are many degrees of visual impairments ranging from difficulty in reading small characters through to complete blindness and colour blindness. Here are the testing tools and techniques for testing web application for vision impairment.

Test making text larger.
Test that magnifying the screen (Zoom) should have a linear layout.
Text should have sufficient colour contrast with the background (greater than 4.5:1 for WCAG 2.1 level AA). Use tools such as WCAG Color contrast checker or similar.
Test with Keyboard only navigation
- Using the tab, enter, and space bar, navigate the page and ensure each input and interaction can be triggered.
- The keyboard focus is never trapped in a loop.
- Tab order is logical and follows the visual order of elements on the page.
- Check that the focus is always visible when moving through the page with the tab key.
- Model dialog boxes need to trap the keyboard. When a modal dialog box is triggered, the keyboard focus need to immediately move to the first actionable element in the modal.
- Check that the focus never goes to elements that won’t be available to somebody using a mouse.
Check that all form inputs have explicit labels and descriptions. These attributes are used by Screen readers.
Check that all relevant images use an ‘img’ tag.
Check that all images have ‘alt’ attributes.
Test using screen readers like JAWS, NVDA, ChromeVox, VoiceOver of mac.
Speech recognition tools such as Dragon Naturally Speaking.
Colour blindness simulator such as Color blinding chrome extension.

Hearing impairment

Hearing disabilities include mild to moderate hearing impairment in one or both ears. It includes hard of hearing or deafness. Here are the testing techniques for hearing impairments.

Subtitles/captions are provided for the video content.
The content is in simple English.
Text transcripts of audio content are provided.
Check that accuracy of captions.
Check that the captions are synchronised with the audio.
Summary of audio and video content is provided.
Check that the audio doesn’t play automatically.

Cognitive, learning and neurological disabilities

The functional cognitive disabilities can range from memory, problem solving, attention, reading, linguistic and verbal, math and visual comprehensions. Here are the testing techniques for cognitive disabilities.

Check that clear instructions to navigate through the website are provided
Colour contrast for foreground and background (greater than 4.5:1 for WCAG 2.1 level AA)
Simple user journeys.
Progress indicators are provided.
Error messages should be explanatory and should be in line with the fields.
Elements are highlighted appropriately.
Clear headings are provided
- Identify visual ‘heading’ elements.
- Check that all visual ‘heading’ elements use an ‘<h>’ tag.
- Verify that all sub heading elements have a higher number.

Physical or motor disabilities

Physical or motor disabilities are weakness and limitations of muscular control. Here are the testing techniques:

Use of speech recognition software like Dragon Naturally Speaking.
All content on the page should be Keyboard accessible
- All content must be keyboard accessible
- Tab order must be logical
- No keyboard traps
Headings should be used to create a logical outline of the page to allow for quick navigation to page sections.
Test with alternative keyboards.
Provide shortcuts for filling up the forms (e.g. postcode lookup).

Speech impairments

Speech disabilities include the inability to produce speech that recognisable to other people or software. The testing techniques are mostly similar to vision and hearing disabilities.

Automation Testing

Axe-core is the accessibility engine for automated testing of HTML based User Interface (UI) of a website for accessibility violations. There are different implementations available for Axe such as Axe chrome extension, Axe firefox extension, Axe-webdriverjs, Reack-axe, Axe-selenium-java, Axe-selenium-csharp, Protractor-accessibility plugin.

I have also created an Accessibility Empathy Lab at Triad Group PLC for Accessibility testing of Web applications. You can read more about that in my blog post – Creating Triad’s Accessibility Empathy Lab.

Do you do any accessibility testing in your project, if so how do you approach it?

How to get ROI from Test Automation in Agile projects

What levels of automation do we need to support Continuous Integration/Continuous Deployment (CI/CD) pipelines?
How can we get a return on investment (ROI) from our automation testing?
How do you implement automation testing in agile projects?
Why are our automation tests flaky and keep failing?
How should I choose an automation testing tool for my project?
These are some common questions I get asked by organisations I work with.

In this article, I will share some key things that needs to be considered when creating a test automation strategy to support CI/CD pipelines and deliver a quick ROI from automation testing in Agile projects.

Why use automation testing?

In Agile software development small teams work on product increments and deploy to production as often as possible. In continuous deployment test automation becomes even more important due to its ability to provide quick feedback to the development team on the health of the application. In order to get quick feedback automated tests need to be executed as often as possible, ideally after every code change that is pushed to the repository. Tests need to be fast, delivering consistent and reliable results.

The major challenge for QAs on any agile project is the amount of Regression testing that needs to be performed before each release; this can be addressed using automated tests.

Prepare a test automation strategy

The test automation strategy should be result-oriented and must fit well with the team and organisation as a whole. There is no one-size-fits-all when it comes to this strategy, so you need to tailor it to each particular project/application, ensuring that it serves the intent and helps to attain the team’s objectives.

It’s a good idea to consider the team size and environments available for testing while preparing a test automation strategy.

Define objectives

It’s important to define objectives for your test automation strategy. Well defined objectives will help you set up parameters to measure its success. The objectives could be different – get fast feedback for the developers, reduce testing efforts in later sprints, enhance confidence in the code through test coverage (Unit, Integration, User Interface (UI), Performance, etc.), faster time to market, cost-effective testing, reusable tests, support continuous integration and much more.

Select suitable automation tools

There are many commercial and free open-source test automation tools available on the market. Choice of the tool will depend on your testing objectives and team requirements.

Here are some key parameters to consider when shortlisting a tool:

Compatibility of the tool with the project tech stack (e.g. support for containerisation)
Internal and external support for the tool (Example implementation, blog posts, etc.)
Users of the tool (QA, dev, PM, etc.)
Easy interface to create and maintain tests
Support for different testing techniques (BDD, TDD, etc.)
Support for different testing levels (Unit, Integration, UI)
Support for running tests in headless mode (To speed up execution of UI automation tests)
Cost of licensing, maintenance and training (open source vs paid)

Create automation sub-tasks

Create test automation sub-tasks under the User Stories so that the automation test scripts can be implemented at the end of the current iteration or at the beginning of the next. In addition, this will help in tracking the progress of automation tests and the test coverage.

Build automation tests

Here are a few things to be considered when it comes to creating automation test scripts:

Reusable and maintainable test scripts

Reusability is one of the key reasons for considering test automation. It is important to consider this aspect in your automation test strategy. Reuse of test scripts will make the process cost-effective and speeds up the automation process. It is important to create a reusable framework that can be optimised for testing across other device platforms or for other projects. Due consideration should be given to concerns like maintainability and execution time.

Automation Test pyramid

As part of the test automation strategy, we need to implement the automation test pyramid to increase the test levels.

Automated Unit Testing

Unit tests are written by developers for any new feature that is developed. Test automation starts at the unit level. These tests will test each testable unit separately.
Unit tests can be run on the developer’s local environment as well as the CI/CD environment.
Unit testing provides the greatest ROI for the team as they are very quick to run, easy to maintain and modify, and give quick feedback to the developer about the quality of the code.

Automated API / integration testing

Integration testing forms the next level up from unit tests to test the functionality of each service (API) independently. These tests are executed at the API layer independently of the User Interface (UI).
Mocks and stubs will be used to factor out the dependence on other services or 3^rd party systems.
These tests are executed once Unit tests are run and passed.
Integration tests can be created by both Developers and Automation Testers.
These tests are fast to execute as they do not depend on UI and can be run in local environments and in the CI/CD environment.

User Interface (UI) automation testing

The UI automation tests cover typical user flows, journeys, and end-to-end scenarios.
These automation tests simulate a user’s interaction with the application through the UI which provides good and meaningful scenarios, however it is prone to issues such as: a dependency on html locators, slow page load-times, limited testing, etc. So, it is a good strategy to keep the number of these tests to a minimum.

Version control

It’s a good idea to employ the same version control (Github, Bitbucket, etc.) tools as the developers, to stay in line with the project tech stack. All unit tests and integration tests can live in the main repository with the application. UI automation tests may be better held in a separate remote repository as these tests depend on third party tools.

Run automation tests in CI/CD pipelines

A CI/CD pipeline helps to automate steps in the software delivery process, such as initiating code builds, running automated tests, and deploying to different environments such as Integration (System test), Staging (Pre-production) and Production.

A pipeline procedure is triggered when code is committed to a repository hosted somewhere like GitHub or Bit bucket. This pipeline can build the application, execute unit tests and report the results.

Once unit tests have run successfully it is deployed to an integration (system test) environment where automated integration (API) tests and UI smoke tests can be executed. Once all tests have passed then a build will be triggered to deploy it to the staging (pre-production) environment where a full set of regression tests are run.

Report results

Even with good test scripts it is difficult to find bugs through automation if there is no adequate reporting available. Clear and comprehensive reporting helps us to reach a conclusion after script execution has completed. The reporting format might be different for each tool but should be easily understood. It is also helps to include screenshots in test reports. There are many third party reporting tools such as Allure Test Report, etc. are available in the market.

Conclusion

The main challenge for the QAs on any agile project is the amount of regression testing that needs to be performed before each release. This can be addressed using automation testing.

Your test automation strategy should be integrated into your Software Development Life Cycle (SDLC) and should include different levels of testing such as unit, integration and UI.

Think about the test automation pyramid while creating your test scripts. Creating more UI tests will make your automation slow, it will become out dated quickly, and be difficult to maintain in a long run.

Test automation is a long term investment. You can get a quick ROI when you run your test scripts in CI/CD pipelines as part of the release process. You can see the benefits of automation testing when you select the tools that will fit into the project tech stack and the way the team works. One should think about reusability and maintainability while developing test scripts and should have clear and comprehensive reports.