In my experience working as a Senior QA consultant, I have seen many organisations using the term Quality Assurance (QA) and Testing interchangeably. The main reason seems to be that the project management and key stakeholders are not aware of the differences between Quality Assurance and Testing.
However, there are some significant differences and, as I explain them in this article, I’ll also point out why it matters that your team and wider business understand them accurately.
Quality Assurance
Quality Assurance serves to ensure that appropriate processes and procedures are in place (and are being adhered to) that enable the delivery of a product or service to an agreed level of quality. Quality Assurance is a much wider topic than Testing because it covers more than just the outputs of software delivery (the end product), it also covers the inputs (how the product is being developed), in order to improve the likelihood of a positive outcome.
QA is a proactive process that works out ways to prevent possible bugsin the process of software development. QA is integrated in a software development lifecycle (SDLC), and it requires the whole project team to be involved including Stakeholders, Business Analysts, Developers and Testers. The QA process helps to improve the productivity of the project team by specifying and establishing the requirements for both software development process and quality standards.
In Agile projects, Quality Assurance is led by Quality Analysts (QAs) to incorporate QA activities throughout the development lifecycle.
The first part in a properly planned and executed Quality Assurance process is the Quality assurance strategy. Itdefines the approach and activities to be carried out during software development to achieve the defined quality of the application under development. QA should have a holistic view of the project while creating the QA strategy and must address all aspects with respect to software projects from requirements capture to development.
A well-defined QA strategy should consider the following areas;
Governance, financial reporting and stakeholder engagement and risk management
Project team skill assessment and training requirements
Communication and collaboration
Methodologies
Document control
Requirements gathering process and definition of non-functional requirements
Application / Service Architecture
Testing strategy
Test Environments (e.g. QA, Staging, UAT and production)
Continuous integration and continuous delivery (CI/CD pipelines)
Version control and branching strategy
Design standards and reviews
Coding standards, code quality checks and reviews
Quality Control
Next up is Quality Control (QC) – a set of activities used to ensure quality in a product or a service. The goal of QC is to ensure a proper implementation of the processes defined in Quality Assurance strategy. The QC activities deal with examination of quality of the “end product” compared to QA that deals with the process to develop the product.
The main focus of QC is to validate that the product meets the specifications and requirements of the customer. If an issue or problem (bug / defect) is identified, it needs to be fixed before the product is delivered to the customer. It is a reactive process that helps to confirm that the product works as expected.
Quality Control activities can be performed by Software Testers with specialist testing skills.
Quality Control activities include:
Walkthroughs
Testing
Inspection
Review
Software Testing
As the development process draws to a close, you arrive at Software Testing or simply Testing as one way in which Quality Control can be implemented. Testing involves validating the product against specifications and customer requirements, finding and reporting defects. It includes various testing techniques such as functional, non-functional, acceptance testing to detect software issues. Besides, the goal of the software testing is also to make sure that the detected defects are fully fixed without any side effects before the product is released to the customer.
Testing activities include:
Test planning
Test case design
Test execution
Defect reporting
Test reporting
Conclusion
QA and Testing are not the same concepts – QA is the strategy that encompasses Testing but much more and involves a much wider set of stakeholders. While Testing is focussed on code quality within a technical arena.
However, they do have the same goal; to ensure development and delivery of a high-quality product to the customer. Yet, when understood and implemented correctly, they focus on different things and use different methods and techniques to reach that goal.
Being aware and observant of these differences enables businesses to create a better understanding of ‘quality’ across a development team and while improving productivity from a wider set of different skills.
It is critical to choose the right test automation tool if you want a quick return on investment from your automation tests. Yet, selecting the right test automation tool is frequently challenging, especially for applications like Dynamics 365.
The Dynamics applications are somewhat complex web applications that contain concepts such as “configurableelements”, “nested iFrames” and “dynamiclocators”. These concepts make it difficult to use conventional test automation tools such as Selenium WebDriver.
In this article, I will discuss different tools that are available for automated testing with Dynamics CRM applications.
Organisations have different business processes that come in many shapes and sizes. Dynamics 365 is flexible and can be configured according to the organisation’s requirements, yet this flexibility comes with risks that need to be considered, such as:
Do these new configurations meet the customer’s specifications?
Have these changes introduced regression issues to the existing system?
Have we documented the process correctly?
Does the application still work as expected after Microsoft updates?
Manual testing helps identify issues in new configurations, and automated tests are useful in identifying regression issues.
Testing also helps to enhance the usability of the Dynamics 365 application being configured, which hopefully translates into happy stakeholders all round – from developers to end users.
Why use test automation?
Regression testing is a repetitive task of re-running existing tests to ensure that previously developed and tested applications still perform as expected after a change or an update. It is a very time-consuming process if performed manually.
Test automaton is used to speed-up the regression testing. Automated tests are version controlled, reusable and can be executed anytime as part of CI/CD pipelines or as scheduled jobs. Business readable automation tests written in BDD Gherkin syntax can be used to document application behaviour.
Dynamics 365 test automation challenges
As mentioned above, Dynamics applications are relatively complex web applications that consist of “configurable elements”, “dynamic locators” and “nested iFrames”. It is important to choose an automation tool that can handle such applications and features.
The good news is that there are certainly tools available in the market.
Test automation tools for Dynamics applications
Here are the four best test automation tools that are compatible with Dynamics applications. In no specific order;
Microsoft EasyRepro
EasyRepro was created by Microsoft to provide Dynamics customers with the ability to facilitate automated UI testing for their organisation. It provides an easy to use set of commands that make setting up UI tests quick and easy. The functionality provided covers the CRM commands that end users would perform on a typical workday, and Microsoft are working to extend that coverage to more functionality.
EasyRepro is built from the open source Selenium WebDriver, which is used widely within the test automation industry. The EasyRepro framework is open source and available on GitHub. It can also execute automation tests in headless browser mode.
EasyRepro supports .NET framework and is available as a NuGet package.
Executive Automats is a no-code automated testing suite for Dynamics. It is a cross-platform regression testing tool designed and developed to increase MS Dynamics 365/CRM deployments speed. It also works with other web-based applications such as SharePoint, SAP, Outlook.
Executive Automats contains features for assertions, validations, try-catch, conditional checks, loops, negative testing, scheduler and reporting. It supports on-premise installation and Azure DevOps integrated API.
You can also record user actions and parameterise them to create automation tests. Executive Automats can also be used for performance testing of Dynamics CRM applications.
LEAPWORK is a no-code automation platform that makes it easy to build and maintain test automations. It contains drag and drop components to create automation tests.
The LEAPWORK automation platform uses Selenium as an engine for automating websites and web applications, but all the Selenium code for automating tests is generated ‘under the hood’. The tester does not, at any point, have to worry about reading or writing a single line of code.
The LEAPWORK components (building blocks) can be used to automate form fields of any kind in Dynamics applications.
Model a system under test as BPMN-style (Business Process Modelling Notation) flowcharts.
Automatically generate test cases from the model, optimising testing for time and risk.
Define test data at the model level and generate data at the same time as test cases.
Export test automation scenarios, automatically generating coverage-focused automation scripts.
Export test cases to test case management tools such as Jira, ALM, Octane, etc.
Analyse test results and manage existing artefacts, with visual dashboards and a file management system that introduces traceability between test assets.
Automated tests can be executed across open source, commercial or homegrown frameworks. You can build flowcharts using a range of drag and drop importers and accelerators, and automatically generate optimised test cases.
Test Modeller supports many different programming languages and automation frameworks, covering web apps, mobile apps, API testing, performance testing and database validation.
TestModeller.io also provides a UI Recorder, as an extension to Google chrome, Firefox and Internet explorer.
The UI Recorder automatically captures tester activity on the browser and imports it into Test Modeller. These recorded tests can be executed on Google chrome, Firefox, Microsoft Edge and Safari.
For Dynamics 365 this means creating models that automatically generate C# code, which can be plugged directly into an automation framework for Dynamics 365, such as EasyRepro.
The table below provides a comparison of the Dynamics 365 automation tools based on key features:
Features
Microsoft EasyRepro
Executive Automats
LEAPWORK
TestModeller.io
Licence Type
Open source
Paid
Paid
Paid
Test development platform
Windows (requires .NET framework)
Cross-platform
Windows
Cross-platform
Application under test
Dynamics 365
Dynamics 365, SharePoint, SAP, Outlook and other web applications
Dynamics 365, desktop, Citrix, web and SAP applications
Dynamics 365, web and mobile applications
Coding skills required
Yes
No
No
Yes
Programming languages
C# .NET
No-code automation
No-code automation
C# .NET, Java, JavaScript
Learning curve
High
Medium
Medium
High
Continuous integration
Popular CI/CD tools (e.g. Jenkins, Azure DevOps)
Azure DevOps (VSTS)
Popular CI/CD tools
Popular CI/CD tools
Installation type
Nuget package added to unit test project
SaaS / Cloud, On-premise
Cloud, On-premise
Web-Based / Cloud / SaaS
Multi-browser support
Yes
Yes
Yes
Yes
Type of testing
UI functional testing
UI functional and performance testing
UI functional testing
UI functional, API and performance testing
Record user actions
No
Yes
Yes
Yes
Conclusion
It is always important to choose the right tool for the type of application you are automating. As I have shown, there are a number of commercial and open source test automation tools available for Dynamics 365, but their features vary considerably. Dynamics 365 is a complex application to automate, due to the nature of its configurable elements. However, knowing the automation tool options available for such applications gives you a head start when developing test automation frameworks.
How understanding stakeholders optimises your testing strategy and the software you end up with.
In my experience working as a Senior QA consultant at Triad, I have seen many organisations consider Quality Assurance (QA) only as an additional cost in the project and ignore the intrinsic value that it brings to software quality.
The main reason seems to be that the project managers and key stakeholders are not be aware of the benefits of QA and the value it adds to the development team.
In this article, I will explain why it is important to know who your stakeholders are and how to optimise your testing strategy to include the interests of these stakeholders.
Because better engagement leads to a better testing strategy that results in better software.
Quality Assurance in Agileprojects
In Agile Software development, it is important to create a test strategy that is results-oriented and fits well with the team and organisation as a whole. In Agile, Quality Assurance is about integrating testing into every stage of the software development life cycle, hence the testing strategy should focus on achieving this.
As with other aspects of software development, to create an effective test strategy you need to know who your stakeholders are. You need to understand their requirements, interests, concerns and goals at the outset, and be prepared to accommodate change. In turn, this will make buy-in from the wider team easier to secure, and they will be interested in the testing outcomes.
Who are the tester’s stakeholders?
Most commonly they are those people who have an interest in the outcome of tests and the information that QAs provide: developers, designers, product owners, project managers, users and sponsors.
Developers
Developers are the people who build the product. These stakeholders need to know where their product fails so they can fix defects [close to their source]. For them, it is important to identify issues early in the development process, because the cost of remediation increases over time, and the later defects are found the less flexibility a project has to respond to them.
The test strategy should contain good development practices like feature branching, pull requests, peer reviews and quality measures such as unit test coverage, integration test coverage, automated smoke/regression tests, performance tests, security tests and code quality checks. These measures provide a safety net giving developers greater confidence to make changes in the application.
Designers
Designers are responsible for designing the User Interface (UI) and User experience (UX) of a product. These stakeholders also need to know where their design fails so they can fix the issues. It is important to identify any design issues as early as possible, typically before the development starts.
The test strategy should include testing techniques such as design reviews, usability testing, accessibility testing, etc. to address the interests of these stakeholders.
Product owners and Business Analysts (BAs)
Product owners and BAs may work together to develop user stories and discuss priorities. They are interested in understanding whether a feature meets their needs and expectations. The QA team should work collaboratively with the POs and BAs to understand the requirements (user stories) and effectively communicate them to the wider team.
A well-defined user story should include background, acceptance criteria and non-functional requirements for that feature. The test strategy should contain activities to review that the user stories are testable and to create acceptance tests to verify that the implemented feature meets the acceptance criteria.
Project Management
These stakeholders need to know the status of deliverables (i.e. application features): are they available, acceptable, and reasonably free of defects? If there is a problem, the manager may need to re-plan or at least manage expectations.
It is useful to include the test deliverables (i.e. test scenarios, test cases, requirement traceability matrix (RTM)), schedules and quality measures in your QA strategy to prove the quality of the application being developed.
Sponsors
These stakeholders need evidence that the system can support their business goals and that the risk of failure is acceptable. It is important to understand the business goals and product risks and include them in the test strategy.
Demonstrate an understanding of the business goals and risks in your testing strategy, it will also help you in defining the right test approach.
Users
These stakeholders need evidence that the system will work for them. As a QA, you should always think as an end-user.
Your test strategy should address real-time use case scenarios such as testing in different browsers, operating systems, devices, usability testing and accessibility testing.
Conclusion
As QAs we need to embrace that we are accountable to stakeholders; you need to engage, consult, and communicate with a range of stakeholders and reflect their needs into your testing strategy. Testing approaches should focus towards meeting their needs at all times.
However, even once the testing strategy is agreed and being put into practice, for QA’s, it is important to communicate effectively with those stakeholders. Sometimes, this will take the form of documentation, shared wikis, Kanban or Scrum boards. If your test strategy addresses the concerns of the stakeholders, then they will feel involved and will support the QA team during critical times.
The best test strategies are the result of exploration, thinking and collaboration.
For this project, we wanted to strengthen our in-house penetration testing (pen test) capability to enable us to prove the security of our web application from the outset, rather than having to wait for the results of our independent pen test towards the end development. Being relatively new to penetration testing, we wanted to choose a tool that was easy to setup and could find as many vulnerabilities as possible. Having considered several free and paid tools, we chose OWASP Zed Attack Proxy (ZAP) due to reasons given above and expanded on below.
In this article, I will demonstrate how to setup and use OWASP ZAP to test the security of a typical web application.
Before I continue, I feel obligated to warn you that you should use this tool only with an application you’re hosting yourself, or one you’ve been given explicit permission to test, as ZAP attempts to modify data and insert malicious scripts in the web application.
What is OWASP?
The Open Web Application Security Project (OWASP) is an open, online community that creates methodologies, tools, technologies and guidance on how to deliver secure web applications. It is an international collaborative initiative comprised of both individuals and corporations. The project aims to standardise security approaches in web development and spread associated knowledge.
What is OWASP ZAP?
OWASP ZAP (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers. It can help to find security vulnerabilities in web applications. It’s also a great tool for experienced pen testers and beginners.
ZAP can scan through the web application and detect issues related to:
SQL injection
Broken Authentication
Sensitive data exposure
Broken Access control
Security misconfiguration
Cross Site Scripting (XSS)
Insecure Deserialization
Components with known vulnerabilities
Missing security headers
Why we chose OWASP ZAP?
As it is designed to be used by people with a wide range of pen testing experience, it was ideal for our team who were new to penetration testing.
ZAP is a free open-source tool which is easy to setup and use. As it is used by the wider community, there is a lot of help available online through the ZAP blog and other articles to help you setup and use the tool.
ZAP is cross platform i.e. you can install it in Windows, Linux or Mac OS.
ZAP can be run in a Docker container, which suited our project tech stack. Also, its functionality is scalable with many diverse extensions published on GitHub.
ZAP Jenkins plugin can be setup to run the scans as part of CI / CD pipelines.
How it works
ZAP is what is known as a “man-in-the-middle proxy.” It stands between the browser and the web application. While you navigate through all the features of the website, it captures all actions. Then it attacks the website with known techniques to find security vulnerabilities.
As ZAP spiders the web application, it constructs a map of the web applications’ pages and the resources used to render those pages. Then it records the requests and responses sent to each page and creates alerts if there is something potentially wrong with a request or response.
Setting up ZAP
To begin with, you need to download and install OWASP ZAP scanner and set it up correctly. ZAP is platform agnostic so you can install it on Windows, Linux or Mac OS. You need Java 8+ installed on your Windows or Linux system.
For the purposes of this article, I’m going to concentrate on its use on Windows.
Starting ZAP
Once setup you can start ZAP by clicking the ZAP icon on your Windows desktop or from the start menu.
When the app launches, it asks you whether you want to save the session or not. If you want to use the current run configuration or test results later, you should save the session for later. For now let’s select “No, I do not want to persist this session at this moment in time”.
OWASP ZAP start-up dialog
Once you click the “Start” button, the ZAP UI will be launched.
ZAP UI
Spidering the web application
Spidering a web application means crawling all the links and getting the structure of the application. ZAP provides two spiders for crawling web applications;
Traditional ZAP spider: The traditional ZAP spider discovers links by examining the HTML in responses from the web application. This spider is fast, but it is not always effective when exploring an AJAX web application.
AJAX spider: This is more likely to be effective for AJAX applications. This spider explores the web application by invoking browsers which then follow the links that have been generated. The AJAX spider is slower than the traditional spider.
Automated scan
This option allows you to launch an automated scan against an application just by entering the URL. If you are new to ZAP, it is best to start with Automated Scan mode.
To run a Quick Start Automated Scan:
Start Zap and click the large ‘Automated Scan’ button in the ‘Quick Start’ tab.
Enter the full URL of the web application you want to attack in the ‘URL to attack’ text box.
Click the ‘Attack’ button.
ZAP Automated scan window
Once you click the ‘Attack’ button, ZAP will start crawling the web application with its spider and passively scan each page it finds. Then ZAP will use the active scanner to attack all of the discovered pages, functionality and parameters.
Exploring the web application manually
Spiders are a great way to explore the basic site, but they should be combined with manual exploration to be more effective. This functionality is very useful when your web application needs a login or contains things like registration forms, etc.
You can launch browsers that are pre-configured to proxy through ZAP via the Quick Start tab. Browsers launched in this way will also ignore any certificate validation warnings that would otherwise be reported.
ZAP manual explore window
To Manually Explore the web application:
Start ZAP and click on the large ‘Manual Explore’ button in the Quick Start tab.
Enter the full URL of the web application to be explored in the ‘URL to explore’ text box.
Select the browser you would like to use and click the ‘Launch Browser’ button.
This will launch the selected browser with a new profile. Now explore all of the targeted web applications through this browser. ZAP passively scans all the requests and responses made during your exploration for vulnerabilities, continues to build the site tree, and records alerts for potential vulnerabilities found during the exploration.
What is passive scanning?
Passive scans only scan the web application responses without altering them. It does not attack or insert malicious scripts to the web application, so this is a safe scan; you can use it if you are new to security testing. Passive scanning is good at finding some vulnerabilities and as a way to get a feel for the basic security of a web application.
What is active scanning?
Active scan attacks the web application using known techniques to find vulnerabilities. This is a real attack that attempts to modify data and insert malicious scripts in the web application.
Active scans put the application at risk, so do not use active scanning against web applications you do not have permission to test.
Running automated scan againt the web application
Inspecting the test results
Once the scan is completed, ZAP generates a list of issues that are found during the scan. These issues can be seen on the Alerts tab that is located in the bottom pane. All the issues are marked with colour coded flags. You can also generate an HTML scan report through the ‘Report’ menu option on the top of the screen.
ZAP scan report risk categories
Summary
ZAP is a free open source platform agnostic security testing tool that scans through your web application to identity any security vulnerabilities as possible. It is a great tool for experienced pen testers, as well as beginners.
ZAP spiders the web application under test and scan for any known vulnerabilities.
For beginners it is easy to start with Automated Scan that will crawl the given URL with spider and passively scan each page it finds. You can do a more in-depth scanning by exploring the web application manually.
ZAP generates the scan report in the form of Alerts that are marked with colour coded flags. You can even download HTML reports from the “Report” menu option.
ZAP can also be integrated into CI/CD pipeline using ZAP Jenkins plugin.
Often when we talk to Test Automation Engineers about Robotic Process Automation (RPA) the first thing we hear is “It is same as test automation with Selenium or similar tools”. I should confess that this is what I thought of it when I heard about RPA for the first time. However, my view has changed since I started learning RPA with UiPath. In this article, I will present the differences I have noticed between the test automation and robotic process alongside some views on how to maximise the benefit from different aspects.
What is Robotic Process Automation (RPA)?
Robotic Process Automation also known as Process Automation or Intelligent Automation helps in automating repetitive tasks without using human power. RPA is a kind of technology that consists of software bot, or a “robot” to emulate and integrate the actions of a human interacting within digital systems to execute a business process. The RPA bots can do a lot of things such as entering data, logging into applications, accomplishing tasks, reading and sorting emails, log out, etc.
RPA is primarily used in applications that require repetitive actions. Processes that require complex calculations, accuracy and precision can be assigned to RPA robots to get the task done efficiently. RPA robots utilize the user interface to capture data and manipulate applications just like humans do. RPA tools like UiPath incorporate elements of Artificial Intelligence (AI) and can be used to automate any structured data.
In summary, RPA technology is used to reduce or eliminate the need for humans to perform high-volume IT support, workflow, and back-office processes, such as those found in finance, accounting, supply chain management and customer service.
What can RPA tools do?
RPA tools are used by business users to automate any business process or functionality. They can be used to automate the process in web, desktop, mobile applications, mainframe and SAP based applications and also the applications that are running on virtual machines (Citrix based) etc. Unlike test automation tools, RPA is normally used only in production environments.
In my opinion, RPA tools can also be used for test automation, however, this would be an expensive option. RPA tools are more powerful than the conventional test automation tools. It is easy to setup and use RPA tools with no coding experience required, hence, they suited and mostly used by business users. Most of these tools have ready to use features like “Robotic Enterprise Framework” (RE Framework) in UiPath that speeds up process design.
Test automation is used as part of the development lifecycle to help the development team get quick feedback about the quality of their software. It is mostly used by the QA team to perform Regression testing. Automation testing tools like Selenium webdriver are used to test the functionality of a web application through a browser interface during the development phase of an application. Test automation is popular in Agile software development to increase team velocity.
In order to get quick feedback, automated test scripts need to be executed as often as possible, ideally after every code change that is pushed to the repository. You can read more about test automation in my blog post “How to optimise test automation on Agile projects”.
Here are the few differences between RPA and test automation;
Test automation is automating the software product and RPA is automating the business process and repetitive tasks.
Most of the test automation tools help to automate current web pages, unlike RPA tools that can automate all the backend processes.
The automation tests run in test environments such as Integration, Staging, etc. whereas RPA tools are used in production environment.
Test automation tools are used by QA team to get a quick feedback about the quality of the software. RPA tools are used by business users to speed up the processes.
Basic or no coding is required for RPA as it is Wizard-driven process, whereas automation tools like Selenium required proficient coding knowledge.
High domain knowledge is required for RPA compared to test automation.
Test automation is part of a software development lifecycle, whereas RPA has its own development lifecycle (Analysis, development, testing and support & maintenance).
There are a number of opensource test automation tools available in the market. When it comes to RPA tools, UiPath community version is free but the rest are paid-for tools.
Test automation tools have limitations when it comes to the type of application they can support, on the other hand RPA tools can automate any application including web, desktop, Citrix, etc.
Conclusion
Hopefully this article has explained how test automation and process automation tools both have a role in delivering automation solutions. The way I see it, the main difference is the environment in which they are used and the type of applications they can automate.
Test automation is used by QA team to verify the application under development as part of development life cycle, on the other hand RPA is used by business users to automate the repetitive tasks.
RPA uses bots (robots) to emulate the human actions to complete a process. Test automation is applied to verify the quality of the product whereas RPA is applied to the process associated with the business environment. RPA tools can be used for test automation, but it will be an expensive option.
Once this distinction is understood better, I believe that the opportunities for RPA in particular become easier to identify and more numerous across a typical business. The analysis above is based on my experience with Test automation and RPA.
As Agile becomes the default approach to software development, it’s easy to start going through the motions without paying too much attention to the underlying rationale and beginning to let some of the steps slip. In my experience as a Test Automation lead working with a variety of Triad clients helping them produce higher quality software efficiently, Agile meetings or “ceremonies” are a key part of successful Agile development.
In this article, I’m going to run through the rationale for them and then focus in on the different aspects that maximise the Quality Assurance aspect of a development project, although doing these well will undoubtedly benefit the entire project.
Perhaps Agile ceremonies’ loftiest and more critical role is that they should bring transparency to the team and set a common goal and vision. These ceremonies help the team solicit feedback, assess their progress, and align what they are doing with the needs of the client. These ceremonies should ensure that everyone in the team are in-sync, making it important for the whole team including Business Analysts/product owners, the development team (developers and testers), delivery managers/scrum masters and stakeholders are involved. Agile ceremonies play a key role in improving the product and process quality – everyone should believe and commit to that and deliver their own participation.
The Agile ceremonies that are particularly important for Quality Assurance:
Backlog refinement meeting
Backlog refinement is the act of adding detail, estimating, and prioritising the items in the Product backlog. This is an ongoing process in which the Product owner, stakeholders and the development team collaborate to steer the product in a way that delivers the most value as early as possible.
During the product backlog refinement, items are reviewed, revised and acceptance criteria are updated depending on the common understanding of the requirements. It is advised to conduct the refinement meetings on a regular basis, ideally every two weeks. This helps to improve the quality of the requirements and to have a prioritised product backlog ready for development.
In Agile Scrum the backlog refinement needs to be done before the team goes into the Sprint planning meeting. This gives an opportunity to verify that the user stories meet the ‘definition of ready’.
A ‘definition of ready’ is the set of criteria agreed amongst the team in order for the story to be considered ready for development. A definition of ready could stipulate for example, that the story should contain a proper description and a summary, that acceptance criteria are complete, relevant screenshots or wireframes have been attached, and business rules have been fully documented.
The team can also do Bug triage as part of this meeting and add any bugs into the backlog depending upon the priority. Bug triage is a process where each bug is prioritised based on its severity, frequency, risk, etc.
In my experience, most large-scale issues that arise in software development projects start due to poor or gaps in requirements, refinement meetings are used to improve the quality of the requirements.
Sprint planning meeting
The product owner, stakeholders and development team meet and review the prioritised backlog. The team discuss each user story to get a clear understanding about how to implement the user story.
The Sprint planning meeting happens before every Sprint and the goal of Sprint planning is to answer the questions “What are we going to work on, and how we are going to do it?” The team creates a Sprint backlog with the list of items from the prioritised product backlog that they plan to work on during the specific Sprint.
The development team will add more technical detail about the implementation of the user stories and create sub-tasks such as implementing backend changes, front-end changes, creating automation test scripts, etc. under each user story so that everyone in the team know what they need to do in order to complete that user story.
The planning meeting helps the team to understand the requirements and to know what they need to do in order to implement the requirement. The most successful projects take prioritisation, with all views considered, extremely seriously. When it comes to the QA phase of a project if just the technical agenda has been reflected cracks and change control issues often arise. So, again active engagement throughout the project is key to ensuring a high-quality output and smooth QA process.
Daily stand-up meeting
A time boxed meeting (ideally 10 to 15 minutes) happens every day where the product owner, stakeholder and the development team meet to give an update of their work. Each person provides highlights of the previous day, planned work for the day, and any blockers that may hinder them, their team or the delivery.
It’s called a stand-up because the team stands the whole time, encouraging the meeting to be speedy as it is time-boxed to 15 minutes. The stand-up meetings are typically held in the same location and at the same time each day.
Daily stand-up meeting helps to get transference in the team and to see how the team is progressing to meet the sprint goal.
Sprint review / show and tell / demos
It is advised to have regular Sprint reviews, often called show & tell sessions. These should be held at the end of every sprint (typically every two weeks) to showcase what has been achieved so far, what is likely to be the goal for the next phase, and to highlight any risks/issues. This meeting is focussed on the product, rather than the process and the Product owner, stakeholders and development team should all participate.
These show and tells, give an opportunity for the development team to demonstrate the work they have completed so far and to get feedback from stakeholders and end users. The feedback from this meeting can be useful for the team to reprioritise the product backlog, add new requirements into the product backlog, and to alter and align the teams working process to fulfil the client needs.
Retrospective meeting
This is the meeting where the development team and product owner get together to discuss how the last phase of development was handled. This focussed on the process, rather than the product. The retrospective will be used to gather thoughts on how the sprint went, and then discuss actions to improve the team’s welfare/ways of working/process etc.
The team should identify what they felt went well (celebrating success), what didn’t go so well (what are the opportunities), and what they can do to improve the process next time around (Even Better If).
The retrospective is an important mechanism that allows a team to continuously evolve and improve throughout the life of a project. During this meeting efforts are made by the team to find effective solutions to problems and to develop action plans.
Other stakeholders ‘may’ be invited to the retrospective, but it is unconventional to have ‘management’ attend. This is because it is a meeting for the team to discuss issues in a safe environment without being led by management or feel held back from expressing concerns.
Conclusion
Agile ceremonies are an important part of Agile development. In Agile, requirements and solutions evolve through collaboration between self-organised cross functional teams and stakeholders. This makes it important to have a regular interaction between QAs, developers, product owner, stakeholders and the end users through daily stand-ups, planning meetings, backlog refinement meetings, regular demos and retrospectives.
These ceremonies bring a lot of clarity into the requirements and a better understanding about the work that is expected from the team. Conducting these ceremonies correctly will help the team to improve the quality of the process and the product, constantly.
In my experience, most large scale issues that arise in projects start when these meetings are not conducted effectively and when the participants in these meetings have skipped or not fully engaged. As such, it is worth reflecting on how well your Agile ceremonies are running and ensuring everyone understands the different purposes they each perform plus feel equipped to participate fully.
The biggest challenge in Agile testing is to know how to integrate testing into the Agile process. In this article I will present how I have successfully integrated testing into the Agile artefacts.
What is Agile software development?
Agile software development refers to software development methodologies based around the idea of iterative development, where requirements and solutions evolve through collaboration between self-organising cross functional teams and stakeholders. The focus of the Agile manifesto is “Continuously delivering working software while allowing for and supporting changing requirements”.
More and more organisations are adopting Agile processes due to the benefits of faster application development cycles and quicker turnaround. However, shorter and faster development cycles are generally questioned for quality, hence it is important to integrate testing into the process rather than as an activity that occurs at the end of the development phase. That is where Quality Assurance (QA) comes in.
Quality throughout the development cycle
QA activities need to be integrated into the entire development cycle in order to assure the quality occurring in the software. Quality must be on the minds of all and built into the team’s day-to-day activities rather than occurring at the end. It should be built into project processes from initial requirements gathering, through to development and testing activities.
Quality in User Stories
A user story is a description of a software feature from an end user perspective. It describes the type of user, what they want and why. A user story helps to create a simplified description of the requirement.
User stories should take account of INVEST principles (but not necessarily be rigidly held to them as not all stories will satisfy all criteria, but they should satisfy most).
Importantly, A user story may be refined many times before being implemented. A good user story should contain details such as a description, Acceptance criteria, background material and relevant screenshots or wireframes.
Acceptance criteria are the conditions that have to be fulfilled in order to mark the story as “Done”, which is exposed to users, stakeholders and the team. They contain different scenarios explaining the workflow of the feature and can be written using BDDGherkin syntax following “Given, When, Then” format. The acceptance criteria should cover the positive (happy path) and negative scenarios. It is also advised to include any specific non-functional requirements that are associated with the user story.
Definition of ready
A Definition of ready is the set of criteria agreed amongst the team in order for the story to be considered ready for development. A definition of ready could stipulate for example that the story should contain a proper description, acceptance criteria are complete, relevant screenshots or wireframes have been attached, and business rules have been fully documented.
Definition of done
Definition of Done (DoD) is a shared understanding of what it means for a user story to be complete. Each Agile team will have its own DoD checklist of features and activities. Basically, a team agrees on a list of criteria which must be met before product increment can be considered “done”.
The idea of DoD is that it ensures everyone on the team knows exactly what is expected of everything the team delivers. It ensures transparency and quality fit for the purpose of the product and organisation.
Here is the checklist for definition of done:
Coding is completed for the presumed functionality
Peer code review performed
Project builds without errors
Unit tests written and passing
Integration tests written and passing
Project deployed to a test environment
Testing performed and issues resolved
Accessibility tests passed
Feature meets the acceptance criteria
Project documentation updated
Relevant automation test scripts created, peer reviewed and checked into version control
Tested on devices/browsers listed in the project assumptions
Each team should collaborate and come up with the definition of done that suits its unique environment.
Documentation
It is always useful to maintain project documents such as User research out comes, Technical implementations, Requirements, Service standards, etc. in a central repository such as Confluence or SharePoint. This documentation can be used for knowledge sharing and can create a living document for the team.
Rest APIs can be designed and documented with tools such as Swagger. Swagger files can also be used to validate the API endpoints.
Source code should be self-explanatory. Whilst comments serve to document the code, it is advised to use comments only where it adds value and to avoid liberally commenting code that is already readable.
Manual and Automation test scripts can be written using BDD Gherkin syntax following “Given, When, Then” format which can be used as a living document for the application.
Test Environment
It is advised to have the test environments setup and deployments tested even before the developers start writing code. This ensures that there is an environment available to test the code once the code is developed. You can have different environments such as local, Integration (QA or System test environment), Staging (pre-production) and Production.
Local environments are used by the developers and testers to run the application in their local machines for development and testing.
The Integration environment is where the code is deployed through Continuous Integration / Continuous Deployment (CI/CD) pipelines for System testing. This environment is used by the QA team to perform manual testing and to run automated Integration (API) tests and Smoke tests ideally every time a code change is pushed to the remote repository.
Ticket workflow is an assortment of statuses, which are used to track the progress of a ticket. The below table illustrates the ticket workflow that can be used in Agile Scrum model or in Kanban model. This workflow board can be created in any project management tool like Jira or Azure DevOps.
To Do
In Dev
Blocked
Ready for code review
In code review
Awaiting QA
In QA
Done
To Do: This is the first state, indicating it is ready to be picked up by a developer. The user story in this state should meet the definition of ready.
In Dev: This state indicates that the feature is under development.
Blocked: This state indicates that the respective story is being blocked and cannot proceed, for example if further information is required to progress or a dependency exits.
Ready for code review: Once the feature is developed, the developer moves the ticket to this state indicating that the ticket is ready for review by a peer.
In code review: The ticket which is currently under peer review is moved to this state with the assignee specified. If there are any comments, then relevant comments are posted against the ticket and assigned back to the appropriate developer.
Once the review comments are fixed, the ticket is moved to “Ready for code review“, then moved to “In code review” by a peer. Finally if it has passed the review, then the code is approved by the peer and merged, ready for QA.
Awaiting QA: Tickets in this state are ready for testing.
In QA: Tickets in this state will be picked by the QA team and undergo thorough manual testing. If the acceptance criteria are not met, then it has failed QA and is moved back to “To Do” and assigned back to the developer with relevant comments, steps to reproduce the issue, relevant screenshots and expected behaviour.
After fixing any defects, the ticket undergoes all the above steps and is retested.
Done: After the ticket has passed QA, it is moved to “Done” with any relevant comments, and closed. In principal, a feature should be moved to “Done” by, or in collaboration with the Product Owner to indicate the Product Owner has accepted this feature as complete (as opposed to the development team just saying it is complete). Depending on the project team structure, an additional status ‘Ready for PO review’ could be introduced.
Exploratory Testing
Exploratory testing is about discovery, investigation and learning. As the name suggests, it’s about testers exploring the application to identify potential edge cases.
Exploratory testing is the simultaneous process of test design and test execution. It is also complementary to test automation; that is while automated checks are checking for regression issues, exploratory testing focuses on new features which have been developed.
Test Automation
The major challenge for QAs on any Agile projects is the amount of regression testing that needs to be performed before each release; this can also be addressed using automation tests.
In order to get quick feedback, automated tests need to be executed as often as possible, ideally after every code change that is pushed to the repository. Tests need to be fast, delivering consistent and reliable results.
It is important to prepare a test automation strategy that is result-oriented and fits well with the team and organisation as a whole. You can read more about creating an affective Automation test strategy in my blog post “How to get ROI from Test Automation in Agile projects”.
Communication
No matter how good the process is, if there is a lack of communication amongst the team members then it is difficult to produce a high quality product. Co-locating the team in a shared workspace will improve face-to-face communication and helps to build trust among the team members. Wherever co-locating is not possible you can use team collaboration tools such as Skype, Slack, Microsoft Teams or similar.
In Agile where requirements and solutions evolve through collaboration between self-organising cross functional teams and stakeholders. This makes it important to have a regular interaction between QAs, developers, the BA/Product Owner and stakeholders through daily stand-ups, planning meetings, backlog refinement meetings, regular demos and retrospectives.
Conclusion
Your team can enjoy the benefits of the Agile development process when you produce a high-quality product frequently and repeatedly. To achieve this, the QA process needs to be integrated into the development cycle and should be in the minds of all team members. It should be part of the team’s day-to-day work rather than something that occurs at the end of development. It is also equally important to have effective communication amongst the team to improve the quality of the product.
Application
Programming Interface (API) is a set of rules and mechanisms by which one
application or component interacts with another. API can return data in a
convenient format such as JSON, XML, etc. APIs for web applications are often
referred as Web services.
Testing these APIs directly as part of integration testing to determine if they meet the expectations for functionality, reliability, performance, and security is known as API Testing.
API
testing is performed at the business layer and can validate application
logic. This is different from browser based Graphical User
Interface (GUI) testing. API testing doesn’t concentrate on the look and
feel of an application.
Why is API testing important?
From past few years the software
development process has evolved using micro services and CI/CD pipelines to
support frequent release of the working software to the end users. Hence
increasing the use of Integration testing (API testing). According to Google
trends, the interest in API testing has increased considerably from last
five years.
API testing is sending different
requests to the API server and verifying the responses from the server. API
testing requires a tool/framework that can send requests to the API and display
the response from server.
API
testing tools
There are number of open-source and
commercial tools available for API testing. The below mind map shows the 10
best tools for REST API testing.
Mindmap showing top 10 API testing tools
Having the right process, tools and solutions for API automation testing are more critical than ever. Finding the perfect tool is a tough job. It is important to carefully consider the project requirements, pros and cons of each tool. It is a good idea to try and create a POC with more than one tool from your shortlist. This gives you a better knowledge of the tool and it’s suitability for your project.
Application
Programming Interface (API) is a set of rules and mechanisms by which one
application or component interacts with another. API can return data in a
convenient format such as JSON, XML, etc. APIs for web applications are often referred
as Web services.
Testing these APIs directly as
part of integration testing to determine if they meet the expectations for
functionality, reliability, performance, and security is known as API Testing. API
Testing can be performed manually and also with automation tools. Automated
API testing is crucial for implementing testing in Continuous
Integration/Continuous Deployment (CI/CD) environment.
API testing is performed at the business layer and can validate application logic. This is different than browser based Graphical User Interface (GUI) testing.
A typical three tier application
What are RESTful services?
Representational
Stage Transfer (REST) is an approach and an architectural style that
defines a set of constraints to be used for creating Web services. The word “Representational”
means that a representation of the resources is transferred i.e. when a client
request a resource, the server do not sent the actual resource, but rather a
representation of that resource in a format that is readable by the client. The
resources can be a list of users, pictures, documents, videos, posts, web
pages, etc. that can be transferred through web. REST relies on a stateless
communications protocol, most commonly, Hypertext
Transfer Protocol (HTTP). REST can structure data in HTML, XML or
YAML formats; however JSON is the most
widely used data structure in RESTful services.
The web services that implement
REST architecture are typically referred as RESTful services.
What does RESTful web services do?
RESTful services performs 4 basic
operations through HTTP protocol:
Retrieving data – Retrieve data from the given
URI
Create new data – Create a new entry in the
database
Update data – Update existing entities in the
server
Delete data – Delete a given resource at the
specified URI
These operations are also called
as CRUD (Create, Read, Update and Delete).
HTTP requests
In REST architecture, the client
makes HTTP request to the server in order to retrieve or modify data. A HTTP request
generally consists of:
An optional request body containing data to be sent to the server. The request body is a structured data such as XML, YAML or JSON. GET and DELETE request methods usually don’t need a message body.
HTTP request methods
The request method indicates the
operation to be performed on the resource identified by the given request URI. The
method should always be given in uppercase. The HTTP request methods are
generally called HTTP verbs.
GET – GET method is used to retrieve
data from server at the specified resource/URI. GET should only extract data
and should have no other effect on the data.
POST – POST method is used to create new entity. It can also be used to send data to the web server to create or update a resource (e.g. customer information, file upload, etc. using HTML forms). The POST request requires a URI, header and a message body.
PUT – PUT method is used to send data to the API to create or update resource. The difference is that PUT requests are idempotent, meaning calling the same PUT request multiple times will always produce the same result. The PUT request requires a URI, header and the message body.
DELETE – DELETE method is used to
delete the resource at the specified URI.
HTTP
Response
When a client sends a valid HTTP
request to the server, the server sends back the HTTP response relates to that
request. The aim of the response is to inform the client that its request has
been processed; or else inform the client that an error occurred in processing
its request.
An HTTP response contains:
Status code
HTTP headers
A response body
Status
code
The HTTP response from the server contain
status
code to inform the client about the status of its request. There are
many status codes available which are grouped in five categories:
Category
Description
Example
1xx: informational
Communicates transfer protocol-level information
100 continue
2xx: Success
Indicates that the client’s request was processed successfully
200 OK 201 Created 204 No Content
3xx: Redirection
Indicates that the client must take some additional action in order to complete their request
300 Multiple choice 302 Found
4xx: Client Error
Indicates that there is an error in the request from the client
400 Bad Request 401 Unauthorized 403 Forbidden
5xx: Server Error
Indicates that there is a problem at the server side
500 Internal Server Error 501 Not Implemented 502 Bad Gateway
HTTP
Headers
The response from the server contain
different types of HTTP headers. The
headers contains information that a client can use to find out more about the
response, and about the server that sent it. These headers also contains
information about the response body such as ‘Content-type’, ‘Content-length’, ‘Date’, etc.
Response
body
For a response to a successful
request, the response
body contains the resource requested by the client (like content of an
HTML form or a document, etc.) or some information about the status of the
request. Not all responses have a body. The ‘content type’ and ‘content length’
of the body are specified in the HTTP header of the response.
For a response to an unsuccessful
request, the response body might provide further information about the reasons
for the error.
How
to test REST API?
API testing is sending different
requests to the API server and verifying the responses from the server. This is
different from browser based User Interface (UI) testing. API testing requires
a tool/framework that can send requests to the API and display the response
from server.
There are number of open-source and
commercial tools available for API testing. Postman, SoapUI, Katalon Studio and Advanced
REST client are the popular open-source tools for API testing. Swagger is an open-source tool for API
documentation and can also be used to validate the API endpoints.
Testing
REST API with Postman
To get started, download and install latest version of
Postman.
Postman landing page
Create and send HTTP Request
You can launch the Postman from the start menu or the desktop shortcut. Once the Postman is launched you can start creating the request by clicking on the new tab with + icon.
Select
GET request from the request method dropdown
Once a successful GET request is
sent to http://httpbin.org/anything,
the server has sent back the HTTP response. The response contains Status-code, Header and Response body.
The status code is set to 200
OK, which means that the request was processed successfully.
The response headers contains information about the response body such
as:
Content-Type: application/json – indicates that the data in the response body is
in JSON format.
Date – date and time the message was sent.
Content-Length: 279 – indicates the size of the data in the response body
The response headers also
contains other information that is required by the client to process and
display the response to the user.
The response body contains the resource requested by the client from
the given URL. In this case the response body is some data presented in JSON
format.
Read more
about using Postman for API testing here.
Agile software development refers to software development methodologies
based around the idea of iterative development, where requirements and
solutions evolve through collaboration between self-organising cross functional
teams and stakeholders. The focus of the Agile
manifesto is “Continuously delivering working
software while allowing for and supporting changing requirements”.
More
and more organisations are adopting Agile processes due to the benefits of
faster application development cycles and quicker turnaround. However, shorter
and faster development cycles are generally questioned for quality, hence it is
important to integrate testing into the process rather than as an activity that
occurs at the end of the development phase.
To
achieve this it’s important to create a Test strategy that is result-oriented
and that fit well with the team and organisation as a whole. There is no
one-size-fit-all when it comes to the strategy, so you need to tailor it to
each particular project/application.
In
this article I intend to introduce the Agile Test Quadrants that are defined by
Brain Marick, which can be used as a starting point for your Agile test
strategy.
What are Agile Testing Quadrants?
Quadrant 1 – Technology-facing tests that support the team
Quadrant 1 consists of test cases
that are technology driven and are performed to support the development team.
Q1 is associated with Automation testing with focus on code quality, and covers
tests such as Unit tests, Integration/API
tests and Component tests. These tests are fast to execute, easy
to maintain and modify, and gives rapid feedback to the development team about
the quality of the code. The tests in this Quadrant plays an important role
when you want to develop an application in a Continuous
Integration and Continuous Deployment (CI/CD) environment. Some example
frameworks and tools used in this quadrant are Junit, Nunit, Xunit, RestSharp, RestAssured, Jenkins, Visual Studio, Eclipse, etc.
Quadrant 2 – Business-facing tests that support the team
Quadrant 2 consists of tests that are
business driven and are performed to support the development team. Q2 is
associated with Automation & Manual testing with focus on requirements, and
covers tests such as Functional, Examples, Story tests, Prototypes and
Simulations. Skilled testers can collaborate with the stakeholders and clients
to understand the business requirements and use that knowledge to validate the
product. Tools and frameworks such as BDD Cucumber, Specflow, Selenium, Protractor, etc. can
be used.
Quadrant 3 – Business-facing tests that critique the product
Quadrant 3 consists of business
facing, system or user acceptance level tests such as Exploratory testing,
Scenario based testing, Usability Testing, User Acceptance testing, demos and
Alpha/Beta Testing. Mostly manual testing methods are used to evaluate the
application based on the user requirements. The skilled testers can pair with
the customers to perform User Acceptance testing.
Quadrant 4 – Technology-facing tests that critique the product
Quadrant 4 consists of technology-driven test cases that critique the product. This quadrant focuses on the non-functional requirements such as Performance, Load, Stress, Scalability, Security, Reliability, Maintainability, Compatibility, etc. Automation tools such as Jmeter, Taurus, Blazemeter, BrowserStack, OWASP ZAP, etc. can be used.
These quadrants helps the teams to plan their testing. There are no hard and fast rules about what goes in what quadrant and no mandate to follow the quadrants in a sequential manner as explained above. A tester may start from any of the quadrants as per their requirements, priority, risks and their own choice. You can always customise the quadrants according to your project and team requirements and can act as a guide to build your Agile Test Strategy.
