A practical guide for Web Accessibility Testing

What is web accessibility?

Web accessibility means that websites, tools and technologies are designed and developed so that people with disabilities can use them. Web accessibility encompasses all disabilities that affect access to the web, including vision, hearing, cognitive, learning, neurological, physical/motor and speech, as well as neurodiverse conditions such as Dyslexia, Autism, Developmental Coordination Disorder (DCD) and more…

Why accessibility matters?

The web has become an essential part of society, but 1 in 5 people have some form of disability that makes it more difficult for them to work in a digital world than the rest. It has, therefore, become increasingly important to make your web applications accessible by all, including people with disabilities.

Web accessibility also benefits people without disabilities; such as people using different devices, people with changing abilities due to ageing, temporary disabilities, situational limitations such as in bright sunlight or in an environment where they cannot listen to audio, and people with a slow internet connection or limited bandwidth.

Web content Accessibility Guidelines (WCAG) 2.1 design principles

The WCAG 2.1 are an internationally recognised set of recommendations for improving web accessibility. They explain about how to make digital services, websites and apps accessible to everyone, including users with impairments to their:

  • Vision – like severely sight impaired (blind), slight impaired (partially sighted) or colour blind people.
  • Hearing – like people who are deaf or hard of hearing
  • Mobility – like those who find it difficult to use a mouse or keyboard
  • Thinking and understanding – like people with dyslexia, autism or learning difficulties

WCAG 2.1 design principles

WCAG 2.1 is based on 4 design principles:

  1. Perceivable – the information and user interface components must be presented to the user in a way that they can recognise and use the service with the senses that are available to them
  2. Operable – the users can find and use the content, regardless of how they choose to access it (e.g. using keyboard or voice commands)
  3. Understandable – the users can understand the content and how the service works
  4. Robust – the content can be interpreted reliably by a wide variety of user agents (e.g. assistive technologies, different browsers etc.)

Web Accessibility testing techniques

Here are the accessibility testing techniques for different types of disabilities.

Vision impairment

There are many degrees of visual impairments ranging from difficulty in reading small characters through to complete blindness and colour blindness. Here are the testing tools and techniques for testing web application for vision impairment.

  • Test making text larger.
  • Test that magnifying the screen (Zoom) should have a linear layout.
  • Text should have sufficient colour contrast with the background (greater than 4.5:1 for WCAG 2.1 level AA). Use tools such as WCAG Color contrast checker or similar.
  • Test with Keyboard only navigation
    • Using the tab, enter, and space bar, navigate the page and ensure each input and interaction can be triggered.
    • The keyboard focus is never trapped in a loop.
    • Tab order is logical and follows the visual order of elements on the page.
    • Check that the focus is always visible when moving through the page with the tab key.
    • Model dialog boxes need to trap the keyboard. When a modal dialog box is triggered, the keyboard focus need to immediately move to the first actionable element in the modal.
    • Check that the focus never goes to elements that won’t be available to somebody using a mouse.
  • Check that all form inputs have explicit labels and descriptions. These attributes are used by Screen readers.
  • Check that all relevant images use an ‘img’ tag.
  • Check that all images have ‘alt’ attributes.
  • Test using screen readers like JAWS, NVDA, ChromeVox, VoiceOver of mac.
  • Speech recognition tools such as Dragon Naturally Speaking.
  • Colour blindness simulator such as Color blinding chrome extension.

Hearing impairment

Hearing disabilities include mild to moderate hearing impairment in one or both ears. It includes hard of hearing or deafness. Here are the testing techniques for hearing impairments.

  • Subtitles/captions are provided for the video content.
  • The content is in simple English.
  • Text transcripts of audio content are provided.
  • Check that accuracy of captions.
  • Check that the captions are synchronised with the audio.
  • Summary of audio and video content is provided.
  • Check that the audio doesn’t play automatically.

Cognitive, learning and neurological disabilities

The functional cognitive disabilities can range from memory, problem solving, attention, reading, linguistic and verbal, math and visual comprehensions. Here are the testing techniques for cognitive disabilities.

  • Check that clear instructions to navigate through the website are provided
  • Colour contrast for foreground and background (greater than 4.5:1 for WCAG 2.1 level AA)
  • Simple user journeys.
  • Progress indicators are provided.
  • Error messages should be explanatory and should be in line with the fields.
  • Elements are highlighted appropriately.
  • Clear headings are provided
    • Identify visual ‘heading’ elements.
    • Check that all visual ‘heading’ elements use an ‘<h>’ tag.
    • Verify that all sub heading elements have a higher number.

Physical or motor disabilities

Physical or motor disabilities are weakness and limitations of muscular control. Here are the testing techniques:

  • Use of speech recognition software like Dragon Naturally Speaking.
  • All content on the page should be Keyboard accessible
    • All content must be keyboard accessible
    • Tab order must be logical
    • No keyboard traps
  • Headings should be used to create a logical outline of the page to allow for quick navigation to page sections.
  • Test with alternative keyboards.
  • Provide shortcuts for filling up the forms (e.g. postcode lookup).

Speech impairments

Speech disabilities include the inability to produce speech that recognisable to other people or software. The testing techniques are mostly similar to vision and hearing disabilities.

Automation Testing

Axe-core is the accessibility engine for automated testing of HTML based User Interface (UI) of a website for accessibility violations. There are different implementations available for Axe such as Axe chrome extension, Axe firefox extension, Axe-webdriverjs, Reack-axe, Axe-selenium-java, Axe-selenium-csharp, Protractor-accessibility plugin.

I have also created an Accessibility Empathy Lab at Triad Group PLC for Accessibility testing of Web applications. You can read more about that in my blog post – Creating Triad’s Accessibility Empathy Lab.

Do you do any accessibility testing in your project, if so how do you approach it?

How to get ROI from Test Automation in Agile projects

What levels of automation do we need to support Continuous Integration/Continuous Deployment (CI/CD) pipelines?

How can we get a return on investment (ROI) from our automation testing?

How do you implement automation testing in agile projects?

Why are our automation tests flaky and keep failing?

How should I choose an automation testing tool for my project?

These are some common questions I get asked by organisations I work with.

In this article, I will share some key things that needs to be considered when creating a test automation strategy to support CI/CD pipelines and deliver a quick ROI from automation testing in Agile projects.

Why use automation testing?

In Agile software development small teams work on product increments and deploy to production as often as possible. In continuous deployment test automation becomes even more important due to its ability to provide quick feedback to the development team on the health of the application. In order to get quick feedback automated tests need to be executed as often as possible, ideally after every code change that is pushed to the repository. Tests need to be fast, delivering consistent and reliable results.

The major challenge for QAs on any agile project is the amount of Regression testing that needs to be performed before each release; this can be addressed using automated tests.

Prepare a test automation strategy

The test automation strategy should be result-oriented and must fit well with the team and organisation as a whole. There is no one-size-fits-all when it comes to this strategy, so you need to tailor it to each particular project/application, ensuring that it serves the intent and helps to attain the team’s objectives.

It’s a good idea to consider the team size and environments available for testing while preparing a test automation strategy.

Define objectives

It’s important to define objectives for your test automation strategy. Well defined objectives will help you set up parameters to measure its success. The objectives could be different – get fast feedback for the developers, reduce testing efforts in later sprints, enhance confidence in the code through test coverage (Unit, Integration, User Interface (UI), Performance, etc.), faster time to market, cost-effective testing, reusable tests, support continuous integration and much more.

Select suitable automation tools

There are many commercial and free open-source test automation tools available on the market. Choice of the tool will depend on your testing objectives and team requirements.

Here are some key parameters to consider when shortlisting a tool:

  • Compatibility of the tool with the project tech stack (e.g. support for containerisation)
  • Internal and external support for the tool (Example implementation, blog posts, etc.)
  • Users of the tool (QA, dev, PM, etc.)
  • Easy interface to create and maintain tests
  • Support for different testing techniques (BDD, TDD, etc.)
  • Support for different testing levels (Unit, Integration, UI)
  • Support for running tests in headless mode (To speed up execution of UI automation tests)
  • Cost of licensing, maintenance and training (open source vs paid)

Create automation sub-tasks

Create test automation sub-tasks under the User Stories so that the automation test scripts can be implemented at the end of the current iteration or at the beginning of the next. In addition, this will help in tracking the progress of automation tests and the test coverage.

Build automation tests

Here are a few things to be considered when it comes to creating automation test scripts:

Reusable and maintainable test scripts

Reusability is one of the key reasons for considering test automation. It is important to consider this aspect in your automation test strategy. Reuse of test scripts will make the process cost-effective and speeds up the automation process. It is important to create a reusable framework that can be optimised for testing across other device platforms or for other projects. Due consideration should be given to concerns like maintainability and execution time.

Automation Test pyramid

As part of the test automation strategy, we need to implement the automation test pyramid to increase the test levels.

Automated Unit Testing

  • Unit tests are written by developers for any new feature that is developed. Test automation starts at the unit level. These tests will test each testable unit separately.
  • Unit tests can be run on the developer’s local environment as well as the CI/CD environment.
  • Unit testing provides the greatest ROI for the team as they are very quick to run, easy to maintain and modify, and give quick feedback to the developer about the quality of the code.

Automated API / integration testing

  • Integration testing forms the next level up from unit tests to test the functionality of each service (API) independently. These tests are executed at the API layer independently of the User Interface (UI).
  • Mocks and stubs will be used to factor out the dependence on other services or 3rd party systems.
  • These tests are executed once Unit tests are run and passed.
  • Integration tests can be created by both Developers and Automation Testers.
  • These tests are fast to execute as they do not depend on UI and can be run in local environments and in the CI/CD environment.

User Interface (UI) automation testing

  • The UI automation tests cover typical user flows, journeys, and end-to-end scenarios.
  • These automation tests simulate a user’s interaction with the application through the UI which provides good and meaningful scenarios, however it is prone to issues such as: a dependency on html locators, slow page load-times, limited testing, etc. So, it is a good strategy to keep the number of these tests to a minimum.

Version control

It’s a good idea to employ the same version control (Github, Bitbucket, etc.) tools as the developers, to stay in line with the project tech stack. All unit tests and integration tests can live in the main repository with the application. UI automation tests may be better held in a separate remote repository as these tests depend on third party tools.

Run automation tests in CI/CD pipelines

A CI/CD pipeline helps to automate steps in the software delivery process, such as initiating code builds, running automated tests, and deploying to different environments such as Integration (System test), Staging (Pre-production) and Production.

A pipeline procedure is triggered when code is committed to a repository hosted somewhere like GitHub or Bit bucket. This pipeline can build the application, execute unit tests and report the results.

Once unit tests have run successfully it is deployed to an integration (system test) environment where automated integration (API) tests and UI smoke tests can be executed. Once all tests have passed then a build will be triggered to deploy it to the staging (pre-production) environment where a full set of regression tests are run.

Report results

Even with good test scripts it is difficult to find bugs through automation if there is no adequate reporting available. Clear and comprehensive reporting helps us to reach a conclusion after script execution has completed. The reporting format might be different for each tool but should be easily understood. It is also helps to include screenshots in test reports. There are many third party reporting tools such as Allure Test Report, etc. are available in the market.

Conclusion

The main challenge for the QAs on any agile project is the amount of regression testing that needs to be performed before each release. This can be addressed using automation testing.

Your test automation strategy should be integrated into your Software Development Life Cycle (SDLC) and should include different levels of testing such as unit, integration and UI.

Think about the test automation pyramid while creating your test scripts. Creating more UI tests will make your automation slow, it will become out dated quickly, and be difficult to maintain in a long run.

Test automation is a long term investment. You can get a quick ROI when you run your test scripts in CI/CD pipelines as part of the release process. You can see the benefits of automation testing when you select the tools that will fit into the project tech stack and the way the team works. One should think about reusability and maintainability while developing test scripts and should have clear and comprehensive reports.