Quality Assurance in a DevOps World

DevOps is a journey.  While many people focus purely on releasing more frequently or adopting new tools, often the more important part of the process is the culture shift required to truly embrace DevOps principles.  In my experience, Quality Assurance (QA) teams can be resistant to these changes.  While it may require a change in how testing is done, enabling continuous testing inside your CI/CD pipelines can free quality assurance engineers up for the tasks they are best suited for.

Continuous Testing

Any effort to create CI/CD pipelines for an organization should include testing.  The most common cases run in these pipelines are unit tests.  These are incredibly valuable and should not be overlooked, but these are not the only tests that can be run continuously.  Any repetitive tests with predictable results are prime targets for automation.  Manually running test cases can be tedious and time consuming.  This can lead to mistakes or important steps being skipped.  More importantly, this is a poor use of a QA engineer’s time.  Repetitive tasks are what computers excel at.  Converting your organization’s integration and acceptance tests can increase confidence in your code while reducing burden on the QA team.

What to Convert?

Saying automating your tests is a good idea is one thing, but it’s unlikely you will be able to convert your manual test cases to automated ones overnight.  When deciding what to convert, you will want to focus on the lowest barrier for entry and the highest impact.  Unit tests are top priority.  They answer the basic question of “does this section of code do what I intended?” and are often already part of development’s process.  Next would be integration tests of the various subsystems to your application.  Even in an application has a user interface there are often APIs, databases, or other systems that do most of the actual logic.  Wrapping these in functional tests can give you confidence in your system without touching the front end.  User interaction automation is often the most difficult to automate but is often the best way to truly end to end tests your application.  These should focus on the most basic end to end tests first, relying on your other tests to cover edge cases.

How to Automate?

There are many tools in the marketplace to help with testing.  Many of them promise to enable automation with limited coding.  Your success may vary, but as a rule the more control over your process you have the better.  With limited set up any programming or scripting language can be effective for testing.  Often it can be efficient to create these in same language your code is written in so developers can be leaned on support for creation and maintenance, but it comes down to your organizations comfort level.  I’ve found PowerShell to be effective tool for setting up quick tests as well which are easily run in any CI/CD pipeline.  For front end tests I would recommend utilizing an open source framework like Selenium or Windows Application Driver.  These have support from the community and give you control of your tests in your own code repository.  Even if you go with a different tool for API or front-end testing, keeping any scripts or project files in source control will give you benefit of version control.  Treating QA as code gives you a single source of truth for effective tests in the same way you have for development code.

Role of the manual tester

Not everyone who works in QA is ready to instantly start coding automated tests.  While training in these skills is worthwhile, there is a time investment involved.  There may be value in enlisting the development team or an automation engineer to set up the initial framework.  However, the veteran QA engineer still has a valuable part to play in this process.  They can be Identifying which tests are the most vital, what areas of the system are the most vulnerable, and where the most time would be saved during this manual process.  When they are involved early in a development effort, they can be identifying the test cases required for new projects and creating the script for any automation that will support it, whether or not they write it themselves.

Most importantly of all is exploratory testing.  Automation is great for performing repetitive tasks.  Where an experienced manual tester shines is in stressing an application in unexpected ways.  These are the people who often know where an application is likely to break, where the holes in other tests might be, and how a user might interact with the application.  That isn’t to say this must be entirely manual either.  While these tests are valuable, any bugs that are found will need steps to reproduce and specifics about what happened.  A tool such as the Test & Feedback extension of Azure DevOps will allow you to record your actions while conducting exploratory testing.  If any issues are found, you can create an exact bug using the information that is recorded.