Continuous Testing for delivering Continuous Value
Traditionally, testing used to happen in an environment called “TEST” environment where the testers will be testing all the features developed, run automated tests, do manual and exploratory testing and identify issues. Identified issues were fixed on priority and then release it to “STAGE” or “UAT” environment for Stakeholder and/or Product-Owner’s sign-off where User-Acceptance testing (UAT) will be carried out. Once signed-off, application was released to “Production”.
In early 2010’s concept of Shift-Left came meaning carrying out early-testing during development phase. Let me explain how to do this shift-left and shift-right testing and can we go further or extreme left and right. Also, how it is important for Continuous Testing.
When someone says shift-left and shift-right, we assume or need to say we are at middle/centre which I think is TEST environment where testers used to test and sign-off. In this environment testing should include all the Automated Functional, Automated Non-functional and Exploratory testing. Non-Functional testing will be ALL that is applicable to the application. Largely, Security and Performance testing is involved. Automated testing is carried out for asserting our expectations or checking the application for known behaviours. Exploratory testing for uncovering new information or covering unknowns in all areas of testing, for e.g. In functional — Unit, Integration and UI level, in non-functional — Security, Performance, Accessibility and many other depending on the application.
Shift-right into an environment typically is STAGE or UAT where the concerned stakeholder/product owner will sign-off, by carrying out User-Acceptance testing. Also, why not Shift-right and do all the testing here as well, like in test environment? Do all the automated functional and non-functional testing along-with exploratory testing to ensure quality and making sure application works as expected.
Shift-left testing is mainly doing early testing during the development phase i.e., in development environment — where code is written and build. Why not and do all the testing here as well, like in test environment? Run all the automated functional and non-functional testing while the code is being written to identify early issues during development. Pair with developer and do a pair-development (feature developed and tested at the same-time). Also, do exploratory testing session for identifying unknowns and/or anything else is getting impacted by the code changes. This way high-quality code will be built and will reduce cascading down the unwanted behaviours to later stages/environments.
Can we go further left? Yes, Why Not? We can do testing in Requirement or Planning phase. During requirement analysis or User-Story writing we can do testing by Example Mapping, refining Acceptance-Criteria, Asking Questions and Challenging Requirements. This will clear all the doubts/assumptions and everyone will have better and shared understanding about the User Story. Also, not to forget the purpose of testing here which will ensure that the right-product is built. Asking questions like what is the purpose of this feature/user-story, why we need to build this, is there any better way of doing it and so on.
Example Mapping can be done as explained in Specification by Example by Gjoko Adzic.
Can we go further right? Can we test after Stage or UAT which is Testing in Production — PROD environment?
Of course, We Can! Do all your testing — automated functional and non-functional, User-Acceptance Testing, Exploratory. Also, Analytics and Monitoring for continuous feedback and improvements. Testing needs to be carried-out in a controlled way without affecting End-user’s data.
As Charity Majors in her “Testing in Production” talks describe doing observability instead of monitoring, about unknown-unknowns instead of known-unknowns. We can’t monitor everything just concentrate on critical paths and do observability to discover unknown-unknowns.
In Summary, Continuous testing right from requirement phase to production and taking feedback from production to requirement, this loop of continuous testing is key to deliver CONTINUOUS VALUE. Continuous Testing will ensure right-product is built during requirements phase and highest-quality product delivered and feedback taken for continuous improvement delivering CONTINUOUS VALUE.
Below figure depicts how Continuous testing can be applied to DevOps:
