Test Pyramid as a Risk Filter
Introduction
Based on Mike Cohn’s, there are so many forms and interpretations of Test Automation Pyramid. All point out the same thing it should have more number of unit tests, followed by service (consisting of API, Integration, Component) tests and finally UI tests.
There’s this another way of looking Test Pyramid as a Bug-Filter.
But none mention about non-functional testing. Does that mean non-functional testing are not automated or part of test automation? For me test automation stands for — tests which are automated, run by a tool by executing test-scripts and asserted pass or fail depending on our expectations.
At least it should say Functional Test Automation Pyramid. But I can see functional test automation is the most widely implemented form of automation and it has kind-of become synonymous with test automation.
Also, Manual & Exploratory Testing bit is not captured properly. It has just been added at the end. Exploratory Testing can happen at any level.
I will now explain how I view this test pyramid or rather triangle. So here is my interpretation.
Software Testing is the process of identifying RISKS and not about finding BUGS. Based on this, Imagine the test pyramid to be upside down as a risk filter identifying risks for the application.
Risk Filter consists of Functional Testing, Security Testing, Performance Testing, Other types of testing (Visual UI, Accessibility, A/B, Usability and many more) and Manual/Exploratory testing. These are the balanced test model to achieve quality.
Functional Testing consists of Unit, Service (Component, Integration, API — Application Programming Interface) and UI (User Interface) testing. Other testing can be Visual UI, Accessibility, A/B testing, usability testing, and many other types of testing.
Newly developed application or feature enters into the risk filter with the purpose of identifying risks — both Functional and Non-functional, meaning carrying out functional tests in UI, Service and Unit level and identifying functional risks. Similarly Performance testing for identifying Performance risks, Security testing for Security risks, likewise Other types of testing identifying Other risks. So all these risks identified are by automation — asserting our expectations or checking the KNOWN behaviours. These are our known or identified risks.
What about Unknowns/new information? These will be identified by Exploratory Testing where the application/feature will be explored. So all that remains in the filter or pyramid is Manual/Exploratory testing identifying unknown risks in all the areas of testing, for e.g. in functional (including Unit, API, UI), Security, Performance and Others. Manual/Exploratory testing can be done at each and every level and also in various types of testing. Say for example at unit-level trying out negative test-data, at API level too we can do exploratory testing, similarly in case of security testing writing SQL queries to identify vulnerabilities, likewise in case of accessibility testing manually checking the correct tab-flow etc. and so on.
Do we need to fix all of the identified risks? Absolutely NOT. Team (Developer/QA/BA) and Product Owner/Stakeholder will prioritise it on a scale of say high, medium, low etc. and fixed as per the priority.
Once all the prioritised risks are fixed then the Stakeholder/Product Owner can release the application to production with KNOWN risks.
Similarly, Functional Test Pyramid will look like below. The automated part and manual/exploratory part are relative. As one increases or decreases the other will be impacted. By removing the lines this test pyramid becomes flexible (automated part can expand and/or shrink) and will suit any kind of applications.
Risk Filter applied to different Applications or Industries
This filter is agile in the sense depending upon application some of the testing will require more thorough testing compared to other types and hence can be expanded or stretched accordingly, vice versa if less testing is required then it can contract or shrink. Functional Test, Performance, Security, and Other testing depending upon the application will get allocated part of the filter accordingly and all that remains in the filter is manual and exploratory testing to complete the balanced test model.
Let’s look at some applications and see how this filter model looks and works.
Health Care Application
Where security of the patient information is more important. Performance might not be of that importance. The filter for that application will look something like this:
Banking/Finance Application
Where both security and performance is important — both will be of equal size. Filter for Banking application will look like this:
Gaming Application
Performance testing might take precedence compared to security. The filter will look like this:
Some applications might not require any Other testing. In that case filter is flexible and will look like this (Other testing can be removed):
