TL;DR
We spent a great deal of time testing software at various stages over the years. While the fine art of testing has evolved a great deal over time many of the types remain similar or have evolved to keep up and are explained below.
These days nearly all type of testing can and should be automated which makes testing more efficient and reliable. That said not everything can or should be automated and better left to humans that are equipped with brains and eyes and creativity to sneak around where automation cannot go.
While our testing days are behind us, NRM could still assist with bringing any type of testing to your situation. So, if you need to improve your delivery flow with greater confidence in the quality of what you are doing contact us for a quick consultation and we will see what we can do.
Test Categories
These broad categories will likely resonate with test categories that you may have heard of. Chances are good though that there are some additional types of testing that you may not be aware of.
| Category | Examples |
|---|---|
| Functional | Unit, Integration, System, Regression, Acceptance, End-to-End |
| Non-Functional & Experiential | Performance, Load, Security, Usability, Compatibility (user configuration), Accessibility, Last Mile |
| Resilience | Fail-over, Chaos Monkey, Stress (surges, Denial of Service (DoS)), Session Affinity, Load Balance, Cache Mechanisms |
| Specialized/Other | Exploratory, API, Mobile, Alpha, Beta, A/B, Localization, Mutation, Fuzz |
Test Types
The various types of testing that exist are explained below along with where they are typically applied.
| Testing Type | Description | Applies to |
|---|---|---|
| Ad-hoc | Manual unscripted testing to discover defects by exploring the application without specific direction. | A quick sanity test often associated with the last step before release. All other tests have been run, and results are understood but there could still be gaps in testing that this might reveal. |
| Exploratory (Ad-Hoc+) | Tour driven manual ad-hoc made popular by James Whitaker with his book Exploratory Software Testing. | Similar to Ad-Hoc but more fun because effort is influenced by a theme or persona. |
| API | Validates application programming interfaces for correctness and reliability. | Code |
| Mobile App Testing | Focuses on mobile device-specific functionality and user experience. | Broad range of device configurations typically spotty connectivity and small screen sizes. |
| Alpha Testing | Internal testing before public release, often by developers or QA. | A form of Ad-hoc |
| Beta Testing | Real-world testing by a select group of end users before general release. | Real world ad-hoc customer experience or acceptance. Might leverage telemetry and/or feedback loops to capture details. |
| A/B Testing | Compares two versions to determine which performs better. | |
| Localization and Globalization Testing | Ensures proper functionality across languages and regions. | Typically, global apps or those that need to support multiple languages. |
| Mutation Testing | Alters code to check if tests detect the changes. | Code robustness |
| Fuzz Testing | Alters parameters applied to APIs and services to check for error handling and robustness. Typically pass/fail based on number of cycles survived without incident. | API robustness |
| Back-end Testing | Inputs random or unexpected data to test robustness. | |
| Destructive Testing | Intentionally tries to break the software to find failure points. | Ideal for anything where a break could wreak havoc or has serious consequences. |
| Dynamic Configuration Testing | Validates use of feature switches, default states as well as supporting infrastructure and architecture. | Validates capabilities to alter product behaviors without re-releasing. Handy for A/B testing, Beta testing or geopolitical change. |
| Pre-Deployment Testing | Leverages Infrastructure as code concepts to set proper base state configuration often from an image or container then validates it is correct. | Confirms that your base operating environment is what it should be. Could block downstream testing saving time and effort working with a tainted system. |
| Unit Testing | Code level testing designed to defend a unit of code against changes elsewhere. Often used in concert with Pair Programming & Test Drive Development to build more reliable code. | Reveals breaking changes as close as possible to the source. Often required to run all unit tests prior to final check-in. |
| Last mile | A form of configuration testing for online services outside of lab-controlled situations. An external configuration test designed to capture network performance as it relates to local customer experience. | Validates customer experience related to network performance beyond our control. |
| Sanity testing | Another way to confirm basic functionality. Often targeted to specific area of concern. | One final look-see to make sure everything as it should be. Aka preserve sanity by eliminating doubt. |
| Smoke Testing | Often down stream of Pre-Deployment where system connections and basics are confirmed. Usually used as a precursor to deeper testing where a “Fail” might block additional test effort on an incorrect or suspicious installation. | |
| Halo Testing | A focused effort to test the surrounding area of a feature or change. Helps to ensure there are no unintended consequences of a change. | Ideal for risky changes that could have side effects. |
| Integration Testing | Aims to validate that distinctly different components can work together as part of a larger system. | |
| Regression Testing | Often associated with a specific functionality change like a fixed defect. Aims the ensure that change solved the problem and that there are no deleterious side effects. | Validating fixed defects. |
| Black Box | This type of testing is conducted with no knowledge of the internal workings of a system or the code involved. | |
| White Box | Unlike Black Box this type of testing is performed with access to the code so that how you conduct this testing is influenced by what you can see in the code. |