Confirmation tests are meant to test that an expected outcome did happen.
They differ from normal tests in the fact that what they are testing is often expected to be unwanted behavior
and hence confirm existence via failing instead of passing.
They can either be system- or unit tests.
While some can temporarily exist as whitebox tests they always end deleted or as blackbox tests.
The most common use case is to confirm a bug report.
As an example lets pose that you are developing a RPG with some sort of buying and selling.
Now you receive a bug report from your play testers that they were able to find a scenario, where they bought something and did not receive the item, but the currency got deducted.
This occurred while they were repeatedly smashing the buy button.
You can now first write a test that checks that the buy button works by pressing it once per frame.
The test is green so you don't have found the bug, but it's nice to know that this works.
Next step is to test, if more inputs than frames causes the issue and yes, the test fails and you have confirmed the existence of the bug.
After fixing it you have not only fixed a bug.
You have also ensured that it will not get re-introduced later on in the development process.
This increases your confidence in refactoring core parts of your system since you know that you don't introduce already known bugs with refactors.
As written in the summary, confirmation tests can start of as whitebox tests to help you program the right functionality.
If you know what steps are involved in implementing a function (lets say map
, filte
, and reduce
in this example) but want to ensure the ordering of the operations,
you could start of writing a test that confirms the correctness of each step.
Assert that the filter, filters for the correct entities.
Then, that the map transforms them to the correct format.
And finally that the reduce aggregates them in the correct way.
The final tests will be a black box that tests the function, while each step of writing the test was effectively a whitebox tests.
This form of confirmation testing is rare to happen.
In most cases your code should be easy to write, which is why the example seams contrived.
If that's untrue, you are better of fixing your code structure instead of implementing new features through confirmation testing.