Introduction to Mutation Testing with StrykerPriyanka JaggaBlockedUnblockFollowFollowingJun 3In this post I will be looking at mutation testing.
What it is, as well as an example to show what it can do, limitations to consider and how it can be added to a Javascript project using Stryker.
The example used in this post can be found here.
What is Mutation Testing?“Quis custodiet ipsos custodes” which in English means “Who will guard the guards!!”.
Mutation testing is a way of testing the quality of your tests by introducing changes into application code and seeing if our test suite detects them.
It is a type of white box testing which is mainly used for unit testing.
This is done by making extremely small changes to the code, so it does not affect the overall objective of the program.
This helps assess the quality of the test cases, which should be robust enough to fail when the mutated code has been injected.
This method is also known as fault-based testing, as it involves creating a fault in the program.
“Mutation testing is based on two hypotheses.
The first is the competent programmer hypothesis.
This hypothesis states that most software faults introduced by experienced programmers are due to small syntactic errors.
The second hypothesis is called the coupling effect.
The coupling effect asserts that simple faults can cascade or couple to form other emergent faults.
”- wikipediaWhy Mutation Testing?Traditional test coverage (i.
e line, statement, branch, etc.
) only looks at the code executed by our tests.
It does not check that our tests are actually able to detect faults in the executed code.
One of the more extreme examples of poor test coverage are tests with no assertions.
Fortunately, these are uncommon in most code bases.
Much more common, is code that is only partially tested by a test suite.
Partially tested code can still execute all of its branches, but may not cover all the scenarios.
How does it work?Faults are introduced into the source code by creating multiple versions of the code, each version is called a mutant.
Each mutant contains a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.
Test cases are applied to the original program and also the mutant program.
Compare the results of the original and mutant program.
If the original program and mutant programs generate the different output, then that the mutant is killed by the test case.
Hence the test case is good enough to detect the change between the original and the mutant program.
If the original program and mutant programs generate the same output, mutant is kept alive.
In such cases, more effective test cases need to be created that kill all mutants.
Suggested toolsStryker — JavascriptPIT — Java or KotlinIn this post, we’re going to talk about Stryker and how it can be used for mutation testing.
Automate Mutation Testing with StrykerMutation testing can be extremely time consuming and complicated to execute manually.
If we consider the number of lines in our code, then given the number of mutants that need to be generated to ensure all the code is sufficiently covered, it can take up a lot of manual effort.
Sounds complicated?.Don’t worry, with Stryker, it makes this process a lot easier and faster to run.
It will only mutate our source code, making sure there are no false positives.
Example:Here is an example from stryker blog, you need to check the age of a user for a credit card application tool.
Users can only proceed if they’re over 18.
Say, we write the following function to confirm if a user can continue:Stryker will then modify the code by locating the return statement and changing it into different variations, such as:return user.
age > 18;return user.
age < 18;return false;return true;These modifications are called mutants.
After the mutants have been generated, they are applied one by one and our tests are executed against them.
If at least one of our tests fail, then the mutant is killed.
And that’s what we want!.If no tests fail, then the mutant has survived.
The better our tests, the fewer mutants should survive.
Stryker can output the results in a variety of different formats.
One of the easiest to read is the clear-text reporter (default reporter for Stryker).
This is the output from Stryker using the clear-text reporter:Mutant killed: /Path/filePath.
js: line 1:9Mutator: BinaryOperatorreturn user.
age >= 18;return user.
age > 18;Mutant survived: /Path/filePath.
js: line 1:9Mutator: RemoveConditionalsreturn user.
age >= 18;return true;The clear-text reporter will output how exactly our code was modified and which mutator was used.
It will then tell us if a mutant was killed, meaning that at least one test failed, or if it survived.
In the output above, you can see the second mutation has been marked as a survivor.
This means a test is most likely missing, one that explicitly tests for an age lower than 18.
Try it yourself with same exampleClone the repository:git clone https://github.
com/pjagga/mutation-testing-strykergit git checkout step1npm installThis contains the function ageIsOldEnough and one unit test.
Run the test using :npm run testYou will notice the test has run and passed.
Let’s check the coverage using:npm run coverageCoverage is 100% and everything passes.
Now, let’s see what mutation testing has to say about it.
Let’s start with installing stryker-clinpm i -g stryker-clistryker initChoose the following options in the questionnaire:?.Do you want to install Stryker locally?: npm?.Are you using one of these frameworks?.Then select a preset configuration.
None/other?.Which test runner do you want to use?.If your test runner isn't listed here, you can choose "command" (it uses your npm testcommand, but will come with a big performance penalty) jest?.What kind of code do you want to mutate?.javascript?.[optional] What kind transformations should be applied to your code?.(Press to select, to toggle all, to invert selection)?. More details