Skip to main content
Suppose a refund policy says, for example, that a manager’s signed approval is valid until 12:00:00. A test checks that the approval works at 11:59 and fails at 12:05. Both pass. Now someone changes one character in the code, so the approval is still accepted at exactly 12:00:00. Every test still passes. Nobody notices, because no test ever asked about the exact instant. That is the kind of gap I wanted to find. Parmana had thousands of passing tests. That number told me the tests ran. It did not tell me whether they would notice if the code gave a wrong answer.

​
Counting tests is not the same as checking them

Line coverage has the same problem. A line can be run by a test that never looks at what it returned. In most software that is a quality issue. In code that decides whether an agent may move ₹1,50,000, a check that is executed but never really tested is a check you are trusting on faith.

​
The idea: break the code on purpose

Mutation testing turns the question around. A tool makes one small deliberate change to the source, such as turning <= into <, or forcing a condition to true, and runs the package’s own tests against it. If a test fails, the change was caught. If every test still passes, it survived, and a survivor points at behaviour no test pins down. The share caught is the mutation score.

​
What I did

The first run, on 2026-10-05 with Stryker, covered the five packages that guard a decision: the envelope verifier, approval, policy, the execution gateway and crypto. It is slow, because every change reruns the whole package suite, so it runs by hand rather than in CI. The first scores were humbling: 89.7% for the envelope verifier, 77.7% for approval, 50.3% for policy, 67.6% for the gateway and 65.4% for crypto. The survivors pointed at real gaps:
  • Approval scope. Only two of the comparators that bound what an approval covers, equal and less than or equal, were tested. A policy using any other one would have relied on untested code.
  • Rule operators. What each operator does at its boundary, or with a value of the wrong type, was largely untested.
  • The gateway. A configured lifetime limit or key revocation could be dropped on its way to the envelope verifier and nothing would fail.
  • Session binding. No single field of the session a connector must match was tested on its own.
Two findings changed behaviour, not just tests. First, signal types were not checked before rules ran. An amount sent as the text “150000” makes every numeric comparison false, so a rule written as “refuse if the amount is over X, otherwise approve” would not refuse it. The shipped refund policy still refused such a request through a later rule and the approval check, and no bypass was found, but the gap was real. Now a request whose signals do not have the declared types is refused before any rule runs. Second, the execution chain check skipped any Execution missing its chain fields as old data, so an edited Execution with its chain signature removed passed that check. The record’s own signature still caught the edit, but the chain check now treats a partly chained Execution as a break. Each survivor that mattered for security got a test, and I checked each new test by putting the change back by hand and watching it fail. The expiry case from the opening is one of them: refused at the exact instant, accepted a millisecond before. There was also a lesson about the measurement itself. Crypto was first measured on Node 22, where the post-quantum tests skip themselves, so every change in those paths counted as missed. The tool now refuses to run below Node 24. Re-run on Node 24 after the fixes: envelope verifier 94.0%, approval 93.9%, policy 96.0%, crypto 85.0%, gateway 83.6%.

​
What it does not do

  • A score is not a proof. It says the tests notice small changes of the kinds the tool makes. It says nothing about a design mistake, a missing check, or an attack that needs many changes at once.
  • Not every survivor can be killed. Some changes are equivalent: removing a guard that a later check repeats changes nothing anyone can observe. Those are listed as such, not forced with artificial tests. Others are in advisory code, such as the warnings about overlapping rules, or in message wording.
  • Only the package’s own tests count. Integration tests elsewhere cannot see the changes, so real protection is somewhat higher than the score, and the score cannot show it.
  • It is not continuous. It runs by hand, so a later change can open a new gap until the next run.
  • Five packages, not all of them. Storage, the runtime and the API were not part of this run.
  • It is not an outside review. It was run as part of Parmana’s own development process. The full results, every gap and what was and was not changed, are public so anyone can rerun it with npm run mutation.

​
The takeaway

A test suite is a claim about the code. Mutation testing checks that claim by lying to it. The useful output was not the percentage but the list of survivors: each one a sentence of the form “this check could be wrong and nothing would tell you”. For a system whose job is to say no, that list is worth more than any test count. Read more: Chapter 22: Testing philosophy, Claims and evidence.