Summary
Assertion inversion testing is a technique for detecting hollow or non-functional test assertions, especially in end-to-end (E2E) test suites. The method works by inverting a test's primary assertion (e.g., changing toBeVisible() to not.toBeVisible()) and re-running the test. If the test remains green with the inverted assertion, the original assertion is hollow—it cannot fail and therefore provides no real coverage. The technique is implemented in the open-source tool playwright-mutation-gate for Playwright tests, but the principle applies to any test framework with explicit assertions.
Key Points
- Hollow assertions are statements that always pass regardless of the application's actual state (e.g., missing
awaiton async assertions, typos in selectors, guarded assertions that skip, or tests that are skipped entirely). - Assertion inversion flips the logic of the main assertion (e.g.,
toBeVisible→not.toBeVisible) and reruns the test. A surviving inversion (test still passes) proves the assertion is hollow. - AI-generated tests exacerbate the problem: LLMs often produce syntactically correct but semantically empty assertions, creating "coverage theater" that requires manual auditing.
- The technique enforces one primary assertion per test, making the test's purpose explicit and the mutation tractable.
- The gate produces three verdicts: Killed (inversion failed – assertion is alive), Survived (inversion passed – assertion is hollow, gate exits with error), and Cannot-mutate (e.g., skipped tests – a deliberate informational signal).
- Limitations: does not check strength of assertions (only hollowness), and a killed mutation may fail for reasons other than the inverted line (e.g., implicit Playwright actions). However, a surviving inversion is always definitive: the test cannot fail on that assertion.