Tests should try to refute the claim

Agents have this habit of sounding extremely confident. It creates a lull, a sense of safety that can be completely inappropriate. The reviewer told me everything checked out, the tests were green and life was good. Well. Until I actually checked the output.

Most of the tests confirmed the happy path and added a few obvious edge cases. Nobody had tried to show that the wrong user could read an object, a replayed request could charge twice, or an interrupted save could leave corrupt state marked as complete.

To find out whether the problem was mere politeness, I pushed the reviewer to the other extreme and asked it to “be malicious”. The experiment failed differently: it produced a longer report full of confident findings for attack surfaces the program did not have. The whole approach needed some serious rethinking.

I built severe-testing around two questions. Before designing a test, what observation would prove the claim false? Before reporting a failure, what evidence shows that it can really happen? A large part of the single skill file keeps speculation out of the findings, because inventing a plausible attack surface is much easier than showing that the code has one.

Give the test something to disprove

“Add tests” invites a tour of the implementation. Naming the claim changes the work. For an upload path, the claim may include authorisation, file-size limits, MIME handling, storage boundaries and cleanup after a partial failure. For an agent tool, it may include tool scope, resistance to indirect instructions and whether untrusted output can influence a privileged action.

Once those promises are explicit, the test plan can become properly rude:

• Can a user read another user’s object by changing its id?
• Can a symlink carry a write outside the workspace?
• Can replaying a request create a second charge?
• Can an interrupted save leave corrupt state marked complete?
• Can a tool return a secret that subsequently enters the model prompt?

Those are findings an engineer can investigate. “Authorisation”, “concurrency” and “prompt injection” are only areas in which a finding might exist.

The test also needs an oracle capable of disagreeing with the code: a permission matrix, schema validator, state-machine invariant, normalised diff, reference library, accessibility tree or mathematical identity. This becomes particularly important with generated code because the same model can write an implementation and a test which share the same misunderstanding. Agreement between them proves very little.

The skill separates broad surfaces into independent lenses—authorisation, injection, concurrency, resource exhaustion, supply chain, secrets and privacy—so one plausible line of attack does not crowd out the others. Destructive and malicious cases stay inside disposable fixtures, sandboxes, test tenants or authorised staging. “Adversarial” is a stance towards the claim, not permission to be reckless with somebody else’s system.

An attacker’s imagination also needs supervision

There is an awkward second problem: tell an agent to be malicious and it becomes exceptionally inventive about attacks the code cannot possibly suffer. It will flag SSRF in a pure function with no network access, prompt injection in code that never calls a model, and a race where no mutable state is shared. A serious-looking report assembled from imaginary attack surfaces is worse than a short one; after a few of those, people learn to skim the whole review.

About thirty of the skill’s 122 lines are therefore devoted to restraint. A concern with no reachable path or reproducer stays out of the findings. Risks already handled by a higher layer have to demonstrate that the layer can be bypassed. Ten-gigabyte inputs do not count when the only route is a bounded command-line argument. The agent still records genuine uncertainty, but it does not promote uncertainty into a bug for the sake of returning a fuller report.

Every candidate finding receives a score:

Score	What the evidence supports
0	The failure mode does not apply to this code.
25	The risk is plausible, but there is no concrete path or reproducer.
50	The failure reproduces under contrived conditions.
75	It reproduces reliably under realistic conditions.
100	It has been demonstrated in the real runtime with captured evidence.

Scores of 50 and above enter the report with the conditions and evidence attached. A 25 remains visible as untested risk without being presented as a defect; a zero disappears. When a change claims to fix a demonstrated bug, the regression test needs to reach 75 or 100. As the skill puts it, “a regression test that does not actually fail on the broken code is not evidence.”

The repository installs as a symlink under ~/.codex/skills/ or ~/.claude/skills/. One small fact is demonstrated directly:

$ grep -o 'adveserial\|vunerability' SKILL.md | sort | uniq -c
      2 adveserial
      1 vunerability

The misspellings are deliberate trigger terms. People invoke tools in the middle of work, spelling mistakes included, and activation metadata should recognise the request they actually make. I kept my own hurried spellings as the test cases.