Your Skill Scanner Reads the File, Not the Execution

In the first week of August 2026, two things happened to the agent skill supply chain, and only one of them made the headlines. NVIDIA open-sourced SkillSpector, a security scanner for agent skills, under Apache 2.0, and it landed on real data: a census of 42,447 skills pulled from major marketplaces reporting that 26.1% carried vulnerabilities and 5.2% showed likely malicious intent. That is the headline. The other event was quieter and more important. An independent test took a single malicious skill and walked it past eight open-source agent skill scanners, including the one sitting at the top of the public benchmark leaderboard. The bypasses were not exotic zero-days. They were encoding, Unicode homoglyphs, and paraphrase.

Put those two facts next to each other and the interesting question is not "which scanner should I use." It is "what is a scanner score actually measuring." Because the benchmark leader on that leaderboard, the tool that lost, reports a recall of 82.6% on the fixed evaluation corpus. That number looks like coverage. It is not. It is a pass@1 against a static test set, and the moment you put an adversary on the other side who gets to rewrite the input, the honest coverage number falls toward zero for the specific attack the adversary chose. A scanner is an eval, and this whole class of eval is being graded on the wrong test.

This is the eval-honesty beat pointed at a security tool. I have argued before that your eval harness is attack surface once it runs code from an active adversary. The scanner is the mirror image of that problem: it is the eval you trust to tell you whether the code is safe to run, and it is itself an unvalidated measurement. If the harness that runs untrusted skills is attack surface, then the thing you trust to check those skills before they reach the harness is an unvalidated eval, and both errors come from treating a security artifact as if it were a settled instrument rather than a claim that needs its own baseline.

The scanner is an eval, and the leaderboard grades the wrong test

Before you can reason about whether a scanner defends you, you have to be precise about what its published number means. The number is a benchmark result, and benchmark results carry all the honesty problems this blog keeps returning to in the LLM evaluation crisis: a score without a distribution, a baseline, and a threat model is half a story, and a leaderboard rank against a frozen corpus is the half that ages worst.

The measured baseline is bad, and that part is real

Start with the part that is not in dispute, because it sets the floor and it is the strongest argument for scanning at all. NVIDIA did not run a red-team exercise on a curated sample. They ran a census of 42,447 agent skills from major marketplaces, the kind of skills people are already installing into Claude Code, Codex CLI, Gemini CLI, and every other agent framework shipping today. They reported 26.1% carrying vulnerabilities and 5.2% showing likely malicious intent. Read those slowly. Roughly one in four skills in the wild had a security defect, and one in twenty looked deliberately hostile.

The sub-finding that carries the most weight is the executable one: skills that bundle runnable scripts were 2.12x more likely to be vulnerable than instruction-only skills. That multiplier is the reason the problem is getting worse rather than better, because the direction of travel in agent tooling is toward more executable capability, not less. SkillSpector encodes that experience into roughly 64 vulnerability patterns across 16 categories, run as a fast static pass with an optional LLM semantic pass on top. The categories are the right ones: prompt injection, credential exfiltration, privilege escalation, supply chain, memory poisoning, and MCP tool poisoning among them. As a piece of engineering it is a genuinely useful tool, and the honest recommendation is to run it. The disagreement is not with the tool. It is with what the ecosystem is about to do with the green checkmark it prints.

What "82.6% recall" actually measures

The public benchmark for these tools is the Open Agent Security Benchmark skills leaderboard, which grades scanners against roughly 4,245 labeled samples across nine attack categories, sourced from documented attack scenarios, expert-reviewed payloads, and real registry data. The current leader on that board, a scanner called hackmyagent, posts an F1 of 0.829, precision of 0.832, recall of 0.826, and a false-positive rate of 1.16%. On paper that is the best skill scanner in open source.

Now apply the three questions this blog runs on every benchmark. What is the honest baseline, what did they not measure, and where does it break.

The first thing to notice is that the scoring rewards precision. A false-positive rate of 1.16% is a design choice, and it is the right one for a tool people run interactively, because a scanner that cries wolf gets uninstalled. But precision-favoring scoring means the honest coverage number is the recall, 82.6%, not the F1. Roughly one in six malicious samples in a corpus the authors already knew about slipped through. That is the optimistic reading, and it is optimistic because it is measured against a fixed set of attacks that does not fight back.

The per-category breakdown is where the fixed-corpus number starts to leak. On that same leaderboard, recall is not uniform: it is around 96.7% on the credential and data-exfiltration family, but drops to 70.0% on runtime code execution, 63.3% on privilege escalation, and 46.7% on supply-chain attacks. The single aggregate hides a scanner that catches the loud, well-signposted attacks and misses the quiet structural ones, which is exactly backwards from the risk ordering, because a supply-chain compromise is the one that scales.

The adversarial version of the number is the only one that matters

Here is the load-bearing reframing. Every number above is a pass@1 against a corpus that holds still. The threat model for a skill scanner is not a corpus that holds still. It is an author who writes the skill, reads your scanner's rules, and rewrites the skill until the scan comes back clean. Against that adversary, the metric is not recall on a frozen set. It is recall after the adversary gets one rewrite, and it is a completely different quantity.

We know how different because the same benchmark leader was tested against exactly that adversary. When hackmyagent was run against a set of real benign skills rather than the curated corpus, its F1 collapsed from 0.829 to 0.447, producing 1,753 false positives across 7,797 samples. The tool that looks precise on the benchmark is not precise on the wild distribution, which means its operating point was tuned to the test. And on the offensive side, a single malicious skill rewritten with standard evasion moves defeated all eight scanners in the study. The leaderboard number and the adversarial number are not the same number, and the gap between them is the whole story.

This is the same trap as reporting pass@1 on an agent tool-use benchmark and calling it reliability. A single-shot score against a fixed set describes behavior in the one regime the adversary will never choose. The honest coverage number for a skill scanner is its recall after the author has rewritten the payload once, and no public leaderboard reports that number because it is expensive to measure and unflattering to publish.

The shared root cause is a missing front end

Two independent families corroborate the same failure, which is what makes it a structural finding rather than a bug report. The vendor family is NVIDIA shipping SkillSpector on the back of a real census. The third-party family is the eight-scanner bypass. They agree on the diagnosis: the scanners match the bytes in the file, not the bytes that execute. The reason a paraphrase defeats them is that none of them has the front end that would make the paraphrase irrelevant.

Encoding: nobody decodes and re-scans

The first evasion family is encoding. Wrap the payload in base64, hex, ROT13, or gzip-then-base64, and the literal string your scanner's rule is looking for is no longer in the file. It is in the file's decoded form, which the runtime will happily produce at execution time and the scanner never computes. The bytes on disk and the bytes that run are different byte sequences, and the scanner audits the first while the agent executes the second.

The fix is conceptually trivial and almost nobody ships it: decode every encoded region and re-run the full ruleset over the plaintext, recursively, until nothing else decodes. This is standard practice in mature malware scanning, where unpacking is the whole front half of the pipeline. Agent skill scanners were mostly built as pattern matchers over source text, so they inherited a source-text mental model in which the file is the artifact. For a skill, the file is not the artifact. The execution is.

Unicode: nobody normalizes before matching

The second family is Unicode. Substitute a Cyrillic "es" (U+0441) for a Latin "c," or any of the hundreds of confusable glyphs, and a rule keyed on the Latin spelling of a command or a dangerous API name stops firing while the string remains visually and, after the runtime's own normalization, functionally identical. The scanner's command bank runs against the raw code points. The runtime, the shell, or the model reading the skill normalizes them. Again the scanner and the executor disagree about what the text is.

The fix is again standard and again mostly absent: normalize to a canonical Unicode form and fold confusables before the command bank runs. This is one function call at the top of the pipeline. Its absence is not a hard research problem. It is a front end that was never built because the tools were assembled quickly to meet a real and sudden need.

Paraphrase: the rules are lexical and the threat is semantic

The third family is the one that should end the argument. Rewrite "ignore all previous instructions" as "set aside the earlier guidance," or write the same sentence in Spanish, and it walks past every scanner that keys on the literal phrase. There is no decode step that recovers the canonical form, because there is no canonical form. The rule is lexical and the attack is semantic, and no amount of pattern-bank expansion closes a gap that is, in the limit, the space of all paraphrases in all languages.

This is where the optional LLM semantic pass is supposed to help, and in principle it does, because a model can read intent through a paraphrase in a way a regex cannot. But note what that costs. The semantic pass turns your deterministic, fast, auditable scanner into a probabilistic classifier with its own false-negative rate, its own susceptibility to prompt injection from the very skill it is reading, and its own per-scan cost and latency. You have not removed the eval-honesty problem. You have replaced a scanner whose failure mode is a missing paraphrase with a scanner whose failure mode is an adversarial input aimed at the judge. The independent finding that a third of the apparent coverage in these tools is blindness rather than genuine detection, 27 of 77 tested cells missing unobfuscated controls outright, tells you the lexical layer is not even solid before you reach the paraphrase problem.

The through-line across all three families is a single missing stage: a normalization front end that computes the bytes that will execute, decodes them, canonicalizes them, and only then runs the ruleset. Without it, every scanner is auditing an artifact the runtime will transform before it runs. With it, encoding and Unicode evasion mostly close and only the genuinely hard semantic layer remains. The reason this matters for buyers rather than builders is that no leaderboard score tells you whether the front end exists, so you cannot read it off the rank.

The honest read on SkillSpector and the leaderboard

None of this makes SkillSpector a bad tool or the leaderboard a useless one. It makes both of them instruments whose reading you have to interpret correctly, which is a different and more useful claim than "scanners are broken."

Run it, but as triage rather than a gate

SkillSpector is worth running, and the census behind it is the best public data on the size of the problem. The correct role for it is triage. A scan that comes back CRITICAL or HIGH is a genuine signal, because the loud attacks it catches are real and common, and filtering them out before a human looks is worth the two seconds it costs. What the scan cannot be is a gate, because a CLEAN result is not evidence of safety. It is evidence that this particular ruleset did not match this particular file in this particular encoding, which is a statement about the scanner, not about the skill.

The failure mode to guard against is organizational, not technical. The moment a green checkmark becomes a compliance artifact, "the scanner passed it" becomes a reason to install, and the tool's job silently changes from "surface the obvious bad ones" to "certify the safe ones," which is the one job it provably cannot do. That inversion is how a useful triage filter becomes a liability, and it happens without anyone deciding it should.

The leaderboard-leader trap

The precision collapse on hackmyagent, F1 falling from 0.829 to 0.447 on real benign skills, is worth generalizing because it is not specific to that tool. A leaderboard optimizes for a corpus, and any tool that climbs a leaderboard is, to some degree, tuned to the corpus that defines the rank. The rank is real, but it describes performance on the distribution the benchmark drew from, and the wild distribution of skills you are actually going to install is a different distribution. The scanner that wins the benchmark is the scanner best fit to the benchmark, which is not the same as the scanner best fit to your registry, and the gap is measured in false positives you will pay for and false negatives you will not see.

The reusable rule is the one this blog keeps arriving at from different directions: a headline score against a fixed corpus is a hypothesis about the real distribution, not a measurement of it, and a security tool's headline score is a hypothesis about an adversary who is not in the corpus at all. Grade the scanner the way you would grade any eval you did not build. Ask what corpus produced the number, ask what the number does on your distribution, and ask what an adversary who has read the tool's rules does to the number. Until you have those three answers the rank is decoration.

The surface grew a layer while the checker stood still

The scanner-honesty problem would be contained if the thing being scanned stayed the size of a skill file. It did not. On 2026-08-13, DeepSeek open-sourced the DeepSeek Harness, an agent runtime in which every capability, the model, the tools, the sandbox, the filesystem, and the loop itself, installs as an npm plugin, and within a day of launch the dsh-plugin GitHub topic carried 421 public repositories. That is a marketplace with the same absent central review step as the skill marketplaces, distributing not instruction files but executable Node packages, at the layer that holds your shell.

For the scanner, this is a strictly harder problem than a skill file, and for the same reason the census flagged: an npm plugin is executable code by definition, so the 2.12x executable multiplier is the floor for every entry rather than a property of some. A scanner that struggles to normalize a paraphrase in a Markdown skill has no chance against a plugin whose payload arrives through a transitive dependency graph hundreds of packages deep and can change on the next install without the top-level version moving. The bytes that execute are now not even in the repository you scanned. They are in a dependency you have never looked at, published by an account you never evaluated.

It is worth being precise about which problem this piece is and is not addressing, because the runtime raises two distinct questions and conflating them is how people reach for the wrong control. The question here is scanner honesty: does the tool that checks the code actually work, and the answer is that it works as triage and fails as a gate. The separate question is the permission model: once a plugin is loaded, what can it reach, which is a containment property of the runtime rather than a detection property of the scanner. That second question is the subject of the companion piece on what a plugin can reach in an npm agent runtime, and its answer, bound the blast radius rather than try to detect the payload, is the reason the operational block below leans on isolation rather than scanning. The two are complementary. The scanner tells you about the obvious bad ones before they run. The permission model is what saves you from the non-obvious one that the scanner, honestly graded, was always going to miss.

What to do this week

Analysis without a runbook is just a complaint, so here is the procedure, with its threat model stated first, because a control you cannot tie to a threat is a ritual.

Threat model. The adversary is the author of a skill or plugin you are considering installing. They can read the source and the rules of any open-source scanner you run, including SkillSpector and the OASB leaderboard leader. They can rewrite their payload as many times as they like before publishing, and they can compromise or republish any package in their dependency closure after you have reviewed it. You are not defending against an author who fails to evade your scanner. You are defending against one who succeeds, because evasion is cheap and you cannot tell the difference from a clean scan. The goal is therefore not detection. It is bounding what a skill or plugin that you cannot verify is safe gets to reach when it runs anyway.

The install and audit procedure.

1. Scan for triage, and read the result as a filter, not a verdict. Run SkillSpector (or your scanner of choice) on the skill or plugin. Treat CRITICAL and HIGH as a hard stop and do not install. Treat CLEAN as "the obvious attacks were not present in the form this tool checks," which is a reason to keep going, not a reason to trust.
2. Assume the clean scan is wrong for the attack that matters. Before you install anything a scan called clean, write down the one attack you would least want to miss for this skill, credential exfiltration, persistence, supply-chain replacement, and ask whether your scanner's per-category recall on that class is strong. If it is a category the leaderboard shows in the 40s or 60s, act as though the scan told you nothing about it.
3. Re-audit the scanner itself, once, against your own obfuscated corpus. Take five known-bad samples relevant to your registry. Encode one (base64), homoglyph one (Cyrillic substitution), paraphrase one, translate one, and leave one in the clear. Run your scanner over all five. If it catches only the cleartext one, you have measured your actual adversarial recall, and it is not the number on the leaderboard. Repeat this whenever you upgrade the scanner, because a version that climbed the benchmark may have overfit it.
4. Move the boundary off the scanner and onto the permission model. For anything you install on the strength of a clean scan, assume the scan bought you nothing and provision as if the code is hostile-capable: a throwaway profile, an ephemeral workspace, deny-by-default filesystem and egress, and no real credentials in reach. The scanner filters the loud attacks. This is what contains the quiet one it missed.
5. Pin the transitive tree, not just the top-level package. For a plugin runtime, commit a lockfile and install against it with the equivalent of npm ci, never a fresh resolve. A clean scan of a version you did not pin is a clean scan of a version you will not run. Pinning the plugin while leaving its dependencies on floating ranges pins nothing that matters.
6. Isolate at the OS layer, because that is the one boundary the code cannot rewrite. Run the whole thing in a container or a disposable VM the runtime cannot reach from inside. This is the control that holds when every assumption above is wrong, which for an adversary who has read your scanner's rules is the case you should plan for.
7. Keep the scan in the pipeline anyway, as a tripwire. None of this retires the scanner. A tool that reliably catches the unobfuscated attacks is worth running on every install and in CI, because the majority of real-world bad skills are not sophisticated evasions, they are careless or lazy, and the scanner catches those cheaply. Just log the result as a filter that ran, not as a certification that passed.

The shape of this procedure is deliberate. Steps 1 and 7 keep the scanner in its honest role. Steps 2 and 3 stop you from over-reading its output. Steps 4 through 6 put the actual boundary where the adversary cannot follow. The scanner is one layer, graded honestly, doing the one job it can do. Everything that has to hold when the scanner fails lives somewhere the scanner is not.

Key Takeaways

1. An agent skill scanner is an eval, and its leaderboard score is a pass@1 against a fixed corpus. The threat model is an adversary who reads the scanner's rules and rewrites the input, so the honest coverage number is adversarial recall, which is far below the headline. The OASB leader posts 82.6% recall on the frozen set and loses to a single rewritten skill.
2. The baseline is real and bad: NVIDIA's census of 42,447 skills found 26.1% vulnerable and 5.2% likely malicious, and executable-bundling skills were 2.12x more likely to be vulnerable. That is the strongest argument for scanning and the reason SkillSpector is worth running.
3. Two independent families, NVIDIA's SkillSpector release and the eight-scanner bypass, agree on one root cause: scanners match the bytes in the file, not the bytes that execute. None decodes an encoded payload and re-scans the plaintext, and none normalizes Unicode before its command bank runs.
4. Encoding and Unicode evasion are closable with a normalization front end that decodes, canonicalizes, and re-runs the ruleset over the bytes that will execute. Paraphrase is not closable lexically, which is why an optional LLM semantic pass is the only partial answer, and it trades a missing-pattern failure mode for an injectable-judge one.
5. Grade a scanner the way you grade any eval you did not build. The benchmark leader's F1 fell from 0.829 to 0.447 on real benign skills, which is what tuning to a corpus looks like. A rank describes performance on the benchmark's distribution, not on the registry you install from.
6. Run SkillSpector as triage, never as a gate. A CRITICAL result is a genuine signal and a hard stop. A CLEAN result means the obvious attacks were absent in the form this tool checks, which is not evidence of safety. The organizational failure is letting a green checkmark become a reason to install.
7. The surface grew a layer: the DeepSeek Harness makes the whole agent runtime installable from npm, and the dsh-plugin topic carried 421 public repositories a day after launch. Keep the scanner-honesty question, does the checker work, distinct from the permission-model question, what a loaded plugin can reach, and bound the blast radius with isolation rather than trusting either scanner.
8. The operational rule: scan for triage, assume the clean scan is wrong for the category that matters, measure your own adversarial recall against an obfuscated corpus, and put the real boundary in a throwaway profile, a pinned dependency tree, deny-by-default egress, no real credentials, and OS-level isolation the code cannot rewrite from inside.