Agent Skills Are a Procedure, Not Knowledge

The pitch for agent skills is that they teach a model something it does not know. You write a SKILL.md, drop it in a directory, and the agent becomes capable of a task it would otherwise fail. A controlled study published in August 2026 says that story is mostly wrong about the mechanism, and getting the mechanism right changes what you should put in a skill file, how big a skill library should get, and what you are buying when you install a skill from a marketplace.

The stakes are no longer one vendor's feature. Anthropic published the SKILL.md format as an open standard at agentskills.io in December 2025, and OpenAI and Microsoft adopted it almost immediately. On October 22, 2026, Anthropic handed the standard to the Agentic AI Foundation, the Linux Foundation home MCP already lives in, and put numbers on the ecosystem: more than 70 agent platforms integrated, about a million skills published, nearly 280 million installs. Skills are the default way the industry packages "how to do this task" for an agent, which makes the question of what they actually contribute worth answering carefully.

The paper is "Demystifying Agent Skills: Why They Work, Until They Don't" (arXiv 2608.14036, Jiang, Huang, Xing, Wu, Gao, Cao, Wang, Liu and Li, submitted 2026-08-14). It moves past aggregate success rates and asks, under controlled ablations, what agent skills actually do. The headline finding is blunt: skills help mainly by procedural anchoring, stabilizing what the agent does, and almost never by injecting knowledge the model lacked. Of the labeled skill mechanisms, procedural anchoring accounts for 65.7% and explicit knowledge injection for 4.5%. A skill is not a fact the model was missing. It is a procedure the model already could execute but was not reliably choosing to.

What the study actually did

Most claims about skills rest on a before-and-after success rate. The best of those is SkillsBench (arXiv 2602.12670, Li et al.), which runs 87 tasks under matched no-skills and curated-skills conditions across 18 model-harness configurations and finds curated skills raise the average pass rate from 33.9% to 50.5%, a gain of 16.6 points. That is a useful, honest measurement of whether skills help. It cannot tell you why, because the comparison is against an agent that has nothing, so the skill's procedural shape and the mere presence of relevant experience are fused into one number.

The Demystifying study's design is built to pull those apart. The authors normalized 8,135 trial records and retained 238 valid unique labels from 240 open-coded trajectory records, across Terminal-Bench 2.0, Terminal-Bench-Pro and SkillsBench, with two agent harnesses (Codex paired with GPT-5.3-Codex and GPT-5.4, and Gemini CLI paired with Gemini-3.1-Pro-Preview). The load-bearing move is the ablation that isolates the representation. They hold the source trajectory pool constant and distill the same past experience two ways: once into Workflow Memory (cleaned trace logs of what happened) and once into a distilled SKILL.md (a procedural artifact), then evaluate both on matched tasks against a Raw baseline with neither. Because the underlying experience is identical across conditions, any difference between Workflow Memory and Skill is attributable to the representation, not the information content. This is the eval honesty discipline applied to a thing the field usually measures with a single delta: hold everything constant except the one variable whose effect you are trying to name.

That design separates two hypotheses a raw success rate fuses together. If skills worked by injecting knowledge, Workflow Memory carrying the same facts would do nearly as well as the skill. If skills worked by stabilizing procedure, the distilled SKILL.md would beat Workflow Memory specifically, because the distillation into "do this first, reach for this tool, verify this, avoid this mistake" is the thing that anchors execution. The data land on the second: skills improve over Workflow Memory by 6.06 points in matched comparisons, and the qualitative labels attribute the wins overwhelmingly to procedural anchoring.

Procedural anchoring, and the honest baseline

Procedural anchoring fixes the parts of a trajectory that are underdetermined: what to do first, which tool to reach for, what to check before proceeding, and which known mistake to avoid. The model already has the capability. What it lacks, run to run, is a stable policy for deploying that capability, and a good SKILL.md supplies the policy. That is why the effect shows up as steadier execution rather than as newly unlocked tasks.

Run the honest-baseline move and the marketing deflates. The relevant baseline for a skill is not "the agent with nothing," which is the comparison behind SkillsBench's 16.6 points. It is "the agent with the same experience represented as workflow memory," and against that baseline the skill format buys 6.06 points. Different benchmarks and configurations, so do not subtract one number from the other, but the direction is unambiguous: a meaningful share of what skills appear to deliver against a bare agent is simply having relevant past experience available at all. The paper's appendix runs the cheapest baselines too, on 26 Terminal-Bench 2.0 tasks with five trials each: a bare agent solved 50.0%, a generic short plan 47.7%, a generic test-first template 59.2%, workflow memory 62.3%, and the distilled skill 79.2%. A one-paragraph "write the test first" instruction recovers about a third of the skill's gain on that slice, which is the clearest evidence in the paper that the value is procedural. The part uniquely attributable to the skill format, the procedural distillation, is real and smaller. If you were sold skills as knowledge and you are seeing modest, variance-reducing gains, this is why: you bought a procedure, and a procedure stabilizes a capability you already had.

The label distribution has a second number worth reading. Beyond anchoring (65.7%) and knowledge injection (4.5%), the paper labels 2.8% of cases as failure warnings, 13.9% as no meaningful use, and 9.4% as counterproductive. A skill was twice as likely to make things worse as to supply a missing fact. That is not an argument against skills. It is an argument against treating them as free.

This also sharpens a claim we have made about agent memory: memory is a write path, not a bigger retriever, and the shape you write it in matters. The same experience distilled into a procedure outperforms the same experience dumped as logs, which means the value was never in retaining more. It was in retaining it in an executable shape.

Where skills break, and what a big pool actually costs

A method that helps is only half a result. The study's more operationally important contribution is the failure map, consolidated into three high-level categories (successful procedural anchoring, execution-layer and verification failures, and invocation, applicability and boundary failures) over twelve fine-grained skill-use modes, and one curve in it should change how you run a skill library, though not in the way a skim suggests.

In the execution study, the agent (Gemini CLI with Gemini-3.1-Pro-Preview, and Codex with GPT-5.4) gets the whole candidate pool in its environment with no preselection, and the authors parse which skills it actually opened. Actual-use precision, the share of skills it used that were the task's ground-truth skill, falls from 29.6% at a pool of 5 to 3.3% at a pool of 100. Recall stays at 54% to 74% even at 100, so the agent usually does read the right skill; it reads a pile of others alongside it.

And here is the part a skimmed abstract will get wrong: task success did not fall. Averaged across the two pairings it moved from 36.4% to 39.3% as the pool grew, and for Codex it rose. The authors' own reading is that "exact ground-truth skill invocation is neither sufficient nor strictly necessary for success," because related skills that are not the annotated one still supply useful procedure. That is consistent with everything above. If skills were knowledge, the wrong skill would carry the wrong facts and success would drop. Because skills are procedure, a neighbouring procedure often anchors execution nearly as well.

So the honest version of the pool-size warning is narrower than "big libraries break agents," and more useful. What collapses is attribution. At a pool of 100 you can no longer say which skill did the work on a given run, which means you cannot evaluate a skill, cannot tell a stale one from a useful one, and cannot catch the 9.4% that actively hurt. What hurts identification most is not raw size but confusability: in the offline diagnostic, an embedding retriever's top-1 precision held at 84% to 93% on 100-skill pools of random or dissimilar distractors, and fell to 53% when the distractors were semantically similar. Ten near-duplicate "deploy" skills are a bigger problem than a hundred unrelated ones.

The failure modes the paper names are where the residual risk lives: brittle assumptions (the procedure encodes conditions that no longer hold), incompatible contexts (invoked for a task it was not built for), and insufficient adaptation (followed too rigidly when the situation needed deviation). A larger, more confusable pool gives the first two more chances to fire, and the precision collapse means you will not see it in your logs unless you go looking. SkillsBench adds a compatible signal at the level of a single skill: focused skills with at most three modules outperform larger or exhaustive bundles. This is the same lesson tool-use evaluation teaches about large tool sets: past some count, the binding constraint is selection, and selection is the part nobody measures.

What the marketplace is actually selling

Put the two findings together and skill marketplaces look different than the listing pages imply. We covered the security side when NVIDIA's census of 42,447 marketplace skills reported roughly a quarter carrying vulnerabilities and a scanner ecosystem that could not reliably catch them. That was the question of whether a skill is safe. This is the prior question of whether a skill is worth having.

If skills work by procedural anchoring rather than knowledge injection, then what a marketplace mostly sells is a procedure you could have written from your own traces, plus a selection problem you now own. The procedure is real value when the task is one you have not done enough to distill yourself, and installing it is faster than writing it. But it is a policy for a capability you had, which means its value is bounded by how well it matches your task and how cleanly it coexists with the skills already in your pool. Import a hundred community skills and, by the study's own curve, the one that fits your task becomes one of many the agent reads, and you lose the ability to say which of them is doing the work or doing harm. The install decision is not "is this skill good." It is "does this skill's procedure match my task well enough to justify the attribution and confusability cost it adds to every other skill in my pool."

This is also the capability answer underneath the supply-chain question about agent runtimes. Install is code execution by design, and install is also an audit cost by design: every skill you add makes every skill already there harder to attribute and evaluate. The reason to keep a skill library small is not only that a smaller attack surface is safer. A smaller pool is one you can still evaluate, skill by skill, and an unevaluated skill is a liability whether or not it is malicious.

What the study does not settle

The discipline that makes this paper worth citing is the same discipline you owe its numbers. Three caveats keep the findings from being over-read.

First, the mechanism percentages come from open coding: 238 labels over 240 records, assigned by annotators reading trajectories. That is a careful qualitative method, and it is a judgment layer, not a measurement. "65.7% anchoring versus 4.5% knowledge injection" is a strong directional signal about which mechanism dominates, and it is not a physical constant. The claim that anchoring dominates is robust to a fair amount of labeling wobble. A second-decimal reading of the split is not.

Second, the results come from terminal and coding-style benchmarks (Terminal-Bench 2.0, Terminal-Bench-Pro, SkillsBench) with a handful of frontier models from two labs. That is where skills are used most, and it is also a domain where frontier models already hold most of the relevant knowledge. SkillsBench's own domain spread is a warning against generalizing. In its current version (v4, June 2026), curated skills add 28.8 points in natural science, 24.1 in media and content production, and 18.9 in cybersecurity, against 11.6 in software engineering and 9.7 in mathematics; the authors attribute the spread to procedural knowledge that is underrepresented in pretraining. (The February preprint reported a far wider +4.5 to +51.9 spread on an earlier task set; quote the current one.) In a domain where the model genuinely lacks facts, knowledge injection may matter more than 4.5% suggests. Treat these numbers as a prior to check on your own tasks, not a spec.

Third, the 29.6% to 3.3% curve comes from two agent-model pairings browsing pools built from SkillsBench annotations, and "ground truth" there means the skill the benchmark annotated, not the only skill that could help. That is exactly why success held while precision fell, and it means the curve measures attribution, not damage. A better selection front end, or a library with fewer near-duplicates, would move the absolute numbers. It will not repeal the shape: a larger, more confusable pool is harder to select from and harder to audit. Plan for the shape, and measure your own breakpoint.

An operational block: running a skill library that does not rot

The findings convert cleanly into a policy. The through-line: a skill is a procedure, its value is variance reduction on a capability you already have, and its enemy is a pool too large to audit.

1. Write skills as procedures, not explainers. A SKILL.md should encode what to do first, which tool to use, what to verify, and which known failure to avoid. If a draft skill reads like documentation of a concept, it is in the wrong format. Cut anything that is not an instruction the agent should follow.
2. Distill from your own traces before importing. The 6.06-point gain came from distilling real trajectories into a procedure. Your own successful runs are the highest-quality source, and a skill distilled from them matches your task by construction. Do not confuse this with asking the model to write skills from nothing: in SkillsBench v4, skills an agent authored before attempting the task landed 8 to 11.5 points below no skills at all. Reach for a marketplace only for tasks you have not run enough to distill.
3. Cap the pool and measure selection precision alongside success. Log which skills the agent actually opened on each task and check them against the one you would have chosen. When precision falls while success holds, you have not broken anything yet, but you have lost the ability to attribute results to skills, and every later evaluation step gets blind. Treat that point as your effective capacity.
4. Scope skills tightly, name their preconditions, and deduplicate. Brittle assumptions and incompatible contexts are selection-into-the-wrong-situation errors, and near-duplicate skills are what made selection hardest. The skill's description field is what the agent selects on, so make it state when the skill applies and when it does not. Do not put boundaries in a skill: "never touch production" in a SKILL.md is a procedure the agent may follow, not a control, and scope has to be enforced below the model.
5. Allow deviation. Insufficient adaptation is a skill followed too rigidly. Write skills as strong defaults with explicit permission to depart when the situation differs, not as scripts to execute verbatim.
6. Retire skills like dependencies. A skill whose assumptions have gone stale is worse than no skill, because it will still be selected and followed. Review the library on a schedule and treat a growing pool as a liability to justify, not an asset to accumulate.
7. Evaluate a skill by its marginal effect on the whole pool. Before adding a skill, run a paired comparison (with and without it) across every task the agent handles, not just its target task. A skill that helps one task and degrades selection for ten is a net loss, and its isolated before-and-after number will hide that.

The scaffold, isolated

There is a larger point here that goes past skills. An agent pointed at a task is a model plus a scaffold, and the field has spent two years unable to say which part of the scaffold does the work, because everything moves together. Skills are a clean instance of scaffold in a file, and this study holds the experience constant to isolate what the format contributes. The answer, that the format buys procedural stability rather than new capability, is what the agents-as-models frame predicts: the model already contains the capability, and the scaffold's job is to deploy it reliably.

That reframing is worth carrying to the next scaffold component sold as a capability upgrade. Memory, tools, skills and elaborate prompt structures are, on current evidence, mostly variance-reduction machinery for capabilities the base model already has. When a vendor tells you a scaffold feature makes your agent capable of something new, run the test this paper ran: hold the model and the underlying information constant, change only the feature, and see whether the ceiling moved or only the variance. Usually, it is the variance.

Key Takeaways

1. Agent skills work through procedural anchoring, not knowledge injection. In the Demystifying Agent Skills study, procedural anchoring accounts for 65.7% of labeled skill mechanisms and explicit knowledge injection for 4.5%. A skill stabilizes a capability the model already has.
2. The honest baseline is workflow memory, not nothing. Curated skills beat a no-skills agent by 16.6 points on SkillsBench, but holding the source experience constant, a distilled SKILL.md beats the same experience stored as workflow memory by 6.06 points. The smaller number is what the skill format itself contributes.
3. Skills are not free. 9.4% of labeled skill uses were counterproductive and 13.9% had no meaningful effect, so a skill was twice as likely to hurt as to supply a missing fact.
4. Selection precision collapses as the pool grows; success did not. Actual-use precision fell from 29.6% with 5 skills to 3.3% with 100 while task success held near 36 to 39%, because a neighbouring procedure often anchors execution nearly as well. That is more evidence that skills are procedure, not facts.
5. What a big library loses first is attribution. At 100 skills you cannot say which one did the work, so you cannot evaluate, retire, or catch the harmful ones. Similar-looking skills hurt identification far more than raw count, and focused skills outperform exhaustive bundles.
6. A marketplace mostly sells a procedure you could have distilled, plus a selection problem you now own. Judge an install by whether its procedure matches your task well enough to justify the cost it imposes on every other skill in the pool.
7. Write scoped procedures, distill from your own traces, cap the pool, and retire stale skills. Evaluate each skill with a paired with-and-without comparison across the whole task mix, not its isolated before-and-after number.
8. The result generalizes to scaffolding. Hold the model and the information constant, change only the scaffold feature, and check whether the ceiling moved or only the variance. Usually it is the variance.