Public repositories now host agent skills — SKILL.md files packaging reusable procedural knowledge — by the hundreds of thousands. We wanted an answer to the practical question underneath that abundance: do they actually make agents better? Our new paper, SkillCorpus, is our attempt to measure it end to end.

SkillCorpus Research · 4 min read · Read the paper on arXiv

The bottleneck was never supply

It is tempting to treat the open skill ecosystem as a scaling problem: more skills, more capability. Spend any time in it and a different picture emerges. The artifacts are fragmented across channels, heavily redundant, and uneven in quality — and whether they help on real work was, until now, genuinely unclear.

So the hard part isn't finding more skills. It is knowing which ones are unique, useful, safe, permissively licensed, and relevant to the task in front of the agent. That is a curation and retrieval problem, and no amount of additional supply solves it.

821,000 in, 96,401 out

SkillCorpus is a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale. The curation stage is the one whose arithmetic tells the story: roughly 821,000 crawled skills enter a multi-stage filtering pipeline and 96,401 come out the other side.

Nearly nine in ten candidates are discarded. That ratio is the empirical answer to "are more skills better" — most of what exists is duplicated, broken, or not safe to hand an agent. What survives is organised by a 16-class taxonomy and scored on three quality facets — utility, robustness, and safety — kept deliberately separate so a skill that is useful but unsafe cannot average its way into the corpus.

Curation is only half the system

A clean corpus that the agent cannot search is inert. SkillCorpus pairs the curated library with a fine-tuned retrieval-and-selection stack that matches task-relevant skills to the task at hand.

Tuning retrieval on the corpus itself matters because a SKILL.md is not a generic document — it has structure and declared intent that off-the-shelf embeddings flatten. And the agent's context budget is finite, so the cost of poor matching isn't just a missed skill; it is the right skill crowded out by near-misses.

Evaluated across conditions, not just on average

The evaluation is deliberately wide, because a result that holds for one harness and one model tells you very little about deployment. SkillCorpus is tested end to end across three benchmarks — SkillsBench, GDPVal, and QwenClawBench — two harnesses, and two open backbones, with a frontier robustness check on top.

Integrating SkillCorpus produced consistent gains on all three benchmarks, with the largest pooled improvement at +7.5 percentage points on SkillsBench. Consistency across that grid is the part we would emphasize over the headline number: the gain is a property of the corpus-and-retrieval system rather than of one favourable configuration.

The more interesting result: where the gains stop

An operational analysis traces those gains to two boundaries, and naming them is arguably the paper's most useful contribution.

The first is a coverage boundary. Improvement tracks how densely the curated corpus covers the task at hand. Where coverage is thick, agents improve sharply; where a task falls into a thin region, there is simply less worth retrieving, and no retrieval quality compensates for absence.

The second is a harness boundary. The same skill delivers more when the harness around it can execute deeply enough to actually use it. The loader and runtime are not neutral plumbing — they are part of the capability, and they cap it.

Together those two boundaries convert "skills help" into something you can plan against: a model of when and how much they help. To our knowledge this is the first end-to-end account of when a curated, retrieval-served community corpus improves real agent tasks — and where it does not.

Which is why the conclusion is narrower than the ecosystem's enthusiasm suggests. More skills are not automatically better. Curation, retrieval, and the harness are what determine whether skills become capability.

Read the paper

SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents — Yanze Wang, Pengfei Yao, Tianyi Sun, Chuanrui Hu, Yan Xiao, Xiaotian Luo, Yunyun Han, Yifan Chen, Jun Sun, Yafeng Deng. Available on arXiv. The dataset, models, and code will be released upon acceptance.

We introduced this corpus earlier as Skill Hub, which carries the full per-configuration breakdown and an additional security benchmark that fall outside the paper's three primary benchmarks.

Reported gains are pooled across the evaluated harness and backbone configurations for each benchmark; the +7.5 pp figure is the pooled result on SkillsBench.