Of course, the classic response to these - even WITH the evidence is often "yOu'Re dOiNg iT wRonG". Does anyone actually have proof - where using skill.md is arguably better than not?
Edit: Fixed company name, added link to Vercel's claim
[0] https://vercel.com/blog/agents-md-outperforms-skills-in-our-...
For the former I'd be interested in learning more about that. From a harness perspective the difference would be the inclusion of the description in the system prompt, and an additional tool call to return the skill. While that's certainly less efficient than adding the context directly I'd be surprised if it degraded task performance significantly.
I tend to be quite focussed with my Skill/Tool usage in general though, inviting them in to context when needed rather than increasing the potential for model confusion.
Sorry, I miquoted the company, it was Vercel, not Cursor.
"A compressed 8KB docs index embedded directly in AGENTS.md achieved a 100% pass rate, while skills maxed out at 79% even with explicit instructions telling the agent to use them. Without those instructions, skills performed no better than having no documentation at all."
https://vercel.com/blog/agents-md-outperforms-skills-in-our-...
Example: a plan/act division, with the harness keeping state of which mode is active, and while in "plan mode", removing/disabling tools that can write data. Cue a mishandled timeout or an UI bug that prevents switching to "act mode", and suddenly the agent is spinning for 10 minutes questioning the nature of their reality, as the basic tools it needs to write code inexplicably ceased to exist, then opting for empirical experimentation and eventually figuring out a way to reimplement "search/replace" using shell calls or Python or whatever alternative wasn't properly sandboxed by the harness writers...
Part of this is just bugs in code, but what irks me is watching the LLM getting gaslighted or plain confused by rules of reality changing underneath it, all because the harness state wasn't made observable to the agent, or someone couldn't be arsed to have their error messages and security policies provide feedback to the LLM and not just the user.
For Claude Code I add the tooling into either CLAUDE.md or .claude/INSTRUCTIONS.md which Claude reads when you start a new instance. If you update it, you MUST ask Claude to reread the file so it knows the full instructions.
It's a great idea: really neat take on programmability, and can be reloaded while the agent is running without tweaking the harness, etc -- lots of benefits.
`pi` has a great skills implementation too.
I think skills might really shine if you take a minimal approach to the system prompt (like `pi`) -- a lot of the times, if I want to orchestrate the agent in some complex behavior, I want to start fresh, and having it walk through a bunch of skills ... possibly the smaller the system prompt, the more likely the agent is to follow the skills without issue.
Challenges (apart from the time taken) are coming up with a good enough gene representation that captures the essence of the problem, building an efficient fitness function, and avoiding local maxima - i.e. a solution that is almost but not quite good enough, but from where you can't breed a better solution.,
Alternatively, I’ve had less luck with purely documentation skills. They seem to be loaded less reliably when they’re not linked to actions the agent wants to take, and it is frustrating to watch the agent try to figure something out when the docs are one skill load away.
Documentation-based skills don’t really work in practice. They tend to waste tokens instead of adding value.
CLI skills are also redundant when the CLI already provides clear built-in help messages. Those help messages are usually up to date, unlike separate skills that need to be maintained independently.
If the CLI itself is confusing (and would likely be confusing for humans as well) then targeted skills can serve as a temporary workaround, a kind of band-aid.
Where skills truly shine is when agents need to understand non-generic terms and concepts: unique product names, brand-specific terminology, custom function names, and other domain-specific language.
1. Skills let the agent know the CLI is available because they get an entry in the context window.
2. They let you provide a ton of organisational knowledge and processes that the agent would have a hard time figuring out from the CLI alone.
3. It is just more efficient to provide quick information in a skill than it is to require an agent to figure out every detail from CLI help messages alone every single time.
don't forget these skills are just text that goes into the llm for it to read, interpret, and then produce text that then gets executed in bash. The more intricate and specific the skill definition the more likely the model is to miss something or not follow it exactly.
For example, on Proposal: AgentFile — Declarative Agent Composition from Skills + Filesystem-Native Skill Delivery
Imo a bad idea, but alas.
Isn’t python just an option ?
1. Using the skills frontmatter to implement a more complex YAML structure, so e.g.
requires:
python:
- pandas>=2.1
skills:
- huggingface-cli@1.x
skills.lock
The idea that agent harnesses should primarily have their functionality dictated by plaintext commands feels like a copout around programming in some actually useful, semi-opinionated functionality (not to mention that it makes capability-discoverability basically impossible). For example, Claude Code has three modes: plan, ask about edits, and auto-accept edits. I always start with a plan and then I end up with multiple tasks. I'd like to auto-accept edits for a step at a time and the only way to do that reliably is to ask CC to do that, but it's not reliable—sometimes it just continues to go into the next step. If this were programmed explicitly into CC rather than relying on agent obedience, we could ditch the nondeterminism and just have a hook on task completion that toggles auto-complete back to "off."
Are we at an ideal balance where Claude Code is pulling things in proactively enough... without bringing in irrelevant skills just because the "vibes" might match in frontmatter? Arguably not. But it's still a powerful system.
Interestingly, I've seen Claude do `./lookup.sh relevant-macro` without any prompting by me. Probably due it being mentioned in the compaction summary.
I think it's more along the lines of acknowledging the fast-paced changes in the field, and refusing to cast into code something that's likely to rapidly evolve in the near future.
Once things settle down into tested practices, we'll see more "permanent" instrumentation arise.
So yeah, I believe "it's a phase" but in a sense that it's a development phase, just like planning or prototyping.
If one can accept failure to follow instructions, then the world is open. That condition does not really comport with how we think about machines. Nevertheless, it is the case.
Right now, a productive split is to place things that you need to happen into tooling and harnessing, and place things that would be nice for the agent to conceptualize into skills.
I have not had much success with skills that have tree based logic (if a do x, else do y), they just tend to do everything in the skill (so will do both x and y).
But just as "hey follow this outline of steps a,b,c" it works quite well in my experience.
Also, be aware that when you add new instructions if you don't tell claude to reread these files, it will NOT have it in its context window until you tell it to read them OR you make a new CC session. This was a bit frustrating for me because it was not immediately obvious.
Use a structured workflow that loops on every task and includes a pause for user confirmation at the end. Enforce it with a hook. I'm not sure if you can toggle auto-accept this way, but I think the end result is what you're asking for.
I use this with great success, sometimes toggling auto-accept on when confidence is high that Claude can complete a step without guidance, and toggling off when confidence is low and you want to slow down and steer, with Claude stopping between the steps. Now that prompt suggestions are a thing, you can just hit enter to continue on the suggested prompt to continue.
Referencing them in AGENTS/CLAUDE.md has increased their usage for me.