I literally hold a "retro" with my agent after a session.

> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.

It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.

Listen to your agent's whinging.

Ive been building my own ai voice agent harness from scratch for close to a year now and following good software architecture practices is a good start. So those rules have to be coded in agents.md somewhere also really helps to have a "vision" section or "spirit of the project" section that describes what the end goal vaguely looks like and things I care about in achieving for the project. This prevents agent from being brittle and "single minded" about its work. But yeah vibes most of the time is how I've also been doing it, but I did find one very important thing that has really sped up my work. So I figure I'd share it here. And that advise is to ignore front end design at all costs until the very end and you are ready to launch. UI related woes kill any type of inertia and are responsible for most of the issues. So I decided abandon all UI stuff until the very end and just focus on iterative refinement, cutting, and other back-end related work and its been great ever since. Ideas can be had, tested, validated/invalidates and you ' get stuck on the optimization thought loop. Recently I have started to consider how i can fully automate the development process as the capabilities are there but designing the instructions on how to do this well and how to handle niche cases without getting me involved takes careful planning in writing out the guide so that will be interesting to see once i get there.

I have Opus grade each of my repos, and each subsystem of them, and do it every weekend.

Since the version of opus has changed over time, so have the evaluations. I was going to constrain it, but since they edit the same files, I can alter those (which it uses as summaries and starting points).

Just working on my own stuff so YMMV, but from a cost reduction standpoint, the two most outsized ROI wins came from making sure the agent runs all terminal commands in quiet and using a codebase index mcp to speed up understanding and reduce full file reads [0]. I've also found the latter also dramatically improved the quality of output.

[0] https://github.com/DeusData/codebase-memory-mcp

Trying something like this in a reusable way at https://github.com/mateffy/gesetz

It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.

But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.

Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?

maybe a bit counter-intuitive but:

I found that removing

  - system prompt
  - skills
  - agents.md
  - mcps

+ reducing tools to just 1 (sh)

gives better results than having 'more' of them

(e.g. look at these traces to see more vs less in action:)

https://smolenv.com/t/nested-template-includes-60636/

not saying the right context does not help

(it definitely does!, but it's not trivial to provide the right context)

  - same task result (passed)
  - finished faster
  - fewer tokens, less cost
  - fewer requests for inference
  - fewer tool calls
  - less peak RAM

You've just described what the AI industry refers to as "evals", a collection of which forms a particular benchmark. I definitely encourage you to define your own evals, because the public benchmarks are often either saturated (largely solved and only going to see small improvements going forward) or seemingly not predictive of real world performance. I could also go further and speculate that they are in the training data, although credible benchmarks avoid this, I'm just not sure how successful they are.

> The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.

I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).

You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.

[1] https://github.com/backnotprop/plannotator

[flagged]

Is there any reason to think that training weights has peaked rather than is accelerating? It feels like now they are increasingly able to pick some low hanging fruit by using the models in order to improve themselves and test optimizations.

What date was the peak? If it is today it’s not something you can know so I assume you think the peak was many months ago.

If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.

Eagerly waiting for the TormentBench.

What is the fitness function , eval?

Very interesting, will give it a demo.

I'm curious to learn more about your experiences here, what you've learned, if you've enjoyed using the REPL with agents

The dream of the AI lisp machine ! Did you build in the functionality that the agent can on the fly rewrite its own (harness)-code?

There is a really good video by the author of pi.dev (which I have used to build some of my own harnesses): https://www.youtube.com/watch?v=RjfbvDXpFls

The basis of the argument is that the labs are constantly pushing updates to their system prompts that are used in claude code or codex, which are exceptionally bloated and change the sand beneath your feet with every update.

By rolling your own harness, as long as you keep up with the latest advances and changes in the ecosystem, you capture a lot of the 'control' that LLM-based development feels like it strips from you.

Obvious disclaimer that I use pi.dev when I am aiming for consistency, not absolute quality. Custom harnesses on pi are what I ship, claude code is still my fallback if I need to make sure a PR is the highest absolute quality

> Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts?

They're trying. I see a few main avenues:

1. Fitting the models to their specific harness, so that if you want peak model performance, you're stuck with their harness. But this only works if there aren't alternative models that are similar in capability or good enough that don't have that restriction.

2. Locking you out of the harness so that you eventually are just letting it do "stuff" with your data and system, although you don't get to see what the stuff is or why it's doing it. This is the pattern discussed in that Earendil blog post, "The session you cannot take with you." Encrypted reasoning tokens, secret agent prompts, perhaps eventually not even seeing what files are being read or what data is being sent back to their server. This way, you are also shackled to their harness because nothing is portable. But it only works if you trust them implicitly and don't have alternative models and harnesses that don't treat you this way.

3. Tying subscription pricing to the use of their harness, so that it's financially punitive to use another one. This is what Anthropic does. But again, it only works if there aren't alternative models and harnesses that work similarly well for you and don't have that restriction.

4. Marketing. Anthropic is leaning into this one heavy, from what I can tell, based on the constant ads I see for Claude Code. Can it work? People do overpay for things like vodka in fancy bottles that are chemically identical to the cheap stuff. Is Veblen-good AI a trillion dollar business, though?

I think the pressure from open source innovation in models and model tooling is going to make it a tough row to hoe. But I'm biased, as I'm actively rooting for Openthropic's demise.

I don't think so. A basic consumer-oriented harness is a commodity (zero moat: you can ask one harness to write you another).

State-of-the-art models have more of the workflow sensibilities built in, and don't need as much help from the harness.

Where harness helps the most is very customised personal workflows (not a textbox for a prompt, but an end-to-end IDE for how you develop software) But then it's best to have your own, rather than some closed-source rigid product.

Harnesses help smaller models, and very tightly hand-holding harnesses are needed for models runnable locally. I think that is very valuable to users, but avoiding paying for the biggest models is the opposite of what the frontier labs want.

Blame Descartes? I don't know. Extended theories of cognition all get shot down, even analytic philosophers who are accused of being more "continental" still primarily hold mind-body dualisms

[flagged]

Agree with the sibling comment that an example would be useful. I'm also curious what kinds of tasks you have in mind. Theoretically almost every knowledge work task boils down to the file editing that agent harnesses already are built around (which probably explains the second half of your comment).

I haven't understood a bit.

Can you make an example?

What do you mean please give more information.

It's engineering management I suppose.