Continue reading on Unthread

some thoughts on harness engineering and model capability seems pretty likely at least for the next year or so we'll see continual releases from frontier labs that are increasingly tuned to work well with some sort of agent harness (claude code, codex, et. al.). i also think it's likely that those harnesses will get smaller and smaller, with increasing amounts of autonomy given to the agent. but one thing i don't see a lot of posting about is the idea of 'harnesses as affordance', and the tensions there. let's take a simple example -- error handling. models will fail to use tools correctly. they fuck up parameters, they try different things, whatever. this exploration thru failure is a perfectly natural way for the models to engage with tools; you wouldn't necessarily expect the model to nail everything perfectly the first time (i hope you wouldn't, at least). contrast this to the UX of a modern SaaS app -- errors are ruthlessly quieted and hidden from users. we build in all sorts of workarounds to massage invalid state into something that looks valid, we swallow up transient failures and retry, we do as much as possible to preserve the appearance that things are fine. the only errors that should be leaked to the user are ones that are truly unrecoverable without human intervention. an extremely common request i get from the field re: agent workflows is "this isn't demoable, it looks bad when it fails." to which i counter, "why? it succeeds eventually, doesn't it?" i tend to believe that the idea of polish-at-all-costs (where 'polish' is just 'hide the error') has trained people to expect the appearance of perfection over the reality of complex systems. i do not think it's good to hide tool call failures from users. i think people should be able to look at the output from a tool and get as much information about what the model did, why it failed, etc. this is part of the human feedback loop; it can help them connect the model actions to their prompts, and help them refine their thinking. hiding these failures eliminates a learning opportunity for users. it is ultimately a social change, not a technical change, that is going to resolve this tension. but i do think we need to grapple with it. are there better ways that we can inspire this feedback loop with people -- eg, when models fail tool calls, can we start some sort of introspection loop to have the model self-interrogate why it failed, and also consider it from the perspective of the initial prompts? stuff like that. the other half of it, though, is to help guide users to think about model interactions differently than they do other software interactions, which will ultimately take time.