MENU
MODELSJune 20264 min READ

The best part is the part I understand least

Building an agent means deciding where to trust the model and where to write hard rules. I give the model more room than I used to, but I’m careful about what reaches the user.

Every agent mixes model judgement with deterministic code. I keep revisiting where the boundary should sit, and over time I’ve moved it further towards the model than feels immediately comfortable.

A good model can weigh a messy situation in a way my hand-written rules rarely do. My rules capture the cases I anticipated; the model can respond to details I didn’t think to encode. When I’ve constrained its behaviour too tightly, the result has usually become narrower and less useful. Giving it room has produced better surprises.

I find that uncomfortable. The most valuable behaviour in the system is also the behaviour I can’t fully explain. I can test it across examples and observe that it works, but I can’t reduce the judgement to a neat set of rules.

Why I keep the deterministic parts anyway

I still keep plenty of deterministic structure for two practical reasons. First, it gives me something inspectable: behaviour I can reproduce, debug, and change one piece at a time. Second, it controls cost and latency. Plain code should do the cheap, predictable work so the model can be reserved for decisions that benefit from it. The result is easier to understand and cheaper to run.

The hardest constraint is not about the model

The harder constraints are often about the person using the product. A model will happily be more thorough, more proactive, and more present than anyone wants. Much of the design work is deciding how much of that capability should reach the user, and when.

I saw this in both Akarii and Omaru. Akarii lives in a team chat and decides when to speak. It was reasonably good at spotting moments when it could help; the harder task was teaching it not to interrupt. Omaru can produce a great deal of work for someone building a business, so we have to decide what is genuinely useful rather than exposing everything the model could generate. In both products, I trust the model with local judgement and use hard constraints to protect the person on the other end.

Where I keep the dial

My current rule of thumb is to let the model handle contextual judgement, keep plain code around anything that needs to be cheap or inspectable, and place the firmest boundaries around user impact. Capability and a good experience are separate questions.

I’ve made mistakes in both directions. Sometimes I wrote brittle rules because they made me feel in control. Other times I let the model run too far because its early results were impressive.

I still lean towards giving the model room to judge. I’m less willing to delegate the decision about what a person should have to see, approve, or tolerate. That boundary belongs to the product team.

012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789FROM MOCKS TO SHIPPED012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789TASTE IS NOT A MYSTERY012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789CAN THE AGENT TALK BACK?012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789THE BEST PART IS THE PART I UNDERSTAND LEAST012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789SHOW THE WORK, KEEP THE MAGIC012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789EVERY AGENT IS A CODING AGENT012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789AN AGENT WITH ITS OWN TIME012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789A LOOP FOR THINGS YOU CAN’T MEASURE012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789NOTHING WORKS UNTIL THE READING DOES012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789MEMORY IN THE LANGUAGE THE AGENT SPEAKS
The best part is the part I understand least | kuoloon chong