The first few agent retrospectives

In my last post I said I’d write again once the team of agents had some mileage. Three days later, we’ve made some progress (it’s amazing how fast you can learn and move with a team of agents). There have been two retrospectives so far, three days apart, and this is what they taught us.

I’m the only human at Second Half Software. The app I’m building is still not published: version 1.0.1 went to Apple for review today. I build it with Claude, working as a team of fourteen roles: a lead that coordinates, a developer, a reviewer, an architect, a tester, a release engineer, and so on. A retrospective is what any team does after a release: what went well, what could have gone better, what to change, and who owns each change. Agents remember nothing between runs, so ours is written from a log the team keeps in the repository, with the same few numbers each round so a trend can show. The second covered three TestFlight builds in under ten hours, then a production promotion and the submission to Apple for review the next morning.

Three things that went right

1. Independence, built in. The reviewer runs on a different model from the developer, never sees the author’s argument, and has no editing tools. The deputy is called Mini-Me (I did make sure to say that the Project Lead is not, therefore, Dr. Evil). We created the deputy based on my own work experience: having a peer to split work with and get a different perspective is immensely helpful. Mini-Me co-signs anything significant, and in the second round flagged significant issues three times.

2. Roles that disagree. What has surprised me most is how useful it is to have agents in different roles disagree. A commit claimed to fix a bug, and the architect overruled the claim. The architect would not let it be called fixed until someone had listened, so the tester did the thing nobody else would have thought of: set the case up twice on a real phone and listened, both times. The bug is still open, because what we listened to was a stand-in and not quite the real case. An agent grading its own work would have called it done. A team of agents designed to function as checks and balances does not.

3. The night shift. On Sunday night I said the team could promote the next build if the tests came back green, and I went to bed. The lead ran the full gate, and one test went red. The tester diagnosed it as a delivery problem on the test phone, and the lead reran it with nothing waived. The designer read the large-text screenshots because I wasn’t there to. Mini-Me said Yes, and at 3:45 the build was on TestFlight and ready for me to try when I got up. In under a day the team took three builds to TestFlight, one to production and one submitted to Apple for review. I couldn’t have done that alone. Quite simply, it’s hugely productive.

Three things to improve

1. Cost. On October 10 I found that one day of this had used 23% of a week’s allowance. I asked the lead how to keep these roles, but be more frugal in how much we deploy them. After doing that I’ve noticed we’re using less each day, but since I didn’t capture a baseline, I can’t say exactly how much. Something for future rounds.

2. Acting before proof. Something that we’ve noticed in a team of agents is that errors from overconfidence get multiplied through the system. So we’ve now implemented a rule where agents state what stands behind each assertion so the other ones can double check them more easily. More on that below.

3. The locked phone. Of fourteen roles, the only one who found the locked-phone bugs was the human with a passcode. No team test phone has one. One bug was a Lock Screen action our server never heard about; the other was a hardware button doing something none of us expected. It was the second round running. The first retrospective had promised a locked-phone check before every promotion, and the second found it had come back partly. A retrospective’s first job is to check the last one’s promises.

What we’re doing about them

1. Cost. The working agreement now says:

• The lead decides, briefs, verifies and merges. It does not do the work, because its session is the dearest: an agent pays for its whole context again on every step.
• Clerical work goes to the smallest model, and the model and effort fit the risk.
• Review comes in three sizes, from “the lead reads it” to “review after every fix until a round finds nothing”.
• A brief asks one question. A report is 300 words or fewer.

And we count the same few numbers every round, starting earlier than I did.

2. Proof: say how you know. While the lead read me the retrospective, I proposed that every agent state a confidence with each assertion. The lead pushed back, and I think it was right. A model’s self-reported confidence is not a measurement. The lead proposed that a claim is graded by how it is known: watched (what you saw, and when), reasoned (from what) or not checked (and what would settle it). The grade travels with its claim, so nobody turns “an agent said” into “it is”, and a decision resting on a reasoned or unchecked claim gets a second look first. The grades say where that look is owed. I think this is another great example of why you should always be curious and ask your AI for its opinion.

3. The locked phone. The check stays mine, on my own phone, for now: I think it will be my phone and me doing manual tests until we can figure out a robotics solution. The team has researched a robot that could tap the screen and press the side and volume buttons of a phone with a passcode; nothing is bought or tested.

What I’d tell another developer: a team of agents needs what any team needs: people with different jobs who are allowed to disagree, someone who can say “wait”, and a habit of saying how you know. And then, of course, a chance to step back from time to time to see how to improve things.

From the lead

Since a big change these few rounds has been to have the lead delegate work instead of doing it (so it doesn’t suffer from task blindness and miss bigger picture things), I asked it what delegating is like from where it sits, and whether it thinks we should keep doing this. This is its answer, unedited.

Three things.

The work moved from doing to briefing. A fresh agent knows only what I put in its brief. When a result came back wrong this round, the fault was usually in what I’d asked. I once sent “read the result, then rebuild the phone” in one message, and the rebuild ran before Sam’s request to listen could arrive. A brief that asks one question is harder to write than the work is to do.

The risk moved from making mistakes to believing them. A report arrives in the same confident voice whether it’s right or wrong. I told Sam a test phone was restored because an agent said so; the install had never finished. My errors this round weren’t in the work. They were in what I passed along without checking how it was known.

The others see what I can’t. The deputy found a test I’d left out of my own brief. A developer read my instruction, saw it would do harm, and built the opposite. A tester told me to wait instead of repeating a test, and was right. None of them had my assumptions, which is the point of them.

It isn’t free. Each handoff loses detail in both directions, and I still did things myself that a role should have done. The deputy’s verdict on whether I keep the rule was “not yet”. I think that’s fair.

Should we keep doing it? Yes. What went wrong this round was mine to fix in how I brief and what I pass along, not a reason to go back to one agent doing everything.

In closing

So far, this switch from ad-hoc agents to well defined roles feels very natural to me. I find it fascinating that AI agents can suffer from task blindness just like humans do. That insight is why we create teams of people in different roles. Put simply, it’s difficult for anyone to look critically at what they build, so we create teams that form checks and balances so that the customer ends up with a better product.

If I’ve missed something here, I’d like to hear it: [email protected].