The fleet's biggest bottleneck is the fleet

At about 21:56 UTC on October 11, the login keychain on my laptop locked, and the agent fleet that lives there made it much worse. Clankie's service spawned a separate security find-generic-password process for every credential read, around ten at once, and each one raised its own password dialog. I typed my password again and again. The dialogs kept coming.

The crash reports show securityd aborting twice and a WindowServer watchdog timeout. The restart wiped five requests from the merge queue. Three messages I'd sent in the #house-hunting Discord room while the Mac was down failed, and nothing retried them once the service came back.

The fleet filed all of that itself. It files most of its bugs. The Clankie project in Linear started on August 16 and now holds about 650 issues; from late September on, nearly every one was written by the Clankie app, not by me. That's the point. In the words of one issue, "i want my clankie to be fully hands off for things like completing linear issues": notice its own breakage, write it up with evidence, and fix it.

Part of the October 5 spike is bookkeeping (see the note on the data at the end). The urgent slice is the honest signal, and it clusters in the last week.

I read the whole project end to end expecting a story about making agents smarter. What I found was a story about one Mac with eighteen cores. The fleet's biggest bottleneck is the fleet.

The first post covered who Clankie is and the second the team he runs. This one is about what that team does to the laptop it lives on.

Five words you'll need

• Seat (or pane): one agent, a Claude Code or Codex session running in a terminal pane on my laptop. Clankie hires them and leads them.
• Worktree: each seat's own checkout of the repo, so seats can edit in parallel without stepping on each other's files.
• The gate: the full test and typecheck run a change must pass before it lands on main.
• The queue: clankie integrate, the one door onto main. It batches changes, runs the gate once per batch, and bisects failures.
• Receipt: the record attached to every message between agents so nothing is sent twice or silently lost.

One Mac, eighteen cores

Every seat shares one laptop, one filesystem and one repo. That's why they can hand each other files, and it's why they can ruin each other's day. Every typecheck, test run, build and iOS simulator lands on the same 18 cores, and Clankie's own service has to stay responsive on top of them. The tracker reads like a list of places where that budget leaked.

The service starved itself. On October 5 Clankie's service sat near 99% of a core for about five hours. A CPU profile put 45% of the time in walking Codex session trees synchronously and 44% in spawning processes. The cause was a fleet refresh with about fifteen callers and no coalescing, so with nine long-running Codex seats it ran back to back. Once callers shared one in-flight refresh and discovery was cached, the service dropped to 2.9–6.4% CPU and /health answered in about a millisecond.

The builders starved everyone. The same night, a burst of seats each started a typecheck in its own worktree, at 1.6–2.5 GB apiece, within two minutes, while six iOS simulators were booted. Load average hit 856, about 48 times the core count, and swap reached 35 GB. Worker calls to Linear timed out. The fix was a heavy lane: expensive commands wait for one of a few slots.

The fix became the incident. Slots stayed held by dead processes. Simulators were handed out by retry polling rather than a queue, so grants came out of order, and one bad waiter could block the whole line. On October 10 the load ratio was 0.52 with 50 GB free, yet every slot was taken and 13 jobs sat waiting. Slots became a ceiling, and admission now depends on measured pressure.

One process locked everyone out. Also on October 10, every fleet proof on the Mac was refused for about 22 minutes: worker tool calls, message_clankie, even the queue's own tests. One worker had run find across every worktree and was holding about 32,700 file descriptors. The helper that proves which process owns a socket gives up when any same-user process holds 16,384 or more. Now it skips a process it can't list unless that process could be the one being proven, records the skip, and agents are told to bound their searches.

Each fix exposed the next load source. For a stretch, the Mac was the bottleneck for its own agents.

Every stall also stopped the agents that would have fixed the next one.

The merge gate sets the speed limit

Once the agents could run, the next ceiling was landing their work. On one early-October day, a single worker ran ten full gates over about three and a half hours and landed nothing, while main took 221 direct pushes in 24 hours. After that I made the queue the only way onto main, for everyone, my own panes included.

The queue then needed the same hardening as everything else. A deploy restart killed the process holding its lock, and the queue refused every request until someone cleared it by hand; locks are now reclaimed when their owner is dead. A hold with no expiry blocked deploys for over fifty minutes without even stopping direct pushes. Retries went to the back of the line and re-ran repos that had already passed.

Across the most recent 101 batches, landing took a median of 8.3 minutes, with a p90 of 16.5 and a max of 70. That beats three and a half hours of nothing, but it's still the fleet's throughput ceiling. In one batch, Vitest reported 624 seconds, and 508 of them were module import.

The tests were nearly as big as the code. On October 9, Clankie had about 249k lines of tests against 261k lines of source. Most breakage came from agent-written tests that pinned incidental detail, so I asked for most of the unit tests to go. Pruning has rules: every deleted test gets a reason, and nothing that is the only coverage of a trust boundary goes.

Agents write tests faster than I'll ever read them. Review discipline has to live in the queue and its rules, because it can't live in me.

A receipt without an expiry is a lock

The most repeated failure in the project isn't about CPU. It's delivery.

Receipts are what let agents promise to deliver every message exactly once. The cost is that one unconfirmed delivery can fence off every message after it. On October 5, a message_clankie request timed out, or was killed by a deploy restart, and left a fence on its pane. From then on, every Codex worker's report got the same answer: "the original receipt is unresolved; no replacement was sent." About three and a half hours of finished work disappeared behind it.

The fix gives each attempt an instance id and a deadline. If a new instance takes over or the deadline passes, the fence is sealed as "not sent" and the next message goes through.

The same shape kept coming back. A false "already answered" acknowledgement left a worker waiting for a reply it never got. A long brief to a worker on my PC failed with sendmsg: Message too long and blocked every follow-up until the lead settled it by hand.

Exactly-once delivery is a promise you can keep only if every uncertain state has an owner and a way out.

Who gets to act as whom

A shared machine also means shared authority. Several issues are decisions about that, not bugs.

I chose "it just works" for connected accounts. Any pane in a session Clankie leads gets standing access to every verified connected tool through two meta-tools: one searches the catalog, one calls a tool. The issue states the trade-off plainly: anything running in a linked session can act as Clankie's connected account. I considered per-process proof and decided the friction wasn't worth it on a machine I own.

Then an agent spoke as me. A Claude session in my own profile had Claude Code's Linear connector signed in as me, and it posted evidence comments under my name. That put words in Linear's history I never wrote. Worse, it tripped Clankie's owner-only wake rule, so the lead was woken by an agent's own output dressed up as mine. Linear writes from worker sessions now go through Clankie's tools, so the tracker always records the Clankie app as the author, and a hook blocks the direct connector.

Permissions ran into the same problem from the other side. A worker's auto-mode safety check kept refusing its commands. Meanwhile the lead kept telling it the blocker was already answered, when no answer had reached it. I first called the fix "yolo"; it shipped as maximum trust mode, one owner switch that runs every harness Clankie launches without its permission guardrails, mapped to each harness's own flag.

A fleet meant to work without me can't stall on a prompt nobody will see. It also can't sign my name to things I didn't say.

Keeping the tracker honest

The tracker is part of the system, not a report on it, so it has to be true. On October 9, more than a hundred issues sat in In Progress, many abandoned for days or weeks after a worker hit its context limit or was closed. I couldn't tell what was actually happening. The rules that came out of it:

• In Progress means a live seat is working on it right now.
• Done means landed, with an evidence comment.
• Todo means paused or orphaned, with a one-line note on why it stopped.

Clankie moves an issue when its seat closes or hands off, and a daily check flags In Progress issues with no live owner.

A smaller incident stays with me more. close_worker_pane closed a worker while a prompt I'd typed sat unsent in its input box, though the tool promises to refuse in that case. Undo didn't bring the pane back; the instruction had to be rebuilt from the close record. Draft detection now recognizes a non-empty input line in both Claude Code and Codex and keeps any draft verbatim.

A tool that destroys a person's half-written text has its priorities backwards. The text is worth more than the pane.

Back to the keychain

The keychain fix is the kind I trust. Each credential is read once and cached in memory. Only one read per credential can be in flight, with backoff. The CLI fails fast with "keychain locked." On a scratch keychain, ten concurrent reads became one security process, and reads against a locked keychain spawned none and failed in 29 ms. Reviewing its own change, the worker also caught a follow-up bug: a failed locked write could leave a pending revision behind.

The issue is careful not to claim the flood caused the crashes, only that it was the dominant keychain load and that it blocked me from recovering.

As of October 11 the fix still hadn't landed. Unrelated gate failures kept blocking it: a Cargo registry HTTP/2 framing error, and a test that took 60 seconds under load and 4.4 seconds alone. So did the retry fix for the dropped Discord messages. That's the whole post in one issue: the fleet found the bug, proved the fix, and then waited on the machine and the gate.

I think of Clankie as a character with a memory and a voice, and he is one. But most of the last six weeks went into plumbing that lets a dozen agents share one machine without starving it, losing each other's messages, or speaking as me. The project's trust boundaries live in that plumbing, and it's what lets me leave the room.

The fleet even shaped this post. I read the project through Clankie's own Linear bridge, the same one from "Who gets to act as whom," and it refused to page past somewhere between the 75th and 150th result of any filter. The fleet's limits decided what I could see of the fleet's limits.

P.S. A sibling project, Clankie Work, is building a tracker native to the fleet: an evidence store, an agent API, and guided reviews. When I'm writing rules to patch a tool's idea of truth, status and receipts from the outside, the tool is probably the wrong one. More on that in its own post.

A note on the data

• Coverage. Because the list endpoint refused deep pages, I worked around it with state, priority and label filters. Older Done issues with no label or parent may be missing, probably a modest number from September.
• Detail. Descriptions in the list are truncated, so the incident details come from the few dozen issues read in full.
• Counts. Counts come from agent-written digests, not a Linear export. The digests flagged 88 of the 107 unprioritized issues in the week of October 5 as retroactive records of work that had already landed.
• Share of effort. A digest estimated that about 60% of high-priority issues are about machine load. That's an agent's judgment, not a measurement, and I've left it out of the argument.