Building a multi-agent orchestrator inside a desktop app

An Agentz agent with everything enabled has a lot of tools: memory stores, a knowledge graph, SQL, documents, a code sandbox, stock data, scheduling. Handing an LLM sixty tool definitions and hoping it picks well is a known way to get an agent that's mediocre at everything. This week the app finished moving to a different shape: one orchestrator that owns the conversation, delegating to specialist sub-agents that each own a domain.

What died: utility agents

The previous design had "utility agents" — globally registered helpers in an always-on map, steered by a shared config object of store flags whose prompts pleaded things like "ALWAYS respect tool_config — never search disabled stores." If you've ever written a prompt that begs the model to respect configuration, you already know how reliably that goes. They also weren't per-agent: every agent got the same utilities whether it wanted them or not.

The replacement makes the configuration structural. Each specialist is enabled or disabled per agent in a config table, with optional per-agent model and provider overrides. A disabled specialist doesn't exist as far as that agent's LLM is concerned — no tool definition, no prompt, nothing to "respect".

How delegation works

Eight specialists exist so far: memory, knowledge graph, database, documents, code, stock, portfolio, and scheduling. For each enabled one, the registry generates a delegation tool named specialist_<id> whose input is essentially a task briefing:

{ "task": "...", "context": "...", "async": false }

The important trick is what the orchestrator loses: any tool belonging to a specialist's tool groups is filtered out of the orchestrator's own tool set while that specialist is enabled. The orchestrator can't run SQL itself; it has to brief the database specialist. Without that forcing function, models happily bypass the specialists they were given.

When a delegation tool fires, the specialist executor spins up a genuinely separate agent: its own system prompt, its own tool set resolved from its tool groups, its own LLM loop with up to 50 tool iterations, and its own model. Model selection inherits sensibly — a specialist can pin a model, otherwise it falls back to a per-provider default, which for Anthropic is Haiku. That last bit matters more than it looks: the orchestrator can run on an expensive model for judgment while a memory-search specialist burns pennies.

Specialists can also run async: the delegation returns a task ID immediately, the orchestrator keeps working, and a dedicated await_specialist_results tool collects results later with a timeout. And because specialists get delegation tools too, they can call each other — which immediately raises the question every multi-agent system must answer: what stops A calling B calling A forever? Two guards: the call stack travels with each request and any specialist already on it is rejected (Circular call detected: memory -> graph -> memory), and recursion depth is capped at three.

Making the invisible visible

The part I'm happiest with isn't the delegation — it's that you can watch it. Every tool start/complete event carries a depth and the identity of the specialist running it. The renderer's hierarchy tracker folds this event stream into a live tree:

orchestrator
├── specialist_memory          ✓ 2.1s
│   ├── search_memory          ✓
│   └── get_facts              ✓
└── specialist_stock           ⟳ running (async)
    └── get_stock_price        ⟳

Parent resolution is the fiddly part: a tool event at depth 1 attaches to whichever specialist node is active; a nested specialist at depth 2 attaches to the active specialist one level up. It's a heuristic — events are a flat stream, and the tree is reconstructed, not transmitted — and getting it wrong produces trees that lie, which is worse than no tree.

As of this week the trace also persists: a serialised execution tree is stored alongside each assistant message, so reopening a conversation shows not just what the agent said but the full delegation tree of how it got there. For debugging a misbehaving agent, this has already paid for itself: "why did it answer without checking memory" is now a thing you can see rather than infer.

Every specialist LLM call is also metered — provider, model, input and output tokens, computed cost — and written to a usage table. Multi-agent systems multiply LLM calls quietly; a single user message can fan out into half a dozen specialist loops. The first day of cost tracking was clarifying in the way bank statements are.

Honest rough edges

It's a week old and it shows in places. Specialist responses are hard-capped at 2048 output tokens, which is fine for "search memory" and cramped for "summarise this document". The usage tracker opens a fresh database connection per subagent call. And the hierarchy tracker still contains a code path for the old utility_ prefix that no longer exists anywhere — dead code as a memorial.

There's also an obvious missing piece: specialists run one at a time. The orchestrator briefing two specialists sequentially, when their tasks are independent, is pure wasted wall-clock. Parallel delegation is next on the list.

Why bother, in one paragraph

The naive alternative — one agent, all tools — degrades in three ways as tools grow: tool choice gets worse, the system prompt becomes a treaty negotiation between domains, and everything runs on the most expensive model because something in the conversation needs it. The orchestrator-specialist split fixes all three with one structure: each prompt stays small and single-minded, each tool set stays shortlisted, and each domain runs on the cheapest model that handles it. The price is machinery — delegation plumbing, recursion guards, a tree visualiser — but the machinery is written once. Prompts that beg, you rewrite forever.