Architecting resilient swarms with Kafka event streams

A swarm of AI agents looks like an orchestration problem. In practice it is a message-delivery problem. Every agent is a consumer that can die halfway through a side effect, and the orchestrator has to be able to resume without duplicating work and without losing it.

Most designs that fail here make the same mistake: treating the broker as a transport detail. Once delivery semantics are an architectural decision rather than a configuration, each individual agent ends up inventing its own recovery strategy — and they always diverge.

The delivery contract that actually matters

Three properties, in order of practical importance: no accepted message may be lost; no message may be processed with a duplicated effect; and ordering must hold within an aggregate, never globally.

1. Set acks=all with a replication factor of at least 3 and min.insync.replicas=2. A broker that acknowledges a write without an fsync is a future incident.
2. Derive partition keys from the aggregate, never from an arrival timestamp. The key is what preserves per-aggregate order while spreading load.
3. Record the offset and the side effect in the same transaction. It is the only way to make an operation genuinely idempotent rather than conventionally so.

Partitioning is a domain decision

Round-robin spreads the load and destroys ordering. Key-per-aggregate preserves ordering and creates hot spots. For an agent swarm the right answer is key-per-aggregate with an optional salt layer, applied only when one specific aggregate starts to dominate traffic.

# partition key: aggregate + optional salt
def partition_key(aggregate_id: str, salt: str | None = None) -> bytes:
    if salt:
        return f"{aggregate_id}#{salt}".encode()
    return aggregate_id.encode()

Idempotent consumers in practice

Idempotency does not come from a library. It comes from a database constraint or a de-duplication record consulted before the effect takes place. With AI agents there is one extra layer: the model response must be addressed by the hash of the prompt plus the model version, otherwise a rebalance re-runs inference you already paid for.

Rebalance, DLQ, and what to do when an agent goes quiet

A rebalance is normal, not a failure. Treating it as an error is the failure. Set the session timeout above your worst-case processing time, apply exponential backoff with jitter on retry, and use a per-partition dead-letter queue with retention long enough to audit.

Finally, monitor the re-processing rate as a first-class metric. Zero consumer lag with a high de-duplication rate is a system silently discarding work — the worst possible failure mode, because every number still looks healthy.