id: 0004-per-client-rate-limit
type: decision
owner: justin
reviewed: 2026-10-15
status: accepted

0004 Per-client rate limit in a Durable Object

Context

The tools are cheap, but not free, and a model stuck in a loop will call one hundreds of times a minute. The usage counters in KV cannot stop that: they are approximate and eventually consistent by design ([[0003-usage-counts-in-kv]]). A limit has to be exact at the moment it is checked, which needs a single writer per client.

Workers has a rate limiting binding, but its counters are per location and approximate, and it cannot tell us how long a refused client should wait.

Decision

• One RateLimiter Durable Object per client, named by cf-connecting-ip. It holds a token bucket: 30 tool calls in a burst, refilling one token every two seconds.
• Only tools/call spends tokens, and only after the arguments pass the input schema. initialize and tools/list are free.
• A refused call returns an isError tool result that says how many seconds to wait, in line with [[0002-tool-errors-are-results]]. Not an HTTP 429: most clients surface a 429 as a transport failure the model never reads, or retry it at once.
• When the bucket is full again the object deletes its storage from an alarm, so idle clients cost nothing.
• If the Durable Object call fails, the tool call is admitted and the error logged.

Consequences

• Every tool call now makes one RPC to a Durable Object before it runs. The object lives near the client's first call, so that hop is usually short.
• Clients behind one NAT or proxy share a bucket. Without auth there is nothing better to key on; if the server ever takes tokens, the key should become the token.
• Failing open means a broken limiter lets a loop through. We prefer that to failing every call when the limiter has a bad minute.
• The stateless transport ([[0001-stateless-streamable-http]]) is unchanged: the limiter keeps state per client, not per session.