Limit tool calls per client with a Durable Object #1
OpenJustin Torre wants to merge 2 commits into main from rate-limit-tool-calls
JTJustin Torreopened this
A model stuck in a loop can call the same tool hundreds of times a minute, and until now nothing stopped it. This adds an exact per-client limit: each caller IP gets a token bucket of 30 tool calls that refills one every two seconds.
What changes
New RateLimiter Durable Object (src/rate-limiter/), one per cf-connecting-ip. The bucket arithmetic is a pure function in bucket.ts; the object only loads, saves and returns the decision. An alarm deletes its storage once the bucket would be full again, so idle clients leave nothing behind.
admit hook in createServer, asked once per tools/call after the arguments pass the input schema. initialize, tools/list and calls with bad arguments are never charged.
Refusals are tool results. A limited client gets isError: true with the wait, for example "Wait 2 seconds before the next tool call", so the model can back off. We do not answer 429; see the decision for why.
Fails open. If the Durable Object call throws, the tool call goes through and the error is logged.
Refused calls are not counted in /stats.
Architecture and docs
system.yaml gains the rate-limiter component and the mcp-server -> rate-limiter RPC edge; the runtime view includes it.
The tool-call flow gets the token check between validation and the tool running, and a new flow, rate-limited-call, shows the refused path end to end.
New decision 0004-per-client-rate-limit and concept rate-limits; tool-design, http-transport, the client guide and the README are updated.
Testing
test/rate-limit.test.ts covers the bucket: a full start, refusal with the wait rounded up, refill up to the capacity, and a clock that steps backwards.
test/worker.test.ts sends 31 calls from one IP and checks that the 31st is refused, that another IP still gets through, that the first IP gets through again after two seconds, and that a broken limiter admits the call.
Checked under wrangler dev: the 31st call from curl is refused, and two seconds later the next one succeeds.
npm test, npm run typecheck, rebasely kb check and rebasely arch check all pass.
Worth a look
Clients behind one NAT share a bucket. With no auth there is nothing better to key on yet.
Every tool call now makes one RPC to a Durable Object before it runs.
A model stuck in a loop can call the same tool hundreds of times a minute, and until now nothing stopped it. This adds an exact per-client limit: each caller IP gets a token bucket of 30 tool calls that refills one every two seconds.
What changes
RateLimiterDurable Object (src/rate-limiter/), one percf-connecting-ip. The bucket arithmetic is a pure function inbucket.ts; the object only loads, saves and returns the decision. An alarm deletes its storage once the bucket would be full again, so idle clients leave nothing behind.admithook increateServer, asked once pertools/callafter the arguments pass the input schema.initialize,tools/listand calls with bad arguments are never charged.isError: truewith the wait, for example "Wait 2 seconds before the next tool call", so the model can back off. We do not answer 429; see the decision for why./stats.Architecture and docs
system.yamlgains therate-limitercomponent and themcp-server -> rate-limiterRPC edge; the runtime view includes it.rate-limited-call, shows the refused path end to end.0004-per-client-rate-limitand conceptrate-limits;tool-design,http-transport, the client guide and the README are updated.Testing
test/rate-limit.test.tscovers the bucket: a full start, refusal with the wait rounded up, refill up to the capacity, and a clock that steps backwards.test/worker.test.tssends 31 calls from one IP and checks that the 31st is refused, that another IP still gets through, that the first IP gets through again after two seconds, and that a broken limiter admits the call.wrangler dev: the 31st call from curl is refused, and two seconds later the next one succeeds.npm test,npm run typecheck,rebasely kb checkandrebasely arch checkall pass.Worth a look