@devries-cameron20 / composio Mirror · read-only / commits / acb7dc025
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 07:15 UTC

fix(harness): retry rate-limited reads and deletes in examples provisioning (#5164)

This PR: - makes `scripts/examples-provision.mjs` wait and retry when staging answers 429, for `GET` and `DELETE` only; a create is never retried - honors `Retry-After` (capped at 60s), otherwise backs off 2s, 4s, 8s, 16s, 32s, up to 5 retries - moves the `fetch` call into a `request()` helper; the timeout, `redirect: 'error'` and error messages are unchanged ## Context The nightly `Examples Live` sweep has failed 61 of its last 61 runs before running any example. Provisioning dies on the `--gc` listing of MCP servers: `GET /api/v3.1/mcp/servers?limit=100&page_no=6 -> 429` (an HTML rate-limit page from the edge). Because GC never gets past listing, the leaked `examples-*` MCP configs it should delete keep piling up, which adds pages and makes the 429 more likely on every run. ## How has this been tested? Ran `node scripts/examples-provision.mjs --gc` against a stubbed `fetch` that serves three pages of stale `examples-*` MCP servers and rate-limits on purpose: - page 2 answered 429 with `Retry-After: 1`, then 429 without it: the script waited 1s, then 4s, then read the page - a `DELETE` answered 429 once: retried after 1s and succeeded; all three configs were deleted - a `POST` answered 429: failed immediately, with no retry

Alberto Schiabel committed Oct 30, 2026 at 19:34 UTC acb7dc0252a4d9494a66e20bbbddbb72ba24c94f
1 file changed +30 -6
scripts/examples-provision.mjs
+30 -6
@@ -80,8 +80,37 @@ if (!API_KEY) {
80 process.exit(1);
81 }
82
83 +// The backend's edge rate-limits bursts of requests with a 429. Listing and
84 +// deleting are safe to repeat, so wait and retry those; never retry a create.
85 +const RETRYABLE_METHODS = new Set(['GET', 'DELETE']);
86 +const MAX_RATE_LIMIT_RETRIES = 5;
87 +
88 +const rateLimitDelayMs = (res, attempt) => {
89 + const retryAfter = Number(res.headers.get('retry-after'));
90 + if (Number.isFinite(retryAfter) && retryAfter > 0) return Math.min(retryAfter, 60) * 1000;
91 + return Math.min(2 ** (attempt + 1), 32) * 1000;
92 +};
93 +
94 async function api(method, path, body) {
84 - const res = await fetch(`${BASE_URL}${path}`, {
95 + for (let attempt = 0; ; attempt++) {
96 + const res = await request(method, path, body);
97 + if (res.status === 429 && RETRYABLE_METHODS.has(method) && attempt < MAX_RATE_LIMIT_RETRIES) {
98 + await res.body?.cancel();
99 + const delayMs = rateLimitDelayMs(res, attempt);
100 + report(`rate limited on ${method} ${path}; retrying in ${delayMs / 1000}s`);
101 + await new Promise(resolve => setTimeout(resolve, delayMs));
102 + continue;
103 + }
104 + const text = await res.text();
105 + if (!res.ok) {
106 + throw new Error(`${method} ${path} -> ${res.status}: ${text.slice(0, 300)}`);
107 + }
108 + return text ? JSON.parse(text) : undefined;
109 + }
110 +}
111 +
112 +function request(method, path, body) {
113 + return fetch(`${BASE_URL}${path}`, {
114 method,
115 headers: { 'x-api-key': API_KEY, 'content-type': 'application/json' },
116 body: body === undefined ? undefined : JSON.stringify(body),
@@ -92,11 +121,6 @@ async function api(method, path, body) {
121 // stripped), so a 3xx would hand the key to whatever host it names.
122 redirect: 'error',
123 });
95 - const text = await res.text();
96 - if (!res.ok) {
97 - throw new Error(`${method} ${path} -> ${res.status}: ${text.slice(0, 300)}`);
98 - }
99 - return text ? JSON.parse(text) : undefined;
124 }
125
126 // Two pagination shapes are in play. Most v3.1 collections (auth_configs,