Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 03:37 UTC
fix(docs): page toolkit data through `@composio/client` (#5481)
## Summary Toolkit pages use cursor pagination so Next.js can cache each response separately. The meta-tool reference generates offline from its checked-in schema snapshot. - Live lists of toolkits, tools, and trigger types use `@composio/client` and `@composio/client/pagination`. - `generate:toolkits` follows `next_cursor`, so the next sync can include all 1,208 `microsoft_365` tools. The committed snapshot has 1,000. - Docs - Toolkit Parity checks the 30 largest deployed `.md` pages by production tool count and the 30 largest by snapshot count, plus `/api/toolkits/instant`, every six hours. It opens one tracking issue per outage and sends a Slack alert. - Live toolkit render traversals abort after 15 seconds and fall back to the snapshot. - Oxlint rejects raw `fetch`, `fetcher`, and `fetchWithRetry` calls to `/api/v3*`. - For docs runtime configuration, replace `COMPOSIO_API_BASE` with the host-only `COMPOSIO_BASE_URL`. ## Context Tableau's toolkit grew from 10 to 436 tools. The snapshot has listed all 436 since Nov 13, yet `docs.composio.dev/toolkits/tableau.md` kept listing the old 10 for a week. The Vercel API key was not the cause. It is a valid production key, and a staging key would have answered with a 401, which falls back to the snapshot. The toolkit page and the `.md` route fetched each toolkit's tools in one request. Next.js stores a fetch response in the Data Cache only when the entry is at most 2 MiB, and the entry holds the body base64-encoded, so a response has to stay under about 1.5 MiB. Tableau's response is 3.26 MB. Vercel logged it on every request: ``` Failed to set Next.js data cache for https://backend.composio.dev/api/v3/tools?toolkit_slug=TABLEAU&toolkit_versions=latest&include_pricing=true&limit=1000, items over 2MB can not be cached (4343788 bytes) ``` When that write fails, Next.js keeps serving the entry it already had. Here that was the 10-tool response from before the upgrade, so every background refresh downloaded the new list and then discarded it. The Data Cache outlives deployments, so redeploying changed nothing. The same warning shows up for ACTIVE_CAMPAIGN, TWITTER, WORKDAY and PAGERDUTY. The snapshot generator had a separate cut-off. It asked for `limit=1000` and never read `next_cursor`, so `microsoft_365` (1208 tools) stopped at 1000. Two alternatives don't solve it. - Dropping the fields the pages don't render. `/tools` has no field selection, and the Data Cache stores the response as received. Even reduced to the rendered fields, `microsoft_365` measures 2.25 MB. - Raising the 2 MiB limit. It's fixed in Next.js and only a custom `cacheHandler` lifts it, which would replace Vercel's Data Cache with storage we run. This PR stays within Vercel's design. ## Implications - Live catalog reads and toolkit generation use v3.1. In the original production check, v3 and v3.1 returned byte-identical bodies for tools, trigger types, toolkits, and the changelog. - A large toolkit now takes several requests on a cache miss (18 for Tableau), each cached as its own entry. Runtime list requests retry up to twice, and the whole traversal aborts after 15 seconds. - Tool lists request 25 items per page, toolkit lists 500, and trigger lists 50, which is the cap `/v3.1/triggers_types` documents. - Tableau's largest page is 1.23 MB, a 1.64 MB entry, or 78% of the limit. One Tableau tool accounts for most of that page, so a smaller page size barely helps. If a single tool grows past about 1.5 MiB, no page size can make it cacheable. The parity check alerts at 85%. - The parity run observes up to 60 toolkits, 30 ranked by production count and 30 by snapshot count. Ranking on one side alone let a collapsed snapshot count drop the affected toolkit out of the check, which is the page most worth watching. The second ranking roughly doubles the run's request volume. - The HTML toolkit page now shows triggers for the latest toolkit version, as the `.md` route already did. - Toolkit generation, `/api/toolkits/instant`, and the snapshot-miss lookup use production. Toolkit generation rejects non-production `COMPOSIO_API_BASE` settings. - Meta-tool generation runs without a session or API credentials. Automatic schema refreshes depend on a production Platform HTTPS export configured through the `COMPOSIO_META_TOOLS_SCHEMA_URL` repository variable. Until that export is available, Docs - Update Data renders the reference from the checked-in `public/data/meta-tools.json`. - Meta-tool page titles are escaped into YAML frontmatter, and the schema rejects a multi-line `displayName`, so an export cannot terminate the frontmatter and inject content into the compiled page. - The stale Tableau entry sits under a URL nothing requests anymore, so no cache purge is needed. - `scripts/fetch-openapi.mjs` and `app/api/proxy` still use `fetch`. The first downloads OpenAPI documents, which aren't client operations, and the second forwards whatever request the browser playground sends. ## Review follow-up Six defects turned up in review on the first revision and are fixed on this branch. They change behavior beyond what the original description covered. - A failed page in a paged traversal now fails the generator run. The first revision logged the error, wrote `tools: []`, and exited 0, which would have republished a zero-tool page — the exact failure this PR exists to end. - The parity checker ranked its watch list by the snapshot's own counts. It now ranks on production counts too and unions both, so neither source can shrink the set. - Live render calls passed no abort signal, so a slow API could hold a render through 49 sequential pages. They now bound the traversal at 15 seconds. - Trigger lists were requesting `limit=1000` from an endpoint whose cap is 50. They now page at 50. - The toolkit changelog parsed with an `as` cast and an `Array.isArray` guard. It parses with zod, like every other payload read here. - Meta-tool titles reached YAML frontmatter unescaped while descriptions were escaped. Titles are escaped now, and a multi-line `displayName` is rejected. ## Verification - The production pagination check returned all 1,208 `microsoft_365` tools, 436 Tableau tools, 46 GitHub trigger types, and 1,829 toolkits, including 89 Instant toolkits. - The original live parity check flagged the stale Tableau page (10 of 436 tools) and passed GitHub and Twitter. After the Markdown counter fix, all 50 largest snapshot-backed pages matched their expected counts, including all 370 Datadog MCP tools. - Offline meta-tool generation reproduced the JSON and all seven reference pages byte for byte. Local tests verified HTTPS export imports and rejection of an empty export. - GitHub CI on `f5e3e2e16` is green, including Anti-slop, docs TypeScript validation, link checks, and docs tests. On the same head, the eight touched suites report 94 passing tests, and `tsc --noEmit` and `lint:links` are clean. - The original head `84434a252` passed the same jobs: 825 static and 273 integration docs tests, one integration test skipped. [Anti-slop](https://github.com/ComposioHQ/composio/actions/runs/44587069134/job/135782777866) · [Docs tests](https://github.com/ComposioHQ/composio/actions/runs/44587069403/job/135782778572) - The parity job still reports the committed snapshot as behind production until the next Docs - Update Data run, because `microsoft_365` sits at the 1,000-item cap. The message now names the snapshot rather than blaming the generator.