Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 19:45 UTC
fix(docs): paginate toolkits fetch so the full catalog reaches the site (#3745)
This PR: - fixes the `docs.composio.dev/toolkits` search returning 0 results for toolkits like Workday that exist in the catalog - paginates `fetchToolkits()` in `generate-toolkits.ts` by following `next_cursor`: `GET /api/v3/toolkits` silently caps `limit` at 1000 per page and defaults to usage ordering, so the previous single-request fetch shipped only the top-1000 toolkits — about half of the ~2.1k catalog - dedupes across pages by slug (pages can overlap when the catalog shifts between cursor fetches) and throws instead of silently truncating if the catalog ever outgrows the page guard, mirroring the dashboard's `listToolkits` procedure - fixes the `docs-update-data.yml` workflow, broken since the sessions-first docs rewrite (https://github.com/ComposioHQ/composio/pull/3637) moved the generated meta-tools MDX to `docs/content/toolkits/meta-tools/`: `create-pull-request` aborted on the stale `docs/content/reference/meta-tools/` pathspec, so every scheduled data sync since then failed at the Create Pull Request step and docs data went stale - updates the request-volume notes in `fetch-with-retry.ts` and `scripts/README.md` (~3000 → ~6500 requests per run); the existing 429 backoff already absorbs the larger run ## Context The toolkits landing page filters a build-time snapshot (`public/data/toolkits-list.json`) client-side, so every toolkit missing from the snapshot is unsearchable and its detail page has no data. The snapshot regenerates via the scheduled `docs-update-data.yml` workflow; with both fixes in place the automated data PR picks up the full catalog and the "Browse N toolkits" count grows from 1000 to the real catalog size. Verified offline by running the script against a stubbed paginated backend (1500 toolkits across 2 pages): all pages are fetched, ordering is preserved, no duplicates. `bun run types:check` and eslint pass. A manual `workflow_dispatch` of the fixed workflow on this branch produced the full-catalog data PR.