@devries-cameron20 / composio Mirror · read-only / commits / bf2977814
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 09:45 UTC

docs: establish Context7 ownership and retrieval regression checks (#4872)

## Summary Addresses [DEVREL-39: Establish ownership and regression checks for Context7](https://linear.app/composio/issue/DEVREL-39/establish-ownership-and-regression-checks-for-context7). - Replace the stale “100+”/“equip's” description with count-free positioning from [production docs](https://docs.composio.dev/llms.txt): managed authentication, tool discovery, and tool execution. Public pages disagree on counts, so no new hard count is introduced. - Verify `next` is the GitHub default and is present in public retrieved source URLs. Leave `branch` omitted (Context7 defaults to the repository default), and leave **all inclusion/exclusion rules unchanged**. - Add the official `$schema` and split the overlength tool-version rule into rules below Context7's [255-character limit](https://context7.com/schema/context7.json). Preserve the v3.1 policy and clarify that scopes/required is v3.1-only. Production [REST guidance](https://docs.composio.dev/reference.md) agrees with the policy. The previous long rule was absent from returned library rules; this observation does not prove why it was omitted. - Add 15 source-backed developer questions, a dependency-free Node runner, six offline regression tests, and an opt-in manual Actions workflow that retains JSON retrieval evidence even on failure. Only offline tests join the default test command. - Add `CONTEXT7.md`: explicit human-owner TODO, claimed-settings verification checklist, manual refresh after each successful production docs deployment, and a same-suite before/after evidence gate for any future folder change. No SDK runtime, generated files, folder rules, or Linear issue state/comments changed. No Context7 login, claim, refresh request, or dashboard setting change was attempted. ## Verification Passed: - `node --test test/context7-check.test.mjs`: **6/6** tests. Covers stale facts, alternatives/whitespace, metadata/rule false positives, malformed/pending/rate-limited retrievals, continuing after errors, credential redaction, and owner-rule length limits. - `pnpm run test:context7` passed before the additional schema-limit test; the final six-test suite passed via the direct command above. - `pnpm typecheck`: **14/14 tasks** successful. - `node_modules/.bin/tsc --allowJs --checkJs --noEmit --target es2023 --module nodenext --types node --skipLibCheck scripts/context7-check.mjs test/context7-check.test.mjs`: passed against final files. - `pnpm run lint`: passed with existing warnings. JavaScript is excluded by the root lint configuration; additionally linted the new `.mjs` files with a temporary configuration inheriting root rules and clearing `ignorePatterns`: no errors, two expected console-output warnings. Temporary config removed. - `node ts/scripts/validate-agent-skills.mjs`: 19 canonical skills/guidance invariants validated. - `node ts/scripts/test-skill-routing.mjs`: 19/19 probes passed. Not green: - `pnpm run test` initially stopped because this environment lacked mise. Installed the repository-pinned mise/Node/Bun/pnpm through the existing installer. Re-running under that pinned toolchain passed `test:toolchain`, then stopped in the unchanged installer test with `FAIL: Windows guidance (missing: Windows is not supported. Use WSL)`. Remaining default-suite stages were not reached. No unrelated installer edits made. - The live retrieval baseline exits **1** because four expected-fact groups are missing, not because the runner or network failed. These are existing retrieval gaps, not proof this PR introduced a regression. - The new GitHub Actions workflow has not run; it is dispatch-only and does not refresh the index. ## Live baseline Command: `node scripts/context7-check.mjs --output /tmp/devrel-39-context7-report.json` - Public API: [GET /api/v2/context](https://context7.com/docs/api-reference/context/get-documentation-context), `libraryId=/composiohq/composio`, `type=json`, default reranking; no API key. - Started: `2026-10-14T23:57:32.102Z`; completed: `2026-10-14T23:57:52.367Z`. - Suite SHA256: `fac50cbc31f667e80e375d40db37b58243151c3dc9de10a04da4f87a38e282a9`. - Existing public index, **not refreshed for this PR**. Public sources use `blob/next`; Context7 did not supply an indexed commit SHA. The local JSON report retains full returned snippet bodies, source URLs, expected facts, timestamps, and the local config snapshot. Local config changes are not claimed to be live. ```text PASS python-install-current-sdk FAIL typescript-install-current-sdk missing [["Node.js 22.22.3"],["ESM-only"]] PASS python-initialize-api-key PASS typescript-initialize-api-key PASS project-api-key-environment FAIL auth-config-versus-connected-account missing [["COMPOSIO_MANAGE_CONNECTIONS"]] PASS python-connect-user-account PASS typescript-connect-user-account PASS python-create-session-tools PASS typescript-create-session-tools PASS execute-tools-through-session PASS select-session-toolkits PASS current-rest-api-base FAIL rest-latest-version-defaults missing [["latest toolkit version"],["GET /tools/{tool_slug}"],["POST /tools/execute/{tool_slug}/input"],["POST /tools/scopes/required"]] FAIL rest-explicit-pinned-version missing [["remains supported"]] 15 queries: 11 passed, 4 failed, 0 errors ``` Representative retrieved evidence: - TypeScript installation returned `npm install @composio/core @composio/openai-agents @openai/agents` from `README.md`, but omitted the requested runtime/ESM requirements from the [production quickstart](https://docs.composio.dev/docs/quickstart.md). - The auth question returned connected-account precedence from `docs/content/docs/configuring-sessions.mdx`, but not `COMPOSIO_MANAGE_CONNECTIONS` from the [authentication guide](https://docs.composio.dev/docs/authentication.md). - The v3.1 default question returned `curl 'https://backend.composio.dev/api/v3/tools?toolkit_slug=googledrive'` from `docs/content/docs/tools-direct/toolkit-versioning.mdx` and direct SDK version-check examples from `ts/docs/api/tools.md`. It did not retrieve the complete five-endpoint policy documented in the current [REST reference](https://docs.composio.dev/reference.md). - The pinned-version question returned `00000000_00` from the v3.1 changelog, but omitted the continued-support fact for v3. During suite development, removed incidental full-command/package-order and example-user-ID/quote assertions before freezing the suite. No expected API policy, runtime requirement, or connection-management fact was removed to hide the four remaining gaps. ## Manual owner/admin steps required 1. Assign the accountable human, backup, and Context7 teamspace in `CONTEXT7.md`. 2. Sign into Context7 and open https://context7.com/composiohq/composio/admin. If already claimed, verify the owning teamspace and your owner/admin access. If unclaimed, click **Claim Library**, copy its generated `url` and `public_key`, merge those fields into the **existing** root `context7.json` without overwriting settings, merge to `next`, then click **Claim Library** again to verify. Never use or commit the secret API key as a claim key. 3. Inspect **Configuration** and **Versions**. Confirm the effective description, branch (`next` or empty/default resolving to `next`), unchanged folders `docs/content`, `ts`, `python`, existing exclusions, REST rules, and versions. Record the date/settings evidence in this PR. Resolve non-folder discrepancies and leave folder/exclusion changes behind the comparison gate. 4. After a successful canonical production docs deploy serves the merged revision, confirm `next` already exists in Context7's branch/version configuration. Supply an owner-authorized `CONTEXT7_API_KEY` securely and trigger the documented [refresh API](https://context7.com/docs/api-reference/refresh/refresh-a-library): ```bash curl --fail-with-body --request POST \ 'https://context7.com/api/v1/refresh' \ --header "Authorization: Bearer ${CONTEXT7_API_KEY:?Set an owner-authorized Context7 key}" \ --header 'Content-Type: application/json' \ --data '{"libraryName":"/composiohq/composio","branch":"next"}' ``` 5. A 200 means refresh started, not finished. Wait for completion, confirm the indexed branch/revision if exposed, record any unavailable revision explicitly, then rerun the baseline or dispatch **Context7 Retrieval Check**. Attach the resulting report and review every FAIL/ERROR. 6. For automatic post-deploy refreshes, first identify the canonical deployment success signal and its production revision. Observed `Production` deployments reported preview hostnames or empty environment URLs, so this PR does **not** wire a potentially wrong automatic trigger. Until that signal is verified, the owner performs step 4 after each successful production docs deploy. Preview, failed deploy, and push events are not refresh triggers. ## Acceptance checklist - **Partial:** Configuration and claimed-library settings reflect production docs. Repository description/rules/schema checked and updated; effective claimed settings and published metadata require authenticated human verification and refresh. - **Done (documentation):** Accountable owner placeholder and refresh trigger/how-to documented. Owner TODO is explicitly allowed by the issue; no person has been assigned automatically. Refresh is a manual after-success policy, not deployed automation. - **Done:** Repeatable fixed query set detects missing retrieval facts. Baseline above records 11 PASS / 4 FAIL / 0 ERROR; offline behavior tests pass. - **Done / not applicable:** Folder-rule changes require before/after evidence. No include/exclude changes in this PR; the controlled comparison procedure is documented. Designated reviewer: @jkomyno. Please review the full diff before submitting/merging. No changeset: no published TypeScript package changed.

Brendan O'Leary committed Oct 15, 2026 at 09:05 UTC bf2977814fe413e852754dda959bd40dff01838b
7 files changed +553 -4
.github/workflows/context7-check.yml new
+35
@@ -0,0 +1,35 @@
1 +name: Context7 Retrieval Check
2 +
3 +# Run after the owner confirms a production deploy and completed index refresh.
4 +# Never part of unit tests, and never refreshes or changes library settings.
5 +on:
6 + workflow_dispatch:
7 +
8 +permissions:
9 + contents: read
10 +
11 +jobs:
12 + retrieval:
13 + runs-on: ubuntu-latest
14 + timeout-minutes: 20
15 + steps:
16 + - name: Checkout code
17 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18 + - name: Set up repository toolchain
19 + uses: ./.github/actions/setup-node-pnpm-bun
20 + with:
21 + enable-caching: 'false'
22 + - name: Test runner offline
23 + run: node --test test/context7-check.test.mjs
24 + - name: Check public retrieval
25 + env:
26 + CONTEXT7_API_KEY: ${{ secrets.CONTEXT7_API_KEY }}
27 + run: node scripts/context7-check.mjs --output context7-baseline.json
28 + - name: Retain retrieval evidence even on failure
29 + if: always()
30 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
31 + with:
32 + name: context7-${{ github.sha }}
33 + path: context7-baseline.json
34 + if-no-files-found: warn
35 + retention-days: 30
CONTEXT7.md new
+99
@@ -0,0 +1,99 @@
1 +# Context7 ownership and retrieval checks
2 +
3 +Keep Context7's Composio index aligned with production documentation, then check that developers can retrieve concrete SDK examples.
4 +
5 +Use the repository's pinned Node.js toolchain. After a completed refresh, capture a baseline from the repository root:
6 +
7 +```bash
8 +node scripts/context7-check.mjs --output context7-baseline.json
9 +```
10 +
11 +Public retrieval can run without a key, subject to Context7's quota. If needed, supply `CONTEXT7_API_KEY` through your shell's secret manager or CI secrets. Never commit a key or put one in a command example.
12 +
13 +## Ownership and verified settings
14 +
15 +- **TODO: accountable owner:** name one human who can access the library admin panel.
16 +- **TODO: backup and teamspace:** name a backup human and the Context7 teamspace that owns access.
17 +- **UNVERIFIED:** claimed ownership, team permissions, and effective admin settings require the owner's authenticated inspection. No claim, dashboard change, or refresh was performed for this change.
18 +
19 +The repository and public production documentation were checked. Preserve this inventory:
20 +
21 +| Setting | Checked state |
22 +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
23 +| Library | `/composiohq/composio` |
24 +| Description source | [Production documentation index](https://docs.composio.dev/llms.txt): authentication, tool discovery, and tool execution for AI agents. Use a count-free description; app/toolkit counts differ across public pages. |
25 +| Branch | GitHub default branch is `next`. `context7.json` omits `branch`, so Context7 defaults to the repository default. Public retrieved sources include `/blob/next/docs/content/`. Effective admin branch remains **UNVERIFIED**. |
26 +| Included folders | `docs/content`, `ts`, `python`, unchanged |
27 +| Excluded folders | `[]`, unchanged |
28 +| Excluded filenames | `CHANGELOG.md`, `changelog.md`, `CHANGELOG.mdx`, `changelog.mdx`, `LICENSE.md`, `license.md`, `CODE_OF_CONDUCT.md`, `code_of_conduct.md`, unchanged |
29 +| Version configuration | `previousVersions: []`, unchanged |
30 +
31 +Keep the API rules in `context7.json` aligned with production:
32 +
33 +- Current REST base: `https://backend.composio.dev/api/v3.1`. The previous `/api/v3` remains supported and frozen with pinned tool-version defaults.
34 +- Only these five v3.1 endpoints default to the latest toolkit version rather than `00000000_00`: `GET /tools`, `GET /tools/{tool_slug}`, `POST /tools/execute/{tool_slug}`, `POST /tools/execute/{tool_slug}/input`, and `POST /tools/scopes/required`.
35 +- Request the pinned version explicitly with `version=00000000_00`, or `toolkit_versions=00000000_00` for `GET /tools`.
36 +- TypeScript `@composio/core` and Python `composio` already use v3.1 through the generated client. The migration guidance concerns direct REST calls.
37 +
38 +The tool-version policy previously occupied one rule longer than the [Context7 schema's 255-character limit](https://context7.com/schema/context7.json). It is now split into shorter rules with the same policy, including the v3.1-only scopes endpoint. Offline tests enforce this limit; `$schema` also exposes it to editors. The REST policy is checked against the [production API reference](https://docs.composio.dev/reference.md).
39 +
40 +## Claim and verify the library
41 +
42 +Follow Context7's [library claiming instructions](https://context7.com/docs/howto/claiming-libraries.md):
43 +
44 +1. Sign in as the accountable human and open [Composio's admin page](https://context7.com/composiohq/composio/admin), or choose **Manage** on its dashboard card.
45 +2. If unclaimed, click **Claim Library** and copy the modal's generated `url` and `public_key`.
46 +3. Merge those two fields into the **existing root `context7.json`**. Do not replace the file with the modal's minimal snippet. Preserve the description, folders, exclusions, API rules, and version settings. The `url` must exactly match `https://context7.com/composiohq/composio`; use the actual generated public key, not a placeholder. This public claim key is not the secret API key.
47 +4. Commit and merge that configuration to the default branch, `next`. Once the file is publicly reachable there, return to the modal and click **Claim Library** to verify ownership.
48 +5. If already claimed, confirm the owning teamspace and your owner/admin role instead of claiming again. Arrange backup access.
49 +6. Inspect **Configuration** and **Versions**. Record the effective description, branch, include/exclude settings, custom rules, and versions, including any dashboard overrides. Compare them with the inventory above. Resolve mismatches without changing folder/exclusion rules, then record the verification date, owner, and evidence in the issue or PR.
50 +
51 +Ownership verification here means access and settings verification, not a claim that Context7's separate verified badge was granted.
52 +
53 +## Refresh after a production documentation deploy
54 +
55 +**Manual operational policy:** the owner refreshes after **each successful production documentation deploy**, once the canonical production site serves that deployment. Never refresh merely because of a push, a preview deploy, or a failed deploy.
56 +
57 +There is no automatic refresh hook in this change. Deployment metadata is ambiguous: deployments labeled `Production` have reported preview hostnames or empty environment URLs. Before wiring automation, verify the canonical documentation deployment service, its success signal, and how that signal identifies the revision actually serving production. An environment label or hostname alone is not sufficient evidence.
58 +
59 +Confirm that the `next` branch already exists in Context7's **Versions** or branch configuration before requesting a branch-specific refresh. The [current refresh API](https://context7.com/docs/api-reference/refresh/refresh-a-library) refreshes an existing library/version; it does not create the branch entry.
60 +
61 +With an owner-authorized key supplied securely as `CONTEXT7_API_KEY`, run:
62 +
63 +```bash
64 +curl --fail-with-body --request POST \
65 + 'https://context7.com/api/v1/refresh' \
66 + --header "Authorization: Bearer ${CONTEXT7_API_KEY:?Set an owner-authorized Context7 key}" \
67 + --header 'Content-Type: application/json' \
68 + --data '{"libraryName":"/composiohq/composio","branch":"next"}'
69 +```
70 +
71 +A `200` response means the refresh **started**, not that indexing finished. Wait for Context7 to report completion and confirm the indexed branch/revision before running the baseline command. Record the production deploy revision, refresh request/completion times, indexed revision/branch, and effective settings. If Context7 doesn't expose a revision, explicitly record that limitation and retain the completion status and retrieved source evidence; don't invent a revision.
72 +
73 +## Run and retain retrieval evidence
74 +
75 +```bash
76 +# Offline runner tests, with no Context7 requests.
77 +node --test test/context7-check.test.mjs
78 +
79 +# Live retrieval baseline, after indexing completes.
80 +node scripts/context7-check.mjs --output context7-baseline.json
81 +```
82 +
83 +The **Context7 Retrieval Check** GitHub Actions workflow (`.github/workflows/context7-check.yml`) is manually dispatched. It checks retrieval, not deployment success or refresh completion, and does not refresh the library.
84 +
85 +The suite is `scripts/context7-queries.json`. The runner calls `GET https://context7.com/api/v2/context` with `libraryId`, `query`, and `type=json`, without bypassing reranking. Each query prints `PASS`, `FAIL`, or `ERROR`, followed by totals; any failure or error exits nonzero.
86 +
87 +Each expected-fact group is required. Any literal alternative in a group may match, case-insensitively with normalized whitespace, in a retrieved snippet's **body**. Groups may match different snippets. Titles and source URLs don't count as facts.
88 +
89 +Attach the JSON report to the PR. It preserves query IDs and text, expected facts, production sources, retrieved snippet bodies, the timestamps, the suite's SHA256, and the local configuration snapshot. Retain console totals and the deployment/index/settings record alongside it. Distinguish missing facts (`FAIL`) from retrieval or request errors (`ERROR`); review the actual snippets before drawing conclusions. A failing first baseline identifies existing retrieval gaps, not proof of a newly introduced regression. These are lexical checks, not a semantic correctness guarantee.
90 +
91 +## Gate folder or exclusion changes on evidence
92 +
93 +Don't narrow indexing to `docs/content` or change exclusions based on intuition. Before proposing a folder-rule change:
94 +
95 +1. Freeze the query suite and record its SHA256. After a completed refresh, capture `context7-before.json` using the baseline command with `--output context7-before.json`. Record the indexed revision/branch and all effective settings.
96 +2. Record the proposed include/exclude diff. Keep the source revision, branch, API rules, description, query suite, and retrieval options constant so the comparison isolates the folder change. If those differ, don't treat the reports as a controlled comparison.
97 +3. With owner approval, apply the candidate settings and refresh under the policy above. Wait for completion, then run the **same suite with the same hash** using `--output context7-after.json`. Record the resulting indexed revision/branch and settings.
98 +4. Attach both reports, retrieved snippet bodies, refresh evidence, and the settings diff. Review every failure and error, including regressions in SDK examples and version guidance. Don't weaken expected facts to make the candidate pass.
99 +5. Keep the existing folder/exclusion rules unless the evidence supports the proposed change. If the suite needs correction, review that correction separately and rerun both baselines with the new identical suite/hash.
context7.json
+6 -3
@@ -1,6 +1,7 @@
1 {
2 + "$schema": "https://context7.com/schema/context7.json",
3 "projectTitle": "Composio",
3 - "description": "Composio equip's your AI agents & LLMs with 100+ high-quality integrations via function calling",
4 + "description": "Composio connects AI agents to apps with managed authentication, tool discovery, and tool execution.",
5 "folders": ["docs/content", "ts", "python"],
6 "excludeFolders": [],
7 "excludeFiles": [
@@ -15,8 +16,10 @@
16 ],
17 "rules": [
18 "The current Composio REST API version is v3.1, served at https://backend.composio.dev/api/v3.1. Prefer it for new code and new examples.",
18 - "https://backend.composio.dev/api/v3 is the previous REST API version. It is frozen with pinned tool-version defaults and remains supported — existing v3 integrations keep working and do not need to migrate.",
19 - "On v3.1, GET /tools, GET /tools/{tool_slug}, POST /tools/execute/{tool_slug}, POST /tools/execute/{tool_slug}/input, and POST /tools/scopes/required default their version parameter to the latest toolkit version instead of the pinned 00000000_00. Code that needs the v3 pinned default must pass it explicitly: version=00000000_00, or toolkit_versions=00000000_00 on GET /tools. This version-default change is limited to these five endpoints.",
19 + "https://backend.composio.dev/api/v3 is the previous REST API version. It is frozen with pinned tool-version defaults and remains supported. Existing v3 integrations keep working and do not need to migrate.",
20 + "On REST v3.1, GET /tools and GET /tools/{tool_slug} default to the latest toolkit version. To pin it, use toolkit_versions=00000000_00 for GET /tools or version=00000000_00 for GET /tools/{tool_slug}.",
21 + "On REST v3.1, POST /tools/execute/{tool_slug}, POST /tools/execute/{tool_slug}/input, and POST /tools/scopes/required default to the latest toolkit version. Pass version=00000000_00 to pin it explicitly.",
22 + "Only those five endpoints change version defaults. POST /tools/scopes/required is v3.1-only; the other four also exist on v3, where omission selects 00000000_00.",
23 "TypeScript (@composio/core) and Python (composio) SDK callers are already on v3.1 through the generated client; this only affects direct REST calls."
24 ],
25 "previousVersions": []
package.json
+3 -1
@@ -25,12 +25,14 @@
25 "build:packages": "turbo build --filter=./ts/packages/**",
26 "cli:record": "turbo run record --filter=./ts/packages/cli",
27 "record": "turbo run record",
28 - "test": "pnpm run test:toolchain && pnpm run test:install-sh && pnpm run test:release-workflow && pnpm run test:issue-triage && pnpm run test:provider-compatibility && turbo test --filter=./ts/packages/** --filter=!@e2e-tests/* && pnpm run test:examples",
28 + "test": "pnpm run test:toolchain && pnpm run test:install-sh && pnpm run test:release-workflow && pnpm run test:issue-triage && pnpm run test:provider-compatibility && pnpm run test:context7 && turbo test --filter=./ts/packages/** --filter=!@e2e-tests/* && pnpm run test:examples",
29 "test:install-sh": "bash test/install-sh-release-resolution.test.sh && bash test/install-sh-atomic-replace.test.sh",
30 "test:release-workflow": "bun run test/release-workflow.test.ts",
31 "test:issue-triage": "bun test test/issue-triage.test.ts",
32 "test:provider-compatibility": "bun test test/provider-compatibility.test.ts",
33 "test:toolchain": "bun run test/mise-bun-pin.test.ts",
34 + "test:context7": "node --test test/context7-check.test.mjs",
35 + "check:context7": "node scripts/context7-check.mjs",
36 "test:examples": "tsx ts/scripts/validate-examples.ts",
37 "test:e2e": "turbo test:e2e --filter='@e2e-tests/*'",
38 "test:e2e:node": "turbo test:e2e:node --filter='@e2e-tests/node-*'",
scripts/context7-check.mjs new
+131
@@ -0,0 +1,131 @@
1 +#!/usr/bin/env node
2 +import { readFile, writeFile } from 'node:fs/promises';
3 +import { parseArgs } from 'node:util';
4 +
5 +/**
6 + * @typedef {{id: string, query: string, source: string, expect: string[][]}} Query
7 + * @typedef {{source?: string, text: string, generationDate?: string}} Snippet
8 + * @typedef {{status: 'PASS'|'FAIL'|'ERROR', missing?: string[][], snippets?: Snippet[], error?: string}} Result
9 + * @typedef {{codeId?: string, codeTitle?: string, codeDescription?: string, generationDate?: string, codeList: {code: string}[]}} CodeSnippet
10 + * @typedef {{codeSnippets: CodeSnippet[], infoSnippets: {pageId?: string, content: string}[], rules?: object}} ContextResponse
11 + */
12 +
13 +const endpoint = 'https://context7.com/api/v2/context';
14 +/** @param {string} text */
15 +const normalize = text => text.toLowerCase().replace(/\s+/g, ' ').trim();
16 +
17 +// Only retrieved content counts as evidence, never titles, URLs, or injected rules.
18 +/**
19 + * @param {Query} query
20 + * @param {ContextResponse} payload
21 + * @returns {Result}
22 + */
23 +export function evaluate(query, payload) {
24 + if (!Array.isArray(payload.codeSnippets) || !Array.isArray(payload.infoSnippets)) {
25 + throw new Error('Invalid Context7 response: expected codeSnippets and infoSnippets arrays');
26 + }
27 + const snippets = [
28 + ...payload.codeSnippets.flatMap(snippet => {
29 + if (!Array.isArray(snippet.codeList)) throw new Error('Invalid Context7 codeList');
30 + return snippet.codeList.map(example => ({
31 + source: snippet.codeId,
32 + text: example.code,
33 + generationDate: snippet.generationDate,
34 + }));
35 + }),
36 + ...payload.infoSnippets.map(snippet => ({ source: snippet.pageId, text: snippet.content })),
37 + ];
38 + if (snippets.some(snippet => typeof snippet.text !== 'string')) {
39 + throw new Error('Invalid Context7 snippet content');
40 + }
41 + const content = snippets.map(snippet => normalize(snippet.text));
42 + const missing = query.expect.filter(
43 + alternatives => !alternatives.some(term => content.some(text => text.includes(normalize(term))))
44 + );
45 + return { status: missing.length ? 'FAIL' : 'PASS', missing, snippets };
46 +}
47 +
48 +/**
49 + * @param {{libraryId: string, queries: Query[]}} suite
50 + * @param {{apiKey?: string, fetchImpl?: (url: URL, init: RequestInit) => Promise<Response>, log?: (line: string) => void}} options
51 + */
52 +export async function runSuite(suite, { apiKey, fetchImpl = fetch, log = console.log } = {}) {
53 + const results = [];
54 + for (const query of suite.queries) {
55 + const url = new URL(endpoint);
56 + url.search = new URLSearchParams({
57 + libraryId: suite.libraryId,
58 + query: query.query,
59 + type: 'json',
60 + }).toString();
61 + /** @type {Result} */
62 + let result;
63 + try {
64 + const response = await fetchImpl(url, {
65 + headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : {},
66 + signal: AbortSignal.timeout(60_000),
67 + redirect: 'error',
68 + });
69 + // 202 means indexing is pending, not a successful retrieval.
70 + if (response.status !== 200) {
71 + throw new Error(`Context7 HTTP ${response.status}; no retrieval evidence`);
72 + }
73 + result = evaluate(query, await response.json());
74 + } catch (error) {
75 + // Don't print server bodies, request headers, or credentials.
76 + result = {
77 + status: 'ERROR',
78 + error:
79 + error instanceof Error
80 + ? error.message.replaceAll(apiKey || '\0', '[REDACTED]')
81 + : 'Retrieval failed',
82 + };
83 + }
84 + results.push({ ...query, ...result });
85 + log(
86 + `${result.status} ${query.id}${result.missing?.length ? ` missing ${JSON.stringify(result.missing)}` : ''}${result.error ? `: ${result.error}` : ''}`
87 + );
88 + }
89 + const totals = { PASS: 0, FAIL: 0, ERROR: 0 };
90 + for (const result of results) totals[result.status]++;
91 + log(
92 + `${results.length} queries: ${totals.PASS} passed, ${totals.FAIL} failed, ${totals.ERROR} errors`
93 + );
94 + return { libraryId: suite.libraryId, endpoint, totals, results };
95 +}
96 +
97 +if (import.meta.main) {
98 + try {
99 + const { values } = parseArgs({ options: { output: { type: 'string' } } });
100 + const suiteText = await readFile(new URL('./context7-queries.json', import.meta.url), 'utf8');
101 + const suite = JSON.parse(suiteText);
102 + const suiteSha256 = Array.from(
103 + new Uint8Array(await crypto.subtle.digest('SHA-256', new TextEncoder().encode(suiteText))),
104 + byte => byte.toString(16).padStart(2, '0')
105 + ).join('');
106 + const startedAt = new Date().toISOString();
107 + const report = await runSuite(suite, { apiKey: process.env.CONTEXT7_API_KEY });
108 + if (values.output) {
109 + await writeFile(
110 + values.output,
111 + `${JSON.stringify(
112 + {
113 + startedAt,
114 + completedAt: new Date().toISOString(),
115 + suiteSha256,
116 + config: JSON.parse(
117 + await readFile(new URL('../context7.json', import.meta.url), 'utf8')
118 + ),
119 + ...report,
120 + },
121 + null,
122 + 2
123 + )}\n`
124 + );
125 + }
126 + process.exitCode = report.totals.FAIL || report.totals.ERROR ? 1 : 0;
127 + } catch (error) {
128 + console.error(error instanceof Error ? error.message : 'Context7 check failed');
129 + process.exitCode = 1;
130 + }
131 +}
scripts/context7-queries.json new
+161
@@ -0,0 +1,161 @@
1 +{
2 + "libraryId": "/composiohq/composio",
3 + "queries": [
4 + {
5 + "id": "python-install-current-sdk",
6 + "query": "How do I install the current Composio Python SDK for OpenAI Agents, and which Python version is required?",
7 + "source": "https://docs.composio.dev/docs/quickstart.md",
8 + "expect": [
9 + ["pip install", "uv add"],
10 + ["composio"],
11 + ["composio-openai-agents"],
12 + ["openai-agents"],
13 + ["Python 3.10"]
14 + ]
15 + },
16 + {
17 + "id": "typescript-install-current-sdk",
18 + "query": "How do I install Composio for the OpenAI Agents SDK in TypeScript, and what Node.js version and module format does it require?",
19 + "source": "https://docs.composio.dev/docs/quickstart.md",
20 + "expect": [
21 + [
22 + "npm install @composio/core @composio/openai-agents @openai/agents",
23 + "pnpm add @composio/core @composio/openai-agents @openai/agents",
24 + "bun add @composio/core @composio/openai-agents @openai/agents",
25 + "yarn add @composio/core @composio/openai-agents @openai/agents"
26 + ],
27 + ["Node.js 22.22.3"],
28 + ["ESM-only"]
29 + ]
30 + },
31 + {
32 + "id": "python-initialize-api-key",
33 + "query": "How do I import and initialize Composio in Python with an explicit project API key?",
34 + "source": "https://docs.composio.dev/docs/authentication/manually-authenticating.md",
35 + "expect": [["from composio import Composio"], ["Composio(api_key="]]
36 + },
37 + {
38 + "id": "typescript-initialize-api-key",
39 + "query": "How do I import and initialize the Composio TypeScript client with an explicit project API key?",
40 + "source": "https://docs.composio.dev/docs/configuring-sessions.md",
41 + "expect": [
42 + [
43 + "import { Composio } from '@composio/core'",
44 + "import { Composio } from \"@composio/core\""
45 + ],
46 + ["new Composio({ apiKey:"]
47 + ]
48 + },
49 + {
50 + "id": "project-api-key-environment",
51 + "query": "Which environment variable configures the Composio project API key, and how do I initialize the Python OpenAI Agents provider after loading dotenv?",
52 + "source": "https://docs.composio.dev/docs/quickstart.md",
53 + "expect": [
54 + ["COMPOSIO_API_KEY"],
55 + ["load_dotenv()"],
56 + ["Composio(provider=OpenAIAgentsProvider())"]
57 + ]
58 + },
59 + {
60 + "id": "auth-config-versus-connected-account",
61 + "query": "What is the difference between a toolkit auth config and a user's connected account, and how does the session manage connections?",
62 + "source": "https://docs.composio.dev/docs/authentication.md",
63 + "expect": [
64 + ["auth config"],
65 + ["credentials"],
66 + ["connected account"],
67 + ["COMPOSIO_MANAGE_CONNECTIONS"]
68 + ]
69 + },
70 + {
71 + "id": "python-connect-user-account",
72 + "query": "How do I manually connect a user's Gmail account in a Python Composio session, display the Connect Link, and wait for authorization?",
73 + "source": "https://docs.composio.dev/docs/authentication/manually-authenticating.md",
74 + "expect": [
75 + ["session.authorize(\"gmail\")"],
76 + ["connection_request.redirect_url"],
77 + ["connection_request.wait_for_connection("]
78 + ]
79 + },
80 + {
81 + "id": "typescript-connect-user-account",
82 + "query": "How do I manually connect a user's Gmail account in a TypeScript Composio session, display the Connect Link, and wait for authorization?",
83 + "source": "https://docs.composio.dev/docs/authentication/manually-authenticating.md",
84 + "expect": [
85 + ["await session.authorize(\"gmail\")"],
86 + ["connectionRequest.redirectUrl"],
87 + ["connectionRequest.waitForConnection("]
88 + ]
89 + },
90 + {
91 + "id": "python-create-session-tools",
92 + "query": "How do I create a Composio tool-router session for my application's user in Python and obtain the session's framework-formatted tools?",
93 + "source": "https://docs.composio.dev/docs/how-composio-works.md",
94 + "expect": [
95 + ["composio.sessions.create(user_id=", "composio.create(user_id="],
96 + ["tools = session.tools()"]
97 + ]
98 + },
99 + {
100 + "id": "typescript-create-session-tools",
101 + "query": "How do I create a Composio tool-router session for my application's user in TypeScript and obtain the session's framework-formatted tools?",
102 + "source": "https://docs.composio.dev/docs/how-composio-works.md",
103 + "expect": [["await composio.create("], ["await session.tools()"]]
104 + },
105 + {
106 + "id": "execute-tools-through-session",
107 + "query": "How should Python and TypeScript execute COMPOSIO_SEARCH_TOOLS through the session tool router rather than the direct tools execution path?",
108 + "source": "https://docs.composio.dev/docs/how-composio-works.md",
109 + "expect": [
110 + ["session.execute(\"COMPOSIO_SEARCH_TOOLS\", arguments="],
111 + ["await session.execute(\"COMPOSIO_SEARCH_TOOLS\","],
112 + ["can only be called inside a tool-router session"]
113 + ]
114 + },
115 + {
116 + "id": "select-session-toolkits",
117 + "query": "How do I restrict a Composio session to GitHub, Gmail, and Slack instead of exposing every toolkit, in Python and TypeScript?",
118 + "source": "https://docs.composio.dev/docs/configuring-sessions.md",
119 + "expect": [
120 + [
121 + "toolkits=[\"github\", \"gmail\", \"slack\"]",
122 + "toolkits={\"enable\": [\"github\", \"gmail\", \"slack\"]}"
123 + ],
124 + [
125 + "toolkits: [\"github\", \"gmail\", \"slack\"]",
126 + "toolkits: { enable: [\"github\", \"gmail\", \"slack\"] }"
127 + ]
128 + ]
129 + },
130 + {
131 + "id": "current-rest-api-base",
132 + "query": "What is the current Composio REST API base URL for new integrations, and which header authenticates a project API key?",
133 + "source": "https://docs.composio.dev/reference.md",
134 + "expect": [["https://backend.composio.dev/api/v3.1"], ["x-api-key"]]
135 + },
136 + {
137 + "id": "rest-latest-version-defaults",
138 + "query": "On Composio REST v3.1, what happens when the toolkit version is omitted on GET /tools, GET /tools/{tool_slug}, POST /tools/execute/{tool_slug}, POST /tools/execute/{tool_slug}/input, and POST /tools/scopes/required?",
139 + "source": "https://docs.composio.dev/reference.md",
140 + "expect": [
141 + ["latest toolkit version"],
142 + ["v3.1"],
143 + ["GET /tools/{tool_slug}"],
144 + ["POST /tools/execute/{tool_slug}/input"],
145 + ["POST /tools/scopes/required"],
146 + ["toolkit_versions"]
147 + ]
148 + },
149 + {
150 + "id": "rest-explicit-pinned-version",
151 + "query": "How do I preserve the old Composio REST v3 pinned toolkit version when calling v3.1, and does the previous v3 API remain supported?",
152 + "source": "https://docs.composio.dev/reference.md",
153 + "expect": [
154 + ["00000000_00"],
155 + ["remains supported"],
156 + ["/api/v3"],
157 + ["version", "toolkit_versions"]
158 + ]
159 + }
160 + ]
161 +}
test/context7-check.test.mjs new
+118
@@ -0,0 +1,118 @@
1 +import assert from 'node:assert/strict';
2 +import { readFile } from 'node:fs/promises';
3 +import { test } from 'node:test';
4 +import { evaluate, runSuite } from '../scripts/context7-check.mjs';
5 +
6 +const query = {
7 + id: 'current-api',
8 + query: 'Which REST API should new code use?',
9 + source: 'https://example.test/current-api',
10 + expect: [['https://backend.composio.dev/api/v3.1'], ['x-api-key', 'project API key']],
11 +};
12 +/** @param {string} content */
13 +const payload = content => ({ codeSnippets: [], infoSnippets: [{ content }] });
14 +
15 +test('stale API retrieval fails even when another required fact is present', () => {
16 + const result = evaluate(
17 + query,
18 + payload('Use https://backend.composio.dev/api/v3 with x-api-key.')
19 + );
20 + assert.equal(result.status, 'FAIL');
21 + assert.deepEqual(result.missing, [['https://backend.composio.dev/api/v3.1']]);
22 +});
23 +
24 +test('alternatives ignore case and whitespace, without combining unrelated snippets into one fact', () => {
25 + const result = evaluate(query, {
26 + codeSnippets: [
27 + {
28 + codeId: 'https://example.test/code',
29 + codeList: [{ code: 'https://backend.composio.dev/api/v3.1' }],
30 + },
31 + ],
32 + infoSnippets: [{ content: 'Authenticate with a PROJECT\n API KEY.' }],
33 + });
34 + assert.equal(result.status, 'PASS');
35 + assert.equal(
36 + evaluate(
37 + { ...query, expect: [['project API key']] },
38 + {
39 + codeSnippets: [],
40 + infoSnippets: [{ content: 'project API' }, { content: 'key' }],
41 + }
42 + ).status,
43 + 'FAIL'
44 + );
45 +});
46 +
47 +test('URLs, titles, descriptions, and injected rules cannot mask missing documentation', () => {
48 + const result = evaluate(query, {
49 + codeSnippets: [
50 + {
51 + codeId: query.expect[0][0],
52 + codeTitle: 'x-api-key',
53 + codeDescription: 'x-api-key',
54 + codeList: [{ code: 'obsolete example' }],
55 + },
56 + ],
57 + infoSnippets: [{ pageId: query.expect[0][0], content: 'unrelated content' }],
58 + rules: { libraryOwn: ['Use https://backend.composio.dev/api/v3.1 with x-api-key'] },
59 + });
60 + assert.equal(result.status, 'FAIL');
61 + assert.deepEqual(result.missing, query.expect);
62 + assert.equal(evaluate(query, payload('')).status, 'FAIL');
63 +});
64 +
65 +test('malformed and pending retrievals are errors, not regression passes; later queries still run', async () => {
66 + const responses = [
67 + new Response('{}', { status: 202 }),
68 + Response.json({ codeSnippets: [], infoSnippets: [{ content: null }] }),
69 + new Response('{}', { status: 429 }),
70 + Response.json(payload('Use https://backend.composio.dev/api/v3.1 with x-api-key')),
71 + ];
72 + const result = await runSuite(
73 + {
74 + libraryId: '/synthetic/library',
75 + queries: responses.map((_, i) => ({ ...query, id: `case-${i}` })),
76 + },
77 + {
78 + fetchImpl: async () => {
79 + const response = responses.shift();
80 + assert.ok(response);
81 + return response;
82 + },
83 + log: () => {},
84 + }
85 + );
86 + assert.deepEqual(
87 + result.results.map(item => item.status),
88 + ['ERROR', 'ERROR', 'ERROR', 'PASS']
89 + );
90 + assert.deepEqual(result.totals, { PASS: 1, FAIL: 0, ERROR: 3 });
91 +});
92 +
93 +test('network failures are distinct from missing facts and redact credentials', async () => {
94 + const result = await runSuite(
95 + { libraryId: '/synthetic/library', queries: [query] },
96 + {
97 + apiKey: 'synthetic-secret',
98 + fetchImpl: async () => {
99 + throw new Error('network unavailable synthetic-secret');
100 + },
101 + log: () => {},
102 + }
103 + );
104 + assert.deepEqual(result.totals, { PASS: 0, FAIL: 0, ERROR: 1 });
105 + assert.equal(result.results[0].error, 'network unavailable [REDACTED]');
106 +});
107 +
108 +test('owner rules fit Context7 public schema limits instead of being rejected for length', async () => {
109 + // https://context7.com/schema/context7.json: rules.maxItems and rules.items.maxLength.
110 + const config = JSON.parse(await readFile(new URL('../context7.json', import.meta.url), 'utf8'));
111 + assert.ok(config.rules.length <= 50, 'Context7 accepts at most 50 rules');
112 + for (const rule of config.rules) {
113 + assert.ok(
114 + rule.length >= 1 && rule.length <= 255,
115 + `Invalid Context7 rule length: ${rule.length}`
116 + );
117 + }
118 +});