@devries-cameron20 / composio-1 / commits / d2fe6c011
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

test(docs): revive semantic parity across agent-readable surfaces

Revives the shared-fact design from closed PR #4443, fixes whitespace-only chunk-boundary failures, and covers real-output mutations plus both search corpora.

Brendan O'Leary committed Oct 15, 2026 at 00:23 UTC d2fe6c0110dd2839278ad8414fe13054e4fea2cc
8 files changed +365 -11
docs/agent-guidance/context/markdown-components.md
+1 -1
@@ -21,4 +21,4 @@ Audit for DEVREL-35, 2026-09-10. Scope: components registered in `mdx-components
21
22 ## Remaining checks
23
24 -DEVREL-32 should exercise actual HTTP page output, the full corpus, search ingestion, and external retrieval against the same expected facts. Static converter tests cannot prove an external index refreshed. DEVREL-38 can use the remaining visual and catalog gaps to prioritize task-complete pages. Do not treat this audit as evidence of benchmark improvement.
24 +The [semantic parity suite](./semantic-parity.md) for DEVREL-32 checks actual HTTP page output, the full corpus, and search ingestion against the same expected facts. Its export checker applies those expectations to external retrieval captures. Static tests cannot prove an external index refreshed. DEVREL-38 can use the remaining visual and catalog gaps to prioritize task-complete pages. Do not treat this audit as evidence of benchmark improvement.
docs/agent-guidance/context/semantic-parity.md new
+50
@@ -0,0 +1,50 @@
1 +# Check semantic parity
2 +
3 +The parity suite checks that task instructions survive page rendering, corpus assembly, and search ingestion. Run these commands from `docs/`:
4 +
5 +```bash
6 +bun test tests/static/semantic-parity.test.ts
7 +bun run build
8 +bun run start
9 +```
10 +
11 +With the server running, use another terminal:
12 +
13 +```bash
14 +bun test tests/integration/semantic-parity.test.ts
15 +```
16 +
17 +Set `TEST_BASE_URL` to check a deployed site. The standard docs CI already runs both test directories against a production build.
18 +
19 +## Coverage
20 +
21 +`tests/fixtures/semantic-parity.ts` defines 35 reviewed facts across eight pages: the product chooser, quickstart, Anthropic provider, For You plugins, coding-agent setup, current and legacy REST tools pages, and the standup bot example. The [component audit](./markdown-components.md) informs the coverage: installation commands, package and toolkit selection rules, warnings, executable code, API versions, image captions and links, staged example source, and repository-availability notices.
22 +
23 +The static suite checks both the complete Algolia replacement payload after chunking and the local `/api/search` corpus. The HTTP suite checks actual page Markdown and each page's section in `/llms-full.txt`. A fact on another page cannot satisfy the assertion. Legacy REST pages are checked individually and in search, including their legacy classification. The current full corpus must exclude them.
24 +
25 +Failures name the page, fact ID, and surface that lost it. Mutation checks remove every expected fact from real surface output and verify that its assertion fails. The external checker test also logs a deliberately removed command and its expected failure to CI output. Assertions ignore whitespace differences from line wrapping and paragraph boundaries, but retain command flags, package names, version numbers, and code punctuation. Review the source instructions before changing an expectation. Do not regenerate expectations from converter output.
26 +
27 +## Check an external export
28 +
29 +Export retrieved text or indexed records into this JSON shape. Include all fixture pages, with each page's chunks in source order:
30 +
31 +```json
32 +{
33 + "surface": "Context7 export",
34 + "capturedAt": "2026-09-11T00:00:00Z",
35 + "includesLegacy": false,
36 + "pages": [
37 + { "url": "/docs/quickstart", "content": "Full retrieved page text goes here." }
38 + ]
39 +}
40 +```
41 +
42 +This abbreviated example fails because it omits pages and facts. Set `includesLegacy` to `true` for indexes that include legacy REST pages, such as the Algolia replacement payload. URLs can be site paths or canonical `docs.composio.dev` URLs, including Markdown suffixes and section anchors.
43 +
44 +```bash
45 +bun run check:docs-parity /tmp/docs-export.json
46 +```
47 +
48 +The checker validates the JSON, groups chunks by page, and runs the shared expectations. Missing pages or facts return a nonzero exit status. Preserve the provider name and capture timestamp when recording results. Local tests prove the ingestion payload retains facts; they do not prove that an external service refreshed its index. External captures remain a separate check because PR CI has no external retrieval credentials.
49 +
50 +These assertions check content preservation. They do not measure retrieval ranking or agent task completion, which belong to DEVREL-37.
docs/lib/search-index.ts
+21 -10
@@ -703,7 +703,7 @@ export async function getDocsSearchIndexes(): Promise<SearchIndex[]> {
703 plugins: [lucideIconsPlugin(), openapiPlugin()],
704 });
705
706 - const mdxIndexes = [
706 + const mdxIndexes = await Promise.all([
707 ...docsSource.getPages(),
708 ...examplesSource.getPages(),
709 ...toolkitsSource.getPages(),
@@ -711,15 +711,26 @@ export async function getDocsSearchIndexes(): Promise<SearchIndex[]> {
711 ...fullReferenceSource.getPages(),
712 ].filter((page) =>
713 !isExcludedFromSearch(page.url) && !isHiddenApiTagUrl(page.url)
714 - ).map((page) => ({
715 - id: page.url,
716 - title: page.data.title ?? 'Untitled',
717 - description: page.data.description,
718 - url: page.url,
719 - structuredData: page.data.structuredData,
720 - keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,
721 - } satisfies SearchIndex));
722 -
714 + ).map(async (page) => {
715 + // Structured MDX search data omits code and custom-component instructions.
716 + // Preserve the same converted text used by the agent-readable page surface.
717 + const content = 'getText' in page.data
718 + ? mdxToCleanMarkdown(await page.data.getText('raw'), page.url)
719 + : undefined;
720 + return {
721 + id: page.url,
722 + title: page.data.title ?? 'Untitled',
723 + description: page.data.description,
724 + url: page.url,
725 + structuredData: content === undefined
726 + ? page.data.structuredData
727 + : {
728 + ...page.data.structuredData,
729 + contents: [{ heading: undefined, content }],
730 + },
731 + keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,
732 + } satisfies SearchIndex;
733 + }));
734 const oauthIndexes = getAuthGuideSearchRecords().map((record) => ({
735 id: record.page_id,
736 title: record.title,
docs/package.json
+1
@@ -32,6 +32,7 @@
32 "generate:toolkit-versions": "bun scripts/refresh-toolkit-versions.ts",
33 "generate:meta-tools": "bun scripts/generate-meta-tools.ts",
34 "generate:api-index": "bun scripts/generate-api-index.ts",
35 + "check:docs-parity": "bun scripts/check-docs-parity.ts",
36 "pretest": "bun run generate:kb",
37 "test": "bun test tests/static/",
38 "test:integration": "bun test tests/integration/ --timeout 30000"
docs/scripts/check-docs-parity.ts new
+41
@@ -0,0 +1,41 @@
1 +import { z } from 'zod';
2 +import { readFileSync } from 'node:fs';
3 +import { assertPageFacts, parityPages } from '../tests/fixtures/semantic-parity';
4 +
5 +// Accept retrieved text or an exported index, with chunks in their source order.
6 +const snapshotSchema = z.object({
7 + surface: z.string().min(1),
8 + capturedAt: z.string().datetime(),
9 + includesLegacy: z.boolean(),
10 + pages: z.array(z.object({
11 + url: z.string().refine(value => {
12 + try {
13 + return new URL(value, 'https://docs.composio.dev').origin === 'https://docs.composio.dev';
14 + } catch {
15 + return false;
16 + }
17 + }, 'Expected a canonical docs.composio.dev URL or site path'),
18 + content: z.string(),
19 + })).min(1),
20 +});
21 +
22 +const path = process.argv[2];
23 +if (!path) throw new Error('Usage: bun scripts/check-docs-parity.ts <snapshot.json>');
24 +const snapshot = snapshotSchema.parse(JSON.parse(readFileSync(path, 'utf8')));
25 +const pages = new Map<string, string>();
26 +for (const page of snapshot.pages) {
27 + const url = new URL(page.url, 'https://docs.composio.dev').pathname.replace(/\.mdx?$/, '').replace(/\/$/, '');
28 + pages.set(url, [pages.get(url), page.content].filter(Boolean).join('\n\n'));
29 +}
30 +
31 +const failures: string[] = [];
32 +for (const page of parityPages) {
33 + if (!snapshot.includesLegacy && page.url.startsWith('/reference/v3/')) continue;
34 + try {
35 + assertPageFacts(pages.get(page.url) ?? '', page, snapshot.surface);
36 + } catch (error) {
37 + failures.push(error instanceof Error ? error.message : String(error));
38 + }
39 +}
40 +if (failures.length) throw new Error(failures.join('\n'));
41 +console.log(`${snapshot.surface}: parity passed for snapshot captured ${snapshot.capturedAt}`);
docs/tests/fixtures/semantic-parity.ts new
+102
@@ -0,0 +1,102 @@
1 +import assert from 'node:assert/strict';
2 +
3 +// Reviewed expectations, independent of the converter and its shared constants.
4 +// Keep commands with their arguments and warnings with their limiting condition.
5 +export const parityPages = [
6 + {
7 + url: '/docs',
8 + facts: [
9 + ['platform-choice', 'Platform'],
10 + ['for-you-choice', 'For You'],
11 + ['platform-selection', 'Add Composio into your agent or app.'],
12 + ['for-you-selection', 'Use Composio yourself with agents you already have.'],
13 + ['build-destination', '/docs/quickstart'],
14 + ['use-destination', '/docs/agent-plugins'],
15 + ],
16 + },
17 + {
18 + url: '/docs/quickstart',
19 + facts: [
20 + ['typescript-install', 'npm install @composio/core @composio/openai-agents @openai/agents'],
21 + ['python-install', 'uv add python-dotenv composio composio-openai-agents openai-agents'],
22 + ['package-choice', 'Use @composio/slim for a smaller install with the same API.'],
23 + ['legacy-package-warning', 'it calls deprecated APIs and does not work with sessions'],
24 + ['current-python-package', 'The current Python package is `composio`. The package `composio-core` is the legacy v1 SDK'],
25 + ['python-session-example', 'session = composio.sessions.create(user_id=user_id)'],
26 + ['typescript-session-example', 'const session = await composio.create(userId);'],
27 + ],
28 + },
29 + {
30 + url: '/docs/providers/anthropic',
31 + facts: [
32 + ['typescript-provider-install', 'npm install @composio/core @composio/anthropic @anthropic-ai/sdk'],
33 + ['python-provider-install', 'pip install composio composio_anthropic anthropic'],
34 + ['provider-loop-example', 'results = composio.provider.handle_tool_calls(response=response, session=session)'],
35 + ['provider-version-condition', 'requires `composio` newer than 0.19.0 (Python) or `@composio/core` ≥ 0.17.0 with `@composio/anthropic` ≥ 0.11.0 (TypeScript)'],
36 + ],
37 + },
38 + {
39 + url: '/docs/agent-plugins',
40 + facts: [
41 + ['cli-install', 'curl -fsSL https://composio.dev/install | sh'],
42 + ['login-and-setup', 'composio login\ncomposio setup --target auto'],
43 + ['noninteractive-setup', 'composio setup --target auto --yes'],
44 + ['sdk-alternative', '/docs/quickstart'],
45 + ],
46 + },
47 + {
48 + url: '/docs/agent-setup/clients',
49 + facts: [
50 + ['codex-project-skill', 'npx skills add ComposioHQ/composio --skill composio --agent codex'],
51 + ['codex-prompt', 'Use the $composio skill'],
52 + ['claude-prompt', 'Use the /composio skill'],
53 + ],
54 + },
55 + {
56 + url: '/reference/api-reference/tools',
57 + facts: [
58 + ['current-list-endpoint', '`/api/v3.1/tools`'],
59 + ['toolkit-version-requirement', 'Manual tool execution requires an explicit toolkit version.'],
60 + ['proxy-security-boundary', 'Proxy execute rejects cross-domain requests'],
61 + ['typescript-proxy-example', "const { status, data } = await composio.tools.proxyExecute({\n endpoint: '/repos/composiohq/composio/issues/1',\n method: 'GET',\n connectedAccountId: 'ca_github_user_123',\n parameters: [{ name: 'Accept', value: 'application/vnd.github.v3+json', in: 'header' }],\n});"],
62 + ],
63 + },
64 + {
65 + url: '/reference/v3/api-reference/tools',
66 + facts: [
67 + ['legacy-list-endpoint', '`/api/v3/tools`'],
68 + ['legacy-execute-endpoint', '`/api/v3/tools/execute/{tool_slug}`'],
69 + ],
70 + },
71 + {
72 + url: '/examples/standup-slackbot',
73 + facts: [
74 + ['toolkit-selection', 'Rule of thumb: posting *as the bot* uses `slackbot`; doing something *as a person* uses `slack`.'],
75 + ['selection-image', '![Choosing the Slackbot toolkit, not Slack](/images/standup-slackbot/auth-config-slackbot.png)\n*Pick Slackbot, not Slack*'],
76 + ['staged-setup-example', "const session = await composio.create('default', {\n authConfigs: { slackbot: AUTH_CONFIG },\n });"],
77 + ['repository-availability-warning', 'The standup bot browser is a documentation snapshot; a public repository is not available.'],
78 + ['setup-command', 'npx tsx scripts/setup.ts'],
79 + ],
80 + },
81 +] as const;
82 +
83 +export type ParityPage = (typeof parityPages)[number];
84 +
85 +export function assertPageFacts(text: string, page: ParityPage, surface: string) {
86 + // Line wrapping and paragraph separators can change during ingestion.
87 + // Keep every non-whitespace character, including command flags and API versions.
88 + const normalized = text.replace(/\s+/g, ' ');
89 + const failures = page.facts
90 + .filter(([, expected]) => !normalized.includes(expected.replace(/\s+/g, ' ')))
91 + .map(([id, expected]) => `${surface} ${page.url} lost fact ${id}: ${expected}`);
92 + assert.equal(failures.length, 0, failures.join('\n'));
93 +}
94 +
95 +// Scope corpus assertions to one page so another page cannot hide missing facts.
96 +export function corpusPages(text: string): Map<string, string> {
97 + const headings = [...text.matchAll(/^# [^\n]+ \((\/[^\s)]+)\)\r?$/gm)];
98 + return new Map(headings.map((heading, index) => [
99 + heading[1],
100 + text.slice(heading.index, headings[index + 1]?.index ?? text.length),
101 + ]));
102 +}
docs/tests/integration/semantic-parity.test.ts new
+45
@@ -0,0 +1,45 @@
1 +import { beforeAll, expect, test } from 'bun:test';
2 +import { fetchPage } from './helpers';
3 +import { assertPageFacts, corpusPages, parityPages } from '../fixtures/semantic-parity';
4 +
5 +let corpus: Map<string, string>;
6 +const markdown = new Map<string, string>();
7 +beforeAll(async () => {
8 + const response = await fetchPage('/llms-full.txt', { timeout: 30_000 });
9 + expect(response.status, 'llms-full.txt status').toBe(200);
10 + corpus = corpusPages(await response.text());
11 + await Promise.all(parityPages.map(async page => {
12 + const response = await fetchPage(`${page.url}.md`);
13 + expect(response.status, `HTTP Markdown ${page.url} status`).toBe(200);
14 + expect(response.headers.get('content-type')).toMatch(/text\/(plain|markdown)/);
15 + markdown.set(page.url, await response.text());
16 + }));
17 +});
18 +
19 +for (const page of parityPages) {
20 + const surfaces = [{ name: 'HTTP Markdown', text: () => markdown.get(page.url) ?? '' }];
21 + // Legacy REST pages are individually readable but intentionally absent from the current corpus.
22 + if (page.url.startsWith('/reference/v3/')) {
23 + test(`llms-full.txt excludes legacy page ${page.url}`, () => {
24 + expect(corpus.has(page.url), `Legacy page leaked into llms-full.txt: ${page.url}`).toBe(false);
25 + });
26 + } else {
27 + surfaces.push({ name: 'llms-full.txt', text: () => corpus.get(page.url) ?? '' });
28 + }
29 + for (const surface of surfaces) {
30 + test(`${surface.name} semantic parity ${page.url}`, () => {
31 + assertPageFacts(surface.text(), page, surface.name);
32 + });
33 + for (const [id, expected] of page.facts) {
34 + test(`mutation catches ${surface.name} ${page.url} lost fact ${id}`, () => {
35 + const text = surface.text();
36 + assertPageFacts(text, page, surface.name);
37 + const mutated = text.replace(/\s+/g, ' ')
38 + .split(expected.replace(/\s+/g, ' ')).join('');
39 + expect(() => assertPageFacts(mutated, page, surface.name)).toThrow(
40 + `${surface.name} ${page.url} lost fact ${id}`,
41 + );
42 + });
43 + }
44 + }
45 +}
docs/tests/static/semantic-parity.test.ts new
+104
@@ -0,0 +1,104 @@
1 +import { beforeAll, expect, test } from 'bun:test';
2 +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
3 +import { tmpdir } from 'node:os';
4 +import { join } from 'node:path';
5 +import type { AdvancedIndex } from 'fumadocs-core/search/server';
6 +import { buildCompleteSearchReplacement } from '../../lib/knowledge/search-replacement';
7 +import { getDocsSearchIndexes, type AlgoliaDocsRecord } from '../../lib/search-index';
8 +import { assertPageFacts, corpusPages, parityPages } from '../fixtures/semantic-parity';
9 +
10 +let records: AlgoliaDocsRecord[];
11 +let localIndexes: AdvancedIndex[];
12 +beforeAll(async () => {
13 + // Exercise both production search ingestion and the /api/search fallback corpus.
14 + [records, localIndexes] = await Promise.all([
15 + buildCompleteSearchReplacement({ validateExternal: false }),
16 + getDocsSearchIndexes(),
17 + ]);
18 +});
19 +
20 +for (const page of parityPages) {
21 + const surfaces = [
22 + {
23 + name: 'Algolia ingestion',
24 + text: () => records.filter(record => record.page_id === page.url)
25 + .sort((a, b) => a.position - b.position)
26 + .map(record => record.content).join('\n\n'),
27 + },
28 + {
29 + name: 'local search corpus',
30 + text: () => localIndexes.find(index => index.url === page.url)
31 + ?.structuredData.contents.map(item => item.content).join('\n\n') ?? '',
32 + },
33 + ];
34 + for (const surface of surfaces) {
35 + test(`${surface.name} semantic parity ${page.url}`, () => {
36 + assertPageFacts(surface.text(), page, surface.name);
37 + });
38 + for (const [id, expected] of page.facts) {
39 + test(`mutation catches ${surface.name} ${page.url} lost fact ${id}`, () => {
40 + const text = surface.text();
41 + assertPageFacts(text, page, surface.name);
42 + const mutated = text.replace(/\s+/g, ' ')
43 + .split(expected.replace(/\s+/g, ' ')).join('');
44 + expect(() => assertPageFacts(mutated, page, surface.name)).toThrow(
45 + `${surface.name} ${page.url} lost fact ${id}`,
46 + );
47 + });
48 + }
49 + }
50 +}
51 +
52 +test('legacy REST records remain classified as legacy in the search payload', () => {
53 + const legacy = records.filter(record => record.page_id === '/reference/v3/api-reference/tools');
54 + expect(legacy.length).toBeGreaterThan(0);
55 + expect(legacy.every(record => record.source_type === 'legacy')).toBe(true);
56 +});
57 +
58 +test('facts on an adjacent corpus page cannot mask a missing fact', () => {
59 + const page = parityPages[1];
60 + const pages = corpusPages(`# Quickstart (${page.url})\nMissing install.\n\n---\n\n# Other (/docs/other)\n${page.facts.map(([, text]) => text).join('\n')}`);
61 + expect(() => assertPageFacts(pages.get(page.url)!, page, 'llms-full.txt')).toThrow(
62 + 'llms-full.txt /docs/quickstart lost fact typescript-install',
63 + );
64 +});
65 +
66 +test('external checker preserves ordered line chunks and names missing facts and pages', () => {
67 + const directory = mkdtempSync(join(tmpdir(), 'docs-parity-'));
68 + const path = join(directory, 'snapshot.json');
69 + const snapshot = {
70 + surface: 'test external export', capturedAt: '2026-10-14T00:00:00Z', includesLegacy: true,
71 + pages: parityPages.flatMap(page => records
72 + .filter(record => record.page_id === page.url)
73 + .sort((a, b) => a.position - b.position)
74 + // Regression for #4443: split multi-line facts between adjacent chunks.
75 + .flatMap(record => record.content.split('\n').map(content => ({
76 + url: `https://docs.composio.dev${page.url}.md#chunk`, content,
77 + })))),
78 + };
79 + const run = (input: unknown) => {
80 + writeFileSync(path, JSON.stringify(input));
81 + return Bun.spawnSync([process.execPath, 'scripts/check-docs-parity.ts', path]);
82 + };
83 + try {
84 + const complete = run(snapshot);
85 + expect(complete.exitCode, complete.stderr.toString()).toBe(0);
86 + const missingFact = run({ ...snapshot, pages: snapshot.pages.map(page => ({
87 + ...page,
88 + content: page.content.replaceAll('composio login', ''),
89 + })) });
90 + expect(missingFact.exitCode).not.toBe(0);
91 + expect(missingFact.stderr.toString()).toContain('test external export /docs/agent-plugins lost fact login-and-setup');
92 + console.info(`Expected mutation failure from external checker:\n${missingFact.stderr.toString()}`);
93 + const missingPage = run({ ...snapshot, pages: snapshot.pages.filter(page => !page.url.includes('/quickstart')) });
94 + expect(missingPage.exitCode).not.toBe(0);
95 + expect(missingPage.stderr.toString()).toContain('test external export /docs/quickstart lost fact typescript-install');
96 + const currentOnly = run({ ...snapshot, includesLegacy: false,
97 + pages: snapshot.pages.filter(page => !page.url.includes('/reference/v3/')) });
98 + expect(currentOnly.exitCode, currentOnly.stderr.toString()).toBe(0);
99 + expect(run({ ...snapshot, pages: [{ url: '/docs', content: 42 }] }).exitCode).not.toBe(0);
100 + expect(run({ ...snapshot, pages: [{ url: 'https://example.com/docs', content: '' }] }).exitCode).not.toBe(0);
101 + } finally {
102 + rmSync(directory, { recursive: true, force: true });
103 + }
104 +});