Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 09:45 UTC
test(docs): preserve semantic parity across agent-readable surfaces (#4871)
## Summary Add semantic parity assertions for task-critical documentation facts, not just endpoint availability or text length. Related issue: [DEVREL-32](https://linear.app/composio/issue/DEVREL-32/add-semantic-parity-tests-across-agent-readable-documentation-surfaces). - Define 35 reviewed facts once across eight representative pages: product selection, Platform quickstart, For You plugins, coding-agent clients, Anthropic provider, current and legacy REST tools, and the standup bot example. - Apply the same expectations to HTTP Markdown, page-scoped `/llms-full.txt`, the local `/api/search` corpus, the complete chunked Algolia ingestion payload, and opt-in captured exports. - Remove every fact from real rendered/indexed output to prove detection. Failures name the page, fact ID, and surface. The external-checker test prints a deliberately removed login command and its expected nonzero failure into normal CI output. - Preserve converted page text in the local search corpus. The new assertions exposed that Fumadocs structured search data omitted commands, code, and custom-component instructions there. Existing section-linked prose and generated OpenAPI search data remain intact; a production HTTP regression verifies the existing section destination. - Add `bun run check:docs-parity <snapshot.json>` and contributor guidance. Existing docs static/integration commands and workflows discover the new tests without a duplicate CI job. Live external checks remain opt-in; no external index refresh or retrieval ranking improvement is claimed. ## Relationship to closed PR #4443 This revives the focused implementation from #4443 (commit `2f4e5e84780a8651c64f035435b3cdb1da409b67`) on current `next`, rather than replacing its design. Investigation: read its description, diff, branch, reviews, comments, and GitHub timeline. The author closed it on 2026-09-11 at 14:42:23 UTC without a recorded reason. There is no timeline evidence of supersession or a conflict explanation. The concrete review finding was that inserting blank lines between ordered chunks falsely rejected a multi-line fact; no fix followed on that branch. The commit is not an ancestor of `next`; neither added fixture/checker path has history on `next`, and no merged semantic-parity PR was found. This revision fixes that review finding by ignoring whitespace-only differences while preserving command flags, package names, versions, and code punctuation. A regression checks actual ingestion content split into ordered line chunks, including the two-command login/setup fact and a multi-line proxy example. It also adds real-output mutations, the local search corpus, and audited image/staged-source/repository-warning facts that the original PR did not cover. The requested `master` base does not exist locally or on `origin`. After that was reported, the requester selected `next`. Work is isolated in a new worktree; unrelated dirty work was preserved. ## Verification Commands run from `docs/`: - `bun install --frozen-lockfile` - `bun run test`: **715 passed, 0 failed**, 67 files. - `NODE_OPTIONS=--max-old-space-size=6144 RAYON_NUM_THREADS=2 bun run build`: production build passed, including TypeScript and 2,443 generated static pages. - `bun run start --port 3312` - `TEST_BASE_URL=http://localhost:3312 bun run test:integration`: **178 passed, 1 existing API-key-dependent skip, 0 failed**, 8 files. - `bun run types:check`: passed. - `bun run lint`: passed with 63 warnings. - `bun run lint:links`: 0 errors. - Runtime smoke: production `/api/search?query=composio_openai_agents` retrieved the quickstart provider import, which the old structured corpus omitted. - Runtime section-context regression: matching prose remains linked to `/docs/configuring-sessions#creating-a-session`, alongside the newly indexed code. - Captured all eight production HTTP Markdown responses and ran `bun run check:docs-parity <snapshot.json>`: complete snapshot passed. After removing the quickstart npm installation command, it exited 1 with `production HTTP snapshot /docs/quickstart lost fact typescript-install`. - Reproduced the #4443 chunk-boundary false rejection before the fix and confirmed the same ordered chunks pass afterward. The initial simultaneous broad-test/build attempt timed out while available memory was low. Static tests passed when run alone; the unchanged production build command passed with bounded Node memory and Rust workers. No repository build configuration was changed. No UI changes, new dependencies, generated artifacts, credentials, or customer data. No Linear API writes were performed. ## CI evidence Both workflows passed on final commit `c2ade96e6`: - [Docs - Tests](https://github.com/ComposioHQ/composio/actions/runs/38661432125/job/116183359904): static tests, production build, HTTP integration tests, and the section-destination regression passed. - [Docs - Lint and TypeScript Validation](https://github.com/ComposioHQ/composio/actions/runs/38661432288/job/116183360524): lint, type-check, and production/Twoslash build passed. The `Run static tests` log contains the deliberately broken-fact proof: ```text Expected mutation caught (checker exit 1): test external export /docs/agent-plugins lost fact login-and-setup ``` The checker failure is expected and asserted by the mutation test; the workflow remains green. The docs test workflow also reports a stale KB semantic-artifact warning and keyword fallback. Rebuilding that external embedding artifact is outside this change.
No changes.