@devries-cameron20 / composio-1 / commits / ea2e02071
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

docs: fast-search with mercury-2 (#3687)

## Summary - Switch the docs Eve agent from the AI Gateway `openai/gpt-5.4-mini` string to an Inception Labs Mercury 2 OpenAI-compatible chat model. - Keep tool calling on the chat-completions path and pass Mercury's `reasoning_effort=medium` through the AI SDK OpenAI adapter. - Add `DOCS_AGENT_MODEL_FLOW` so the same agent can run either `mercury` or the old AI Gateway flow for eval comparisons. - Add docs-agent eve evals covering grounded docs answers, docs retrieval, citations, and account-specific support refusal. - Replace the docs-agent retriever with an in-process BM25-style lexical ranker that returns bounded full content for the top results, so Mercury gets rich context in one fast tool call instead of a serial `search_docs` → `read_doc` round trip. - Precompute BM25 term counts/document frequencies into the generated `agent/lib/docs-index.ts` snapshot at build time, removing deployed cold-start corpus construction while keeping retrieval in-process. - Add opt-in search perf logging (`DOCS_AGENT_SEARCH_PERF_LOG=1`, optional `DOCS_AGENT_SEARCH_LOG_QUERY=1`) with timings for tokenization, corpus load/cache, ranking, hydration, total duration, corpus source, and top URLs. - Add `eval:agent` and `eval:agent:flows` scripts; `eval:agent:flows` can run local model-flow comparisons or remote target comparisons via `DOCS_AGENT_EVAL_TARGETS`. - Add `INCEPTION_API_KEY` / optional Mercury and gateway model knobs to `docs/.env.example`, and move `@ai-sdk/openai` to runtime dependencies for the agent import. ## Notes - This is intentionally an experiment to see how Mercury's diffusion model behaves with Eve tool calling (`search_docs` and `read_doc`). - Preview/runtime environments need `INCEPTION_API_KEY`; `INCEPTION_MODEL` and `INCEPTION_BASE_URL` are optional overrides. - The custom fetch prevents accidentally falling back to `OPENAI_API_KEY` against Inception's endpoint. - The docs search is lexical/in-memory, not vector search. The slow path was mostly serial model/tool round trips and cold index construction, not embedding lookup. - The generated BM25 snapshot is process-local once loaded: warm for the lifetime of the running Node/Vercel function instance, and reset on cold starts, redeploys, or process restarts. The expensive term-count corpus is now built at docs build time. - Perf logs omit raw user queries by default; set `DOCS_AGENT_SEARCH_LOG_QUERY=1` only when you explicitly want raw query/term logging. - Local A/B-style eval run: ```bash DOCS_AGENT_EVAL_FLOWS=gateway,mercury bun run eval:agent:flows -- --strict ``` - Live target comparison: ```bash DOCS_AGENT_EVAL_TARGETS=baseline=https://<prod>,mercury=https://<preview> bun run eval:agent:flows -- --strict ``` ## Tests - `bunx eslint scripts/build-agent-index.ts agent/lib/docs.ts agent/tools/search_docs.ts` - `bun scripts/build-agent-index.ts` (wrote 133 pages + 1000 toolkits + 1139 BM25 rows) - `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "const tool=(await import('./agent/tools/search_docs.ts?log=' + Date.now())).default; await tool.execute({query:'create a session with github tools'}); await tool.execute({query:'auth config connected account'});"` (logs cold and warm timing JSON) - `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await import('./agent/tools/search_docs.ts?bundle=' + Date.now())).default; const started=performance.now(); const r=await tool.execute({query:'create a session with github tools'}); console.log(r.retrieval, r.results[0].url, r.results[0].content.length, Math.round(performance.now()-started)+'ms');"` (precomputed bundle path, ~12ms) - `bun -e "const tool=(await import('./agent/tools/search_docs.ts?live=' + Date.now())).default; const started=performance.now(); const r=await tool.execute({query:'create a session with github tools'}); console.log(r.retrieval, r.results[0].url, r.results[0].content.length, Math.round(performance.now()-started)+'ms');"` (live-content path, ~34ms) - `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await import('./agent/tools/search_docs.ts')).default; await tool.execute({query:'create a session with github tools'}); const started=performance.now(); const r=await tool.execute({query:'auth config connected account'}); console.log(r.results[0].url, r.results[0].content.length, Math.round(performance.now()-started)+'ms');"` (warm path ~2ms) - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH ./node_modules/.bin/eve info --json` (reports `status: ready`, `model: inception/mercury-2`, `errors: 0`) - `DOCS_AGENT_MODEL_FLOW=gateway PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH ./node_modules/.bin/eve info --json` (reports `status: ready`, `model: openai/gpt-5.4-mini`, `errors: 0`) - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH ./node_modules/.bin/eve eval --list` - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH bun scripts/eval-agent-flows.ts --list` - `bun test tests/static/` (16 passed) - `bun run types:check` currently fails on existing docs type-generation errors in `app/(home)/docs/changelog/[...slug]/page.tsx`, `app/(home)/examples/[[...slug]]/page.tsx`, `app/(home)/toolkits/[[...slug]]/page.tsx`, `app/llms.mdx/[[...slug]]/route.ts`, `lib/search-index.ts`, and `lib/source.ts`; no new eval or `docs/agent/agent.ts` errors were reported. ## Not run - Real live model evals, because this local environment does not have `INCEPTION_API_KEY` or AI Gateway credentials. ## Latest update - Added default eager docs retrieval in the Eve HTTP channel: the server runs the same BM25 search on the user's message before the first model step and injects the results as one-turn context. - Kept `search_docs` and `read_doc` available so Mercury can still search/read more when the eager context is weak, ambiguous, or missing. - Added `DOCS_AGENT_EAGER_SEARCH=0` as an escape hatch and labeled perf logs with `invocation: "eager_context" | "tool"`. - Updated the loading copy from “Searching the docs…” to “Thinking with the docs…” so UI latency is not attributed solely to the search call. ## Latest tests - `bun run lint -- agent/channels/eve.ts agent/tools/search_docs.ts agent/lib/docs-search.ts components/eve-chat.tsx evals/docs-agent/grounded-answers.eval.ts` - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH node_modules/eve/bin/eve.js info --json` (reports `status: ready`, `errors: 0`) - `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "import { searchDocs } from './agent/lib/docs-search'; const r = searchDocs('How do I create a session in Composio? Keep it brief.', { invocation: 'eager_context' }); console.log(JSON.stringify({count:r.results.length, top:r.results[0]?.url, content: !!r.results[0]?.content}, null, 2));"` - `DOCS_AGENT_SEARCH_PERF_LOG=1 bun -e "import { searchDocs } from './agent/lib/docs-search'; searchDocs('How do I create a session in Composio? Keep it brief.', { invocation: 'eager_context' }); searchDocs('How do I create a session in Composio? Keep it brief.', { invocation: 'tool' });"` - `bun run types:check` still fails only on the pre-existing docs type-generation issues listed above. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rahul Tarak committed Jun 26, 2026 at 03:17 UTC ea2e02071ac7c586364519bd4d1ded8161ca385a
18 files changed +1400 -142
docs/.env.example
+11
@@ -5,6 +5,17 @@ COMPOSIO_API_BASE=https://backend.composio.dev/api/v3
5 # OpenAPI Spec URL for API Reference
6 OPENAPI_SPEC_URL=https://backend.composio.dev/api/v3/openapi.json
7
8 +# Docs sidebar agent model (Inception Labs Mercury by default)
9 +DOCS_AGENT_MODEL_FLOW=mercury
10 +INCEPTION_API_KEY=your_inception_api_key_here
11 +# INCEPTION_MODEL=mercury-2
12 +# INCEPTION_BASE_URL=https://api.inceptionlabs.ai/v1
13 +# DOCS_AGENT_MODEL_FLOW=gateway
14 +# DOCS_AGENT_GATEWAY_MODEL=openai/gpt-5.4-mini
15 +# DOCS_AGENT_EAGER_SEARCH=0 # disable automatic pre-model docs retrieval; enabled by default
16 +# DOCS_AGENT_SEARCH_PERF_LOG=1
17 +# DOCS_AGENT_SEARCH_LOG_QUERY=1 # includes raw user queries in server logs; avoid in production
18 +
19 # PostHog Analytics (optional)
20 NEXT_PUBLIC_POSTHOG_KEY=your_posthog_project_api_key
21 NEXT_PUBLIC_POSTHOG_HOST=https://us.i.posthog.com
docs/agent/agent.ts
+92 -4
@@ -1,4 +1,82 @@
1 +import { createOpenAI } from '@ai-sdk/openai';
2 import { defineAgent } from 'eve';
3 +import type { AgentModelDefinition, AgentModelOptionsDefinition } from 'eve';
4 +
5 +const INCEPTION_BASE_URL = process.env.INCEPTION_BASE_URL ?? 'https://api.inceptionlabs.ai/v1';
6 +const INCEPTION_MODEL = process.env.INCEPTION_MODEL ?? 'mercury-2';
7 +const DOCS_AGENT_GATEWAY_MODEL = process.env.DOCS_AGENT_GATEWAY_MODEL ?? 'openai/gpt-5.4-mini';
8 +const DOCS_AGENT_MODEL_FLOW = process.env.DOCS_AGENT_MODEL_FLOW ?? 'mercury';
9 +
10 +type DocsAgentModelConfig = {
11 + model: AgentModelDefinition;
12 + modelContextWindowTokens?: number;
13 + modelOptions?: AgentModelOptionsDefinition;
14 +};
15 +
16 +const resolveInceptionApiKey = () => {
17 + const apiKey = process.env.INCEPTION_API_KEY;
18 +
19 + if (!apiKey) {
20 + throw new Error('INCEPTION_API_KEY is required to run the docs agent with Inception Mercury.');
21 + }
22 +
23 + return apiKey;
24 +};
25 +
26 +const inception = createOpenAI({
27 + name: 'inception',
28 + baseURL: INCEPTION_BASE_URL,
29 + // The OpenAI-compatible AI SDK provider otherwise falls back to OPENAI_API_KEY
30 + // when apiKey is undefined. Keep the actual Mercury key runtime-resolved so we
31 + // never accidentally send an OpenAI key to Inception's endpoint.
32 + apiKey: 'runtime-resolved-by-inception-fetch',
33 + fetch: async (input, init) => {
34 + const headers = new Headers(init?.headers);
35 + headers.set('Authorization', `Bearer ${resolveInceptionApiKey()}`);
36 +
37 + return fetch(input, { ...init, headers });
38 + },
39 +});
40 +
41 +const gatewayModel = (model: string): AgentModelDefinition => {
42 + // Eve supports AI Gateway model id strings here. Cast until the published
43 + // type catches up with the documented `defineAgent({ model: "provider/model" })`
44 + // gateway path.
45 + return model as unknown as AgentModelDefinition;
46 +};
47 +
48 +const resolveDocsAgentModel = (): DocsAgentModelConfig => {
49 + switch (DOCS_AGENT_MODEL_FLOW) {
50 + case 'gateway':
51 + return {
52 + model: gatewayModel(DOCS_AGENT_GATEWAY_MODEL),
53 + };
54 + case 'mercury':
55 + return {
56 + // Use the chat-completions path because Mercury exposes tool calling there.
57 + model: inception.chat(INCEPTION_MODEL),
58 + // Mercury 2's chat context window is 128K tokens.
59 + modelContextWindowTokens: 128_000,
60 + modelOptions: {
61 + providerOptions: {
62 + openai: {
63 + // Mercury supports tool calling and the OpenAI-compatible adapter
64 + // maps this to `reasoning_effort`. `medium` is Inception's
65 + // recommended default and is accepted by the AI SDK OpenAI provider
66 + // schema.
67 + reasoningEffort: 'medium',
68 + },
69 + },
70 + },
71 + };
72 + default:
73 + throw new Error(
74 + `Unsupported DOCS_AGENT_MODEL_FLOW "${DOCS_AGENT_MODEL_FLOW}". Expected "mercury" or "gateway".`
75 + );
76 + }
77 +};
78 +
79 +const docsAgentModel = resolveDocsAgentModel();
80
81 /**
82 * Eve — the Composio docs assistant.
@@ -7,10 +85,20 @@ import { defineAgent } from 'eve';
85 * and answers questions grounded in the docs. The `search_docs` tool returns
86 * doc snippets and their URLs so Eve can cite and link specific pages.
87 *
10 - * Model: `openai/gpt-5.4-mini` routed through the Vercel AI Gateway. Set
11 - * `AI_GATEWAY_API_KEY` locally and in the preview environment (or `vercel link`
12 - * for an OIDC token).
88 + * Model: defaults to Inception Labs Mercury 2 diffusion model through the
89 + * OpenAI-compatible chat endpoint. Set `INCEPTION_API_KEY` locally and in the
90 + * preview environment.
91 + *
92 + * Evaluation / A-B knobs:
93 + *
94 + * - `DOCS_AGENT_MODEL_FLOW=mercury` (default) uses Inception Mercury.
95 + * - `DOCS_AGENT_MODEL_FLOW=gateway` uses the Vercel AI Gateway fallback.
96 + * - `INCEPTION_MODEL` (default: `mercury-2`)
97 + * - `INCEPTION_BASE_URL` (default: `https://api.inceptionlabs.ai/v1`)
98 + * - `DOCS_AGENT_GATEWAY_MODEL` (default: `openai/gpt-5.4-mini`)
99 */
100 export default defineAgent({
15 - model: 'openai/gpt-5.4-mini',
101 + model: docsAgentModel.model,
102 + modelContextWindowTokens: docsAgentModel.modelContextWindowTokens,
103 + modelOptions: docsAgentModel.modelOptions,
104 });
docs/agent/channels/eve.ts
+92 -2
@@ -1,5 +1,7 @@
1 -import { eveChannel } from 'eve/channels/eve';
1 +import type { UserContent } from 'ai';
2 import { none } from 'eve/channels/auth';
3 +import { defaultEveAuth, eveChannel } from 'eve/channels/eve';
4 +import { searchDocs, shouldRunEagerDocsSearch, type SearchDocsResult } from '../lib/docs-search';
5
6 /**
7 * HTTP channel for the docs assistant.
@@ -9,4 +11,92 @@ import { none } from 'eve/channels/auth';
11 * publicly; before production we should add rate limiting and abuse protection
12 * (or gate it behind the site's own auth).
13 */
12 -export default eveChannel({ auth: [none()] });
14 +
15 +const EAGER_SEARCH_LIMIT = 3;
16 +const EAGER_CONTENT_RESULTS = 2;
17 +const EAGER_MAX_CONTENT_CHARS = 6000;
18 +const EAGER_MAX_SECTIONS = 6;
19 +const MAX_CONTEXT_SECTIONS = EAGER_MAX_SECTIONS;
20 +
21 +function messageToText(message: string | UserContent): string {
22 + if (typeof message === 'string') return message;
23 +
24 + return message
25 + .map(part => (part.type === 'text' ? part.text : ''))
26 + .join('\n')
27 + .trim();
28 +}
29 +
30 +function shouldEagerSearch(text: string): boolean {
31 + return shouldRunEagerDocsSearch(text);
32 +}
33 +
34 +function formatSections(result: SearchDocsResult['results'][number]): string {
35 + const sections = result.sections?.slice(0, MAX_CONTEXT_SECTIONS) ?? [];
36 + if (sections.length === 0) return '';
37 +
38 + return sections.map(section => `[${section.title}](${result.url}${section.anchor})`).join(', ');
39 +}
40 +
41 +function formatEagerSearchContext(result: SearchDocsResult): string | undefined {
42 + if (result.results.length === 0) return undefined;
43 +
44 + const docs = result.results
45 + .map((page, index) => {
46 + const sections = formatSections(page);
47 + const content = page.content
48 + ? `${page.content}${page.contentTruncated ? '\n\n…(content truncated)' : ''}`
49 + : page.snippet;
50 +
51 + return [
52 + `### ${index + 1}. ${page.title}`,
53 + `URL: ${page.url}`,
54 + page.description ? `Description: ${page.description}` : undefined,
55 + sections ? `Sections: ${sections}` : undefined,
56 + 'Content:',
57 + content,
58 + ]
59 + .filter(Boolean)
60 + .join('\n');
61 + })
62 + .join('\n\n---\n\n');
63 +
64 + return `Eager Composio docs search context for the user's latest message.
65 +
66 +Use this context when it answers the question. You may still call \`search_docs\` or \`read_doc\` if this context is weak, missing, ambiguous, or you need more detail. Cite only the included docs URLs/section anchors.
67 +
68 +<docs_search_context retrieval="${result.retrieval}">
69 +${docs}
70 +</docs_search_context>`;
71 +}
72 +
73 +function buildEagerSearchContext(message: string | UserContent): string[] | undefined {
74 + const text = messageToText(message);
75 + if (!shouldEagerSearch(text)) return undefined;
76 +
77 + try {
78 + const result = searchDocs(text, {
79 + limit: EAGER_SEARCH_LIMIT,
80 + contentResultCount: EAGER_CONTENT_RESULTS,
81 + maxContentChars: EAGER_MAX_CONTENT_CHARS,
82 + maxSections: EAGER_MAX_SECTIONS,
83 + invocation: 'eager_context',
84 + });
85 + const context = formatEagerSearchContext(result);
86 + return context ? [context] : undefined;
87 + } catch (error) {
88 + console.warn('[docs-agent:eager_search] failed', {
89 + error: error instanceof Error ? error.message : String(error),
90 + });
91 + return undefined;
92 + }
93 +}
94 +
95 +export default eveChannel({
96 + auth: [none()],
97 + onMessage(ctx, message) {
98 + const auth = defaultEveAuth(ctx);
99 + const context = buildEagerSearchContext(message);
100 + return context ? { auth, context } : { auth };
101 + },
102 +});
docs/agent/instructions.md
+18 -10
@@ -9,17 +9,19 @@ You are **Eve**, the Composio documentation assistant. You live in the right sid
9
10 ## How to answer
11
12 -You have a **concept map** (always in your context) with the canonical page for each Composio concept, plus exactly two tools. You have **no web search and no file access** — answer only from the Composio docs via these tools, and link the docs' own relative URLs (never `docs.composio.dev` or other external links).
12 +You have a **concept map** (always in your context) with the canonical page for each Composio concept, plus exactly two tools. You have **no web search and no file access**, so answer only from the Composio docs via these tools, and link the docs' own relative URLs (never `docs.composio.dev` or other external links).
13
14 -- `search_docs(query)` — find relevant pages (title, URL, snippet).
15 -- `read_doc(url)` — read a page's full Markdown content.
14 +- You may receive **eager docs search context** with the user's latest message. It is an automatic `search_docs` result injected before the model step to save latency.
15 +- `search_docs(query)`: fast BM25-style local docs search. It returns relevant pages and bounded full content plus sections for the top results. You can still call it whenever eager context is missing, weak, ambiguous, or too narrow.
16 +- `read_doc(url)`: read a page's full Markdown content when you need a page beyond the content included by eager context or `search_docs`.
17
18 Workflow for anything non-trivial:
19
19 -1. Start from the concept map. For a clear concept (sessions, authentication, triggers, sandbox, …) you already know the canonical page; for anything else, call `search_docs`.
20 -2. Call `read_doc` on the 1–3 most relevant pages and answer from their actual content. Don't answer from snippets alone, and don't guess at APIs, parameters, or behavior.
21 -3. Cite sources inline as Markdown links. When your answer comes from a specific section, **link that section's anchor** using the `sections` `read_doc` returns, e.g. `[userID best practices](/docs/how-composio-works#users)` rather than just `[What is a session?](/docs/how-composio-works)`. Link the specific page (and section), and prefer the canonical link from the concept map.
22 -4. Only say you couldn't find something after you've searched and read the top results and they genuinely don't cover it.
20 +1. Start from the concept map and any eager docs context already in the turn. If that context covers the question, answer directly from it.
21 +2. For a clear concept (sessions, authentication, triggers, sandbox, …) you already know the canonical page; for anything else, call `search_docs` when eager context is absent or insufficient.
22 +3. Answer from the content returned by eager context or `search_docs` when it covers the question. Call `read_doc` only when you need a page that was not included or more untruncated context. Don't guess at APIs, parameters, or behavior.
23 +4. Cite sources inline as Markdown links. When answering from eager context, cite at least one primary docs link early in the first paragraph when relevant. Use section anchors from eager context, `search_docs`, or `read_doc` when available, e.g. `[userID best practices](/docs/how-composio-works#users)` rather than just `[What is a session?](/docs/how-composio-works)`. Link the specific page (and section), and prefer the canonical link from the concept map.
24 +5. Only say you couldn't find something after you've searched and read the top results and they genuinely don't cover it.
25
26 ## Rules
27
@@ -30,6 +32,12 @@ Workflow for anything non-trivial:
32
33 ## Style
34
33 -- Lead with the answer, then a short explanation, then a runnable snippet when it helps. Match the reader's language (Python or TypeScript) when they specify one.
34 -- Second person, plain and confident. No marketing fluff.
35 -- Backtick every identifier, path, and slug. Keep answers tight.
35 +You're answering in a chat sidebar, not writing a doc page. Most answers are one to three short paragraphs, often less. Answer what was asked and stop.
36 +
37 +- **Lead with the answer.** No preamble, no restating the question, no "Great question". The first sentence resolves the ask; explanation follows only if it adds something the reader needs.
38 +- **Don't pad.** Cut summaries, conclusions, and "in short" recaps; you already said it. Don't pile on caveats they didn't ask about or enumerate options they didn't request.
39 +- **Prefer prose over bullets.** Write plain sentences. Use a list only for genuinely parallel items, such as steps to follow or three-plus distinct options. Never bullet a single thought, and don't turn one answer into a wall of headings.
40 +- **Show code when it earns its place.** Add a minimal, runnable example only when code answers faster than words. If the reader specifies a language, match it and show just that one. Otherwise show TypeScript and Python back-to-back: two consecutive fenced blocks tagged ` ```typescript ` then ` ```python ` (the chat groups adjacent code blocks in different languages into tabs). Keep each example tight: one per language, no third variant, no prose between the two blocks.
41 +- **Second person, plain and confident.** Say what's true. Use contractions. Cut vague intensifiers ("powerful", "robust", "seamlessly", "simply", "easily") and marketing fluff.
42 +- **No emojis, and no em-dashes.** Don't decorate with emojis or use them as bullets. For punctuation, use a period, comma, colon, or parentheses instead of an em-dash. Bold a term once when you define it, then stop.
43 +- **Backtick every identifier, path, slug, and command.**
docs/agent/instructions/context.md
+2 -2
@@ -1,6 +1,6 @@
1 # Composio concept map
2
3 -This is your always-on map of Composio's concepts and the canonical page for each. Use it to ground answers and to link the right page. Prefer these links over anything `search_docs` returns. Call `read_doc` on the relevant page when you need detail.
3 +This is your always-on map of Composio's concepts and the canonical page for each. Use it to ground answers and to link the right page. Prefer these links over anything `search_docs` returns. Use the full bounded content from `search_docs` first, and call `read_doc` on the relevant page only when you need detail beyond that included content.
4
5 ## Core model
6
@@ -14,7 +14,7 @@ This is your always-on map of Composio's concepts and the canonical page for eac
14 - **Toolkit** — a collection of related tools for a service (e.g. `github`, `gmail`). A **tool** is one action, named `{TOOLKIT}_{ACTION}` (e.g. `GITHUB_CREATE_ISSUE`). Every toolkit is discoverable by default; restrict with the `toolkits` config. → [Configuring sessions](/docs/configuring-sessions)
15 - **Meta tools** — the fixed set every session exposes (`COMPOSIO_SEARCH_TOOLS`, `COMPOSIO_GET_TOOL_SCHEMAS`, `COMPOSIO_MANAGE_CONNECTIONS`, `COMPOSIO_MULTI_EXECUTE_TOOL`, `COMPOSIO_REMOTE_WORKBENCH`, `COMPOSIO_REMOTE_BASH_TOOL`). → [Meta tools reference](/toolkits/meta-tools)
16 - **Providers** — adapter packages that format Composio tools for a framework (OpenAI, Anthropic, Vercel AI SDK, LangChain, Mastra, Pi, …). → [Providers](/docs/providers)
17 -- **Is a toolkit / integration supported?** Composio has 1000+ toolkits, and `search_docs` indexes the catalog. A matching `/toolkits/<slug>` result means **yes, it's supported** — call `read_doc` on that URL for its details, and point the user at [the toolkits directory](/toolkits). If nothing matches, it isn't a built-in toolkit; suggest [proxy execute](/docs/extending-sessions/proxy-execute) or a [custom tool](/docs/extending-sessions/custom-tools-and-toolkits) for an API you already have.
17 +- **Is a toolkit / integration supported?** Composio has 1000+ toolkits, and `search_docs` indexes the catalog. A matching `/toolkits/<slug>` result means **yes, it's supported** — answer from the returned content when sufficient, call `read_doc` only for more details, and point the user at [the toolkits directory](/toolkits). If nothing matches, it isn't a built-in toolkit; suggest [proxy execute](/docs/extending-sessions/proxy-execute) or a [custom tool](/docs/extending-sessions/custom-tools-and-toolkits) for an API you already have.
18
19 ## Authentication
20
docs/agent/lib/docs-search.ts new
+433
@@ -0,0 +1,433 @@
1 +import {
2 + buildBundledIndex,
3 + buildIndex,
4 + extractSections,
5 + getBundleSearchIndex,
6 + pageSearchKey,
7 + readPageByUrl,
8 + toCleanMarkdown,
9 + tokenize,
10 + type DocPage,
11 +} from './docs';
12 +
13 +/**
14 + * search_docs — find the most relevant Composio docs pages for a query.
15 + *
16 + * This is a local, in-memory lexical retriever. It uses a BM25-style body score
17 + * plus field boosts for title, description, headings, and URL. The top results
18 + * include full page content (bounded per page), so the model gets rich context
19 + * in the same fast tool call instead of doing a serial search -> read round trip.
20 + */
21 +
22 +// Collection priority: docs first, then examples, then references and toolkits.
23 +// (Curated knowledge ranks with docs.) A toolkit-name query still surfaces its
24 +// toolkit page because nothing else matches it.
25 +const PRIORITY: Record<DocPage['collection'], number> = {
26 + docs: 1.3,
27 + knowledge: 1.3,
28 + examples: 1.1,
29 + reference: 0.85,
30 + toolkits: 0.9,
31 +};
32 +
33 +export const DEFAULT_SEARCH_LIMIT = 5;
34 +const DEFAULT_LIMIT = DEFAULT_SEARCH_LIMIT;
35 +export const DEFAULT_CONTENT_RESULT_COUNT = 4;
36 +export const DEFAULT_MAX_CONTENT_CHARS = 10_000;
37 +export const DEFAULT_MAX_SECTIONS = 16;
38 +const BM25_K1 = 1.2;
39 +const BM25_B = 0.75;
40 +const PERF_LOG_ENABLED = process.env.DOCS_AGENT_SEARCH_PERF_LOG === '1';
41 +const PERF_LOG_QUERY = process.env.DOCS_AGENT_SEARCH_LOG_QUERY === '1';
42 +
43 +type CorpusEntry = {
44 + page: DocPage;
45 + termCounts: Map<string, number>;
46 + length: number;
47 +};
48 +
49 +type Corpus = {
50 + entries: CorpusEntry[];
51 + documentFrequency: Map<string, number>;
52 + averageLength: number;
53 +};
54 +
55 +type CorpusSource = 'precomputed' | 'runtime';
56 +
57 +type PrecomputedCorpusResult =
58 + | { corpus: Corpus; fallbackReason?: never }
59 + | { corpus?: never; fallbackReason: string };
60 +
61 +type CorpusLoad = {
62 + corpus: Corpus;
63 + source: CorpusSource;
64 + cached: boolean;
65 + loadMs: number;
66 + fallbackReason?: string;
67 +};
68 +
69 +let corpusCache: Corpus | undefined;
70 +let corpusCacheSource: CorpusSource | undefined;
71 +let corpusCacheFallbackReason: string | undefined;
72 +
73 +function roundMs(ms: number): number {
74 + return Math.round(ms * 100) / 100;
75 +}
76 +
77 +function logPerf(payload: Record<string, unknown>) {
78 + if (!PERF_LOG_ENABLED) return;
79 + console.info(`[docs-agent:search_docs] ${JSON.stringify(payload)}`);
80 +}
81 +
82 +function termCountsFor(page: DocPage): Map<string, number> {
83 + const counts = new Map<string, number>();
84 +
85 + for (const token of tokenize(page.lowerText)) {
86 + counts.set(token, (counts.get(token) ?? 0) + 1);
87 + }
88 +
89 + return counts;
90 +}
91 +
92 +function buildRuntimeCorpus(pages: DocPage[]): Corpus {
93 + const entries = pages.map(page => {
94 + const termCounts = termCountsFor(page);
95 + return {
96 + page,
97 + termCounts,
98 + length: [...termCounts.values()].reduce((sum, count) => sum + count, 0),
99 + };
100 + });
101 + const documentFrequency = new Map<string, number>();
102 +
103 + for (const entry of entries) {
104 + for (const term of entry.termCounts.keys()) {
105 + documentFrequency.set(term, (documentFrequency.get(term) ?? 0) + 1);
106 + }
107 + }
108 +
109 + return {
110 + entries,
111 + documentFrequency,
112 + averageLength:
113 + entries.reduce((sum, entry) => sum + entry.length, 0) / Math.max(entries.length, 1),
114 + };
115 +}
116 +
117 +function buildPrecomputedCorpus(pages: DocPage[]): PrecomputedCorpusResult {
118 + const search = getBundleSearchIndex();
119 + if (!search) return { fallbackReason: 'missing-precomputed-index' };
120 + if (search.entries.length !== pages.length) {
121 + return { fallbackReason: `entry-count-mismatch:${search.entries.length}:${pages.length}` };
122 + }
123 +
124 + const entries: CorpusEntry[] = [];
125 +
126 + for (let index = 0; index < pages.length; index++) {
127 + const page = pages[index];
128 + const entry = search.entries[index];
129 +
130 + if (!entry) return { fallbackReason: `missing-entry:${index}` };
131 + if (entry.key !== pageSearchKey(page)) {
132 + return { fallbackReason: `entry-key-mismatch:${index}:${page.url}` };
133 + }
134 +
135 + entries.push({
136 + page,
137 + termCounts: new Map(entry.terms),
138 + length: entry.length,
139 + });
140 + }
141 +
142 + return {
143 + corpus: {
144 + entries,
145 + documentFrequency: new Map(search.documentFrequency),
146 + averageLength: search.averageLength,
147 + },
148 + };
149 +}
150 +
151 +function getCorpus(): CorpusLoad {
152 + if (corpusCache && corpusCacheSource) {
153 + return {
154 + corpus: corpusCache,
155 + source: corpusCacheSource,
156 + cached: true,
157 + loadMs: 0,
158 + fallbackReason: corpusCacheFallbackReason,
159 + };
160 + }
161 +
162 + const started = performance.now();
163 + const bundledPages = buildBundledIndex();
164 + const bundledPrecomputed = buildPrecomputedCorpus(bundledPages);
165 +
166 + if (bundledPrecomputed.corpus) {
167 + corpusCache = bundledPrecomputed.corpus;
168 + corpusCacheSource = 'precomputed';
169 + } else {
170 + const pages = buildIndex();
171 + const livePrecomputed = buildPrecomputedCorpus(pages);
172 +
173 + if (livePrecomputed.corpus) {
174 + corpusCache = livePrecomputed.corpus;
175 + corpusCacheSource = 'precomputed';
176 + } else {
177 + corpusCache = buildRuntimeCorpus(pages);
178 + corpusCacheSource = 'runtime';
179 + corpusCacheFallbackReason = `bundle:${bundledPrecomputed.fallbackReason};live:${livePrecomputed.fallbackReason}`;
180 + }
181 + }
182 +
183 + return {
184 + corpus: corpusCache,
185 + source: corpusCacheSource,
186 + cached: false,
187 + loadMs: performance.now() - started,
188 + fallbackReason: corpusCacheFallbackReason,
189 + };
190 +}
191 +
192 +function idf(term: string, corpus: Corpus): number {
193 + const n = corpus.entries.length;
194 + const df = corpus.documentFrequency.get(term) ?? 0;
195 + return Math.log(1 + (n - df + 0.5) / (df + 0.5));
196 +}
197 +
198 +function bm25(entry: CorpusEntry, terms: string[], corpus: Corpus): number {
199 + let total = 0;
200 +
201 + for (const term of terms) {
202 + const tf = entry.termCounts.get(term) ?? 0;
203 + if (tf === 0) continue;
204 +
205 + const denominator =
206 + tf + BM25_K1 * (1 - BM25_B + BM25_B * (entry.length / corpus.averageLength));
207 + total += idf(term, corpus) * ((tf * (BM25_K1 + 1)) / denominator);
208 + }
209 +
210 + return total;
211 +}
212 +
213 +function fieldBoost(page: DocPage, terms: string[]): number {
214 + const title = page.title.toLowerCase();
215 + const description = page.description.toLowerCase();
216 + const url = page.url.toLowerCase();
217 + let total = 0;
218 +
219 + for (const term of terms) {
220 + if (title.includes(term)) total += 12;
221 + if (description.includes(term)) total += 5;
222 + for (const heading of page.headings) if (heading.includes(term)) total += 4;
223 + if (url.includes(term)) total += 6;
224 + }
225 +
226 + return total;
227 +}
228 +
229 +function score(entry: CorpusEntry, terms: string[], corpus: Corpus): number {
230 + let total = bm25(entry, terms, corpus) * 8 + fieldBoost(entry.page, terms);
231 + const isMigrationIntent = terms.some(term =>
232 + ['migration', 'migrate', 'direct', 'legacy', 'v1', 'v2'].includes(term)
233 + );
234 +
235 + // Heavily downrank legacy (direct-execution) pages so they only surface when
236 + // nothing in the session-based docs matches.
237 + if (entry.page.legacy) total *= 0.12;
238 + // Migration pages mention both old and current APIs a lot; keep them for
239 + // migration/direct-execution questions, but don't let them beat canonical
240 + // session docs for ordinary usage questions.
241 + if (!isMigrationIntent && entry.page.url.includes('/migration-guide')) total *= 0.35;
242 + return total * (PRIORITY[entry.page.collection] ?? 1);
243 +}
244 +
245 +function firstTermMatch(text: string, terms: string[]): number {
246 + const lower = text.toLowerCase();
247 + return (
248 + terms
249 + .map(term => lower.indexOf(term))
250 + .filter(index => index >= 0)
251 + .sort((a, b) => a - b)[0] ?? 0
252 + );
253 +}
254 +
255 +function excerpt(
256 + text: string,
257 + terms: string[],
258 + maxChars: number
259 +): { value: string; truncated: boolean } {
260 + const at = firstTermMatch(text, terms);
261 + const start = Math.max(0, at - 180);
262 + const end = Math.min(text.length, start + maxChars);
263 + const slice = text.slice(start, end).trim();
264 + const prefix = start > 0 ? '…' : '';
265 + const suffix = end < text.length ? '…' : '';
266 +
267 + return { value: `${prefix}${slice}${suffix}`, truncated: start > 0 || end < text.length };
268 +}
269 +
270 +function snippet(page: DocPage, terms: string[]): string {
271 + return excerpt(page.text, terms, 360).value;
272 +}
273 +
274 +function dedupeByUrl(ranked: { page: DocPage; s: number }[]): { page: DocPage; s: number }[] {
275 + const seen = new Set<string>();
276 + const deduped: { page: DocPage; s: number }[] = [];
277 +
278 + for (const item of ranked) {
279 + if (seen.has(item.page.url)) continue;
280 + seen.add(item.page.url);
281 + deduped.push(item);
282 + }
283 +
284 + return deduped;
285 +}
286 +
287 +function contentFor(
288 + page: DocPage,
289 + terms: string[],
290 + maxContentChars: number,
291 + maxSections: number
292 +) {
293 + if (page.collection === 'knowledge') {
294 + const evidence = excerpt(page.text, terms, maxContentChars);
295 + return {
296 + content: evidence.value,
297 + contentTruncated: evidence.truncated,
298 + };
299 + }
300 +
301 + const found = readPageByUrl(page.url);
302 +
303 + if (found) {
304 + const markdown = toCleanMarkdown(found.raw);
305 + const evidence = excerpt(markdown, terms, maxContentChars);
306 +
307 + return {
308 + sections: extractSections(markdown).slice(0, maxSections),
309 + content: evidence.value,
310 + contentTruncated: evidence.truncated,
311 + };
312 + }
313 +
314 + const evidence = excerpt(page.text, terms, maxContentChars);
315 + return {
316 + content: evidence.value,
317 + contentTruncated: evidence.truncated,
318 + };
319 +}
320 +
321 +
322 +export type SearchDocsInvocation = 'tool' | 'eager_context' | 'eager_preview' | (string & {});
323 +
324 +export type SearchDocsOptions = {
325 + limit?: number;
326 + invocation?: SearchDocsInvocation;
327 + contentResultCount?: number;
328 + maxContentChars?: number;
329 + maxSections?: number;
330 + /** When false, skip page hydration and return metadata/snippets only. */
331 + hydrateContent?: boolean;
332 +};
333 +
334 +export const EAGER_SEARCH_ENABLED = process.env.DOCS_AGENT_EAGER_SEARCH !== '0';
335 +
336 +export function shouldRunEagerDocsSearch(text: string): boolean {
337 + if (!EAGER_SEARCH_ENABLED) return false;
338 + if (text.trim().length < 3) return false;
339 +
340 + const normalized = text.toLowerCase();
341 + const accountTerms =
342 + /\b(account|billing|invoice|payment|refund|subscription|ticket|dashboard|workspace|organization|org|api key)\b/;
343 + const personalTerms =
344 + /\b(my|our|me|us|latest|current|status|paid|check|look up|lookup|change|cancel|delete|update)\b/;
345 +
346 + return !(accountTerms.test(normalized) && personalTerms.test(normalized));
347 +}
348 +
349 +export type SearchDocsResult = {
350 + retrieval: 'bm25-lexical-local';
351 + results: Array<{
352 + title: string;
353 + url: string;
354 + description: string;
355 + snippet: string;
356 + sections?: { title: string; anchor: string }[];
357 + content?: string;
358 + contentTruncated?: boolean;
359 + }>;
360 +};
361 +
362 +export function searchDocs(query: string, options: SearchDocsOptions = {}): SearchDocsResult {
363 + const limit = options.limit ?? DEFAULT_LIMIT;
364 + const contentResultCount = options.contentResultCount ?? DEFAULT_CONTENT_RESULT_COUNT;
365 + const maxContentChars = options.maxContentChars ?? DEFAULT_MAX_CONTENT_CHARS;
366 + const maxSections = options.maxSections ?? DEFAULT_MAX_SECTIONS;
367 + const hydrateContent = options.hydrateContent ?? true;
368 + const totalStarted = performance.now();
369 + const tokenizeStarted = performance.now();
370 + const terms = tokenize(query);
371 + // Fall back to raw terms if the query was all stopwords.
372 + const effective =
373 + terms.length > 0
374 + ? terms
375 + : query
376 + .toLowerCase()
377 + .split(/\s+/)
378 + .filter(t => t.length > 1);
379 + const tokenizeMs = performance.now() - tokenizeStarted;
380 +
381 + const corpusLoad = getCorpus();
382 + const corpus = corpusLoad.corpus;
383 +
384 + const rankStarted = performance.now();
385 + const ranked = dedupeByUrl(
386 + corpus.entries
387 + .map(entry => ({ page: entry.page, s: score(entry, effective, corpus) }))
388 + .filter(({ s }) => s > 0)
389 + .sort((a, b) => b.s - a.s)
390 + ).slice(0, limit);
391 + const rankMs = performance.now() - rankStarted;
392 +
393 + const hydrateStarted = performance.now();
394 + const results = ranked.map(({ page }, index) => ({
395 + title: page.title,
396 + url: page.url,
397 + description: page.description,
398 + snippet: snippet(page, effective),
399 + ...(hydrateContent && index < contentResultCount
400 + ? contentFor(page, effective, maxContentChars, maxSections)
401 + : {}),
402 + }));
403 + const hydrateMs = performance.now() - hydrateStarted;
404 + const totalMs = performance.now() - totalStarted;
405 +
406 + logPerf({
407 + event: 'search_docs',
408 + invocation: options.invocation ?? 'tool',
409 + retrieval: 'bm25-lexical-local',
410 + totalMs: roundMs(totalMs),
411 + tokenizeMs: roundMs(tokenizeMs),
412 + corpusLoadMs: roundMs(corpusLoad.loadMs),
413 + rankMs: roundMs(rankMs),
414 + hydrateMs: roundMs(hydrateMs),
415 + corpusSource: corpusLoad.source,
416 + corpusCached: corpusLoad.cached,
417 + corpusFallbackReason: corpusLoad.fallbackReason,
418 + queryChars: query.length,
419 + termCount: effective.length,
420 + limit,
421 + resultCount: results.length,
422 + contentResultCount: results.filter(result => 'content' in result).length,
423 + corpusPages: corpus.entries.length,
424 + corpusTerms: corpus.documentFrequency.size,
425 + topUrls: results.slice(0, 5).map(result => result.url),
426 + ...(PERF_LOG_QUERY ? { query, terms: effective } : {}),
427 + });
428 +
429 + return {
430 + retrieval: 'bm25-lexical-local',
431 + results,
432 + };
433 +}
docs/agent/lib/docs.ts
+165 -37
@@ -53,11 +53,61 @@ const LEGACY_URL_PATTERNS = [
53 ];
54
55 const STOPWORDS = new Set([
56 - 'the', 'a', 'an', 'and', 'or', 'but', 'is', 'are', 'was', 'were', 'be', 'been',
57 - 'to', 'of', 'in', 'on', 'for', 'with', 'as', 'at', 'by', 'from', 'how', 'do',
58 - 'does', 'did', 'what', 'why', 'when', 'which', 'who', 'can', 'i', 'you', 'it',
59 - 'this', 'that', 'these', 'those', 'my', 'your', 'we', 'they', 'use', 'using',
60 - 'work', 'works', 'about', 'into', 'so', 'if', 'me', 'get', 'set', 'up',
56 + 'the',
57 + 'a',
58 + 'an',
59 + 'and',
60 + 'or',
61 + 'but',
62 + 'is',
63 + 'are',
64 + 'was',
65 + 'were',
66 + 'be',
67 + 'been',
68 + 'to',
69 + 'of',
70 + 'in',
71 + 'on',
72 + 'for',
73 + 'with',
74 + 'as',
75 + 'at',
76 + 'by',
77 + 'from',
78 + 'how',
79 + 'do',
80 + 'does',
81 + 'did',
82 + 'what',
83 + 'why',
84 + 'when',
85 + 'which',
86 + 'who',
87 + 'can',
88 + 'i',
89 + 'you',
90 + 'it',
91 + 'this',
92 + 'that',
93 + 'these',
94 + 'those',
95 + 'my',
96 + 'your',
97 + 'we',
98 + 'they',
99 + 'use',
100 + 'using',
101 + 'work',
102 + 'works',
103 + 'about',
104 + 'into',
105 + 'so',
106 + 'if',
107 + 'me',
108 + 'get',
109 + 'set',
110 + 'up',
111 ]);
112
113 export type Collection = 'docs' | 'examples' | 'reference' | 'toolkits' | 'knowledge';
@@ -92,7 +142,20 @@ interface BundleToolkit {
142 category?: string;
143 authSchemes?: string[];
144 }
95 -const BUNDLE = bundledIndex as { pages: BundlePage[]; toolkits: BundleToolkit[] };
145 +export interface BundleSearchIndex {
146 + averageLength: number;
147 + documentFrequency: [string, number][];
148 + entries: {
149 + key: string;
150 + length: number;
151 + terms: [string, number][];
152 + }[];
153 +}
154 +const BUNDLE = bundledIndex as {
155 + pages: BundlePage[];
156 + toolkits: BundleToolkit[];
157 + search?: BundleSearchIndex;
158 +};
159
160 /**
161 * Whether the live content tree is on disk. True in dev / local (read fresh
@@ -120,14 +183,14 @@ export function urlFromContentPath(absPath: string): string | undefined {
183 }
184
185 export function isLegacyUrl(url: string): boolean {
123 - return LEGACY_URL_PATTERNS.some((p) => url === p || url.startsWith(`${p}/`));
186 + return LEGACY_URL_PATTERNS.some(p => url === p || url.startsWith(`${p}/`));
187 }
188
189 export function tokenize(query: string): string[] {
190 return query
191 .toLowerCase()
192 .split(/[^a-z0-9_]+/)
130 - .filter((t) => t.length > 1 && !STOPWORDS.has(t));
193 + .filter(t => t.length > 1 && !STOPWORDS.has(t));
194 }
195
196 export function parseFrontmatter(raw: string): {
@@ -143,7 +206,12 @@ export function parseFrontmatter(raw: string): {
206 const m = fm.match(new RegExp(`^${key}:\\s*(.+)$`, 'm'));
207 return m ? m[1].trim().replace(/^["']|["']$/g, '') : '';
208 };
146 - return { title: get('title'), description: get('description'), legacy: get('legacy') === 'true', body };
209 + return {
210 + title: get('title'),
211 + description: get('description'),
212 + legacy: get('legacy') === 'true',
213 + body,
214 + };
215 }
216
217 /** Strip MDX/JSX noise to readable text (keeps prose and inline code words). */
@@ -173,13 +241,24 @@ export function toCleanMarkdown(raw: string): string {
241 * of just the page. Anchor slugs match the docs' heading-id convention.
242 */
243 export function extractSections(markdown: string): { title: string; anchor: string }[] {
176 - return [...markdown.matchAll(/^#{2,4}\s+(.+?)\s*$/gm)].map((m) => {
244 + return [...markdown.matchAll(/^#{2,4}\s+(.+?)\s*$/gm)].map(m => {
245 const title = m[1].replace(/[`*_]/g, '').trim();
178 - const slug = title.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/(^-|-$)/g, '');
246 + const slug = title
247 + .toLowerCase()
248 + .replace(/[^a-z0-9]+/g, '-')
249 + .replace(/(^-|-$)/g, '');
250 return { title, anchor: `#${slug}` };
251 });
252 }
253
254 +export function pageSearchKey(page: Pick<DocPage, 'collection' | 'title' | 'url'>): string {
255 + return `${page.collection}\u0000${page.url}\u0000${page.title}`;
256 +}
257 +
258 +export function getBundleSearchIndex(): BundleSearchIndex | undefined {
259 + return BUNDLE.search;
260 +}
261 +
262 function makePage(args: {
263 collection: Collection;
264 title: string;
@@ -188,8 +267,11 @@ function makePage(args: {
267 legacy: boolean;
268 body: string;
269 }): DocPage {
191 - const headings = (args.body.match(/^#{1,4}\s+(.+)$/gm) ?? []).map((h) =>
192 - h.replace(/^#{1,4}\s+/, '').toLowerCase().trim(),
270 + const headings = (args.body.match(/^#{1,4}\s+(.+)$/gm) ?? []).map(h =>
271 + h
272 + .replace(/^#{1,4}\s+/, '')
273 + .toLowerCase()
274 + .trim()
275 );
276 const text = toPlainText(args.body);
277 return {
@@ -225,7 +307,9 @@ function loadKnowledge(): DocPage[] {
307 const bodyStart = m.index! + m[0].length;
308 const bodyEnd = i + 1 < matches.length ? matches[i + 1].index! : raw.length;
309 const body = raw.slice(bodyStart, bodyEnd).trim();
228 - pages.push(makePage({ collection: 'knowledge', title, description: '', url, legacy: false, body }));
310 + pages.push(
311 + makePage({ collection: 'knowledge', title, description: '', url, legacy: false, body })
312 + );
313 }
314 return pages;
315 }
@@ -251,19 +335,44 @@ function getToolkitMap(): Map<string, Toolkit> {
335 const map = new Map<string, Toolkit>();
336 if (CONTENT_AVAILABLE) {
337 try {
254 - const parsed = JSON.parse(readFileSync(join(APP_ROOT, 'public', 'data', 'toolkits.json'), 'utf8'));
255 - const list: Toolkit[] = Array.isArray(parsed) ? parsed : (parsed.toolkits ?? parsed.items ?? []);
338 + const parsed = JSON.parse(
339 + readFileSync(join(APP_ROOT, 'public', 'data', 'toolkits.json'), 'utf8')
340 + );
341 + const list: Toolkit[] = Array.isArray(parsed)
342 + ? parsed
343 + : (parsed.toolkits ?? parsed.items ?? []);
344 for (const tk of list) if (tk?.slug) map.set(tk.slug, tk);
345 } catch {
346 // no catalog available
347 }
348 }
261 - // Deployed runtime (or missing catalog): use the bundled toolkit snapshot.
262 - if (map.size === 0) for (const tk of BUNDLE.toolkits) if (tk?.slug) map.set(tk.slug, tk);
349 + // Deployed runtime (or partial traced catalog): merge in the bundled toolkit
350 + // snapshot so search/read_doc still cover every generated toolkit page.
351 + for (const tk of BUNDLE.toolkits) if (tk?.slug && !map.has(tk.slug)) map.set(tk.slug, tk);
352 toolkitMap = map;
353 return map;
354 }
355
356 +function loadBundledToolkits(): DocPage[] {
357 + const pages: DocPage[] = [];
358 + for (const tk of BUNDLE.toolkits) {
359 + if (!tk?.slug) continue;
360 + const name = tk.name ?? tk.slug;
361 + const body = `${tk.description ?? ''} Category: ${tk.category ?? ''}. Toolkit slug: ${tk.slug}.`;
362 + pages.push(
363 + makePage({
364 + collection: 'toolkits',
365 + title: `${name} toolkit`,
366 + description: tk.description ?? '',
367 + url: `/toolkits/${tk.slug}`,
368 + legacy: false,
369 + body,
370 + })
371 + );
372 + }
373 + return pages;
374 +}
375 +
376 function loadToolkits(): DocPage[] {
377 const pages: DocPage[] = [];
378 for (const tk of getToolkitMap().values()) {
@@ -277,7 +386,7 @@ function loadToolkits(): DocPage[] {
386 url: `/toolkits/${tk.slug}`,
387 legacy: false,
388 body,
280 - }),
389 + })
390 );
391 }
392 return pages;
@@ -285,6 +394,27 @@ function loadToolkits(): DocPage[] {
394
395 let indexCache: DocPage[] | undefined;
396
397 +export function buildBundledIndex(): DocPage[] {
398 + const pages: DocPage[] = [];
399 + let bundleHasKnowledge = false;
400 + for (const p of BUNDLE.pages) {
401 + if (p.collection === 'knowledge') bundleHasKnowledge = true;
402 + pages.push(
403 + makePage({
404 + collection: p.collection,
405 + title: p.title,
406 + description: p.description,
407 + url: p.url,
408 + legacy: p.legacy || isLegacyUrl(p.url),
409 + body: p.markdown,
410 + })
411 + );
412 + }
413 + if (!bundleHasKnowledge) pages.push(...loadKnowledge());
414 + pages.push(...loadBundledToolkits());
415 + return pages;
416 +}
417 +
418 export function buildIndex(): DocPage[] {
419 if (indexCache) return indexCache;
420 const pages: DocPage[] = [];
@@ -317,24 +447,15 @@ export function buildIndex(): DocPage[] {
447 url,
448 legacy: legacy || isLegacyUrl(url),
449 body,
320 - }),
450 + })
451 );
452 }
453 }
454 } else {
455 // Deployed runtime: content/ isn't on disk, so index the bundled snapshot.
326 - for (const p of BUNDLE.pages) {
327 - pages.push(
328 - makePage({
329 - collection: p.collection,
330 - title: p.title,
331 - description: p.description,
332 - url: p.url,
333 - legacy: p.legacy || isLegacyUrl(p.url),
334 - body: p.markdown,
335 - }),
336 - );
337 - }
456 + pages.push(...buildBundledIndex());
457 + indexCache = pages;
458 + return pages;
459 }
460 pages.push(...loadKnowledge());
461 pages.push(...loadToolkits());
@@ -345,14 +466,15 @@ export function buildIndex(): DocPage[] {
466 /** Resolve a page URL back to its source file and return the raw MDX. */
467 function authSchemesMarkdown(schemes: string[] = []): string {
468 if (!schemes.length) return 'n/a';
348 - return schemes.map((scheme) => `\`${scheme}\``).join(', ');
469 + return schemes.map(scheme => `\`${scheme}\``).join(', ');
470 }
471
472 export function readPageByUrl(url: string): { title: string; raw: string } | undefined {
473 const clean = url.split('#')[0].split('?')[0].replace(/\/$/, '');
474 const parts = clean.split('/').filter(Boolean);
475 const collection = parts.shift();
355 - if (!collection || !COLLECTIONS.includes(collection as (typeof COLLECTIONS)[number])) return undefined;
476 + if (!collection || !COLLECTIONS.includes(collection as (typeof COLLECTIONS)[number]))
477 + return undefined;
478
479 // Toolkit pages are generated from the catalog, not MDX. Synthesize content
480 // so the assistant can confirm support and describe the toolkit.
@@ -377,7 +499,12 @@ export function readPageByUrl(url: string): { title: string; raw: string } | und
499
500 if (CONTENT_AVAILABLE) {
501 const base = join(CONTENT_ROOT, collection, ...parts);
380 - const candidates = [`${base}.mdx`, `${base}.md`, join(base, 'index.mdx'), join(base, 'index.md')];
502 + const candidates = [
503 + `${base}.mdx`,
504 + `${base}.md`,
505 + join(base, 'index.mdx'),
506 + join(base, 'index.md'),
507 + ];
508 for (const candidate of candidates) {
509 try {
510 const raw = readFileSync(candidate, 'utf8');
@@ -387,11 +514,12 @@ export function readPageByUrl(url: string): { title: string; raw: string } | und
514 // try next candidate
515 }
516 }
390 - return undefined;
517 + // Fall through to the bundled snapshot. Vercel may trace only part of the
518 + // content tree, while the generated agent snapshot still has the full docs.
519 }
520
521 // Deployed runtime: serve the page from the bundled snapshot.
394 - const page = BUNDLE.pages.find((p) => p.url === clean);
522 + const page = BUNDLE.pages.find(p => p.url === clean);
523 if (page) return { title: page.title || url, raw: page.markdown };
524 return undefined;
525 }
docs/agent/tools/search_docs.ts
+21 -61
@@ -1,75 +1,35 @@
1 import { defineTool } from 'eve/tools';
2 import { z } from 'zod';
3 -import { buildIndex, tokenize, type DocPage } from '../lib/docs';
3 +import { DEFAULT_SEARCH_LIMIT, searchDocs } from '../lib/docs-search';
4
5 /**
6 * search_docs — find the most relevant Composio docs pages for a query.
7 *
8 - * Returns each match's title, URL, description, and a snippet. Use it to locate
9 - * pages, then call `read_doc` on the best matches to read their full content
10 - * before answering.
8 + * This is a local, in-memory lexical retriever. It uses a BM25-style body score
9 + * plus field boosts for title, description, headings, and URL. The top results
10 + * include full page content (bounded per page), so the model gets rich context
11 + * in the same fast tool call instead of doing a serial search -> read round trip.
12 */
13
13 -// Collection priority: docs first, then examples, then references and toolkits.
14 -// (Curated knowledge ranks with docs.) A toolkit-name query still surfaces its
15 -// toolkit page because nothing else matches it.
16 -const PRIORITY: Record<DocPage['collection'], number> = {
17 - docs: 1.3,
18 - knowledge: 1.3,
19 - examples: 1.1,
20 - reference: 0.85,
21 - toolkits: 0.9,
22 -};
23 -
24 -function score(page: DocPage, terms: string[]): number {
25 - const title = page.title.toLowerCase();
26 - let total = 0;
27 - for (const term of terms) {
28 - if (title.includes(term)) total += 12;
29 - if (page.description.toLowerCase().includes(term)) total += 5;
30 - for (const heading of page.headings) if (heading.includes(term)) total += 4;
31 - if (page.url.toLowerCase().includes(term)) total += 6;
32 - const occurrences = page.lowerText.split(term).length - 1;
33 - total += Math.min(occurrences, 5);
34 - }
35 - // Heavily downrank legacy (direct-execution) pages so they only surface when
36 - // nothing in the session-based docs matches.
37 - if (page.legacy) total *= 0.12;
38 - return total * (PRIORITY[page.collection] ?? 1);
39 -}
40 -
41 -function snippet(page: DocPage, terms: string[]): string {
42 - const lower = page.text.toLowerCase();
43 - const at = terms.map((t) => lower.indexOf(t)).filter((i) => i >= 0).sort((a, b) => a - b)[0] ?? 0;
44 - const start = Math.max(0, at - 100);
45 - const slice = page.text.slice(start, start + 360).trim();
46 - return (start > 0 ? '…' : '') + slice + (start + 360 < page.text.length ? '…' : '');
47 -}
48 -
14 export default defineTool({
15 description:
51 - 'Search the Composio documentation. Returns the most relevant pages with their title, URL, description, and a snippet. Follow up with read_doc on the best matches to read full page content before answering.',
16 + 'Search the Composio documentation with a fast BM25-style local retriever. Returns relevant pages and includes full bounded content for the top matches, so you can answer from the returned context. Call read_doc only if you need a page beyond the included content.',
17 inputSchema: z.object({
53 - query: z.string().min(1).describe('What to look for, e.g. "authentication", "create a session", "trigger webhook verification".'),
54 - limit: z.number().int().min(1).max(12).optional().describe('How many pages to return. Defaults to 8.'),
18 + query: z
19 + .string()
20 + .min(1)
21 + .describe(
22 + 'What to look for, e.g. "authentication", "create a session", "trigger webhook verification".'
23 + ),
24 + limit: z
25 + .number()
26 + .int()
27 + .min(1)
28 + .max(12)
29 + .optional()
30 + .describe('How many pages to return. Defaults to 5.'),
31 }),
56 - async execute({ query, limit = 8 }) {
57 - const terms = tokenize(query);
58 - // Fall back to raw terms if the query was all stopwords.
59 - const effective = terms.length > 0 ? terms : query.toLowerCase().split(/\s+/).filter((t) => t.length > 1);
60 - const ranked = buildIndex()
61 - .map((page) => ({ page, s: score(page, effective) }))
62 - .filter(({ s }) => s > 0)
63 - .sort((a, b) => b.s - a.s)
64 - .slice(0, limit);
65 -
66 - return {
67 - results: ranked.map(({ page }) => ({
68 - title: page.title,
69 - url: page.url,
70 - description: page.description,
71 - snippet: snippet(page, effective),
72 - })),
73 - };
32 + async execute({ query, limit = DEFAULT_SEARCH_LIMIT }) {
33 + return searchDocs(query, { limit, invocation: 'tool' });
34 },
35 });
docs/app/api/docs-agent/eager-search/route.ts new
+34
@@ -0,0 +1,34 @@
1 +import { NextRequest, NextResponse } from 'next/server';
2 +import { searchDocs, shouldRunEagerDocsSearch } from '@/agent/lib/docs-search';
3 +
4 +const EAGER_PREVIEW_LIMIT = 3;
5 +
6 +export async function POST(request: NextRequest) {
7 + try {
8 + const body = (await request.json()) as { message?: unknown };
9 + const message = typeof body.message === 'string' ? body.message.trim() : '';
10 +
11 + if (message.length < 3) {
12 + return NextResponse.json({ sources: [] });
13 + }
14 +
15 + if (!shouldRunEagerDocsSearch(message)) {
16 + return NextResponse.json({ sources: [] });
17 + }
18 +
19 + const result = searchDocs(message, {
20 + limit: EAGER_PREVIEW_LIMIT,
21 + hydrateContent: false,
22 + invocation: 'eager_preview',
23 + });
24 +
25 + return NextResponse.json({
26 + sources: result.results.map(({ title, url }) => ({ title, url })),
27 + });
28 + } catch (error) {
29 + console.warn('[docs-agent:eager_preview] failed', {
30 + error: error instanceof Error ? error.message : String(error),
31 + });
32 + return NextResponse.json({ sources: [] }, { status: 500 });
33 + }
34 +}
docs/bun.lock
+1 -1
@@ -5,6 +5,7 @@
5 "": {
6 "name": "fumadocs",
7 "dependencies": {
8 + "@ai-sdk/openai": "^3.0.9",
9 "@pierre/diffs": "^1.2.11",
10 "@pierre/trees": "^1.0.0-beta.4",
11 "@radix-ui/react-tabs": "^1.1.13",
@@ -41,7 +42,6 @@
42 "devDependencies": {
43 "@ai-sdk/anthropic": "^3.0.11",
44 "@ai-sdk/mcp": "^1.0.6",
44 - "@ai-sdk/openai": "^3.0.9",
45 "@ai-sdk/react": "^3.0.102",
46 "@anthropic-ai/claude-agent-sdk": "^0.2.5",
47 "@anthropic-ai/sdk": "^0.71.2",
docs/components/eve-chat.tsx
+42 -6
@@ -1,10 +1,10 @@
1 'use client';
2
3 -import { useEffect, useRef } from 'react';
3 +import { useEffect, useRef, useState } from 'react';
4 import { usePathname } from 'next/navigation';
5 import { useEveAgent } from 'eve/react';
6 import { Send, X, Sparkles, Square, SquarePen } from 'lucide-react';
7 -import { AssistantMessage, ToolActivity } from './eve-message';
7 +import { AssistantMessage, EagerSourcePreview, ToolActivity, type EagerSource } from './eve-message';
8 import { closeEveChat, useEveChatOpen } from './eve-chat-store';
9
10 const SUGGESTIONS = [
@@ -32,11 +32,13 @@ export function EveChat() {
32 const lastHasAssistantText =
33 lastMessage?.role === 'assistant' &&
34 lastMessage.parts.some((part) => part.type === 'text' && part.text.trim().length > 0);
35 - // Show the loading indicator from submit through the search/read phase, until
36 - // the assistant's text actually starts streaming, so it doesn't flicker off.
35 + // Show the loading indicator from submit through retrieval/model synthesis,
36 + // until the assistant's text actually starts streaming, so it doesn't flicker off.
37 const thinking = isBusy && !lastHasAssistantText;
38 const inputRef = useRef<HTMLTextAreaElement>(null);
39 const scrollRef = useRef<HTMLDivElement>(null);
40 + const previewAbortRef = useRef<AbortController | null>(null);
41 + const [eagerSources, setEagerSources] = useState<EagerSource[]>([]);
42
43 useEffect(() => {
44 if (isOpen) inputRef.current?.focus();
@@ -46,9 +48,37 @@ export function EveChat() {
48 scrollRef.current?.scrollTo({ top: scrollRef.current.scrollHeight });
49 }, [agent.data.messages]);
50
51 + function clearEagerPreview() {
52 + previewAbortRef.current?.abort();
53 + previewAbortRef.current = null;
54 + setEagerSources([]);
55 + }
56 +
57 + function fetchEagerPreview(message: string) {
58 + clearEagerPreview();
59 + const controller = new AbortController();
60 + previewAbortRef.current = controller;
61 +
62 + void fetch('/api/docs-agent/eager-search', {
63 + method: 'POST',
64 + headers: { 'Content-Type': 'application/json' },
65 + body: JSON.stringify({ message }),
66 + signal: controller.signal,
67 + })
68 + .then((response) => (response.ok ? response.json() : null))
69 + .then((data: { sources?: EagerSource[] } | null) => {
70 + if (controller.signal.aborted) return;
71 + setEagerSources(Array.isArray(data?.sources) ? data.sources : []);
72 + })
73 + .catch(() => {});
74 + }
75 +
76 function submit(message: string) {
77 const trimmed = message.trim();
51 - if (trimmed.length > 0 && !isBusy) void agent.send({ message: trimmed });
78 + if (trimmed.length > 0 && !isBusy) {
79 + fetchEagerPreview(trimmed);
80 + void agent.send({ message: trimmed });
81 + }
82 }
83
84 return (
@@ -81,6 +111,7 @@ export function EveChat() {
111 title="New chat"
112 onClick={() => {
113 if (isBusy) agent.stop();
114 + clearEagerPreview();
115 agent.reset();
116 inputRef.current?.focus();
117 }}
@@ -148,6 +179,11 @@ export function EveChat() {
179 </div>
180 </li>
181 ))}
182 + {thinking && eagerSources.length > 0 && (
183 + <li>
184 + <EagerSourcePreview active sources={eagerSources} />
185 + </li>
186 + )}
187 {thinking && (
188 <li className="flex items-center gap-2 text-[13px] text-fd-muted-foreground">
189 <span className="inline-flex gap-1">
@@ -155,7 +191,7 @@ export function EveChat() {
191 <span className="size-1.5 animate-bounce rounded-full bg-fd-muted-foreground/60 [animation-delay:-0.15s]" />
192 <span className="size-1.5 animate-bounce rounded-full bg-fd-muted-foreground/60" />
193 </span>
158 - Searching the docs…
194 + Thinking with the docs…
195 </li>
196 )}
197 </ul>
docs/components/eve-message.tsx
+52
@@ -7,6 +7,58 @@ import remarkGfm from 'remark-gfm';
7 import hljs from 'highlight.js';
8 import { Check, Copy, FileText, Search, Wrench } from 'lucide-react';
9
10 +export type EagerSource = {
11 + title: string;
12 + url: string;
13 +};
14 +
15 +/**
16 + * Renders eager docs search preview links while the assistant is thinking.
17 + */
18 +export function EagerSourcePreview({ active, sources }: { active?: boolean; sources: EagerSource[] }) {
19 + const router = useRouter();
20 + if (sources.length === 0) return null;
21 +
22 + const base =
23 + 'flex flex-wrap items-center gap-x-2 gap-y-1 py-0.5 text-[11px] text-fd-muted-foreground';
24 +
25 + return (
26 + <div className={base}>
27 + <span className="inline-flex shrink-0 items-center gap-1 text-fd-muted-foreground">
28 + <Search className="size-3 text-[var(--composio-brand)]/70" aria-hidden="true" />
29 + <span>{active ? 'Searching docs' : 'Searched docs'}</span>
30 + </span>
31 + {sources.map((source) => {
32 + const href = source.url.startsWith('/') ? source.url : undefined;
33 + const label = source.title || source.url;
34 + if (!href) {
35 + return (
36 + <span key={source.url} className="truncate">
37 + {label}
38 + </span>
39 + );
40 + }
41 +
42 + return (
43 + <a
44 + key={source.url}
45 + href={href}
46 + title={href}
47 + onClick={(event) => {
48 + if (event.metaKey || event.ctrlKey || event.shiftKey || event.altKey || event.button !== 0) return;
49 + event.preventDefault();
50 + router.push(href);
51 + }}
52 + className="group inline-flex max-w-full items-center truncate underline-offset-2 transition-colors hover:text-fd-foreground hover:underline"
53 + >
54 + {label}
55 + </a>
56 + );
57 + })}
58 + </div>
59 + );
60 +}
61 +
62 /**
63 * Renders the agent's tool activity inline — which docs it searched and which
64 * pages it read — so you can see its "thinking". Returns null for non-tool parts.
docs/evals/docs-agent/grounded-answers.eval.ts new
+47
@@ -0,0 +1,47 @@
1 +import { defineEval } from 'eve/evals';
2 +
3 +const cases = [
4 + {
5 + description: 'Explains session creation with current TypeScript APIs.',
6 + prompt:
7 + 'Search the Composio docs, then explain in TypeScript how I create a Composio session for user `alice` with GitHub tools. Include the relevant docs link.',
8 + expectedSignals: [/composio\.(create|sessions\.create)/i, /github/i],
9 + },
10 + {
11 + description: 'Distinguishes auth configs from connected accounts.',
12 + prompt:
13 + 'Search the Composio docs, then explain the difference between an auth config and a connected account in Composio. Keep it short and cite the docs.',
14 + expectedSignals: [/auth config/i, /connected account/i],
15 + },
16 + {
17 + description: 'Explains the session-based MCP setup path.',
18 + prompt:
19 + 'Search the Composio docs, then explain how I expose Composio session tools over MCP. Include the recommended `mcp: true` shape and cite the docs.',
20 + expectedSignals: [/mcp/i, /mcp\s*:\s*true/i],
21 + },
22 +] as const;
23 +
24 +export default cases.map(row =>
25 + defineEval({
26 + description: row.description,
27 + tags: ['docs-agent', 'model-flow', 'grounded-answer'],
28 + metadata: {
29 + suite: 'docs-agent-model-flow',
30 + prompt: row.prompt,
31 + },
32 + async test(t) {
33 + await t.send(row.prompt);
34 +
35 + t.completed();
36 + t.noFailedActions();
37 + // The docs channel now injects eager search context before the model step,
38 + // so a good answer may not need an explicit search_docs tool call.
39 + t.maxToolCalls(3);
40 + t.messageIncludes(/\]\(\/(?:docs|reference)\//);
41 +
42 + for (const signal of row.expectedSignals) {
43 + t.messageIncludes(signal);
44 + }
45 + },
46 + })
47 +);
docs/evals/docs-agent/out-of-scope.eval.ts new
+17
@@ -0,0 +1,17 @@
1 +import { defineEval } from 'eve/evals';
2 +
3 +export default defineEval({
4 + description: 'Refuses account-specific support requests without using docs tools.',
5 + tags: ['docs-agent', 'model-flow', 'policy'],
6 + metadata: {
7 + suite: 'docs-agent-model-flow',
8 + },
9 + async test(t) {
10 + await t.send('Can you check whether my latest Composio invoice has been paid?');
11 +
12 + t.completed();
13 + t.noFailedActions();
14 + t.usedNoTools();
15 + t.messageIncludes(/support|dashboard|billing|account/i);
16 + },
17 +});
docs/evals/evals.config.ts new
+6
@@ -0,0 +1,6 @@
1 +import { defineEvalConfig } from 'eve/evals';
2 +
3 +export default defineEvalConfig({
4 + maxConcurrency: 2,
5 + timeoutMs: 90_000,
6 +});
docs/package.json
+3 -1
@@ -18,6 +18,8 @@
18 "lint:links": "bun scripts/validate-links.ts",
19 "sync:search": "bun scripts/sync-algolia-search.ts",
20 "test:search": "bun scripts/test-algolia-search.ts",
21 + "eval:agent": "eve eval docs-agent --skip-report",
22 + "eval:agent:flows": "bun scripts/eval-agent-flows.ts",
23 "generate:toolkits": "bun scripts/generate-toolkits.ts",
24 "generate:meta-tools": "bun scripts/generate-meta-tools.ts",
25 "generate:api-index": "bun scripts/generate-api-index.ts",
@@ -25,6 +27,7 @@
27 "test:integration": "bun test tests/integration/ --timeout 30000"
28 },
29 "dependencies": {
30 + "@ai-sdk/openai": "^3.0.9",
31 "@pierre/diffs": "^1.2.11",
32 "@pierre/trees": "^1.0.0-beta.4",
33 "@radix-ui/react-tabs": "^1.1.13",
@@ -61,7 +64,6 @@
64 "devDependencies": {
65 "@ai-sdk/anthropic": "^3.0.11",
66 "@ai-sdk/mcp": "^1.0.6",
64 - "@ai-sdk/openai": "^3.0.9",
67 "@ai-sdk/react": "^3.0.102",
68 "@anthropic-ai/claude-agent-sdk": "^0.2.5",
69 "@anthropic-ai/sdk": "^0.71.2",
docs/scripts/build-agent-index.ts
+254 -18
@@ -5,8 +5,12 @@
5 * `public/data/`, so the agent can't read the docs from disk at runtime. This
6 * script snapshots every content page (cleaned Markdown + metadata) and the
7 * toolkit catalog into a generated module the agent imports, so eve bundles it
8 - * into the deployed service. In dev the agent still reads `content/` live; the
9 - * bundle is only used when the content tree isn't on disk.
8 + * into the deployed service. It also precomputes BM25-style lexical term stats
9 + * for the docs assistant search tool, avoiding cold-start corpus construction in
10 + * the deployed function.
11 + *
12 + * In dev the agent still reads `content/` live when the generated search snapshot
13 + * does not match; the bundle is always used when the content tree isn't on disk.
14 *
15 * Self-contained on purpose: it does NOT import `agent/lib/docs.ts` (which
16 * imports the generated file), so it can run from a clean checkout where
@@ -15,22 +19,80 @@
19 *
20 * Run manually with `bun scripts/build-agent-index.ts`.
21 */
18 -import { readdirSync, readFileSync, writeFileSync } from 'node:fs';
22 +import { existsSync, readdirSync, readFileSync, writeFileSync } from 'node:fs';
23 import { join, relative } from 'node:path';
24
25 const ROOT = process.cwd();
26 const CONTENT = join(ROOT, 'content');
27 +const KNOWLEDGE_FILE = join(ROOT, 'agent', 'knowledge.md');
28 const COLLECTIONS = ['docs', 'reference', 'examples', 'toolkits'] as const;
29 const LEGACY_URL_PATTERNS = [
30 '/docs/tools-direct',
31 '/docs/auth-configuration',
32 '/docs/sessions-vs-direct-execution',
33 ];
34 +const STOPWORDS = new Set([
35 + 'the',
36 + 'a',
37 + 'an',
38 + 'and',
39 + 'or',
40 + 'but',
41 + 'is',
42 + 'are',
43 + 'was',
44 + 'were',
45 + 'be',
46 + 'been',
47 + 'to',
48 + 'of',
49 + 'in',
50 + 'on',
51 + 'for',
52 + 'with',
53 + 'as',
54 + 'at',
55 + 'by',
56 + 'from',
57 + 'how',
58 + 'do',
59 + 'does',
60 + 'did',
61 + 'what',
62 + 'why',
63 + 'when',
64 + 'which',
65 + 'who',
66 + 'can',
67 + 'i',
68 + 'you',
69 + 'it',
70 + 'this',
71 + 'that',
72 + 'these',
73 + 'those',
74 + 'my',
75 + 'your',
76 + 'we',
77 + 'they',
78 + 'use',
79 + 'using',
80 + 'work',
81 + 'works',
82 + 'about',
83 + 'into',
84 + 'so',
85 + 'if',
86 + 'me',
87 + 'get',
88 + 'set',
89 + 'up',
90 +]);
91
30 -// The helpers below mirror agent/lib/docs.ts so generated URLs and cleaned
31 -// Markdown match what the live (filesystem) path produces. Keep them in sync.
92 +// The helpers below mirror agent/lib/docs.ts so generated URLs, cleaned
93 +// Markdown, page keys, and BM25 token stats match what the runtime path uses.
94 function isLegacyUrl(url: string): boolean {
33 - return LEGACY_URL_PATTERNS.some((p) => url === p || url.startsWith(`${p}/`));
95 + return LEGACY_URL_PATTERNS.some(p => url === p || url.startsWith(`${p}/`));
96 }
97
98 function urlFromContentPath(absPath: string): string | undefined {
@@ -49,7 +111,19 @@ function urlFromContentPath(absPath: string): string | undefined {
111 return undefined;
112 }
113
52 -function parseFrontmatter(raw: string): { title: string; description: string; legacy: boolean; body: string } {
114 +function tokenize(query: string): string[] {
115 + return query
116 + .toLowerCase()
117 + .split(/[^a-z0-9_]+/)
118 + .filter(t => t.length > 1 && !STOPWORDS.has(t));
119 +}
120 +
121 +function parseFrontmatter(raw: string): {
122 + title: string;
123 + description: string;
124 + legacy: boolean;
125 + body: string;
126 +} {
127 const match = raw.match(/^---\n([\s\S]*?)\n---\n?([\s\S]*)$/);
128 if (!match) return { title: '', description: '', legacy: false, body: raw };
129 const [, fm, body] = match;
@@ -57,7 +131,23 @@ function parseFrontmatter(raw: string): { title: string; description: string; le
131 const m = fm.match(new RegExp(`^${key}:\\s*(.+)$`, 'm'));
132 return m ? m[1].trim().replace(/^["']|["']$/g, '') : '';
133 };
60 - return { title: get('title'), description: get('description'), legacy: get('legacy') === 'true', body };
134 + return {
135 + title: get('title'),
136 + description: get('description'),
137 + legacy: get('legacy') === 'true',
138 + body,
139 + };
140 +}
141 +
142 +function toPlainText(body: string): string {
143 + return body
144 + .replace(/```[\s\S]*?```/g, ' ')
145 + .replace(/<\/?[A-Za-z][^>]*>/g, ' ')
146 + .replace(/!\[[^\]]*\]\([^)]*\)/g, ' ')
147 + .replace(/\[([^\]]*)\]\([^)]*\)/g, '$1')
148 + .replace(/[#>*_`|]/g, ' ')
149 + .replace(/\s+/g, ' ')
150 + .trim();
151 }
152
153 function toCleanMarkdown(raw: string): string {
@@ -77,6 +167,131 @@ interface BundlePage {
167 markdown: string;
168 }
169
170 +interface BundleToolkit {
171 + slug: string;
172 + name?: string;
173 + description?: string;
174 + category?: string;
175 + authSchemes?: string[];
176 +}
177 +
178 +interface SearchPage {
179 + collection: string;
180 + title: string;
181 + description: string;
182 + url: string;
183 + headings: string[];
184 + lowerText: string;
185 +}
186 +
187 +function pageSearchKey(page: Pick<SearchPage, 'collection' | 'title' | 'url'>): string {
188 + return `${page.collection}\u0000${page.url}\u0000${page.title}`;
189 +}
190 +
191 +function makeSearchPage(args: {
192 + collection: string;
193 + title: string;
194 + description: string;
195 + url: string;
196 + body: string;
197 +}): SearchPage {
198 + const headings = (args.body.match(/^#{1,4}\s+(.+)$/gm) ?? []).map(h =>
199 + h
200 + .replace(/^#{1,4}\s+/, '')
201 + .toLowerCase()
202 + .trim()
203 + );
204 + const text = toPlainText(args.body);
205 +
206 + return {
207 + collection: args.collection,
208 + title: args.title || args.url,
209 + description: args.description,
210 + url: args.url,
211 + headings,
212 + lowerText: `${args.title} ${args.description} ${text}`.toLowerCase(),
213 + };
214 +}
215 +
216 +function loadKnowledgePages(): BundlePage[] {
217 + if (!existsSync(KNOWLEDGE_FILE)) return [];
218 +
219 + const raw = readFileSync(KNOWLEDGE_FILE, 'utf8');
220 + const pages: BundlePage[] = [];
221 + const sectionRe = /^##\s+(.+?)\s*(?:\(([^)]+)\))?\s*$/gm;
222 + const matches = [...raw.matchAll(sectionRe)];
223 +
224 + for (let index = 0; index < matches.length; index++) {
225 + const match = matches[index];
226 + const title = match[1].trim();
227 + const url = (match[2] ?? '').trim() || '/docs';
228 + const bodyStart = match.index! + match[0].length;
229 + const bodyEnd = index + 1 < matches.length ? matches[index + 1].index! : raw.length;
230 + const body = raw.slice(bodyStart, bodyEnd).trim();
231 +
232 + pages.push({
233 + collection: 'knowledge',
234 + url,
235 + title,
236 + description: '',
237 + legacy: false,
238 + markdown: body,
239 + });
240 + }
241 +
242 + return pages;
243 +}
244 +
245 +function searchPageForToolkit(tk: BundleToolkit): SearchPage {
246 + const name = tk.name ?? tk.slug;
247 + const body = `${tk.description ?? ''} Category: ${tk.category ?? ''}. Toolkit slug: ${tk.slug}.`;
248 +
249 + return makeSearchPage({
250 + collection: 'toolkits',
251 + title: `${name} toolkit`,
252 + description: tk.description ?? '',
253 + url: `/toolkits/${tk.slug}`,
254 + body,
255 + });
256 +}
257 +
258 +function termCountsFor(page: SearchPage): Map<string, number> {
259 + const counts = new Map<string, number>();
260 +
261 + for (const token of tokenize(page.lowerText)) {
262 + counts.set(token, (counts.get(token) ?? 0) + 1);
263 + }
264 +
265 + return counts;
266 +}
267 +
268 +function buildSearchIndex(searchPages: SearchPage[]) {
269 + const entries = searchPages.map(page => {
270 + const counts = termCountsFor(page);
271 + const terms = [...counts.entries()];
272 +
273 + return {
274 + key: pageSearchKey(page),
275 + terms,
276 + length: terms.reduce((sum, [, count]) => sum + count, 0),
277 + };
278 + });
279 + const documentFrequency = new Map<string, number>();
280 +
281 + for (const entry of entries) {
282 + for (const [term] of entry.terms) {
283 + documentFrequency.set(term, (documentFrequency.get(term) ?? 0) + 1);
284 + }
285 + }
286 +
287 + return {
288 + averageLength:
289 + entries.reduce((sum, entry) => sum + entry.length, 0) / Math.max(entries.length, 1),
290 + documentFrequency: [...documentFrequency.entries()],
291 + entries,
292 + };
293 +}
294 +
295 const pages: BundlePage[] = [];
296 for (const collection of COLLECTIONS) {
297 const dir = join(CONTENT, collection);
@@ -104,14 +319,6 @@ for (const collection of COLLECTIONS) {
319 }
320 }
321
107 -interface BundleToolkit {
108 - slug: string;
109 - name?: string;
110 - description?: string;
111 - category?: string;
112 - authSchemes?: string[];
113 -}
114 -
322 let toolkits: BundleToolkit[] = [];
323 try {
324 const parsed = JSON.parse(readFileSync(join(ROOT, 'public', 'data', 'toolkits.json'), 'utf8'));
@@ -129,11 +336,38 @@ try {
336 // no catalog available
337 }
338
339 +const contentSearchPages = pages.map(page =>
340 + makeSearchPage({
341 + collection: page.collection,
342 + title: page.title,
343 + description: page.description,
344 + url: page.url,
345 + body: page.markdown,
346 + })
347 +);
348 +const knowledgePages = loadKnowledgePages();
349 +pages.push(...knowledgePages);
350 +const knowledgeSearchPages = knowledgePages.map(page =>
351 + makeSearchPage({
352 + collection: page.collection,
353 + title: page.title,
354 + description: page.description,
355 + url: page.url,
356 + body: page.markdown,
357 + })
358 +);
359 +const toolkitSearchPages = toolkits.map(searchPageForToolkit);
360 +const search = buildSearchIndex([
361 + ...contentSearchPages,
362 + ...knowledgeSearchPages,
363 + ...toolkitSearchPages,
364 +]);
365 +
366 // Emit a .ts module (not .json): eve's discovery scans agent/lib/ and only
367 // accepts authored modules there, so a raw .json fails `eve build`. We embed the
368 // snapshot as a JSON string parsed at runtime — a plain string literal keeps the
369 // source fast for tsc and free of escaping pitfalls, and lets eve bundle it.
136 -const json = JSON.stringify({ pages, toolkits });
370 +const json = JSON.stringify({ pages, toolkits, search });
371 const ts =
372 `/* eslint-disable */\n` +
373 `// Auto-generated by scripts/build-agent-index.ts — do not edit by hand.\n` +
@@ -142,4 +376,6 @@ const ts =
376 `export default JSON.parse(\n ${JSON.stringify(json)},\n) as unknown;\n`;
377 const out = join(ROOT, 'agent', 'lib', 'docs-index.ts');
378 writeFileSync(out, ts);
145 -console.log(`[build-agent-index] wrote ${pages.length} pages + ${toolkits.length} toolkits (${Math.round(json.length / 1024)} KB) -> ${out}`);
379 +console.log(
380 + `[build-agent-index] wrote ${pages.length} pages + ${toolkits.length} toolkits + ${search.entries.length} BM25 rows (${Math.round(json.length / 1024)} KB) -> ${out}`
381 +);
docs/scripts/eval-agent-flows.ts new
+110
@@ -0,0 +1,110 @@
1 +#!/usr/bin/env bun
2 +
3 +import { spawnSync } from 'node:child_process';
4 +
5 +const DEFAULT_LOCAL_FLOWS = ['gateway', 'mercury'] as const;
6 +const EVE_BIN = process.env.EVE_BIN ?? './node_modules/.bin/eve';
7 +const extraArgs = process.argv.slice(2);
8 +
9 +type EvalRun = {
10 + name: string;
11 + env?: Record<string, string>;
12 + url?: string;
13 +};
14 +
15 +const parseList = (value: string | undefined): string[] =>
16 + value
17 + ?.split(',')
18 + .map(item => item.trim())
19 + .filter(Boolean) ?? [];
20 +
21 +const parseRemoteTargets = (value: string | undefined): EvalRun[] =>
22 + parseList(value).map(entry => {
23 + const separator = entry.indexOf('=');
24 + if (separator === -1) {
25 + throw new Error(
26 + `Invalid DOCS_AGENT_EVAL_TARGETS entry "${entry}". Use name=https://deployment.example.`
27 + );
28 + }
29 +
30 + const name = entry.slice(0, separator).trim();
31 + const url = entry.slice(separator + 1).trim();
32 +
33 + if (!name || !url) {
34 + throw new Error(
35 + `Invalid DOCS_AGENT_EVAL_TARGETS entry "${entry}". Both name and URL are required.`
36 + );
37 + }
38 +
39 + return { name, url };
40 + });
41 +
42 +const buildRuns = (): EvalRun[] => {
43 + const remoteTargets = parseRemoteTargets(process.env.DOCS_AGENT_EVAL_TARGETS);
44 +
45 + if (remoteTargets.length > 0) {
46 + return remoteTargets;
47 + }
48 +
49 + const flows = parseList(process.env.DOCS_AGENT_EVAL_FLOWS);
50 + const selectedFlows = flows.length > 0 ? flows : [...DEFAULT_LOCAL_FLOWS];
51 +
52 + return selectedFlows.map(flow => ({
53 + name: flow,
54 + env: { DOCS_AGENT_MODEL_FLOW: flow },
55 + }));
56 +};
57 +
58 +const warnForMissingCredentials = (run: EvalRun) => {
59 + const flow = run.env?.DOCS_AGENT_MODEL_FLOW;
60 +
61 + if (flow === 'mercury' && !process.env.INCEPTION_API_KEY) {
62 + console.warn(
63 + '[eval-agent-flows] INCEPTION_API_KEY is not set; Mercury evals will fail or skip model calls.'
64 + );
65 + }
66 +
67 + if (flow === 'gateway' && !process.env.AI_GATEWAY_API_KEY && !process.env.VERCEL_OIDC_TOKEN) {
68 + console.warn(
69 + '[eval-agent-flows] AI_GATEWAY_API_KEY/VERCEL_OIDC_TOKEN is not set; gateway evals will fail or skip model calls.'
70 + );
71 + }
72 +};
73 +
74 +const runEval = (run: EvalRun) => {
75 + const args = ['eval', 'docs-agent', '--skip-report'];
76 +
77 + if (run.url) {
78 + args.push('--url', run.url);
79 + }
80 +
81 + args.push(...extraArgs);
82 +
83 + console.log(`\n## ${run.url ? 'Remote target' : 'Local model flow'}: ${run.name}`);
84 + console.log(`$ ${EVE_BIN} ${args.join(' ')}`);
85 + warnForMissingCredentials(run);
86 +
87 + return (
88 + spawnSync(EVE_BIN, args, {
89 + env: { ...process.env, ...run.env },
90 + stdio: 'inherit',
91 + }).status ?? 1
92 + );
93 +};
94 +
95 +const runs = buildRuns();
96 +let failed = 0;
97 +
98 +for (const run of runs) {
99 + const status = runEval(run);
100 + if (status !== 0) {
101 + failed += 1;
102 + }
103 +}
104 +
105 +if (failed > 0) {
106 + console.error(`\n${failed}/${runs.length} docs-agent eval run(s) failed.`);
107 + process.exit(1);
108 +}
109 +
110 +console.log(`\nAll ${runs.length} docs-agent eval run(s) passed.`);