@devries-cameron20 / composio-1 / commits / 18c3b98de
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

docs: reapply Algolia docs search (#3454)

## Summary - Re-apply the Algolia docs search migration after the previous PR was merged and reverted. - Keep the existing Fumadocs search UI while using Algolia API when search keys are configured, with `/api/search` fallback for local development and tests. - Add a first-party Algolia index builder/sync script that creates section-sized docs records from MDX/OpenAPI/toolkit/changelog content, configures index relevance settings, and replaces index objects without relying on Algolia Crawler. - Add Algolia Insights view/click events and a terminal search relevance test script. - Clean search breadcrumbs so results show labels like `Toolkit` and `Cookbook` instead of duplicated `toolkits > Gmail` formatting. ## Tests - `cd docs && bun run types:check` - `cd docs && bun run sync:search --dry-run` - `cd docs && bunx eslint components/custom-search-dialog.tsx lib/search-index.ts scripts/sync-algolia-search.ts scripts/test-algolia-search.ts` ## Notes - Live Algolia sync requires `ALGOLIA_ADMIN_API_KEY`. - Live search relevance tests require `ALGOLIA_SEARCH_API_KEY` or `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY`.

Rahul Tarak committed May 19, 2026 at 02:27 UTC 18c3b98de69084477a4121297d7ed24a02660f68
10 files changed +1094 -94
.github/workflows/docs-search-sync.yml new
+46
@@ -0,0 +1,46 @@
1 +name: Docs - Sync Algolia Search
2 +
3 +on:
4 + push:
5 + branches:
6 + - next
7 + paths:
8 + - 'docs/**'
9 + workflow_dispatch:
10 +
11 +permissions:
12 + contents: read
13 +
14 +jobs:
15 + sync-search:
16 + runs-on: ubuntu-latest
17 + defaults:
18 + run:
19 + working-directory: ./docs
20 + env:
21 + ALGOLIA_APP_ID: ${{ vars.ALGOLIA_APP_ID || '62HI9PQZ1L' }}
22 + ALGOLIA_ADMIN_API_KEY: ${{ secrets.ALGOLIA_ADMIN_API_KEY }}
23 + ALGOLIA_INDEX_NAME: ${{ vars.ALGOLIA_INDEX_NAME || 'docs_composio_dev_62hi9pqz1l_pages' }}
24 +
25 + steps:
26 + - name: Checkout repository
27 + uses: actions/checkout@v6
28 +
29 + - name: Setup Bun
30 + uses: oven-sh/setup-bun@v2
31 + with:
32 + bun-version-file: '.bun-version'
33 +
34 + - name: Install dependencies
35 + run: bun install
36 +
37 + - name: Validate search index generation
38 + run: bun run sync:search --dry-run
39 +
40 + - name: Sync Algolia search index
41 + if: env.ALGOLIA_APP_ID != '' && env.ALGOLIA_ADMIN_API_KEY != ''
42 + run: bun run sync:search
43 +
44 + - name: Skip sync without Algolia secrets
45 + if: env.ALGOLIA_APP_ID == '' || env.ALGOLIA_ADMIN_API_KEY == ''
46 + run: echo "Skipping Algolia sync because ALGOLIA_APP_ID or ALGOLIA_ADMIN_API_KEY is not configured."
docs/CLAUDE.md
+8
@@ -9,6 +9,8 @@ bun install # Install dependencies
9 bun run dev # Dev server (http://localhost:3000)
10 bun run build # Production build (validates TS code blocks)
11 bun run types:check # Type check
12 +bun run sync:search # Build and sync Algolia docs search index (requires Algolia env vars)
13 +bun run test:search # Test live Algolia search relevance from the terminal
14 ```
15
16 ## Project Structure
@@ -83,6 +85,12 @@ Detailed documentation for Claude is organized in `.claude/`:
85
86 `content/docs/glossary.mdx` defines key Composio terms (auth config, session, toolkit, etc.) using `<Glossary>` and `<GlossaryTerm name="...">` components (`components/glossary.tsx`). The component renders a filterable two-column table. The markdown converter in `lib/source.ts` converts `<GlossaryTerm>` tags to `### Term` headings for LLM-friendly output. When adding new Composio concepts, add a `<GlossaryTerm>` entry and update the `keywords` frontmatter array.
87
88 +## Search
89 +
90 +The docs search dialog uses Algolia in production when `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY` is configured. `NEXT_PUBLIC_ALGOLIA_APP_ID` defaults to `62HI9PQZ1L`, and `NEXT_PUBLIC_ALGOLIA_INDEX_NAME` defaults to `docs_composio_dev_62hi9pqz1l_pages`. Without a search API key it falls back to `/api/search` for local development and tests. Algolia searches request `clickAnalytics`, and the client sends search result view/click events with `search-insights`.
91 +
92 +Run `ALGOLIA_APP_ID=62HI9PQZ1L ALGOLIA_ADMIN_API_KEY=... ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages bun run sync:search` from `docs/` to rebuild the Algolia index. The sync script does not use Algolia Crawler: it creates section-sized records from MDX/OpenAPI/toolkit data, configures searchable attributes, `customRanking`, and `attributeForDistinct`, then replaces the index objects. Use `bun run sync:search --dry-run --samples` to inspect generated records and `ALGOLIA_SEARCH_API_KEY=... bun run test:search "oauth auth config"` to test live relevance.
93 +
94 ## API Versioning
95
96 The API reference supports v3.1 (latest, default) and v3.0 side-by-side:
docs/README.md
+27
@@ -74,6 +74,31 @@ All TypeScript code blocks in MDX files are **type-checked at build time** using
74
75 See `CLAUDE.md` for detailed patterns and troubleshooting.
76
77 +## Search
78 +
79 +Docs search uses Algolia when `NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY` is set; otherwise it falls back to the local Fumadocs `/api/search` endpoint for development and tests. The sync script builds a first-party index from MDX/OpenAPI/toolkit data (no crawler required), splits long pages into section-sized records, configures searchable attributes/custom ranking/distinct, requests `clickAnalytics`, and sends search result view/click events with `search-insights`.
80 +
81 +```bash
82 +NEXT_PUBLIC_ALGOLIA_APP_ID=62HI9PQZ1L # optional; default shown
83 +NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY=...
84 +NEXT_PUBLIC_ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages # optional; default shown
85 +```
86 +
87 +Sync the Algolia index with an admin key:
88 +
89 +```bash
90 +ALGOLIA_APP_ID=62HI9PQZ1L
91 +ALGOLIA_ADMIN_API_KEY=...
92 +ALGOLIA_INDEX_NAME=docs_composio_dev_62hi9pqz1l_pages bun run sync:search
93 +```
94 +
95 +Preview the generated records or test live relevance:
96 +
97 +```bash
98 +bun run sync:search --dry-run --samples
99 +ALGOLIA_SEARCH_API_KEY=... bun run test:search "oauth auth config" "gmail send email"
100 +```
101 +
102 ## Commands
103
104 | Command | Description |
@@ -81,4 +106,6 @@ See `CLAUDE.md` for detailed patterns and troubleshooting.
106 | `bun run dev` | Dev server |
107 | `bun run build` | Production build (validates TS code blocks) |
108 | `bun run types:check` | Type check |
109 +| `bun run sync:search` | Sync docs search records to Algolia |
110 +| `bun run test:search` | Query the configured Algolia index from the terminal |
111
docs/app/api/search/route.ts
+4 -85
@@ -1,89 +1,8 @@
1 -// Use direct imports from collections to avoid top-level await in lib/source.ts
2 -import { docs, reference, cookbooks, toolkits, changelog } from 'fumadocs-mdx:collections/server';
1 import { createSearchAPI } from 'fumadocs-core/search/server';
4 -import { loader, multiple } from 'fumadocs-core/source';
5 -import { lucideIconsPlugin } from 'fumadocs-core/source/lucide-icons';
6 -import { openapiSource, openapiPlugin } from 'fumadocs-openapi/server';
7 -import { openapi } from '@/lib/openapi';
8 -import { getAllToolkitsSync } from '@/lib/toolkit-data';
2 +import { getDocsSearchIndexes } from '@/lib/search-index';
3
10 -// Create loaders directly here to avoid the problematic lib/source.ts import
11 -const docsSource = loader({
12 - baseUrl: '/docs',
13 - source: docs.toFumadocsSource(),
14 - plugins: [lucideIconsPlugin()],
15 -});
16 -
17 -const cookbooksSource = loader({
18 - baseUrl: '/cookbooks',
19 - source: cookbooks.toFumadocsSource(),
20 - plugins: [lucideIconsPlugin()],
21 -});
22 -
23 -const toolkitsSource = loader({
24 - baseUrl: '/toolkits',
25 - source: toolkits.toFumadocsSource(),
26 - plugins: [lucideIconsPlugin()],
27 -});
28 -
29 -// Dynamic toolkit entries from toolkits.json
30 -const mdxToolkitSlugs = new Set(
31 - toolkitsSource.getPages().map((page) => page.slugs.join('/')),
32 -);
33 -
34 -const dynamicToolkitIndexes = getAllToolkitsSync()
35 - .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))
36 - .map((toolkit) => ({
37 - id: `/toolkits/${toolkit.slug}`,
38 - title: toolkit.name,
39 - description: toolkit.description,
40 - url: `/toolkits/${toolkit.slug}`,
41 - structuredData: { headings: [] as never[], contents: [] as never[] },
42 - keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],
43 - }));
44 -
45 -// Changelog entries
46 -const changelogIndexes = changelog.map((entry) => ({
47 - id: `/docs/changelog/${entry.date.replace(/-/g, '/')}#${entry.title}`,
48 - title: entry.title,
49 - description: entry.description ?? '',
50 - url: `/docs/changelog/${entry.date.replace(/-/g, '/')}`,
51 - structuredData: { headings: [] as never[], contents: [] as never[] },
52 - keywords: ['changelog'],
53 -}));
54 -
55 -// Use dynamic indexes to support async OpenAPI page loading
4 +// Fallback/local search endpoint. Production search uses Algolia from the client
5 +// when NEXT_PUBLIC_ALGOLIA_* variables are configured.
6 export const { GET } = createSearchAPI('advanced', {
57 - indexes: async () => {
58 - // Load OpenAPI pages and build full reference source
59 - const openapiPages = await openapiSource(openapi, {
60 - groupBy: 'tag',
61 - baseDir: 'api-reference',
62 - });
63 -
64 - const fullReferenceSource = loader({
65 - baseUrl: '/reference',
66 - source: multiple({
67 - mdx: reference.toFumadocsSource(),
68 - openapi: openapiPages,
69 - }),
70 - plugins: [lucideIconsPlugin(), openapiPlugin()],
71 - });
72 -
73 - const mdxIndexes = [
74 - ...docsSource.getPages(),
75 - ...cookbooksSource.getPages(),
76 - ...toolkitsSource.getPages(),
77 - ...fullReferenceSource.getPages(),
78 - ].map((page) => ({
79 - id: page.url,
80 - title: page.data.title ?? 'Untitled',
81 - description: page.data.description,
82 - url: page.url,
83 - structuredData: page.data.structuredData,
84 - keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,
85 - }));
86 -
87 - return [...mdxIndexes, ...dynamicToolkitIndexes, ...changelogIndexes];
88 - },
7 + indexes: getDocsSearchIndexes,
8 });
docs/bun.lock
+35
@@ -1,5 +1,6 @@
1 {
2 "lockfileVersion": 1,
3 + "configVersion": 0,
4 "workspaces": {
5 "": {
6 "name": "fumadocs",
@@ -7,6 +8,7 @@
8 "@radix-ui/react-tabs": "^1.1.13",
9 "@shikijs/twoslash": "^3.21.0",
10 "@vercel/analytics": "^1.6.1",
11 + "algoliasearch": "^5",
12 "class-variance-authority": "^0.7.1",
13 "clsx": "^2.1.1",
14 "fumadocs-core": "^16.4.2",
@@ -19,6 +21,7 @@
21 "posthog-js": "^1.319.0",
22 "react": "^19.2.3",
23 "react-dom": "^19.2.3",
24 + "search-insights": "^2",
25 "shiki": "^3.20.0",
26 "tailwind-merge": "^3.4.0",
27 },
@@ -95,6 +98,34 @@
98
99 "@ai-sdk/ui-utils-v5": ["@ai-sdk/ui-utils@1.2.11", "", { "dependencies": { "@ai-sdk/provider": "1.1.3", "@ai-sdk/provider-utils": "2.2.8", "zod-to-json-schema": "^3.24.1" }, "peerDependencies": { "zod": "^3.23.8" } }, "sha512-3zcwCc8ezzFlwp3ZD15wAPjf2Au4s3vAbKsXQVyhxODHcmu0iyPO2Eua6D/vicq/AUm/BAo60r97O6HU+EI0+w=="],
100
101 + "@algolia/abtesting": ["@algolia/abtesting@1.18.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-aehCadlWOGvrT91KUIZpC0MbB8KBW9yUuvTJFd2xesR7le/IsT4nJUnjCCZ4ZqZCeTcPHPV5mo//fZ5oxcSVYw=="],
102 +
103 + "@algolia/client-abtesting": ["@algolia/client-abtesting@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-HmXOGBOAOJPounpBzBpuY0zDYeiCpxgHnQmuA7JO6ScukcBdGp3/XM9zJk5pJx/xNGD68mbPGXWpDxGtl6BwDQ=="],
104 +
105 + "@algolia/client-analytics": ["@algolia/client-analytics@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-5oo4+I8iixie9vXhCyNFCzeIr8pqA3FQ//VsLHTDvZAV4ttYOPGvYHGQq5NSalrLx5Jc3dRro/5uDOlnUMcBJg=="],
106 +
107 + "@algolia/client-common": ["@algolia/client-common@5.52.1", "", {}, "sha512-qCDoZfx5MpX7XQzvQ3bC4tSEMkQWQMaF/ABtLuoze03Y/flR563CCSws02qIJ23oX7lxl92LsilZjINVyTdtLw=="],
108 +
109 + "@algolia/client-insights": ["@algolia/client-insights@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-hnGs0/lsFJ2PWDxNBz7pxreXo/Xz7gxYRcfePBUjsH26ad0kU/sgnVZd9LwWBpsQv65z2jlb5dkyaB9WE9M9FQ=="],
110 +
111 + "@algolia/client-personalization": ["@algolia/client-personalization@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-2VxxNc/uBysyKvGeBdSM5n9eIDKH8kWD7wd9/yqbJAiVwU4Yv6tU1LSJusHKrXV/aCu1KW7t9Gug9QyeEmtn/Q=="],
112 +
113 + "@algolia/client-query-suggestions": ["@algolia/client-query-suggestions@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-O6mPtsw3xEfNOe6gWFpYLeAZAIljNa4Hgna3bq15PwyN7nbjTY0wXJFRbzs/0YVf75Br+SbOQUmjKxXYjDiSiQ=="],
114 +
115 + "@algolia/client-search": ["@algolia/client-search@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-gA8oJOV1LnQQkDf91iebNnFInHuW0gRPEgLSOQ7EfipCEjYTHm5swm1DlH9H5RaRw4RrHuzHBegnlzc0MAstcg=="],
116 +
117 + "@algolia/ingestion": ["@algolia/ingestion@1.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-U9zZfc5xIu9wRxZkt+HceJUAD4VKHKbAyLSloJdEyMRmphXeibfrY9cxqIXBcmPeZzGhn3Imb35Dq8l19PkJhw=="],
118 +
119 + "@algolia/monitoring": ["@algolia/monitoring@1.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-a3SGNceHmkQfq77iG8Ka+w1pvwfZa/0lzEIgse30fL0kD+yKnd/dg0dQvSfFPAEt2f21DMcGkDSSeJlO3KdQjQ=="],
120 +
121 + "@algolia/recommend": ["@algolia/recommend@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-z98QEguCFDpxb4S/PyrUK1igqF8tPsdbqOUUO6ON91vJ58w+Gwa6ncrI0oNXSFcrkxA5EqPKPQ2A1PBCn08TYQ=="],
122 +
123 + "@algolia/requester-browser-xhr": ["@algolia/requester-browser-xhr@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-CI7+/0I11QeZM59Uc8whd2or0kqzFVjpaPn9Qpwll/krHcBAxk24WkAQ6WX+IwDVMfpont4YGbKwAmCre3vE8Q=="],
124 +
125 + "@algolia/requester-fetch": ["@algolia/requester-fetch@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-S6bDuw9byfOvm3T71cgdoZgrgnZq6hpdMLkx52Louh57nUAmvGQESz2aojOynQHjbTiV55smvAFbgn0qT4tJrg=="],
126 +
127 + "@algolia/requester-node-http": ["@algolia/requester-node-http@5.52.1", "", { "dependencies": { "@algolia/client-common": "5.52.1" } }, "sha512-tqZXM+54rWo4mk5jL5Z/flE11nPmNEdXwFBM5py9DkOmbjeCNemfVd45FyM97XdzfZ0dl9uOJC6PYn1FpkeyQg=="],
128 +
129 "@alloc/quick-lru": ["@alloc/quick-lru@5.2.0", "", {}, "sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw=="],
130
131 "@antfu/install-pkg": ["@antfu/install-pkg@1.1.0", "", { "dependencies": { "package-manager-detector": "^1.3.0", "tinyexec": "^1.0.1" } }, "sha512-MGQsmw10ZyI+EJo45CdSER4zEb+p31LpDAFp2Z3gkSd1yqVZGi0Ebx++YTEMonJy4oChEMLsxZ64j8FH6sSqtQ=="],
@@ -865,6 +896,8 @@
896
897 "ajv-formats": ["ajv-formats@3.0.1", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],
898
899 + "algoliasearch": ["algoliasearch@5.52.1", "", { "dependencies": { "@algolia/abtesting": "1.18.1", "@algolia/client-abtesting": "5.52.1", "@algolia/client-analytics": "5.52.1", "@algolia/client-common": "5.52.1", "@algolia/client-insights": "5.52.1", "@algolia/client-personalization": "5.52.1", "@algolia/client-query-suggestions": "5.52.1", "@algolia/client-search": "5.52.1", "@algolia/ingestion": "1.52.1", "@algolia/monitoring": "1.52.1", "@algolia/recommend": "5.52.1", "@algolia/requester-browser-xhr": "5.52.1", "@algolia/requester-fetch": "5.52.1", "@algolia/requester-node-http": "5.52.1" } }, "sha512-fHA8+kXTbjagw3jkLiaS7KKrH8qe2DyOsiUhGlN4cdT77PEsfqXZl7ewDk1hsg+pJnPlnE50XtLxjR91iJOpmg=="],
900 +
901 "alien-signals": ["alien-signals@3.1.2", "", {}, "sha512-d9dYqZTS90WLiU0I5c6DHj/HcKkF8ZyGN3G5x8wSbslulz70KOxaqCT0hQCo9KOyhVqzqGojvNdJXoTumZOtcw=="],
902
903 "ansi-regex": ["ansi-regex@6.2.2", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="],
@@ -1999,6 +2032,8 @@
2032
2033 "scroll-into-view-if-needed": ["scroll-into-view-if-needed@3.1.0", "", { "dependencies": { "compute-scroll-into-view": "^3.0.2" } }, "sha512-49oNpRjWRvnU8NyGVmUaYG4jtTkNonFZI86MmGRDqBphEK2EXT9gdEUoQPZhuBM8yWHxCWbobltqYO5M4XrUvQ=="],
2034
2035 + "search-insights": ["search-insights@2.17.3", "", {}, "sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ=="],
2036 +
2037 "section-matter": ["section-matter@1.0.0", "", { "dependencies": { "extend-shallow": "^2.0.1", "kind-of": "^6.0.0" } }, "sha512-vfD3pmTzGpufjScBh50YHKzEu2lxBWhVEHsNGoEXmCmn2hKGfeNLYMzCJpe8cD7gqX7TJluOVpBkAequ6dgMmA=="],
2038
2039 "secure-json-parse": ["secure-json-parse@2.7.0", "", {}, "sha512-6aU+Rwsezw7VR8/nyvKTx8QpWH9FrcYiXXlqC4z5d5XQBDRqtbfsRjnwGyqbi3gddNtWHuEk9OANUotL26qKUw=="],
docs/components/custom-search-dialog.tsx
+194 -9
@@ -1,7 +1,9 @@
1 'use client';
2
3 -import { useEffect, useState } from 'react';
3 +import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
4 import Link from 'next/link';
5 +import { liteClient } from 'algoliasearch/lite';
6 +import aa from 'search-insights';
7 import {
8 SearchDialog,
9 SearchDialogClose,
@@ -15,14 +17,20 @@ import {
17 } from 'fumadocs-ui/components/dialog/search';
18 import { useI18n } from 'fumadocs-ui/contexts/i18n';
19 import { useDocsSearch } from 'fumadocs-core/search/client';
20 +import type { BaseIndex } from 'fumadocs-core/search/algolia';
21 import { BotMessageSquare } from 'lucide-react';
22 import { toggleDecimalWidget, detectMac } from './ask-ai-button';
23
24 function MetaKey() {
25 const [key, setKey] = useState('⌘');
26 +
27 useEffect(() => {
24 - if (!detectMac()) setKey('Ctrl');
28 + if (detectMac()) return;
29 +
30 + const id = window.setTimeout(() => setKey('Ctrl'), 0);
31 + return () => window.clearTimeout(id);
32 }, []);
33 +
34 return key;
35 }
36
@@ -37,17 +45,182 @@ interface CustomSearchDialogProps extends SharedProps {
45 api?: string;
46 }
47
48 +type AlgoliaHit = {
49 + objectID: string;
50 + url?: string;
51 +};
52 +
53 +type AlgoliaSearchResponse = {
54 + hits: AlgoliaHit[];
55 + queryID?: string;
56 +};
57 +
58 +type AlgoliaHitMeta = {
59 + objectID: string;
60 + position: number;
61 + queryID?: string;
62 +};
63 +
64 +type SearchResultItem = {
65 + content: unknown;
66 + breadcrumbs?: unknown[];
67 +};
68 +
69 +const SEARCH_BREADCRUMB_LABELS: Record<string, string> = {
70 + toolkits: 'Toolkit',
71 + toolkit: 'Toolkit',
72 + cookbooks: 'Cookbook',
73 + cookbook: 'Cookbook',
74 + docs: 'Docs',
75 + reference: 'Reference',
76 + 'api-reference': 'API Reference',
77 + changelog: 'Changelog',
78 +};
79 +
80 +function normalizeSearchBreadcrumb(value: string): string {
81 + return SEARCH_BREADCRUMB_LABELS[value.toLowerCase()] ?? value;
82 +}
83 +
84 +function normalizeSearchResult<T extends SearchResultItem>(result: T): T {
85 + if (!result.breadcrumbs) return result;
86 +
87 + const breadcrumbs = result.breadcrumbs.map((breadcrumb) =>
88 + typeof breadcrumb === 'string' ? normalizeSearchBreadcrumb(breadcrumb) : breadcrumb,
89 + );
90 +
91 + const content = typeof result.content === 'string' ? result.content.toLowerCase() : null;
92 + const dedupedBreadcrumbs = breadcrumbs.filter((breadcrumb, index) => {
93 + if (index !== breadcrumbs.length - 1 || content === null || typeof breadcrumb !== 'string') {
94 + return true;
95 + }
96 +
97 + return breadcrumb.toLowerCase() !== content;
98 + });
99 +
100 + return {
101 + ...result,
102 + breadcrumbs: dedupedBreadcrumbs,
103 + };
104 +}
105 +
106 export default function CustomSearchDialog({
107 defaultLinks = [],
108 api = '/api/search',
109 ...props
110 }: CustomSearchDialogProps) {
111 const { locale } = useI18n();
46 - const { search, setSearch, query } = useDocsSearch({
47 - type: 'fetch',
48 - locale,
49 - api,
50 - });
112 + const algoliaHitMetaRef = useRef(new Map<string, AlgoliaHitMeta>());
113 + const insightsInitializedRef = useRef(false);
114 +
115 + const ensureAlgoliaInsights = useCallback((appId: string, searchApiKey: string) => {
116 + if (insightsInitializedRef.current) return;
117 +
118 + aa('init', {
119 + appId,
120 + apiKey: searchApiKey,
121 + useCookie: true,
122 + });
123 + insightsInitializedRef.current = true;
124 + }, []);
125 +
126 + const clientOptions = useMemo(() => {
127 + const appId = process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? '62HI9PQZ1L';
128 + const searchApiKey = process.env.NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY;
129 + const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';
130 +
131 + if (appId && searchApiKey) {
132 + const client = liteClient(appId, searchApiKey);
133 +
134 + return {
135 + type: 'algolia' as const,
136 + client,
137 + indexName,
138 + locale,
139 + onSearch: async (query: string, tag?: string) => {
140 + ensureAlgoliaInsights(appId, searchApiKey);
141 +
142 + const result = await client.searchForHits<BaseIndex>({
143 + requests: [
144 + {
145 + type: 'default',
146 + indexName,
147 + query,
148 + distinct: true,
149 + hitsPerPage: 10,
150 + filters: tag ? `tag:${tag}` : undefined,
151 + clickAnalytics: true,
152 + },
153 + ],
154 + });
155 + const response = result.results[0] as AlgoliaSearchResponse;
156 + const metadata = new Map<string, AlgoliaHitMeta>();
157 +
158 + response.hits.forEach((hit, index) => {
159 + if (!hit.url || metadata.has(hit.url)) return;
160 + metadata.set(hit.url, {
161 + objectID: hit.objectID,
162 + position: index + 1,
163 + queryID: response.queryID,
164 + });
165 + });
166 +
167 + algoliaHitMetaRef.current = metadata;
168 +
169 + const viewedObjectIDs = Array.from(metadata.values())
170 + .slice(0, 20)
171 + .map((hit) => hit.objectID);
172 +
173 + if (viewedObjectIDs.length > 0) {
174 + aa('viewedObjectIDs', {
175 + index: indexName,
176 + eventName: 'Docs Search Results Viewed',
177 + objectIDs: viewedObjectIDs,
178 + });
179 + }
180 +
181 + return result;
182 + },
183 + };
184 + }
185 +
186 + return {
187 + type: 'fetch' as const,
188 + locale,
189 + api,
190 + };
191 + }, [api, ensureAlgoliaInsights, locale]);
192 +
193 + const { search, setSearch, query } = useDocsSearch(clientOptions);
194 + const searchResults = useMemo(() => {
195 + if (query.data === 'empty' || !query.data) return query.data;
196 + return query.data.map(normalizeSearchResult);
197 + }, [query.data]);
198 +
199 + const trackAlgoliaClick = useCallback((href: string) => {
200 + const url = new URL(href, window.location.origin);
201 + const path = `${url.pathname}${url.hash}`;
202 + const hit = algoliaHitMetaRef.current.get(path) ?? algoliaHitMetaRef.current.get(url.pathname);
203 + const indexName = process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? 'docs_composio_dev_62hi9pqz1l_pages';
204 +
205 + if (!hit) return;
206 +
207 + if (hit.queryID) {
208 + aa('clickedObjectIDsAfterSearch', {
209 + index: indexName,
210 + eventName: 'Docs Search Result Clicked',
211 + queryID: hit.queryID,
212 + objectIDs: [hit.objectID],
213 + positions: [hit.position],
214 + });
215 + return;
216 + }
217 +
218 + aa('clickedObjectIDs', {
219 + index: indexName,
220 + eventName: 'Docs Search Result Clicked',
221 + objectIDs: [hit.objectID],
222 + });
223 + }, []);
224
225 return (
226 <SearchDialog
@@ -63,7 +236,19 @@ export default function CustomSearchDialog({
236 <SearchDialogInput />
237 <SearchDialogClose />
238 </SearchDialogHeader>
66 - <div className="search-scroll-container max-md:min-h-0 max-md:flex-1 max-md:overflow-y-auto">
239 + <div
240 + className="search-scroll-container max-md:min-h-0 max-md:flex-1 max-md:overflow-y-auto"
241 + onClickCapture={(event) => {
242 + const target = event.target;
243 + if (!(target instanceof Element)) return;
244 +
245 + const anchor = target.closest('a[href]');
246 + const href = anchor?.getAttribute('href');
247 + if (!href) return;
248 +
249 + trackAlgoliaClick(href);
250 + }}
251 + >
252 {query.data === 'empty' && defaultLinks.length > 0 ? (
253 <div className="flex flex-col p-2">
254 {defaultLinks.map((link) => (
@@ -81,7 +266,7 @@ export default function CustomSearchDialog({
266 ))}
267 </div>
268 ) : (
84 - <SearchDialogList items={query.data === 'empty' ? null : query.data} />
269 + <SearchDialogList items={searchResults === 'empty' ? null : searchResults} />
270 )}
271 </div>
272 <div className="flex items-center justify-between border-t px-3 py-2">
docs/lib/search-index.ts new
+618
@@ -0,0 +1,618 @@
1 +// Use direct imports from collections to avoid top-level await in lib/source.ts.
2 +import { createHash } from 'node:crypto';
3 +import { existsSync, readdirSync, readFileSync } from 'node:fs';
4 +import { join, relative } from 'node:path';
5 +import { docs, reference, cookbooks, toolkits } from 'fumadocs-mdx:collections/server';
6 +import { loader, multiple } from 'fumadocs-core/source';
7 +import { lucideIconsPlugin } from 'fumadocs-core/source/lucide-icons';
8 +import { openapiSource, openapiPlugin } from 'fumadocs-openapi/server';
9 +import type { AdvancedIndex } from 'fumadocs-core/search/server';
10 +import type { BaseIndex } from 'fumadocs-core/search/algolia';
11 +import { openapi } from '@/lib/openapi';
12 +import { mdxToCleanMarkdown } from '@/lib/source';
13 +import { getAllToolkitsSync } from '@/lib/toolkit-data';
14 +
15 +export const ALGOLIA_DEFAULT_APP_ID = '62HI9PQZ1L';
16 +export const ALGOLIA_DEFAULT_INDEX_NAME = 'docs_composio_dev_62hi9pqz1l_pages';
17 +
18 +const MAX_CHUNK_CHARS = 3_800;
19 +const MAX_CHUNK_BYTES = 9_000;
20 +const MAX_TOOL_ALIAS_ITEMS = 80;
21 +const MAX_TOOL_ALIAS_BYTES = 2_500;
22 +
23 +// Create loaders directly here to avoid the problematic lib/source.ts import in the
24 +// fallback route. This route is intentionally still Fumadocs/Orama-backed for local
25 +// development when Algolia env vars aren't configured.
26 +const docsSource = loader({
27 + baseUrl: '/docs',
28 + source: docs.toFumadocsSource(),
29 + plugins: [lucideIconsPlugin()],
30 +});
31 +
32 +const cookbooksSource = loader({
33 + baseUrl: '/cookbooks',
34 + source: cookbooks.toFumadocsSource(),
35 + plugins: [lucideIconsPlugin()],
36 +});
37 +
38 +const toolkitsSource = loader({
39 + baseUrl: '/toolkits',
40 + source: toolkits.toFumadocsSource(),
41 + plugins: [lucideIconsPlugin()],
42 +});
43 +
44 +type SearchIndex = AdvancedIndex & {
45 + keywords?: string[];
46 +};
47 +
48 +export type AlgoliaDocsRecord = BaseIndex & {
49 + objectID: string;
50 + description?: string;
51 + keywords?: string[];
52 + slug?: string;
53 + headings?: string[];
54 + tool_names?: string[];
55 + tool_slugs?: string[];
56 + type: string;
57 + lang: string;
58 + page_rank: number;
59 + toolkit_popularity: number;
60 + section_rank: number;
61 + position: number;
62 + depth: number;
63 + tags?: string[];
64 +};
65 +
66 +function getFrontmatter(source: string): string {
67 + return source.match(/^---\n([\s\S]*?)\n---/)?.[1] ?? '';
68 +}
69 +
70 +function getFrontmatterValue(frontmatter: string, key: string): string | undefined {
71 + const match = frontmatter.match(new RegExp(`^${key}:\\s*(?:"([^"]*)"|'([^']*)'|(.+))$`, 'm'));
72 + const value = match?.[1] ?? match?.[2] ?? match?.[3];
73 + return value?.trim();
74 +}
75 +
76 +function getFrontmatterList(frontmatter: string, key: string): string[] {
77 + const raw = getFrontmatterValue(frontmatter, key);
78 + if (!raw) return [];
79 +
80 + if (raw.startsWith('[') && raw.endsWith(']')) {
81 + return raw
82 + .slice(1, -1)
83 + .split(',')
84 + .map((value) => value.trim().replace(/^['"]|['"]$/g, ''))
85 + .filter(Boolean);
86 + }
87 +
88 + return [raw];
89 +}
90 +
91 +function listContentFiles(dir: string): string[] {
92 + if (!existsSync(dir)) return [];
93 +
94 + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {
95 + const path = join(dir, entry.name);
96 + if (entry.isDirectory()) return listContentFiles(path);
97 + if (entry.isFile() && /\.mdx?$/.test(entry.name)) return [path];
98 + return [];
99 + });
100 +}
101 +
102 +function slugify(value: string): string {
103 + return value
104 + .toLowerCase()
105 + .trim()
106 + .replace(/[<>]/g, '')
107 + .replace(/[`*_~\[\]()]/g, '')
108 + .replace(/&[a-z]+;/g, '')
109 + .replace(/[^\w\s-]/g, '')
110 + .replace(/\s+/g, '-')
111 + .replace(/-+/g, '-')
112 + .replace(/^-|-$/g, '');
113 +}
114 +
115 +function uniqueSlug(value: string, seen: Map<string, number>): string {
116 + const base = slugify(value) || 'section';
117 + const count = seen.get(base) ?? 0;
118 + seen.set(base, count + 1);
119 + return count === 0 ? base : `${base}-${count + 1}`;
120 +}
121 +
122 +function splitIntoChunks(text: string): string[] {
123 + const paragraphs = text
124 + .split(/\n{2,}/)
125 + .map((paragraph) => paragraph.trim())
126 + .filter(Boolean);
127 +
128 + const chunks: string[] = [];
129 + let current = '';
130 +
131 + const pushCurrent = () => {
132 + const trimmed = current.trim();
133 + if (trimmed) chunks.push(trimmed);
134 + current = '';
135 + };
136 +
137 + for (const paragraph of paragraphs) {
138 + const next = current ? `${current}\n\n${paragraph}` : paragraph;
139 + if (next.length <= MAX_CHUNK_CHARS && Buffer.byteLength(next, 'utf8') <= MAX_CHUNK_BYTES) {
140 + current = next;
141 + continue;
142 + }
143 +
144 + pushCurrent();
145 +
146 + if (paragraph.length <= MAX_CHUNK_CHARS && Buffer.byteLength(paragraph, 'utf8') <= MAX_CHUNK_BYTES) {
147 + current = paragraph;
148 + continue;
149 + }
150 +
151 + for (let i = 0; i < paragraph.length; i += MAX_CHUNK_CHARS) {
152 + chunks.push(paragraph.slice(i, i + MAX_CHUNK_CHARS).trim());
153 + }
154 + }
155 +
156 + pushCurrent();
157 + return chunks;
158 +}
159 +
160 +function contentHash(value: string): string {
161 + return createHash('sha1').update(value).digest('hex').slice(0, 10);
162 +}
163 +
164 +function urlFromContentPath(path: string): { url: string; type: string } | undefined {
165 + const rel = relative(join(process.cwd(), 'content'), path).replace(/\\/g, '/');
166 + const withoutExt = rel.replace(/\.mdx?$/, '');
167 + const parts = withoutExt.split('/');
168 + const collection = parts.shift();
169 + if (!collection) return undefined;
170 +
171 + if (collection === 'docs') {
172 + return { url: `/docs/${parts.join('/')}`.replace(/\/index$/, ''), type: 'docs' };
173 + }
174 + if (collection === 'cookbooks') {
175 + return { url: `/cookbooks/${parts.join('/')}`.replace(/\/index$/, ''), type: 'cookbooks' };
176 + }
177 + if (collection === 'reference') {
178 + const url = `/reference/${parts.join('/')}`.replace(/\/index$/, '');
179 + return { url, type: url.startsWith('/reference/v3') ? 'v3-reference' : 'reference' };
180 + }
181 + if (collection === 'toolkits') {
182 + if (parts[0] === 'faq') return undefined;
183 + return { url: `/toolkits/${parts.join('/')}`.replace(/\/index$/, ''), type: 'toolkits' };
184 + }
185 + if (collection === 'changelog') return undefined;
186 +
187 + return undefined;
188 +}
189 +
190 +function slugTokens(url: string): string {
191 + return url
192 + .split(/[/-]/)
193 + .map((part) => part.replace(/[_#?=&.]/g, ' '))
194 + .join(' ')
195 + .replace(/\s+/g, ' ')
196 + .trim();
197 +}
198 +
199 +const typeLabels: Record<string, string> = {
200 + docs: 'Docs',
201 + cookbooks: 'Cookbook',
202 + reference: 'Reference',
203 + 'v3-reference': 'Legacy v3 Reference',
204 + toolkits: 'Toolkit',
205 + changelog: 'Changelog',
206 + 'api-reference': 'API Reference',
207 +};
208 +
209 +function titleizeSlug(value: string): string {
210 + return value
211 + .split('-')
212 + .filter(Boolean)
213 + .map((part) => part.charAt(0).toUpperCase() + part.slice(1))
214 + .join(' ')
215 + .replace(/\bApi\b/g, 'API')
216 + .replace(/\bMcp\b/g, 'MCP')
217 + .replace(/\bSdk\b/g, 'SDK')
218 + .replace(/\bOauth\b/g, 'OAuth');
219 +}
220 +
221 +function breadcrumbsForUrl(url: string, type: string): string[] {
222 + const label = typeLabels[type] ?? titleizeSlug(type);
223 +
224 + if (type === 'toolkits' || type === 'cookbooks' || type === 'changelog') {
225 + return [label];
226 + }
227 +
228 + const parts = url.split('/').filter(Boolean);
229 + const parentParts = parts.slice(1, -1);
230 + return [label, ...parentParts.map(titleizeSlug)].filter(Boolean);
231 +}
232 +
233 +const TOOLKIT_POPULARITY_OVERRIDES: Record<string, number> = {
234 + gmail: 1_000,
235 + github: 980,
236 + slack: 960,
237 + googledrive: 940,
238 + googlecalendar: 930,
239 + notion: 920,
240 + linear: 900,
241 + jira: 890,
242 + hubspot: 860,
243 + salesforce: 850,
244 + resend: 650,
245 + zoho_mail: 520,
246 + mailchimp: 500,
247 + sendgrid: 480,
248 + mailsoftly: 260,
249 +};
250 +
251 +function getToolkitSlugFromUrl(url: string): string | null {
252 + return url.match(/^\/toolkits\/([^/#?]+)/)?.[1] ?? null;
253 +}
254 +
255 +function toolkitPopularity(url: string, type: string): number {
256 + if (type !== 'toolkits') return 0;
257 +
258 + const slug = getToolkitSlugFromUrl(url);
259 + if (!slug) return 0;
260 +
261 + const toolkit = toolkitBySlug.get(slug);
262 + const override = TOOLKIT_POPULARITY_OVERRIDES[slug] ?? 0;
263 + const managedAuthBoost = toolkit?.composioManagedAuthSchemes?.length ? 120 : 0;
264 + const authBoost = toolkit?.authSchemes?.includes('OAUTH2') ? 40 : 0;
265 + const triggerBoost = Math.min((toolkit?.triggerCount ?? 0) * 10, 80);
266 + const toolCountBoost = Math.min((toolkit?.toolCount ?? 0), 100);
267 +
268 + return override + managedAuthBoost + authBoost + triggerBoost + toolCountBoost;
269 +}
270 +
271 +function pageRank(url: string, type: string): number {
272 + // Prefer conceptual docs over generated/reference material when textual
273 + // relevance is otherwise close. Toolkit aliases can still win earlier via
274 + // searchableAttributes when the query matches a tool name/slug exactly.
275 + if (type === 'docs') {
276 + if (url === '/docs' || url === '/docs/') return 2_400;
277 + if (url.includes('/quickstart')) return 2_300;
278 + if (url.includes('/authentication')) return 2_220;
279 + if (url.includes('/tools-and-toolkits')) return 2_180;
280 + if (url.includes('/configuring-sessions')) return 2_120;
281 + return 2_000;
282 + }
283 +
284 + if (type === 'cookbooks') return 1_500;
285 + if (type === 'toolkits') return 1_250;
286 + // Current v3.1 reference should be available, but conceptual docs should
287 + // win whenever both match. Legacy v3 reference is only a last-resort result.
288 + if (type === 'reference') return 650;
289 + if (type === 'api-reference') return 560;
290 + if (type === 'v3-reference') return 25;
291 + if (type === 'changelog') return 300;
292 + return 400;
293 +}
294 +
295 +function recordsFromMarkdownPage(input: {
296 + url: string;
297 + type: string;
298 + title: string;
299 + description?: string;
300 + keywords?: string[];
301 + markdown: string;
302 + breadcrumbs?: string[];
303 + tags?: string[];
304 + toolNames?: string[];
305 + toolSlugs?: string[];
306 +}): AlgoliaDocsRecord[] {
307 + const clean = mdxToCleanMarkdown(input.markdown);
308 + const lines = clean.split('\n');
309 + const headingSlugs = new Map<string, number>();
310 + const headings: string[] = [];
311 + const sections: Array<{ heading?: string; section_id?: string; depth: number; text: string; position: number }> = [];
312 + let currentHeading: string | undefined;
313 + let currentSectionId: string | undefined;
314 + let currentDepth = 0;
315 + let currentLines: string[] = [];
316 + let sectionPosition = 0;
317 +
318 + const flush = () => {
319 + const text = currentLines.join('\n').trim();
320 + if (!text) return;
321 + sections.push({
322 + heading: currentHeading,
323 + section_id: currentSectionId,
324 + depth: currentDepth,
325 + text,
326 + position: sectionPosition++,
327 + });
328 + currentLines = [];
329 + };
330 +
331 + for (const line of lines) {
332 + const match = line.match(/^(#{1,6})\s+(.+)$/);
333 + if (match) {
334 + flush();
335 + const depth = match[1].length;
336 + const heading = match[2].trim();
337 + currentHeading = heading;
338 + currentSectionId = uniqueSlug(heading, headingSlugs);
339 + currentDepth = depth;
340 + headings.push(heading);
341 + currentLines.push(line);
342 + continue;
343 + }
344 +
345 + currentLines.push(line);
346 + }
347 + flush();
348 +
349 + const fallbackText = clean.trim() || [input.title, input.description, ...(input.keywords ?? [])].filter(Boolean).join('\n');
350 + if (sections.length === 0 && fallbackText) {
351 + sections.push({ text: fallbackText, position: 0, depth: 0 });
352 + }
353 +
354 + let recordPosition = 0;
355 + return sections.flatMap((section) => {
356 + const chunks = splitIntoChunks(section.text);
357 + return chunks.map((chunk, chunkIndex) => {
358 + const sectionPart = section.section_id ?? 'overview';
359 + const objectID = `${input.url}__${sectionPart}__${chunkIndex}__${contentHash(chunk)}`;
360 + const sectionRank = Math.max(10, 120 - section.depth * 12 - chunkIndex * 2);
361 +
362 + const position = recordPosition++;
363 + const includeToolkitAliases = input.type === 'toolkits' && position === 0;
364 +
365 + return {
366 + objectID,
367 + title: input.title,
368 + description: input.description,
369 + breadcrumbs: input.breadcrumbs,
370 + url: input.url,
371 + page_id: input.url,
372 + section: section.heading,
373 + section_id: section.section_id,
374 + content: chunk,
375 + keywords: input.keywords,
376 + slug: slugTokens(input.url),
377 + headings,
378 + tool_names: includeToolkitAliases ? input.toolNames : undefined,
379 + tool_slugs: includeToolkitAliases ? input.toolSlugs : undefined,
380 + type: input.type,
381 + lang: 'en',
382 + tags: input.tags,
383 + page_rank: pageRank(input.url, input.type),
384 + toolkit_popularity: toolkitPopularity(input.url, input.type),
385 + section_rank: sectionRank,
386 + position,
387 + depth: section.depth,
388 + } satisfies AlgoliaDocsRecord;
389 + });
390 + });
391 +}
392 +
393 +const toolkitBySlug = new Map(getAllToolkitsSync().map((toolkit) => [toolkit.slug, toolkit]));
394 +
395 +function limitToolkitAliases(values: string[]): string[] {
396 + const aliases: string[] = [];
397 + let bytes = 0;
398 +
399 + for (const value of values) {
400 + const trimmed = value.trim();
401 + if (!trimmed) continue;
402 +
403 + const nextBytes = Buffer.byteLength(JSON.stringify(trimmed), 'utf8');
404 + if (aliases.length >= MAX_TOOL_ALIAS_ITEMS || bytes + nextBytes > MAX_TOOL_ALIAS_BYTES) break;
405 +
406 + aliases.push(trimmed);
407 + bytes += nextBytes;
408 + }
409 +
410 + return aliases;
411 +}
412 +
413 +function getToolkitSearchFields(slug: string): Pick<Parameters<typeof recordsFromMarkdownPage>[0], 'toolNames' | 'toolSlugs' | 'tags'> {
414 + const toolkit = toolkitBySlug.get(slug);
415 + if (!toolkit) return {};
416 +
417 + return {
418 + toolNames: limitToolkitAliases(toolkit.tools.map((tool) => tool.name).filter(Boolean)),
419 + toolSlugs: limitToolkitAliases(toolkit.tools.map((tool) => tool.slug).filter(Boolean)),
420 + tags: [toolkit.category].filter(Boolean) as string[],
421 + };
422 +}
423 +
424 +function getFilesystemRecords(): AlgoliaDocsRecord[] {
425 + const contentDir = join(process.cwd(), 'content');
426 + return listContentFiles(contentDir).flatMap((file) => {
427 + const route = urlFromContentPath(file);
428 + if (!route) return [];
429 +
430 + const source = readFileSync(file, 'utf8');
431 + const frontmatter = getFrontmatter(source);
432 + const title = getFrontmatterValue(frontmatter, 'title');
433 + if (!title) return [];
434 +
435 + const toolkitFields = route.type === 'toolkits'
436 + ? getToolkitSearchFields(route.url.replace(/^\/toolkits\//, ''))
437 + : {};
438 +
439 + return recordsFromMarkdownPage({
440 + url: route.url,
441 + type: route.type,
442 + title,
443 + description: getFrontmatterValue(frontmatter, 'description'),
444 + keywords: getFrontmatterList(frontmatter, 'keywords'),
445 + markdown: source,
446 + breadcrumbs: breadcrumbsForUrl(route.url, route.type),
447 + ...toolkitFields,
448 + });
449 + });
450 +}
451 +
452 +// Dynamic toolkit entries from toolkits.json. We include a compact sample of tool
453 +// names/slugs so queries like "gmail send email" can still find the Gmail toolkit
454 +// page without indexing the full tool catalog as noisy giant records.
455 +const mdxToolkitSlugs = new Set(
456 + toolkitsSource.getPages().map((page) => page.slugs.join('/')),
457 +);
458 +
459 +const dynamicToolkitIndexes = getAllToolkitsSync()
460 + .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))
461 + .map((toolkit) => ({
462 + id: `/toolkits/${toolkit.slug}`,
463 + title: toolkit.name,
464 + description: toolkit.description,
465 + url: `/toolkits/${toolkit.slug}`,
466 + structuredData: { headings: [], contents: [] },
467 + keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],
468 + } satisfies SearchIndex));
469 +
470 +function getDynamicToolkitRecords(): AlgoliaDocsRecord[] {
471 + return getAllToolkitsSync()
472 + .filter((toolkit) => !mdxToolkitSlugs.has(toolkit.slug))
473 + .flatMap((toolkit) => {
474 + const toolsText = (toolkit.tools ?? [])
475 + .slice(0, 60)
476 + .map((tool) => `${tool.slug ?? ''} ${tool.name ?? ''}`)
477 + .join('\n');
478 +
479 + return recordsFromMarkdownPage({
480 + url: `/toolkits/${toolkit.slug}`,
481 + type: 'toolkits',
482 + title: toolkit.name,
483 + description: toolkit.description,
484 + keywords: [toolkit.slug, toolkit.category].filter(Boolean) as string[],
485 + markdown: `# ${toolkit.name}\n\n${toolkit.description ?? ''}\n\n## Available tools\n\n${toolsText}`,
486 + breadcrumbs: breadcrumbsForUrl(`/toolkits/${toolkit.slug}`, 'toolkits'),
487 + ...getToolkitSearchFields(toolkit.slug),
488 + });
489 + });
490 +}
491 +
492 +function getChangelogIndexes(): SearchIndex[] {
493 + const changelogDir = join(process.cwd(), 'content/changelog');
494 + if (!existsSync(changelogDir)) return [];
495 +
496 + return readdirSync(changelogDir)
497 + .filter((file) => file.endsWith('.mdx'))
498 + .flatMap((file) => {
499 + const source = readFileSync(join(changelogDir, file), 'utf8');
500 + const frontmatter = getFrontmatter(source);
501 + const date = getFrontmatterValue(frontmatter, 'date');
502 + const title = getFrontmatterValue(frontmatter, 'title');
503 + if (!date || !title) return [];
504 +
505 + const url = `/docs/changelog/${date.replace(/-/g, '/')}`;
506 +
507 + return [{
508 + id: `${url}#${title}`,
509 + title,
510 + description: getFrontmatterValue(frontmatter, 'description') ?? '',
511 + url,
512 + structuredData: { headings: [], contents: [] },
513 + keywords: ['changelog'],
514 + } satisfies SearchIndex];
515 + });
516 +}
517 +
518 +function getChangelogRecords(): AlgoliaDocsRecord[] {
519 + const changelogDir = join(process.cwd(), 'content/changelog');
520 + if (!existsSync(changelogDir)) return [];
521 +
522 + return readdirSync(changelogDir)
523 + .filter((file) => file.endsWith('.mdx'))
524 + .flatMap((file) => {
525 + const source = readFileSync(join(changelogDir, file), 'utf8');
526 + const frontmatter = getFrontmatter(source);
527 + const date = getFrontmatterValue(frontmatter, 'date');
528 + const title = getFrontmatterValue(frontmatter, 'title');
529 + if (!date || !title) return [];
530 +
531 + return recordsFromMarkdownPage({
532 + url: `/docs/changelog/${date.replace(/-/g, '/')}`,
533 + type: 'changelog',
534 + title,
535 + description: getFrontmatterValue(frontmatter, 'description'),
536 + keywords: ['changelog', date],
537 + markdown: source,
538 + breadcrumbs: breadcrumbsForUrl(`/docs/changelog/${date.replace(/-/g, '/')}`, 'changelog'),
539 + });
540 + });
541 +}
542 +
543 +export async function getDocsSearchIndexes(): Promise<SearchIndex[]> {
544 + // Load OpenAPI pages and build full reference source.
545 + const openapiPages = await openapiSource(openapi, {
546 + groupBy: 'tag',
547 + baseDir: 'api-reference',
548 + });
549 +
550 + const fullReferenceSource = loader({
551 + baseUrl: '/reference',
552 + source: multiple({
553 + mdx: reference.toFumadocsSource(),
554 + openapi: openapiPages,
555 + }),
556 + plugins: [lucideIconsPlugin(), openapiPlugin()],
557 + });
558 +
559 + const mdxIndexes = [
560 + ...docsSource.getPages(),
561 + ...cookbooksSource.getPages(),
562 + ...toolkitsSource.getPages(),
563 + ...fullReferenceSource.getPages(),
564 + ].map((page) => ({
565 + id: page.url,
566 + title: page.data.title ?? 'Untitled',
567 + description: page.data.description,
568 + url: page.url,
569 + structuredData: page.data.structuredData,
570 + keywords: 'keywords' in page.data ? (page.data.keywords as string[]) : undefined,
571 + } satisfies SearchIndex));
572 +
573 + return [...mdxIndexes, ...dynamicToolkitIndexes, ...getChangelogIndexes()];
574 +}
575 +
576 +async function getOpenApiRecords(): Promise<AlgoliaDocsRecord[]> {
577 + const openapiPages = await openapiSource(openapi, {
578 + groupBy: 'tag',
579 + baseDir: 'api-reference',
580 + });
581 +
582 + const openapiOnlySource = loader({
583 + baseUrl: '/reference',
584 + source: openapiPages,
585 + plugins: [lucideIconsPlugin(), openapiPlugin()],
586 + });
587 +
588 + return openapiOnlySource.getPages().flatMap((page) => {
589 + const contents = [
590 + page.data.description,
591 + ...(page.data.structuredData?.headings ?? []).map((heading) => heading.content),
592 + ...(page.data.structuredData?.contents ?? []).map((content) => content.content),
593 + ].filter(Boolean).join('\n\n');
594 +
595 + return recordsFromMarkdownPage({
596 + url: page.url,
597 + type: 'api-reference',
598 + title: page.data.title ?? 'Untitled',
599 + description: page.data.description,
600 + markdown: `# ${page.data.title ?? 'Untitled'}\n\n${contents}`,
601 + breadcrumbs: breadcrumbsForUrl(page.url, 'api-reference'),
602 + });
603 + });
604 +}
605 +
606 +export async function getAlgoliaSearchDocuments(): Promise<AlgoliaDocsRecord[]> {
607 + const records = [
608 + ...getFilesystemRecords(),
609 + ...getDynamicToolkitRecords(),
610 + ...getChangelogRecords(),
611 + ...await getOpenApiRecords(),
612 + ];
613 +
614 + const deduped = new Map<string, AlgoliaDocsRecord>();
615 + for (const record of records) deduped.set(record.objectID, record);
616 +
617 + return Array.from(deduped.values());
618 +}
docs/package.json
+4
@@ -10,6 +10,8 @@
10 "postinstall": "fumadocs-mdx",
11 "lint": "eslint",
12 "lint:links": "bun scripts/validate-links.ts",
13 + "sync:search": "bun scripts/sync-algolia-search.ts",
14 + "test:search": "bun scripts/test-algolia-search.ts",
15 "generate:toolkits": "bun scripts/generate-toolkits.ts",
16 "generate:meta-tools": "bun scripts/generate-meta-tools.ts",
17 "generate:api-index": "bun scripts/generate-api-index.ts",
@@ -20,6 +22,7 @@
22 "@radix-ui/react-tabs": "^1.1.13",
23 "@shikijs/twoslash": "^3.21.0",
24 "@vercel/analytics": "^1.6.1",
25 + "algoliasearch": "^5",
26 "class-variance-authority": "^0.7.1",
27 "clsx": "^2.1.1",
28 "fumadocs-core": "^16.4.2",
@@ -32,6 +35,7 @@
35 "posthog-js": "^1.319.0",
36 "react": "^19.2.3",
37 "react-dom": "^19.2.3",
38 + "search-insights": "^2",
39 "shiki": "^3.20.0",
40 "tailwind-merge": "^3.4.0"
41 },
docs/scripts/sync-algolia-search.ts new
+102
@@ -0,0 +1,102 @@
1 +import { algoliasearch } from 'algoliasearch';
2 +import {
3 + ALGOLIA_DEFAULT_APP_ID,
4 + ALGOLIA_DEFAULT_INDEX_NAME,
5 + getAlgoliaSearchDocuments,
6 +} from '@/lib/search-index';
7 +
8 +function requireEnv(name: string, fallback?: string): string {
9 + const value = process.env[name] ?? fallback;
10 + if (!value) {
11 + throw new Error(`Missing required environment variable: ${name}`);
12 + }
13 + return value;
14 +}
15 +
16 +const dryRun = process.argv.includes('--dry-run');
17 +const showSamples = process.argv.includes('--samples');
18 +const indexName =
19 + process.env.ALGOLIA_INDEX_NAME ?? process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? ALGOLIA_DEFAULT_INDEX_NAME;
20 +
21 +const records = await getAlgoliaSearchDocuments();
22 +
23 +if (records.length === 0) {
24 + throw new Error('Refusing to sync an empty Algolia docs index.');
25 +}
26 +
27 +if (dryRun) {
28 + const uniquePages = new Set(records.map((record) => record.page_id)).size;
29 + console.log(`Prepared ${records.length} Algolia records across ${uniquePages} docs pages for index "${indexName}".`);
30 +
31 + if (showSamples) {
32 + console.log(JSON.stringify(records.slice(0, 5), null, 2));
33 + }
34 +
35 + process.exit(0);
36 +}
37 +
38 +const appId = requireEnv(
39 + 'ALGOLIA_APP_ID',
40 + process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? ALGOLIA_DEFAULT_APP_ID,
41 +);
42 +const adminApiKey = requireEnv('ALGOLIA_ADMIN_API_KEY');
43 +const client = algoliasearch(appId, adminApiKey);
44 +
45 +console.log(`Configuring Algolia index "${indexName}"...`);
46 +await client.setSettings({
47 + indexName,
48 + indexSettings: {
49 + attributeForDistinct: 'page_id',
50 + attributesToRetrieve: [
51 + 'objectID',
52 + 'title',
53 + 'description',
54 + 'section',
55 + 'content',
56 + 'url',
57 + 'section_id',
58 + 'breadcrumbs',
59 + 'page_id',
60 + 'type',
61 + ],
62 + searchableAttributes: [
63 + 'unordered(title)',
64 + 'unordered(keywords)',
65 + 'unordered(slug)',
66 + 'unordered(section)',
67 + 'unordered(headings)',
68 + 'unordered(tool_names)',
69 + 'unordered(tool_slugs)',
70 + 'unordered(description)',
71 + 'unordered(content)',
72 + ],
73 + customRanking: [
74 + 'desc(page_rank)',
75 + 'desc(toolkit_popularity)',
76 + 'desc(section_rank)',
77 + 'asc(position)',
78 + ],
79 + attributesForFaceting: [
80 + 'filterOnly(type)',
81 + 'filterOnly(lang)',
82 + 'searchable(tags)',
83 + ],
84 + attributesToHighlight: ['title', 'section', 'content'],
85 + attributesToSnippet: ['content:30'],
86 + highlightPreTag: '<mark>',
87 + highlightPostTag: '</mark>',
88 + ignorePlurals: true,
89 + minProximity: 1,
90 + removeStopWords: false,
91 + removeWordsIfNoResults: 'lastWords',
92 + typoTolerance: true,
93 + advancedSyntax: true,
94 + },
95 +});
96 +
97 +console.log(`Replacing ${records.length} records in Algolia index "${indexName}"...`);
98 +await client.replaceAllObjects({
99 + indexName,
100 + objects: records.map((record) => ({ ...record })),
101 +});
102 +console.log(`Synced Algolia docs search index "${indexName}".`);
docs/scripts/test-algolia-search.ts new
+56
@@ -0,0 +1,56 @@
1 +import { liteClient } from 'algoliasearch/lite';
2 +import type { AlgoliaDocsRecord } from '@/lib/search-index';
3 +import { ALGOLIA_DEFAULT_APP_ID, ALGOLIA_DEFAULT_INDEX_NAME } from '@/lib/search-index';
4 +
5 +function requireEnv(name: string, fallback?: string): string {
6 + const value = process.env[name] ?? fallback;
7 + if (!value) throw new Error(`Missing required environment variable: ${name}`);
8 + return value;
9 +}
10 +
11 +const appId = requireEnv('ALGOLIA_APP_ID', process.env.NEXT_PUBLIC_ALGOLIA_APP_ID ?? ALGOLIA_DEFAULT_APP_ID);
12 +const searchApiKey = requireEnv(
13 + 'ALGOLIA_SEARCH_API_KEY',
14 + process.env.NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY ?? process.env.ALGOLIA_ADMIN_API_KEY,
15 +);
16 +const indexName = process.env.ALGOLIA_INDEX_NAME ?? process.env.NEXT_PUBLIC_ALGOLIA_INDEX_NAME ?? ALGOLIA_DEFAULT_INDEX_NAME;
17 +const queries = process.argv.slice(2);
18 +
19 +if (queries.length === 0) {
20 + queries.push(
21 + 'quickstart',
22 + 'oauth auth config',
23 + 'create connected account',
24 + 'gmail send email',
25 + 'tool router session search',
26 + 'webhook verification',
27 + );
28 +}
29 +
30 +const client = liteClient(appId, searchApiKey);
31 +
32 +for (const query of queries) {
33 + const result = await client.searchForHits<AlgoliaDocsRecord>({
34 + requests: [
35 + {
36 + type: 'default',
37 + indexName,
38 + query,
39 + hitsPerPage: 5,
40 + distinct: true,
41 + getRankingInfo: true,
42 + clickAnalytics: true,
43 + },
44 + ],
45 + });
46 +
47 + const response = result.results[0];
48 + console.log(`\n## ${query}`);
49 + response.hits.forEach((hit, index) => {
50 + const section = hit.section ? ` — ${hit.section}` : '';
51 + const snippet = hit.content.replace(/\s+/g, ' ').slice(0, 180);
52 + console.log(`${index + 1}. ${hit.title}${section}`);
53 + console.log(` ${hit.url}`);
54 + console.log(` ${snippet}${snippet.length === 180 ? '…' : ''}`);
55 + });
56 +}