Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

next
md 97 lines 4.8 KB
Rendered Raw
1 # KB Freshness Verification
2
3 ## Decision
4
5 Verify published knowledge base guides mechanically against production data on
6 every catalog refresh, rather than relying on the `reviewAfter` calendar alone.
7
8 `docs-update-data.yml` already refreshes `public/data/toolkits.json` and the
9 OpenAPI specs every five hours and on every Apollo production deploy. A new
10 `verify:kb` step runs immediately after that refresh and checks each published
11 guide's mechanical claims against the snapshot the docs site is about to ship.
12
13 Findings never block the data sync. They are written to the job summary and to a
14 single long-lived `kb-freshness` issue, which the job closes when the KB verifies
15 clean.
16
17 An evergreen guide's review deadline is a maintenance reminder: expiry remains
18 an error in this report, but does not block catalog assembly or unpublish the
19 guide. A published time-sensitive guide with an expired review window still
20 blocks catalog assembly until it is reverified or explicitly held for review.
21 Both kinds of published guide must carry valid verification and review dates.
22
23 ## Context
24
25 The KB carries two kinds of claim. Most are mechanically checkable — a tool slug
26 exists, a toolkit is in the catalog, a provider doc link resolves. The rest need
27 human judgement.
28
29 Review dates treat both kinds identically, which has two costs. Checkable claims
30 wait months for a human to confirm something a script can confirm in seconds, and
31 the human queue fills with work that did not need a human. An audit of the source
32 corpus found 36 of 180 cited action identifiers already absent from the catalog;
33 this check found one of them — `CANVAS_GET_ACCOUNTS` — live in a published guide.
34
35 Review dates used to fail late. Every `reviewAfter` expiry threw during catalog
36 assembly, so one overdue evergreen guide broke the docs build. On 2026-11-10,
37 61 evergreen guides expired together and blocked unrelated SDK pull requests
38 during docs installation. Evergreen deadlines now stay in the verifier report;
39 time-sensitive expiry still blocks publication. The verifier also reports
40 guides sharing a review date so the maintenance queue is visible before expiry.
41
42 ## What Is Checked
43
44 | Check | Source of truth | Severity |
45 | --- | --- | --- |
46 | Cited tool and trigger slugs exist | `public/data/toolkits.json` | error |
47 | Linked toolkits exist | `public/data/toolkits.json` | error |
48 | Provider doc links resolve | live HTTP, `--check-links` | error on 404/410 |
49 | Review window not expired | manifest | error |
50 | Guides sharing one `reviewAfter` date | manifest | warning |
51 | Verification age past threshold | manifest | warning |
52
53 Three exclusions keep the report trustworthy enough to act on. A permanently red
54 report is one nobody reads.
55
56 - **Placeholders.** `<COMPOSIO_API_KEY>` and `$TOKEN` share the shape of a tool
57 slug and will never appear in a catalog. A token is treated as a placeholder
58 only when every appearance is wrapped.
59 - **Bare URLs.** Only markdown link targets are probed. Bare URLs in prose and
60 code samples are API hosts (`https://api.ahrefs.com/v3`) where 401 or 404 is
61 the correct response.
62 - **Rate limiting.** 5xx and 429 mean the provider is throttling the probe, not
63 that the page is gone.
64
65 A guide whose subject is a removed identifier declares it in
66 `verifyIgnoreToolSlugs`, so the exemption is reviewable in the manifest rather
67 than hidden in the checker.
68
69 ## `kb/source` Is A Verbatim Snapshot
70
71 The first publish tranche shipped before `articlePath` existed, so those guides
72 were published by rewriting the prose in place inside `kb/source` — turning
73 support-agent voice into customer voice. That silently made the snapshot a
74 derivative of the commit it claims to come from: nine of twenty-five sources no
75 longer matched upstream.
76
77 Provenance is only meaningful if the snapshot is verbatim, and drift detection is
78 impossible against a baseline that was already edited — every future comparison
79 would report permanent false drift on those nine files.
80
81 Those guides now carry `articlePath` like the rest, and their sources are
82 restored to the upstream bytes. Published output is unchanged; the article body
83 renders exactly the prose the source previously supplied. A regression test
84 asserts no published guide renders from the source snapshot.
85
86 ## Consequences
87
88 - Checkable claims are verified on every production deploy instead of quarterly.
89 - Human review narrows to prose that cannot be checked mechanically.
90 - A guide contradicting production is visible within hours, in one tracked issue
91 that closes itself.
92 - The checker must keep working when time-sensitive expiry has tripped the build
93 gate, so it assembles the catalog with a floor date and re-checks all review
94 deadlines against the real clock.
95 - Coverage — questions with no article at all — is out of scope here. That is a
96 demand-side signal and is measured from support ticket volume, not from the
97 catalog.