Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
| 1 | # KB Freshness Verification |
| 2 | |
| 3 | ## Decision |
| 4 | |
| 5 | Verify published knowledge base guides mechanically against production data on |
| 6 | every catalog refresh, rather than relying on the `reviewAfter` calendar alone. |
| 7 | |
| 8 | `docs-update-data.yml` already refreshes `public/data/toolkits.json` and the |
| 9 | OpenAPI specs every five hours and on every Apollo production deploy. A new |
| 10 | `verify:kb` step runs immediately after that refresh and checks each published |
| 11 | guide's mechanical claims against the snapshot the docs site is about to ship. |
| 12 | |
| 13 | Findings never block the data sync. They are written to the job summary and to a |
| 14 | single long-lived `kb-freshness` issue, which the job closes when the KB verifies |
| 15 | clean. |
| 16 | |
| 17 | An evergreen guide's review deadline is a maintenance reminder: expiry remains |
| 18 | an error in this report, but does not block catalog assembly or unpublish the |
| 19 | guide. A published time-sensitive guide with an expired review window still |
| 20 | blocks catalog assembly until it is reverified or explicitly held for review. |
| 21 | Both kinds of published guide must carry valid verification and review dates. |
| 22 | |
| 23 | ## Context |
| 24 | |
| 25 | The KB carries two kinds of claim. Most are mechanically checkable — a tool slug |
| 26 | exists, a toolkit is in the catalog, a provider doc link resolves. The rest need |
| 27 | human judgement. |
| 28 | |
| 29 | Review dates treat both kinds identically, which has two costs. Checkable claims |
| 30 | wait months for a human to confirm something a script can confirm in seconds, and |
| 31 | the human queue fills with work that did not need a human. An audit of the source |
| 32 | corpus found 36 of 180 cited action identifiers already absent from the catalog; |
| 33 | this check found one of them — `CANVAS_GET_ACCOUNTS` — live in a published guide. |
| 34 | |
| 35 | Review dates used to fail late. Every `reviewAfter` expiry threw during catalog |
| 36 | assembly, so one overdue evergreen guide broke the docs build. On 2026-11-10, |
| 37 | 61 evergreen guides expired together and blocked unrelated SDK pull requests |
| 38 | during docs installation. Evergreen deadlines now stay in the verifier report; |
| 39 | time-sensitive expiry still blocks publication. The verifier also reports |
| 40 | guides sharing a review date so the maintenance queue is visible before expiry. |
| 41 | |
| 42 | ## What Is Checked |
| 43 | |
| 44 | | Check | Source of truth | Severity | |
| 45 | | --- | --- | --- | |
| 46 | | Cited tool and trigger slugs exist | `public/data/toolkits.json` | error | |
| 47 | | Linked toolkits exist | `public/data/toolkits.json` | error | |
| 48 | | Provider doc links resolve | live HTTP, `--check-links` | error on 404/410 | |
| 49 | | Review window not expired | manifest | error | |
| 50 | | Guides sharing one `reviewAfter` date | manifest | warning | |
| 51 | | Verification age past threshold | manifest | warning | |
| 52 | |
| 53 | Three exclusions keep the report trustworthy enough to act on. A permanently red |
| 54 | report is one nobody reads. |
| 55 | |
| 56 | - **Placeholders.** `<COMPOSIO_API_KEY>` and `$TOKEN` share the shape of a tool |
| 57 | slug and will never appear in a catalog. A token is treated as a placeholder |
| 58 | only when every appearance is wrapped. |
| 59 | - **Bare URLs.** Only markdown link targets are probed. Bare URLs in prose and |
| 60 | code samples are API hosts (`https://api.ahrefs.com/v3`) where 401 or 404 is |
| 61 | the correct response. |
| 62 | - **Rate limiting.** 5xx and 429 mean the provider is throttling the probe, not |
| 63 | that the page is gone. |
| 64 | |
| 65 | A guide whose subject is a removed identifier declares it in |
| 66 | `verifyIgnoreToolSlugs`, so the exemption is reviewable in the manifest rather |
| 67 | than hidden in the checker. |
| 68 | |
| 69 | ## `kb/source` Is A Verbatim Snapshot |
| 70 | |
| 71 | The first publish tranche shipped before `articlePath` existed, so those guides |
| 72 | were published by rewriting the prose in place inside `kb/source` — turning |
| 73 | support-agent voice into customer voice. That silently made the snapshot a |
| 74 | derivative of the commit it claims to come from: nine of twenty-five sources no |
| 75 | longer matched upstream. |
| 76 | |
| 77 | Provenance is only meaningful if the snapshot is verbatim, and drift detection is |
| 78 | impossible against a baseline that was already edited — every future comparison |
| 79 | would report permanent false drift on those nine files. |
| 80 | |
| 81 | Those guides now carry `articlePath` like the rest, and their sources are |
| 82 | restored to the upstream bytes. Published output is unchanged; the article body |
| 83 | renders exactly the prose the source previously supplied. A regression test |
| 84 | asserts no published guide renders from the source snapshot. |
| 85 | |
| 86 | ## Consequences |
| 87 | |
| 88 | - Checkable claims are verified on every production deploy instead of quarterly. |
| 89 | - Human review narrows to prose that cannot be checked mechanically. |
| 90 | - A guide contradicting production is visible within hours, in one tracked issue |
| 91 | that closes itself. |
| 92 | - The checker must keep working when time-sensitive expiry has tripped the build |
| 93 | gate, so it assembles the catalog with a floor date and re-checks all review |
| 94 | deadlines against the real clock. |
| 95 | - Coverage — questions with no article at all — is out of scope here. That is a |
| 96 | demand-side signal and is measured from support ticket volume, not from the |
| 97 | catalog. |