Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 03:37 UTC
KB Freshness Verification
Decision
Verify published knowledge base guides mechanically against production data on
every catalog refresh, rather than relying on the reviewAfter calendar alone.
docs-update-data.yml already refreshes public/data/toolkits.json and the
OpenAPI specs every five hours and on every Apollo production deploy. A new
verify:kb step runs immediately after that refresh and checks each published
guide's mechanical claims against the snapshot the docs site is about to ship.
Findings never block the data sync. They are written to the job summary and to a
single long-lived kb-freshness issue, which the job closes when the KB verifies
clean.
An evergreen guide's review deadline is a maintenance reminder: expiry remains an error in this report, but does not block catalog assembly or unpublish the guide. A published time-sensitive guide with an expired review window still blocks catalog assembly until it is reverified or explicitly held for review. Both kinds of published guide must carry valid verification and review dates.
Context
The KB carries two kinds of claim. Most are mechanically checkable — a tool slug exists, a toolkit is in the catalog, a provider doc link resolves. The rest need human judgement.
Review dates treat both kinds identically, which has two costs. Checkable claims
wait months for a human to confirm something a script can confirm in seconds, and
the human queue fills with work that did not need a human. An audit of the source
corpus found 36 of 180 cited action identifiers already absent from the catalog;
this check found one of them — CANVAS_GET_ACCOUNTS — live in a published guide.
Review dates used to fail late. Every reviewAfter expiry threw during catalog
assembly, so one overdue evergreen guide broke the docs build. On 2026-11-10,
61 evergreen guides expired together and blocked unrelated SDK pull requests
during docs installation. Evergreen deadlines now stay in the verifier report;
time-sensitive expiry still blocks publication. The verifier also reports
guides sharing a review date so the maintenance queue is visible before expiry.
What Is Checked
| Check | Source of truth | Severity |
|---|---|---|
| Cited tool and trigger slugs exist | public/data/toolkits.json |
error |
| Linked toolkits exist | public/data/toolkits.json |
error |
| Provider doc links resolve | live HTTP, --check-links |
error on 404/410 |
| Review window not expired | manifest | error |
Guides sharing one reviewAfter date |
manifest | warning |
| Verification age past threshold | manifest | warning |
Three exclusions keep the report trustworthy enough to act on. A permanently red report is one nobody reads.
- Placeholders.
<COMPOSIO_API_KEY>and$TOKENshare the shape of a tool slug and will never appear in a catalog. A token is treated as a placeholder only when every appearance is wrapped. - Bare URLs. Only markdown link targets are probed. Bare URLs in prose and
code samples are API hosts (
https://api.ahrefs.com/v3) where 401 or 404 is the correct response. - Rate limiting. 5xx and 429 mean the provider is throttling the probe, not that the page is gone.
A guide whose subject is a removed identifier declares it in
verifyIgnoreToolSlugs, so the exemption is reviewable in the manifest rather
than hidden in the checker.
kb/source Is A Verbatim Snapshot
The first publish tranche shipped before articlePath existed, so those guides
were published by rewriting the prose in place inside kb/source — turning
support-agent voice into customer voice. That silently made the snapshot a
derivative of the commit it claims to come from: nine of twenty-five sources no
longer matched upstream.
Provenance is only meaningful if the snapshot is verbatim, and drift detection is impossible against a baseline that was already edited — every future comparison would report permanent false drift on those nine files.
Those guides now carry articlePath like the rest, and their sources are
restored to the upstream bytes. Published output is unchanged; the article body
renders exactly the prose the source previously supplied. A regression test
asserts no published guide renders from the source snapshot.
Consequences
- Checkable claims are verified on every production deploy instead of quarterly.
- Human review narrows to prose that cannot be checked mechanically.
- A guide contradicting production is visible within hours, in one tracked issue that closes itself.
- The checker must keep working when time-sensitive expiry has tripped the build gate, so it assembles the catalog with a floor date and re-checks all review deadlines against the real clock.
- Coverage — questions with no article at all — is out of scope here. That is a demand-side signal and is measured from support ticket volume, not from the catalog.