@devries-cameron20 / composio-1 / commits / 2367b80d9
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

chore(ci): enforce agent guidance validators in CI (#4447)

This PR: - Add `.github/workflows/agent-substrate.yml` running `pnpm validate:agent-skills` and `pnpm validate:skill-routing` on every push and pull request; both validators previously ran in no CI workflow - No path filters on the trigger: the stale-guidance walk scans every text file in the repo, so any change can affect the result (PR runs restore caches but only `next` pushes save them, per the `setup-node-pnpm-bun` guidance) - Skip `vendor/` directories in the `validate:agent-skills` stale-guidance walk, which was failing on read-only third-party snapshots mentioning other tools' rule conventions - Extend the validator's command scan to `CONTRIBUTING.md` (with a `pnpm dlx` exemption), so its documented commands are checked against `package.json`, `python/Makefile`, and `python/noxfile.py` like the rest of the guidance - Point the routing-test header, root `AGENTS.md`, and `skill-maintenance` reference docs at the new workflow, and add a "Working with AI Coding Agents" section to `CONTRIBUTING.md` covering the inherited agent setup, the two checks, and the routing-probe requirement for skill edits ## Context These two validators are the only checks keeping repo-level agent guidance honest: command names mentioned in guidance are verified against `package.json`, `python/Makefile`, and `python/noxfile.py`, and routing probes assert each skill stays the unique top match for its representative task. Until now nothing enforced either one, and the stale-guidance walk was already red on vendored trees — a failure no guidance owner could fix, which trains people to ignore the check. This makes both checks blocking everywhere they can bite. ## Verification - `pnpm validate:agent-skills` — 19 skills, green, now including `CONTRIBUTING.md` commands - `pnpm validate:skill-routing` — 19 probes over 19 skills, green - Workflow YAML parsed; oxlint and prettier clean on touched files - `Agent Substrate` workflow ran green on this PR (42s) before the trigger change and re-runs on every push

Alberto Schiabel committed Sep 11, 2026 at 17:31 UTC 2367b80d9dde2c6201b1e62e198aeb2ac3a3e4bb
6 files changed +95 -3
.agents/skills/skill-maintenance/references/skill-format.md
+2
@@ -41,6 +41,8 @@ pnpm validate:agent-skills
41 pnpm validate:skill-routing
42 ```
43
44 +Both validators are deterministic, and `.github/workflows/agent-substrate.yml` runs them in CI on every push and pull request — the stale-guidance scan covers the whole repo, so any change can affect the result.
45 +
46 `validate:agent-skills` checks frontmatter, names, reference links, compatibility symlink state, stale guidance references, and known command names.
47
48 `validate:skill-routing` is a deterministic routing smoke test (`ts/scripts/test-skill-routing.mjs`): for representative tasks it asserts the expected skill is the unique top match by distinctive trigger phrases, and fails if any skill has no probe. When you add or rename a skill, or rewrite a `description`, add or update its probe so routing stays covered.
.github/workflows/agent-substrate.yml new
+58
@@ -0,0 +1,58 @@
1 +name: Agent Substrate
2 +
3 +# Guards the repo-level AI-agent guidance — the files coding agents read —
4 +# so it cannot silently drift from reality:
5 +#
6 +# * `pnpm validate:agent-skills` — SKILL.md frontmatter and reference links,
7 +# the .agents/skills <-> .claude/skills compatibility symlink, required
8 +# nested AGENTS.md files, stale guidance references, and every pnpm/make/
9 +# nox command mentioned in guidance checked against its real definition.
10 +# * `pnpm validate:skill-routing` — deterministic routing smoke test: each
11 +# skill keeps a probe asserting it is the unique top match for a
12 +# representative task, so description edits cannot silently break routing.
13 +#
14 +# These checks run nowhere else, so this workflow is their only enforcement
15 +# point. It runs on every push and PR with no path filters: the
16 +# validate:agent-skills stale-guidance walk scans every text file in the
17 +# repo, so any change can affect the result. Both scripts are
18 +# dependency-free Node programs, but the standard toolchain setup is kept
19 +# so the pnpm script wiring is exercised too.
20 +
21 +on:
22 + push:
23 + branches: [master, next]
24 + pull_request:
25 + branches: [master, next]
26 +
27 +concurrency:
28 + group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
29 + cancel-in-progress: ${{ github.event_name == 'pull_request' }}
30 +
31 +permissions:
32 + contents: read
33 +
34 +jobs:
35 + validate:
36 + name: Validate agent guidance
37 + runs-on: ubuntu-latest
38 +
39 + steps:
40 + - name: Checkout Code
41 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
42 +
43 + - name: Setup Node.js, pnpm, Bun
44 + uses: ./.github/actions/setup-node-pnpm-bun
45 + with:
46 + # Only push runs on next save caches: PR-scoped caches are invisible
47 + # to other branches, so saving them costs post-job time and evicts
48 + # reusable default-branch caches from the 10 GB repo quota.
49 + cache-save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/next' }}
50 +
51 + - name: Install dependencies
52 + run: pnpm install --frozen-lockfile
53 +
54 + - name: Validate agent skills and guidance invariants
55 + run: pnpm validate:agent-skills
56 +
57 + - name: Validate skill routing
58 + run: pnpm validate:skill-routing
AGENTS.md
+2
@@ -73,6 +73,8 @@ pnpm validate:agent-skills
73 pnpm validate:skill-routing
74 ```
75
76 +Both validators are dependency-free and deterministic. `.github/workflows/agent-substrate.yml` runs them in CI on every push and pull request — the stale-guidance scan covers the whole repo, so any change can affect the result.
77 +
78 Python commands run from `python/` unless otherwise noted:
79
80 ```bash
CONTRIBUTING.md
+19 -1
@@ -7,6 +7,7 @@ Thank you for your interest in contributing to Composio. This guide covers the r
7 - [Development Setup](#development-setup)
8 - [Project Structure](#project-structure)
9 - [Development Commands](#development-commands)
10 +- [Working with AI Coding Agents](#working-with-ai-coding-agents)
11 - [Coding Standards](#coding-standards)
12 - [Documentation Requirements](#documentation-requirements)
13 - [Pull Request Process](#pull-request-process)
@@ -34,7 +35,7 @@ Use [mise](https://mise.jdx.dev) to install the toolchain:
35 mise install
36 ```
37
37 -pnpm is installed through mise's npm backend. Do not rely on Corepack for this repository.
38 +mise installs pnpm through its npm backend. Do not rely on Corepack for this repository.
39
40 ### Getting Started
41
@@ -149,6 +150,23 @@ import-map targets), so treat their output as advisory: verify a finding is
150 truly unreferenced before deleting, and suppress confirmed false positives via
151 `knip.json` / `vulture_allowlist.py`.
152
153 +## Working with AI Coding Agents
154 +
155 +This repository ships its own agent guidance, and CI keeps it honest. You get it for free — an agent that reads this repo inherits the layout, commands, and guardrails without setup.
156 +
157 +`AGENTS.md` files live at the root and inside each subtree (`ts/`, `python/`, `docs/`, and the packages). Coding agents read the nearest one automatically, so you usually do not need to do anything beyond keeping them accurate when you move code. The canonical skill tree is `.agents/skills/` (with `.claude/skills` as a compatibility symlink): focused, task-scoped skills that route an agent to the right workflow, from `bug-fixing` to `cli-release`.
158 +
159 +Two deterministic checks guard this guidance:
160 +
161 +```bash
162 +pnpm validate:agent-skills # frontmatter, reference links, stale guidance refs, command names
163 +pnpm validate:skill-routing # routing smoke test over skill descriptions
164 +```
165 +
166 +`validate:agent-skills` parses `package.json`, `python/Makefile`, and `python/noxfile.py`, then verifies every command mentioned in guidance actually exists — so guidance cannot recommend a command that was renamed away. Both checks run in CI via `.github/workflows/agent-substrate.yml` whenever agent guidance changes.
167 +
168 +If you add or rename a skill, or rewrite a skill description, run both checks and add a routing probe in `ts/scripts/test-skill-routing.mjs` so routing stays covered. To author or edit a skill, read [`.agents/skills/skill-maintenance/SKILL.md`](.agents/skills/skill-maintenance/SKILL.md) first.
169 +
170 ## Coding Standards
171
172 ### TypeScript
ts/scripts/test-skill-routing.mjs
+2 -1
@@ -9,7 +9,8 @@
9 // obvious match, or another skill grows ambiguous overlap. It also fails if a
10 // skill has no probe, so routing coverage tracks the taxonomy.
11 //
12 -// Run: pnpm validate:skill-routing (belongs in the verify/CI aggregate too).
12 +// Run: pnpm validate:skill-routing (enforced in CI by
13 +// .github/workflows/agent-substrate.yml).
14
15 import fs from 'node:fs';
16 import path from 'node:path';
ts/scripts/validate-agent-skills.mjs
+12 -1
@@ -235,6 +235,15 @@ const ignoredDirs = new Set([
235 '.venv',
236 'dist',
237 'build',
238 + // Vendored trees (see AGENTS.md "Generated And Vendored Paths") are read-only
239 + // third-party snapshots. Their docs legitimately mention other tools'
240 + // rule-file conventions, so the stale-guidance walk must not police them —
241 + // only first-party guidance files are in scope. Invariant: every `vendor`
242 + // directory in this repo is third-party; revisit this skip if a first-party
243 + // one ever appears.
244 + // (Keep this comment free of the stalePatterns literals: this file is
245 + // itself scanned by the walk below.)
246 + 'vendor',
247 ]);
248
249 const ignoredFiles = new Set(['GOAL.md', 'PLAN.md', 'HANDOFF.md', 'RELEASE_NOTES.md']);
@@ -313,7 +322,8 @@ const validatePnpmCommands = (relativePath, text) => {
322 command.startsWith('-') ||
323 command === 'install' ||
324 command === 'exec' ||
316 - command === 'turbo'
325 + command === 'turbo' ||
326 + command === 'dlx'
327 ) {
328 continue;
329 }
@@ -407,6 +417,7 @@ const commandFiles = [
417 return files;
418 }),
419 ...requiredAgentFiles.map(relativePath => path.join(repoRoot, relativePath)),
420 + path.join(repoRoot, 'CONTRIBUTING.md'),
421 ];
422
423 for (const absolutePath of commandFiles) {