@devries-cameron20 / composio-1 / commits / d3b8f8dcc
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

ci: add label-triggered docs agent eval check (V1, informational)

Adding the run-docs-tests label to a PR runs an agent-buildability eval against the PR's Vercel docs preview: deterministic routing (no-eval / Smoke / Deep), isolated agents reading the proposed docs through a rewrite proxy, PASS/FAIL/ERROR + agent docs-feedback posted as a PR comment. Non-blocking; all eval logic lives in ComposioHQ/docs-agent-eval. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Soumya Medapati committed Aug 17, 2026 at 14:13 UTC d3b8f8dcc54a25dbcabe7ef9fbcd9b99f7a559bf
1 file changed +37
.github/workflows/docs-agent-eval.yml new
+37
@@ -0,0 +1,37 @@
1 +# Docs Agent Eval — label-triggered CI check (V1, informational)
2 +#
3 +# Add the `run-docs-tests` label to a docs PR to run an agent-buildability
4 +# eval against the PR's Vercel docs preview: a deterministic router picks
5 +# no-eval / Smoke (1 agent) / Deep (5 agents), fresh isolated agents follow
6 +# the proposed docs through a rewrite proxy, and the result is posted back
7 +# here as a PR comment. `run-docs-tests-deep` forces Deep.
8 +#
9 +# All eval logic lives in ComposioHQ/docs-agent-eval (reusable workflow) —
10 +# see its ARCHITECTURE.md. This check is informational and must NOT be made
11 +# a required status check in V1.
12 +#
13 +# Required repo/org secrets: COMPOSIO_ORG_API_KEY, DEEPSEEK_API_KEY,
14 +# VERCEL_TOKEN (see the PR description for provisioning notes).
15 +name: Docs Agent Eval
16 +
17 +on:
18 + pull_request:
19 + types: [labeled, synchronize, reopened]
20 +
21 +concurrency:
22 + group: docs-agent-eval-pr-${{ github.event.pull_request.number }}
23 + cancel-in-progress: true # newest commit wins; stale evals are superseded
24 +
25 +jobs:
26 + eval:
27 + if: contains(github.event.pull_request.labels.*.name, 'run-docs-tests')
28 + permissions:
29 + contents: read
30 + pull-requests: write # the result comment
31 + # Full-SHA pin required by org policy — bump on docs-agent-eval releases.
32 + uses: ComposioHQ/docs-agent-eval/.github/workflows/eval-pr.yml@88dec5bf58d3f77b598c03eec0e6a0c587a57554
33 + with:
34 + pr_number: ${{ github.event.pull_request.number }}
35 + mode: ${{ contains(github.event.pull_request.labels.*.name, 'run-docs-tests-deep') && 'Deep' || 'Auto' }}
36 + post_comment: true
37 + secrets: inherit