Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 16:00 UTC
ci: add label-triggered docs agent eval check (V1, informational)
Adding the run-docs-tests label to a PR runs an agent-buildability eval against the PR's Vercel docs preview: deterministic routing (no-eval / Smoke / Deep), isolated agents reading the proposed docs through a rewrite proxy, PASS/FAIL/ERROR + agent docs-feedback posted as a PR comment. Non-blocking; all eval logic lives in ComposioHQ/docs-agent-eval. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Soumya Medapati committed
Aug 17, 2026 at 14:13 UTC
d3b8f8dcc54a25dbcabe7ef9fbcd9b99f7a559bf
1 file changed
+37
.github/workflows/docs-agent-eval.yml
new
+37
@@ -0,0 +1,37 @@
1
+# Docs Agent Eval — label-triggered CI check (V1, informational)
2
+#
3
+# Add the `run-docs-tests` label to a docs PR to run an agent-buildability
4
+# eval against the PR's Vercel docs preview: a deterministic router picks
5
+# no-eval / Smoke (1 agent) / Deep (5 agents), fresh isolated agents follow
6
+# the proposed docs through a rewrite proxy, and the result is posted back
7
+# here as a PR comment. `run-docs-tests-deep` forces Deep.
8
+#
9
+# All eval logic lives in ComposioHQ/docs-agent-eval (reusable workflow) —
10
+# see its ARCHITECTURE.md. This check is informational and must NOT be made
11
+# a required status check in V1.
12
+#
13
+# Required repo/org secrets: COMPOSIO_ORG_API_KEY, DEEPSEEK_API_KEY,
14
+# VERCEL_TOKEN (see the PR description for provisioning notes).
15
+name: Docs Agent Eval
16
+
17
+on:
18
+ pull_request:
19
+ types: [labeled, synchronize, reopened]
20
+
21
+concurrency:
22
+ group: docs-agent-eval-pr-${{ github.event.pull_request.number }}
23
+ cancel-in-progress: true # newest commit wins; stale evals are superseded
24
+
25
+jobs:
26
+ eval:
27
+ if: contains(github.event.pull_request.labels.*.name, 'run-docs-tests')
28
+ permissions:
29
+ contents: read
30
+ pull-requests: write # the result comment
31
+ # Full-SHA pin required by org policy — bump on docs-agent-eval releases.
32
+ uses: ComposioHQ/docs-agent-eval/.github/workflows/eval-pr.yml@88dec5bf58d3f77b598c03eec0e6a0c587a57554
33
+ with:
34
+ pr_number: ${{ github.event.pull_request.number }}
35
+ mode: ${{ contains(github.event.pull_request.labels.*.name, 'run-docs-tests-deep') && 'Deep' || 'Auto' }}
36
+ post_comment: true
37
+ secrets: inherit