@devries-cameron20 / composio Mirror · read-only / commits / c0b71a5dc
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 16:00 UTC

ci: label-triggered docs agent eval (V1, informational) (#4158)

## What this adds A **label-triggered, non-blocking CI check** that tests whether a docs PR's proposed content can still guide a fresh coding agent to working Composio code — before merge, against the PR's own Vercel preview. Add the **`run-docs-tests`** label to a docs PR → 1. A **route job** decides if and for which PR to run: label present + PR author is an org member/collaborator. Two entry paths so either order works — label added after the preview built (`pull_request` events), or preview built after the label (`deployment_status` events, which also cover new pushes; `synchronize` is deliberately not used). 2. A deterministic router classifies the change: **no-eval** (assets only) / **Smoke** (1 agent) / **Deep** (5 agents — auth, install, env-var, code-sample, endpoint, or navigation changes). `run-docs-tests-deep` forces Deep. 3. The PR's exact-HEAD preview URL is resolved from **GitHub's deployment records** (written by the Vercel↔GitHub integration, read with the per-run `GITHUB_TOKEN`) — **no Vercel credentials anywhere** (decision 2026-08-19). Isolated agents get it served through a rewrite proxy — they believe they're on docs.composio.dev. 4. Each agent (fresh env, ephemeral Composio project, torn down after) attempts the weather-toolkit task by following the docs. 5. An **independent judge** verdicts every run before teardown — re-executes the agent's script with the run's live key and scans for the v1 anti-patterns (hardcoded slugs, legacy packages, manual execution, /examples cheating). The agent's own success claim is recorded but never trusted; the PR comment shows L1·L2·L3 per run and any judge-overruled self-reports. 6. Result lands here as a PR comment: **PASS / FAIL / ERROR**, run counts, teardown status, and the agents' own docs-friction feedback. New pushes supersede stale runs (`cancel-in-progress`). **This file is a router + one SHA-pinned call** — all eval logic lives in [ComposioHQ/docs-agent-eval-ci](https://github.com/ComposioHQ/docs-agent-eval-ci) (reusable workflow, SHA-pinned per org policy). See its [ARCHITECTURE.md](https://github.com/ComposioHQ/docs-agent-eval-ci/blob/main/ARCHITECTURE.md) for the full component map and design-spec compliance audit. ## Evidence it works (sandbox runs) - End-to-end green Smoke, auto-routed: [run](https://github.com/ComposioHQ/docs-agent-eval-ci/actions/runs/32056739696) (PR #4125 → smoke → PASS 1/1) - End-to-end green Deep, auto-routed: [run](https://github.com/ComposioHQ/docs-agent-eval-ci/actions/runs/32057336110) (PR #4136 → deep via auth-docs signal → PASS 5/5, incl. agent feedback reproducing known docs findings) - The exact `workflow_call` mechanism this shim uses: [run](https://github.com/ComposioHQ/docs-agent-eval-ci/actions/runs/32059660256) - Receipts-based preview resolution live-verified on PR #4174 (exact-SHA → snapshot URL); fork PRs (no Vercel build) correctly produce the explained no-preview ERROR path - Builder (DeepSeek V4 Flash) calibrated 16/16 against production docs before any CI verdicts ## ⚠️ Before merge (repo admin) 1. ~~Secrets~~ **Done 2026-08-19**: `COMPOSIO_ORG_API_KEY` + `DEEPSEEK_API_KEY` exist in this repo's Actions secrets. (`VERCEL_TOKEN` is no longer needed — removed from the design.) Team-owned replacements for the individually-owned keys remain part of the rollout (current owner offboards Aug 31). 2. **Engine repo visibility**: `docs-agent-eval-ci` is currently *internal*; a public repo cannot call an internal reusable workflow, so it must be flipped public (pre-publication scrub done) **before** this merges. 3. **Pin bump**: after the engine's `no-vercel-credentials` PR merges, the `uses:` SHA here gets re-pinned to the engine's merge commit. Labels `run-docs-tests` / `run-docs-tests-deep` are already created. ## Non-goals (V1) Informational only — **do not mark this check required**. No merge blocking, no statistical thresholds, no LLM routing; scenario is the weather-toolkit eval only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Update 2026-08-21: keep the engine PRIVATE via a GitHub App Design change (Sarah): the engine repo stays **private** instead of going public. A public repo can't `uses:` a private repo's reusable workflow, so the `eval` job now runs the engine **inline**: 1. mints a short-lived **GitHub App token** (`actions/create-github-app-token`, SHA-pinned) scoped read-only to `docs-agent-eval-ci` 2. checks the private engine out with it (`path: engine/`) 3. runs the engine on the runner (`route_pr.py` → `run_full_eval.py`); result comment + check status still posted with composio's built-in `GITHUB_TOKEN` **⚠️ This supersedes the earlier public-reusable-workflow design in this PR — needs a fresh review of the `eval` job.** New repo secrets required (added): `DOCS_EVAL_APP_ID`, `DOCS_EVAL_APP_PRIVATE_KEY`, plus existing `COMPOSIO_ORG_API_KEY`, `DEEPSEEK_API_KEY`. No Vercel credentials. Before this can be tested: the GitHub App must be **installed** on ComposioHQ with **Contents: Read** on `docs-agent-eval-ci`. Since the check only fires from the default branch, testing happens after this merges (informational check, non-blocking, safe to merge to test).

Soumya Medapati committed Aug 21, 2026 at 13:13 UTC c0b71a5dc507512d1f98039a34d8a77ce07ee928

No changes.