Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 06:00 UTC
| 1 | # Teach `composio dev playground-execute` for developer projects |
| 2 | |
| 3 | **Decision: kept.** On developer tasks the skill sent agents to the root `composio execute`, which acts on the consumer project. The developer reference now says how a tool runs inside a developer project. |
| 4 | |
| 5 | ## What the runs showed |
| 6 | |
| 7 | Baseline, Claude Code 2.1.310 with `claude-haiku-5-5`, 5 runs per journey and arm (`…-haiku-5-5-baseline`): |
| 8 | |
| 9 | - `developer-auth-configs-and-execute`: 3/5 with the skill, 5/5 without it. Every run with the skill took the root tree (wrong tree 100%). |
| 10 | - `developer-execute-unconnected`: 0/5 with the skill, wrong tree 100%. |
| 11 | |
| 12 | The transcripts show the cause. The agent loads the skill, opens `references/composio-dev.md`, runs `composio dev auth-configs list` correctly, then runs the tool with `composio execute`. The reference never named `composio dev playground-execute`; its only words on executing were "if the user only wants to connect and execute tools as an end user, return to the top-level workflow", and the skill's first step is "start with `composio execute`". |
| 13 | |
| 14 | ## The change |
| 15 | |
| 16 | `ts/packages/cli/skills-src/composio-cli/`: |
| 17 | |
| 18 | - `references/composio-dev.ts`: a new section, "Run Tools In A Developer Project". It says that root `execute` and `link` act on your own account in the consumer project, names `composio dev playground-execute` (same `-d`, `--get-schema`, `--dry-run`), and names `composio dev connected-accounts link <toolkit>` for a missing connection. The examples use Gmail, not the journeys' Slack. The closing sentence now reads "act on their own account rather than in a developer project". |
| 19 | - `index.ts`: one sentence after the pointer to the reference: "In a developer project, run tools with `composio dev playground-execute`, not `composio execute`." |
| 20 | |
| 21 | ## Results |
| 22 | |
| 23 | Skill arm, baseline → change. Same harness (`9ff7364caa93`), skill `72d9d691700d` → `f6f4d7820d90`. |
| 24 | |
| 25 | Haiku, train (`…-dev-playground-execute`): |
| 26 | |
| 27 | | journey | pass@1 | pass^5 | wrong tree | |
| 28 | | ----------------------------- | --------- | ------- | ---------- | |
| 29 | | consumer-slack-message | 40% → 80% | 0% → 0% | 0% → 0% | |
| 30 | | developer-execute-unconnected | 0% → 0% | 0% → 0% | 100% → 0% | |
| 31 | | first-run-login | 40% → 80% | 0% → 0% | 0% → 0% | |
| 32 | |
| 33 | Haiku, holdout (`…-dev-playground-execute-holdout`): |
| 34 | |
| 35 | | journey | pass@1 | pass^5 | wrong tree | |
| 36 | | ---------------------------------- | ----------- | ----------- | ---------- | |
| 37 | | developer-auth-configs-and-execute | 60% → 100% | 0% → 100% | 100% → 0% | |
| 38 | | run-inline-script | 100% → 100% | 100% → 100% | 0% → 0% | |
| 39 | |
| 40 | Opus, the two developer journeys, 3 runs each (`…-opus-baseline` → `…-opus-dev-playground-execute`): |
| 41 | |
| 42 | | journey | pass@1 | pass^3 | mean cost | |
| 43 | | ---------------------------------- | ----------- | ----------- | --------------- | |
| 44 | | developer-execute-unconnected | 0% → 100% | 0% → 100% | $0.084 → $0.079 | |
| 45 | | developer-auth-configs-and-execute | 100% → 100% | 100% → 100% | $0.118 → $0.049 | |
| 46 | |
| 47 | Reading them: |
| 48 | |
| 49 | - The wrong-tree rate on the developer journeys went from 100% to 0% for Haiku, in train and in holdout. This is what the change targets. |
| 50 | - Haiku still fails `developer-execute-unconnected` after the change, now on the right tree: `composio dev connected-accounts link slack --no-browser` prints nothing while it waits, so the agent has no URL to give the person. That is a CLI bug (the URL goes through terminal decoration that is dropped off a TTY), not a skill problem. Opus got past it with `--no-browser --no-wait`. |
| 51 | - The +40 on `consumer-slack-message` and `first-run-login` is 2/5 → 4/5, inside the 95% intervals of both batches. The change does not touch those paths; treat it as noise. |
| 52 | - Opus did not take the wrong tree before the change either; with it, Opus finds the developer path in fewer commands (two instead of three to six on `developer-auth-configs-and-execute`). |
| 53 | |
| 54 | Spend: $0.12 (Haiku baseline, both arms), $0.06 (Haiku with the change, both splits), $0.99 (Opus, both sides). |
| 55 | |
| 56 | ## Repeat it |
| 57 | |
| 58 | From `ts/e2e-tests/cli/agent-journeys`, with the key loaded (see [HILLCLIMB.md](../../HILLCLIMB.md)): |
| 59 | |
| 60 | ```sh |
| 61 | # The kept comparisons |
| 62 | bun run eval.ts compare evals/2026-11-11-dev-playground-execute/2026-11-11T17-53-36-claude-code-claude-haiku-5-5-baseline \ |
| 63 | evals/2026-11-11-dev-playground-execute/2026-11-11T18-14-14-claude-code-claude-haiku-5-5-dev-playground-execute |
| 64 | bun run eval.ts compare evals/2026-11-11-dev-playground-execute/2026-11-11T17-53-36-claude-code-claude-haiku-5-5-baseline \ |
| 65 | evals/2026-11-11-dev-playground-execute/2026-11-11T18-24-23-claude-code-claude-haiku-5-5-dev-playground-execute-holdout |
| 66 | |
| 67 | # Measure the current checkout against the kept result |
| 68 | bun run eval.ts run --model claude-haiku-5-5 --runs 5 --arms skill --label recheck |
| 69 | bun run eval.ts compare evals/2026-11-11-dev-playground-execute/2026-11-11T18-14-14-claude-code-claude-haiku-5-5-dev-playground-execute <new batch id> |
| 70 | ``` |
| 71 | |
| 72 | The baseline batches ran with the skill source at the parent of the commit that made this change. |