@devries-cameron20 / composio Mirror · read-only
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 06:00 UTC

next
md 72 lines 5.07 KB
Rendered Raw
1 # Teach `composio dev playground-execute` for developer projects
2
3 **Decision: kept.** On developer tasks the skill sent agents to the root `composio execute`, which acts on the consumer project. The developer reference now says how a tool runs inside a developer project.
4
5 ## What the runs showed
6
7 Baseline, Claude Code 2.1.310 with `claude-haiku-5-5`, 5 runs per journey and arm (`…-haiku-5-5-baseline`):
8
9 - `developer-auth-configs-and-execute`: 3/5 with the skill, 5/5 without it. Every run with the skill took the root tree (wrong tree 100%).
10 - `developer-execute-unconnected`: 0/5 with the skill, wrong tree 100%.
11
12 The transcripts show the cause. The agent loads the skill, opens `references/composio-dev.md`, runs `composio dev auth-configs list` correctly, then runs the tool with `composio execute`. The reference never named `composio dev playground-execute`; its only words on executing were "if the user only wants to connect and execute tools as an end user, return to the top-level workflow", and the skill's first step is "start with `composio execute`".
13
14 ## The change
15
16 `ts/packages/cli/skills-src/composio-cli/`:
17
18 - `references/composio-dev.ts`: a new section, "Run Tools In A Developer Project". It says that root `execute` and `link` act on your own account in the consumer project, names `composio dev playground-execute` (same `-d`, `--get-schema`, `--dry-run`), and names `composio dev connected-accounts link <toolkit>` for a missing connection. The examples use Gmail, not the journeys' Slack. The closing sentence now reads "act on their own account rather than in a developer project".
19 - `index.ts`: one sentence after the pointer to the reference: "In a developer project, run tools with `composio dev playground-execute`, not `composio execute`."
20
21 ## Results
22
23 Skill arm, baseline → change. Same harness (`9ff7364caa93`), skill `72d9d691700d` → `f6f4d7820d90`.
24
25 Haiku, train (`…-dev-playground-execute`):
26
27 | journey | pass@1 | pass^5 | wrong tree |
28 | ----------------------------- | --------- | ------- | ---------- |
29 | consumer-slack-message | 40% → 80% | 0% → 0% | 0% → 0% |
30 | developer-execute-unconnected | 0% → 0% | 0% → 0% | 100% → 0% |
31 | first-run-login | 40% → 80% | 0% → 0% | 0% → 0% |
32
33 Haiku, holdout (`…-dev-playground-execute-holdout`):
34
35 | journey | pass@1 | pass^5 | wrong tree |
36 | ---------------------------------- | ----------- | ----------- | ---------- |
37 | developer-auth-configs-and-execute | 60% → 100% | 0% → 100% | 100% → 0% |
38 | run-inline-script | 100% → 100% | 100% → 100% | 0% → 0% |
39
40 Opus, the two developer journeys, 3 runs each (`…-opus-baseline` → `…-opus-dev-playground-execute`):
41
42 | journey | pass@1 | pass^3 | mean cost |
43 | ---------------------------------- | ----------- | ----------- | --------------- |
44 | developer-execute-unconnected | 0% → 100% | 0% → 100% | $0.084 → $0.079 |
45 | developer-auth-configs-and-execute | 100% → 100% | 100% → 100% | $0.118 → $0.049 |
46
47 Reading them:
48
49 - The wrong-tree rate on the developer journeys went from 100% to 0% for Haiku, in train and in holdout. This is what the change targets.
50 - Haiku still fails `developer-execute-unconnected` after the change, now on the right tree: `composio dev connected-accounts link slack --no-browser` prints nothing while it waits, so the agent has no URL to give the person. That is a CLI bug (the URL goes through terminal decoration that is dropped off a TTY), not a skill problem. Opus got past it with `--no-browser --no-wait`.
51 - The +40 on `consumer-slack-message` and `first-run-login` is 2/5 → 4/5, inside the 95% intervals of both batches. The change does not touch those paths; treat it as noise.
52 - Opus did not take the wrong tree before the change either; with it, Opus finds the developer path in fewer commands (two instead of three to six on `developer-auth-configs-and-execute`).
53
54 Spend: $0.12 (Haiku baseline, both arms), $0.06 (Haiku with the change, both splits), $0.99 (Opus, both sides).
55
56 ## Repeat it
57
58 From `ts/e2e-tests/cli/agent-journeys`, with the key loaded (see [HILLCLIMB.md](../../HILLCLIMB.md)):
59
60 ```sh
61 # The kept comparisons
62 bun run eval.ts compare evals/2026-11-11-dev-playground-execute/2026-11-11T17-53-36-claude-code-claude-haiku-5-5-baseline \
63 evals/2026-11-11-dev-playground-execute/2026-11-11T18-14-14-claude-code-claude-haiku-5-5-dev-playground-execute
64 bun run eval.ts compare evals/2026-11-11-dev-playground-execute/2026-11-11T17-53-36-claude-code-claude-haiku-5-5-baseline \
65 evals/2026-11-11-dev-playground-execute/2026-11-11T18-24-23-claude-code-claude-haiku-5-5-dev-playground-execute-holdout
66
67 # Measure the current checkout against the kept result
68 bun run eval.ts run --model claude-haiku-5-5 --runs 5 --arms skill --label recheck
69 bun run eval.ts compare evals/2026-11-11-dev-playground-execute/2026-11-11T18-14-14-claude-code-claude-haiku-5-5-dev-playground-execute <new batch id>
70 ```
71
72 The baseline batches ran with the skill source at the parent of the commit that made this change.