Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
| 1 | # Batch 2026-11-11T17-53-36-claude-code-claude-haiku-5-5-baseline |
| 2 | |
| 3 | claude-code 2.1.310, claude-haiku-5-5, 5 run(s) per journey and arm. |
| 4 | Revision e4e5d922e; skill 72d9d691700d; harness 9ff7364caa93. |
| 5 | Spent $0.118 on new runs; 0 run(s) reused. |
| 6 | |
| 7 | | split | journey | arm | passed | pass@1 (95% CI) | pass@k | pass^k | wrong tree | mean cost | infra | |
| 8 | | ------- | ---------------------------------- | -------- | ------ | --------------- | ------ | ------ | ---------- | --------- | ----- | |
| 9 | | train | consumer-slack-message | no-skill | 0/5 | 0% (0%–43%) | 0% | 0% | 0% | $0.002 | 0 | |
| 10 | | train | consumer-slack-message | skill | 2/5 | 40% (12%–77%) | 100% | 0% | 0% | $0.003 | 0 | |
| 11 | | train | developer-execute-unconnected | no-skill | 1/5 | 20% (4%–62%) | 100% | 0% | 20% | $0.004 | 0 | |
| 12 | | train | developer-execute-unconnected | skill | 0/5 | 0% (0%–43%) | 0% | 0% | 100% | $0.003 | 0 | |
| 13 | | train | first-run-login | no-skill | 3/5 | 60% (23%–88%) | 100% | 0% | 0% | $0.001 | 0 | |
| 14 | | train | first-run-login | skill | 2/5 | 40% (12%–77%) | 100% | 0% | 0% | $0.001 | 0 | |
| 15 | | holdout | developer-auth-configs-and-execute | no-skill | 5/5 | 100% (57%–100%) | 100% | 100% | 40% | $0.004 | 0 | |
| 16 | | holdout | developer-auth-configs-and-execute | skill | 3/5 | 60% (23%–88%) | 100% | 0% | 100% | $0.004 | 0 | |
| 17 | | holdout | run-inline-script | no-skill | 5/5 | 100% (57%–100%) | 100% | 100% | 0% | $0.002 | 0 | |
| 18 | | holdout | run-inline-script | skill | 5/5 | 100% (57%–100%) | 100% | 100% | 0% | $0.002 | 0 | |
| 19 | |
| 20 | | split | arm | mean pass@1 | mean pass^k | mean wrong tree | |
| 21 | | ------- | -------- | ----------- | ----------- | --------------- | |
| 22 | | train | skill | 27% | 0% | 33% | |
| 23 | | train | no-skill | 27% | 0% | 7% | |
| 24 | | holdout | skill | 80% | 50% | 50% | |
| 25 | | holdout | no-skill | 100% | 100% | 20% | |
| 26 | |
| 27 | ## Failed checks |
| 28 | |
| 29 | - consumer-slack-message (no-skill): |
| 30 | - 4/5 the last `composio execute` exited 0 |
| 31 | - 5/5 SLACK_SEND_MESSAGE completed once in the whole run |
| 32 | - 5/5 it ran on the consumer project, as consumer-user-e2e-acme-0001 |
| 33 | - 5/5 it went to #general |
| 34 | - 5/5 its text was exactly "deploy finished" |
| 35 | - 5/5 the account was linked on the consumer project |
| 36 | - consumer-slack-message (skill): |
| 37 | - 3/5 the last `composio execute` exited 0 |
| 38 | - 3/5 SLACK_SEND_MESSAGE completed once in the whole run |
| 39 | - 3/5 it ran on the consumer project, as consumer-user-e2e-acme-0001 |
| 40 | - 3/5 it went to #general |
| 41 | - 3/5 its text was exactly "deploy finished" |
| 42 | - 3/5 the account was linked on the consumer project |
| 43 | - developer-execute-unconnected (no-skill): |
| 44 | - 4/5 SLACK_SEND_MESSAGE completed once in the whole run |
| 45 | - 4/5 it ran on the developer project, as pg-test-00000000-0000-4000-8000-00000000a001 |
| 46 | - 4/5 it went to #general |
| 47 | - 4/5 its text was exactly "hello from dev" |
| 48 | - 4/5 the account was linked on the developer project for the test user |
| 49 | - 4/5 the last `composio dev playground-execute` exited 0 |
| 50 | - developer-execute-unconnected (skill): |
| 51 | - 5/5 SLACK_SEND_MESSAGE completed once in the whole run |
| 52 | - 5/5 it ran on the developer project, as pg-test-00000000-0000-4000-8000-00000000a001 |
| 53 | - 5/5 it went to #general |
| 54 | - 5/5 its text was exactly "hello from dev" |
| 55 | - 5/5 the account was linked on the developer project for the test user |
| 56 | - 5/5 the last `composio dev playground-execute` exited 0 |
| 57 | - first-run-login (no-skill): |
| 58 | - 2/5 the model finished by itself, within the turn limit |
| 59 | - 2/5 exactly one CLI login session was created, and it was linked |
| 60 | - 2/5 the login completed: the user API key was fetched for the linked session |
| 61 | - 2/5 the login was stored: a new `composio whoami` exits 0 as the person, in their org |
| 62 | - first-run-login (skill): |
| 63 | - 3/5 the model finished by itself, within the turn limit |
| 64 | - 3/5 exactly one CLI login session was created, and it was linked |
| 65 | - 3/5 the login completed: the user API key was fetched for the linked session |
| 66 | - 3/5 the login was stored: a new `composio whoami` exits 0 as the person, in their org |
| 67 | - developer-auth-configs-and-execute (skill): |
| 68 | - 2/5 SLACK_SEND_MESSAGE completed once in the whole run |
| 69 | - 2/5 it ran on the developer project, as pg-test-00000000-0000-4000-8000-00000000a001 |
| 70 | - 2/5 it went to #general |
| 71 | - 2/5 its text was exactly "hello from dev" |
| 72 | - 2/5 it ran through `composio dev playground-execute` |