@devries-cameron20 / composio Mirror · read-only
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 07:15 UTC

next
md 72 lines 4.7 KB
Rendered Raw
1 # Batch 2026-11-11T17-53-36-claude-code-claude-haiku-5-5-baseline
2
3 claude-code 2.1.310, claude-haiku-5-5, 5 run(s) per journey and arm.
4 Revision e4e5d922e; skill 72d9d691700d; harness 9ff7364caa93.
5 Spent $0.118 on new runs; 0 run(s) reused.
6
7 | split | journey | arm | passed | pass@1 (95% CI) | pass@k | pass^k | wrong tree | mean cost | infra |
8 | ------- | ---------------------------------- | -------- | ------ | --------------- | ------ | ------ | ---------- | --------- | ----- |
9 | train | consumer-slack-message | no-skill | 0/5 | 0% (0%–43%) | 0% | 0% | 0% | $0.002 | 0 |
10 | train | consumer-slack-message | skill | 2/5 | 40% (12%–77%) | 100% | 0% | 0% | $0.003 | 0 |
11 | train | developer-execute-unconnected | no-skill | 1/5 | 20% (4%–62%) | 100% | 0% | 20% | $0.004 | 0 |
12 | train | developer-execute-unconnected | skill | 0/5 | 0% (0%–43%) | 0% | 0% | 100% | $0.003 | 0 |
13 | train | first-run-login | no-skill | 3/5 | 60% (23%–88%) | 100% | 0% | 0% | $0.001 | 0 |
14 | train | first-run-login | skill | 2/5 | 40% (12%–77%) | 100% | 0% | 0% | $0.001 | 0 |
15 | holdout | developer-auth-configs-and-execute | no-skill | 5/5 | 100% (57%–100%) | 100% | 100% | 40% | $0.004 | 0 |
16 | holdout | developer-auth-configs-and-execute | skill | 3/5 | 60% (23%–88%) | 100% | 0% | 100% | $0.004 | 0 |
17 | holdout | run-inline-script | no-skill | 5/5 | 100% (57%–100%) | 100% | 100% | 0% | $0.002 | 0 |
18 | holdout | run-inline-script | skill | 5/5 | 100% (57%–100%) | 100% | 100% | 0% | $0.002 | 0 |
19
20 | split | arm | mean pass@1 | mean pass^k | mean wrong tree |
21 | ------- | -------- | ----------- | ----------- | --------------- |
22 | train | skill | 27% | 0% | 33% |
23 | train | no-skill | 27% | 0% | 7% |
24 | holdout | skill | 80% | 50% | 50% |
25 | holdout | no-skill | 100% | 100% | 20% |
26
27 ## Failed checks
28
29 - consumer-slack-message (no-skill):
30 - 4/5 the last `composio execute` exited 0
31 - 5/5 SLACK_SEND_MESSAGE completed once in the whole run
32 - 5/5 it ran on the consumer project, as consumer-user-e2e-acme-0001
33 - 5/5 it went to #general
34 - 5/5 its text was exactly "deploy finished"
35 - 5/5 the account was linked on the consumer project
36 - consumer-slack-message (skill):
37 - 3/5 the last `composio execute` exited 0
38 - 3/5 SLACK_SEND_MESSAGE completed once in the whole run
39 - 3/5 it ran on the consumer project, as consumer-user-e2e-acme-0001
40 - 3/5 it went to #general
41 - 3/5 its text was exactly "deploy finished"
42 - 3/5 the account was linked on the consumer project
43 - developer-execute-unconnected (no-skill):
44 - 4/5 SLACK_SEND_MESSAGE completed once in the whole run
45 - 4/5 it ran on the developer project, as pg-test-00000000-0000-4000-8000-00000000a001
46 - 4/5 it went to #general
47 - 4/5 its text was exactly "hello from dev"
48 - 4/5 the account was linked on the developer project for the test user
49 - 4/5 the last `composio dev playground-execute` exited 0
50 - developer-execute-unconnected (skill):
51 - 5/5 SLACK_SEND_MESSAGE completed once in the whole run
52 - 5/5 it ran on the developer project, as pg-test-00000000-0000-4000-8000-00000000a001
53 - 5/5 it went to #general
54 - 5/5 its text was exactly "hello from dev"
55 - 5/5 the account was linked on the developer project for the test user
56 - 5/5 the last `composio dev playground-execute` exited 0
57 - first-run-login (no-skill):
58 - 2/5 the model finished by itself, within the turn limit
59 - 2/5 exactly one CLI login session was created, and it was linked
60 - 2/5 the login completed: the user API key was fetched for the linked session
61 - 2/5 the login was stored: a new `composio whoami` exits 0 as the person, in their org
62 - first-run-login (skill):
63 - 3/5 the model finished by itself, within the turn limit
64 - 3/5 exactly one CLI login session was created, and it was linked
65 - 3/5 the login completed: the user API key was fetched for the linked session
66 - 3/5 the login was stored: a new `composio whoami` exits 0 as the person, in their org
67 - developer-auth-configs-and-execute (skill):
68 - 2/5 SLACK_SEND_MESSAGE completed once in the whole run
69 - 2/5 it ran on the developer project, as pg-test-00000000-0000-4000-8000-00000000a001
70 - 2/5 it went to #general
71 - 2/5 its text was exactly "hello from dev"
72 - 2/5 it ran through `composio dev playground-execute`