@cryptotaxi247 / SquadScope
Code Commits Issues Pulls Sessions CI/CD
main
main
md 229 lines 7.05 KB
Rendered Raw
1 # Cooperative Rate Limiting for Multi-Agent Deployments
2
3 > Coordinate API quota across multiple Ralph instances to prevent cascading failures.
4
5 ## Problem
6
7 The [circuit breaker template](ralph-circuit-breaker.md) handles single-instance rate limiting well. But when multiple Ralphs run across machines (or pods on K8s), each instance independently hits API limits:
8
9 - **No coordination** — 5 Ralphs each think they have full API quota
10 - **Thundering herd** — All Ralphs retry simultaneously after rate limit resets
11 - **Priority inversion** — Low-priority work exhausts quota before critical work runs
12 - **Reactive only** — Circuit opens AFTER 429, wasting the failed request
13
14 ## Solution: 6-Pattern Architecture
15
16 These patterns layer on top of the existing circuit breaker. Each is independent — adopt one or all.
17
18 ### Pattern 1: Traffic Light (RAAS — Rate-Aware Agent Scheduling)
19
20 Map GitHub API `X-RateLimit-Remaining` to traffic light states:
21
22 | State | Remaining % | Behavior |
23 |-------|------------|----------|
24 | 🟢 GREEN | >20% | Normal operation |
25 | 🟡 AMBER | 5–20% | Only P0 agents proceed |
26 | 🔴 RED | <5% | Block all except emergency P0 |
27
28 ```typescript
29 type TrafficLight = 'green' | 'amber' | 'red';
30
31 function getTrafficLight(remaining: number, limit: number): TrafficLight {
32 const pct = remaining / limit;
33 if (pct > 0.20) return 'green';
34 if (pct > 0.05) return 'amber';
35 return 'red';
36 }
37
38 function shouldProceed(light: TrafficLight, agentPriority: number): boolean {
39 if (light === 'green') return true;
40 if (light === 'amber') return agentPriority === 0; // P0 only
41 return false; // RED — block all
42 }
43 ```
44
45 ### Pattern 2: Cooperative Token Pool (CMARP)
46
47 A shared JSON file (`~/.squad/rate-pool.json`) distributes API quota:
48
49 ```json
50 {
51 "totalLimit": 5000,
52 "resetAt": "2026-03-22T20:00:00Z",
53 "allocations": {
54 "picard": { "priority": 0, "allocated": 2000, "used": 450, "leaseExpiry": "2026-03-22T19:55:00Z" },
55 "data": { "priority": 1, "allocated": 1750, "used": 200, "leaseExpiry": "2026-03-22T19:55:00Z" },
56 "ralph": { "priority": 2, "allocated": 1250, "used": 100, "leaseExpiry": "2026-03-22T19:55:00Z" }
57 }
58 }
59 ```
60
61 **Rules:**
62 - P0 agents (Lead) get 40% of quota
63 - P1 agents (specialists) get 35%
64 - P2 agents (Ralph, Scribe) get 25%
65 - Stale leases (>5 minutes without heartbeat) are auto-recovered
66 - Each agent checks their remaining allocation before making API calls
67
68 ```typescript
69 interface RatePoolAllocation {
70 priority: number;
71 allocated: number;
72 used: number;
73 leaseExpiry: string;
74 }
75
76 interface RatePool {
77 totalLimit: number;
78 resetAt: string;
79 allocations: Record<string, RatePoolAllocation>;
80 }
81
82 function canUseQuota(pool: RatePool, agentName: string): boolean {
83 const alloc = pool.allocations[agentName];
84 if (!alloc) return true; // Unknown agent — allow (graceful)
85
86 // Reclaim stale leases from crashed agents
87 const now = new Date();
88 for (const [name, a] of Object.entries(pool.allocations)) {
89 if (new Date(a.leaseExpiry) < now && name !== agentName) {
90 a.allocated = 0; // Reclaim
91 }
92 }
93
94 return alloc.used < alloc.allocated;
95 }
96 ```
97
98 ### Pattern 3: Predictive Circuit Breaker (PCB)
99
100 Opens the circuit BEFORE getting a 429 by predicting when quota will run out:
101
102 ```typescript
103 interface RateSample {
104 timestamp: number; // Date.now()
105 remaining: number; // from X-RateLimit-Remaining header
106 }
107
108 class PredictiveCircuitBreaker {
109 private samples: RateSample[] = [];
110 private readonly maxSamples = 10;
111 private readonly warningThresholdSeconds = 120;
112
113 addSample(remaining: number): void {
114 this.samples.push({ timestamp: Date.now(), remaining });
115 if (this.samples.length > this.maxSamples) {
116 this.samples.shift();
117 }
118 }
119
120 /** Predict seconds until quota exhaustion using linear regression */
121 predictExhaustion(): number | null {
122 if (this.samples.length < 3) return null;
123
124 const n = this.samples.length;
125 const first = this.samples[0];
126 const last = this.samples[n - 1];
127
128 const elapsedMs = last.timestamp - first.timestamp;
129 if (elapsedMs === 0) return null;
130
131 const consumedPerMs = (first.remaining - last.remaining) / elapsedMs;
132 if (consumedPerMs <= 0) return null; // Not consuming — safe
133
134 const msUntilExhausted = last.remaining / consumedPerMs;
135 return msUntilExhausted / 1000;
136 }
137
138 shouldOpen(): boolean {
139 const eta = this.predictExhaustion();
140 if (eta === null) return false;
141 return eta < this.warningThresholdSeconds;
142 }
143 }
144 ```
145
146 ### Pattern 4: Priority Retry Windows (PWJG)
147
148 Non-overlapping jitter windows prevent thundering herd:
149
150 | Priority | Retry Window | Description |
151 |----------|-------------|-------------|
152 | P0 (Lead) | 500ms–5s | Recovers first |
153 | P1 (Specialists) | 2s–30s | Moderate delay |
154 | P2 (Ralph/Scribe) | 5s–60s | Most patient |
155
156 ```typescript
157 function getRetryDelay(priority: number, attempt: number): number {
158 const windows: Record<number, [number, number]> = {
159 0: [500, 5000], // P0: 500ms–5s
160 1: [2000, 30000], // P1: 2s–30s
161 2: [5000, 60000], // P2: 5s–60s
162 };
163
164 const [min, max] = windows[priority] ?? windows[2];
165 const base = Math.min(min * Math.pow(2, attempt), max);
166 const jitter = Math.random() * base * 0.5;
167 return base + jitter;
168 }
169 ```
170
171 ### Pattern 5: Resource Epoch Tracker (RET)
172
173 Heartbeat-based lease system for multi-machine deployments:
174
175 ```typescript
176 interface ResourceLease {
177 agent: string;
178 machine: string;
179 leaseStart: string;
180 leaseExpiry: string; // Typically 5 minutes from now
181 allocated: number;
182 }
183
184 // Each agent renews its lease every 2 minutes
185 // If lease expires (agent crashed), allocation is reclaimed
186 ```
187
188 ### Pattern 6: Cascade Dependency Detector (CDD)
189
190 Track downstream failures and apply backpressure:
191
192 ```
193 Agent A (rate limited) → Agent B (waiting for A) → Agent C (waiting for B)
194 ↑ Backpressure signal: "don't start new work"
195 ```
196
197 When a dependency is rate-limited, upstream agents should pause new work rather than queuing requests that will fail.
198
199 ## Kubernetes Integration
200
201 On K8s, cooperative rate limiting can use KEDA to scale pods based on API quota:
202
203 ```yaml
204 apiVersion: keda.sh/v1alpha1
205 kind: ScaledObject
206 spec:
207 scaleTargetRef:
208 name: ralph-deployment
209 triggers:
210 - type: external
211 metadata:
212 scalerAddress: keda-copilot-scaler:6000
213 # Scaler returns 0 when rate limited → pods scale to zero
214 ```
215
216 See [keda-copilot-scaler](https://github.com/tamirdresher/keda-copilot-scaler) for a complete implementation.
217
218 ## Quick Start
219
220 1. **Minimum viable:** Adopt Pattern 1 (Traffic Light) — read `X-RateLimit-Remaining` from API responses
221 2. **Multi-machine:** Add Pattern 2 (Cooperative Pool) — shared `rate-pool.json`
222 3. **Production:** Add Pattern 3 (Predictive CB) — prevent 429s entirely
223 4. **Kubernetes:** Add KEDA scaler for automatic pod scaling
224
225 ## References
226
227 - [Circuit Breaker Template](ralph-circuit-breaker.md) — Foundation patterns
228 - [Squad on AKS](https://github.com/tamirdresher/squad-on-aks) — Production K8s deployment
229 - [KEDA Copilot Scaler](https://github.com/tamirdresher/keda-copilot-scaler) — Custom KEDA external scaler