@samitouri / QOSami-colab-cli-2
Code Commits Issues Pulls Sessions CI/CD
main
main
md 625 lines 18.3 KB
Rendered Raw
1 # Colab CLI: Demo Walkthroughs
2
3 *Captured 2026-05-07 against a live Colab backend with `showboat` 0.6.1.*
4 <!-- showboat-id: a24e677e-5052-4bec-8f82-36eb7a7859f9 -->
5
6 Eleven scenarios that exercise common workflows, plus a final "bridging back to the browser" example. Every `colab` invocation below was actually executed; the text inside each `output` block was captured verbatim from stdout/stderr.
7
8 **Methodology**
9 - Auth: `--auth=adc`. To set up: `gcloud auth application-default login --scopes=openid,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/colaboratory`.
10 - Accelerator: every session uses **CPU**. Provisioning real accelerators is gated by per-account quota and would not work for most readers; the workflows themselves are accelerator-agnostic, so where a demo's narrative mentions a GPU or TPU the prose flags the substitution.
11 - Interactive subcommands — `colab auth`, `colab drivemount`, and unpiped `colab repl` / `colab console` — are **not run** here because they require human interaction at a TTY. Demos that would normally use them include an inline note explaining what they do and the workflow continues with the non-interactive parts.
12 - `enable_update_check` is set to `false` in `~/.config/colab-cli/settings.json` for the duration of recording so the daily upgrade banner doesn't pollute output.
13 - `PYTHONWARNINGS=ignore` is set in the environment to suppress the ADC quota-project warning that `google.auth` emits on every call from end-user credentials.
14
15 **Re-verifiability caveat**: this document is **not** re-verifiable with `showboat verify`. Each `colab new` produces a fresh server-assigned session endpoint (`m-s-...`), so the recorded output never matches a re-run exactly. Treat this as a one-time witness that the workflows succeeded as of the recording date.
16
17 ## Demo 1: Cloud-native scientist
18
19 Provision a session, run a JAX workload over a small dataset, then tear the session down. Demonstrates the headline pattern of `colab new` → `colab exec` → `colab stop`. (A full-fidelity run of this scenario would also call `colab auth` and `colab drivemount` so the JAX code could read from BigQuery and write to Drive — both interactive, see the skip note below — and would request a TPU instead of CPU.)
20
21 ```bash
22 uv run colab --auth=adc new -s research
23 ```
24
25 ```output
26 [colab] Creating session 'research'...
27 [colab] Session READY.
28 ```
29
30 *Skipped:* `colab auth -s research` and `colab drivemount -s research`. Both require interactive TTY consent — `auth` prompts the user to visit an OAuth URL and paste back a verification code; `drivemount` prompts for an Enter keypress after the user grants consent in their browser. Verified separately in `integration/`.
31
32 ```bash
33 uv run colab --auth=adc install -s research jax 2>&1 | tail -20
34 ```
35
36 ```output
37 [colab] Installing packages on research (preferring uv)...
38 Installation Complete (via uv)!
39 ```
40
41 ```bash
42 cat <<'EOF' | uv run colab --auth=adc exec -s research
43 import jax, jax.numpy as jnp
44 import numpy as np
45
46 # (BigQuery substituted with synthetic data — would normally use:
47 # df = bigquery.Client().query('SELECT * FROM bigquery-public-data.ml_datasets.iris LIMIT 100').to_dataframe())
48 data = np.random.RandomState(0).randn(100, 4)
49
50 print('Devices:', jax.devices())
51 w = jax.random.normal(jax.random.PRNGKey(0), (4, 4))
52 out = jax.jit(lambda x, w: x @ w)(jnp.array(data), w)
53 print(f'Processed {len(out)} rows.')
54 EOF
55
56 ```
57
58 ```output
59 Devices: [CpuDevice(id=0)]
60 Processed 100 rows.
61 ```
62
63 ```bash
64 uv run colab --auth=adc stop -s research
65 ```
66
67 ```output
68 [colab] Stopping session 'research'...
69 [colab] Session terminated.
70 ```
71
72 ## Demo 2: Fast iteration on GPU
73
74 A typical model-training cycle: provision → install dependencies → run a training script → check status → download the resulting checkpoint. The script here is a 1-layer linear regression on synthetic data so it finishes in a few seconds on CPU; substitute your real training code and `--gpu A100` for a production run.
75
76 ```bash
77 uv run colab --auth=adc new -s trainer
78 ```
79
80 ```output
81 [colab] Creating session 'trainer'...
82 [colab] Session READY.
83 ```
84
85 ```bash
86 uv run colab --auth=adc install -s trainer torch 2>&1 | tail -5
87 ```
88
89 ```output
90 [colab] Installing packages on trainer (preferring uv)...
91 Installation Complete (via uv)!
92 ```
93
94 ```bash
95 uv run colab --auth=adc exec -s trainer -f /tmp/train.py
96 ```
97
98 ```output
99 Epoch 1/10: loss 14.380
100 Epoch 2/10: loss 11.639
101 Epoch 3/10: loss 9.440
102 Epoch 4/10: loss 7.671
103 Epoch 5/10: loss 6.247
104 Epoch 6/10: loss 5.097
105 Epoch 7/10: loss 4.167
106 Epoch 8/10: loss 3.414
107 Epoch 9/10: loss 2.802
108 Epoch 10/10: loss 2.305
109 Training complete.
110 ```
111
112 ```bash
113 uv run colab --auth=adc download -s trainer /content/model.bin /tmp/model.bin && ls -la /tmp/model.bin
114 ```
115
116 ```output
117 [colab] Downloaded '/content/model.bin' to '/tmp/model.bin'
118 -rw-r----- 1 rtp primarygroup 1877 May 7 23:11 /tmp/model.bin
119 ```
120
121 ```bash
122 uv run colab --auth=adc status -s trainer
123 ```
124
125 ```output
126 [trainer] m-s-kkb-usw1c0-21g32dh850cd4 | Hardware: CPU | Variant: DEFAULT | Status: IDLE
127 Last Execution: /tmp/train.py at 2026-05-07 23:11:32
128 ```
129
130 ```bash
131 uv run colab --auth=adc stop -s trainer
132 ```
133
134 ```output
135 [colab] Stopping session 'trainer'...
136 [colab] Session terminated.
137 ```
138
139 ## Demo 3: Interactive troubleshooting (piped)
140
141 Both `colab console` and `colab repl` accept piped stdin and exit on EOF, so they compose well with shell pipelines and other CLI tools. This demo investigates remote disk usage with a one-shot shell command, lists `/content`, creates and removes a scratch file, and then queries free space from a one-shot REPL.
142
143 ```bash
144 uv run colab --auth=adc new -s debug
145 ```
146
147 ```output
148 [colab] Creating session 'debug'...
149 [colab] Session READY.
150 ```
151
152 *Note:* `colab console` connects to a tmux-wrapped pty on the VM, so even when stdin is piped the raw stdout contains terminal-control bytes (cursor moves, status-line repaints, ANSI color). For programmatic consumption, pipe the output through `grep -a` (force binary-safe) and a regex matching the line(s) you care about, as shown below.
153
154 ```bash
155 echo 'df -h /content' | uv run colab --auth=adc console -s debug 2>&1 | grep -aE 'overlay|/dev/'
156 ```
157
158 ```output
159 overlay 108G 21G 87G 20% /
160 ```
161
162 ```bash
163 uv run colab --auth=adc ls -s debug /content
164 ```
165
166 ```output
167 .config/
168 sample_data/
169 ```
170
171 ```bash
172 echo 'with open("/content/scratch.log", "w") as f: f.write("x" * 1024 * 100)
173 print("created scratch.log (100 KB)")' | uv run colab --auth=adc exec -s debug
174 ```
175
176 ```output
177 created scratch.log (100 KB)
178 ```
179
180 ```bash
181 uv run colab --auth=adc rm -s debug /content/scratch.log
182 ```
183
184 ```output
185 [colab] Deleted /content/scratch.log
186 ```
187
188 ```bash
189 echo 'import shutil; print(shutil.disk_usage("/").free // 2**30, "GB free")' | uv run colab --auth=adc repl -s debug
190 ```
191
192 ```output
193 86 GB free
194 ```
195
196 ```bash
197 uv run colab --auth=adc stop -s debug
198 ```
199
200 ```output
201 [colab] Stopping session 'debug'...
202 [colab] Session terminated.
203 ```
204
205 ## Demo 4: Multi-modal output (plots & notebooks)
206
207 Demonstrates plot redirection (`--output-image`) and notebook execution (`colab exec -f file.ipynb` writes outputs back into `<name>_output.ipynb`).
208
209 ```bash
210 uv run colab --auth=adc new -s reporter
211 ```
212
213 ```output
214 [colab] Creating session 'reporter'...
215 [colab] Session READY.
216 ```
217
218 ```bash
219 cat <<'EOF' | uv run colab --auth=adc exec -s reporter --output-image /tmp/sine.png
220 import matplotlib.pyplot as plt, numpy as np
221 x = np.linspace(0, 10, 100)
222 plt.plot(x, np.sin(x)); plt.title('Sine'); plt.show()
223 EOF
224
225 ```
226
227 ```output
228 <Figure size 640x480 with 1 Axes>
229
230 [Image saved to: /tmp/sine.png]
231 ```
232
233 ```bash {image}
234 ![Sine wave plot generated on the Colab VM and saved to /tmp/sine.png](/tmp/sine.png)
235 ```
236
237 ![Sine wave plot generated on the Colab VM and saved to /tmp/sine.png](3042ab12-2026-05-07.png)
238
239 ```bash
240 uv run colab --auth=adc exec -s reporter -f /tmp/analysis.ipynb && ls /tmp/analysis_output.ipynb
241 ```
242
243 ```output
244 [colab] Parsing notebook '/tmp/analysis.ipynb'...
245 [colab] Executing cell 1/2 - a8850b8f...
246 mean = 18
247 stdev = 13.49
248 [colab] Executing cell 2/2 - c31a0002...
249 rows: 6
250 sum: 108
251 [colab] Saving notebook with outputs to '/tmp/analysis_output.ipynb'...
252 /tmp/analysis_output.ipynb
253 ```
254
255 ```bash
256 uv run colab --auth=adc log -s reporter -o /tmp/reporter.md && wc -l /tmp/reporter.md
257 ```
258
259 ```output
260 [colab] Exported history to '/tmp/reporter.md'.
261 55 /tmp/reporter.md
262 ```
263
264 ```bash
265 uv run colab --auth=adc stop -s reporter
266 ```
267
268 ```output
269 [colab] Stopping session 'reporter'...
270 [colab] Session terminated.
271 ```
272
273 ## Demo 5: Bulk data via GCS
274
275 Pull a batch of objects down from a Google Cloud Storage bucket, transform them on the VM, and pull the results back. A full-fidelity workflow is `colab new --gpu L4` -> `colab auth` (so VM-side `gcloud` works) -> `gcloud storage cp gs://bucket/raw/*.jpg /content/images/` (via piped `colab console`) -> install pillow/torchvision -> process -> download. We skip the auth step here (interactive; the user has to click through OAuth) and substitute synthetic image generation in place of the GCS pull, which keeps the input -> install -> batch-process -> download shape intact.
276
277 ```bash
278 uv run colab --auth=adc new -s data-proc
279 ```
280
281 ```output
282 [colab] Creating session 'data-proc'...
283 [colab] Session READY.
284 ```
285
286 ```bash
287 uv run colab --auth=adc install -s data-proc pillow 2>&1 | tail -3
288 ```
289
290 ```output
291 [colab] Installing packages on data-proc (preferring uv)...
292 Installation Complete (via uv)!
293 ```
294
295 ```bash
296 cat <<'EOF' | uv run colab --auth=adc exec -s data-proc
297 # (would normally pull from GCS via 'gcloud storage cp gs://my-bucket/raw_data/*.jpg')
298 import os, zipfile
299 from PIL import Image, ImageFilter
300 os.makedirs('/content/images', exist_ok=True)
301 os.makedirs('/content/processed', exist_ok=True)
302 # Generate 10 synthetic input images
303 for i in range(10):
304 Image.new('RGB', (64, 64), (i * 25, 100, 200 - i * 15)).save(f'/content/images/img_{i:02d}.jpg')
305 # Process: blur each
306 for src in sorted(os.listdir('/content/images')):
307 img = Image.open(f'/content/images/{src}').filter(ImageFilter.GaussianBlur(2))
308 img.save(f'/content/processed/{src}')
309 # Zip results
310 with zipfile.ZipFile('/content/processed/batch.zip', 'w') as z:
311 for f in sorted(os.listdir('/content/processed')):
312 if f.endswith('.jpg'):
313 z.write(f'/content/processed/{f}', f)
314 print(f'Processed {len(os.listdir("/content/processed")) - 1} images, archived to batch.zip')
315 EOF
316
317 ```
318
319 ```output
320 Processed 10 images, archived to batch.zip
321 ```
322
323 ```bash
324 uv run colab --auth=adc download -s data-proc /content/processed/batch.zip /tmp/batch.zip && ls -la /tmp/batch.zip
325 ```
326
327 ```output
328 [colab] Downloaded '/content/processed/batch.zip' to '/tmp/batch.zip'
329 -rw-r----- 1 rtp primarygroup 7902 May 7 23:19 /tmp/batch.zip
330 ```
331
332 ```bash
333 uv run colab --auth=adc stop -s data-proc
334 ```
335
336 ```output
337 [colab] Stopping session 'data-proc'...
338 [colab] Session terminated.
339 ```
340
341 ## Demo 6: Resource check & subscription
342
343 Inspect a long-running session, then export its history as a notebook for archival. (`colab pay`, which opens `https://colab.research.google.com/signup` in the system browser to manage compute units, would normally fit here too — we don't invoke it because it would pop a browser window in the recording environment.)
344
345 ```bash
346 uv run colab --auth=adc new -s long-running
347 ```
348
349 ```output
350 [colab] Creating session 'long-running'...
351 [colab] Session READY.
352 ```
353
354 ```bash
355 uv run colab --auth=adc status -s long-running
356 ```
357
358 ```output
359 [long-running] m-s-kkb-use4a2-2qvalahyh7yzg | Hardware: CPU | Variant: DEFAULT | Status: IDLE
360 ```
361
362 ```bash
363 echo 'print("hello from session")' | uv run colab --auth=adc exec -s long-running
364 ```
365
366 ```output
367 hello from session
368 ```
369
370 ```bash
371 uv run colab --auth=adc log -s long-running -o /tmp/checkpoint.ipynb && ls -la /tmp/checkpoint.ipynb
372 ```
373
374 ```output
375 [colab] Exported history to '/tmp/checkpoint.ipynb'.
376 -rw-r----- 1 rtp primarygroup 974 May 7 23:20 /tmp/checkpoint.ipynb
377 ```
378
379 ```bash
380 uv run colab --auth=adc stop -s long-running
381 ```
382
383 ```output
384 [colab] Stopping session 'long-running'...
385 [colab] Session terminated.
386 ```
387
388 ## Demo 7: Reproducible research
389
390 Quick exploration via piped repl, file inspection via piped exec, then capture the whole session as a notebook artifact via `colab log -o <name>.ipynb`. The notebook is replayable in the Colab UI.
391
392 ```bash
393 uv run colab --auth=adc new -s pivot
394 ```
395
396 ```output
397 [colab] Creating session 'pivot'...
398 [colab] Session READY.
399 ```
400
401 ```bash
402 uv run colab --auth=adc install -s pivot scipy 2>&1 | tail -3
403 ```
404
405 ```output
406 [colab] Installing packages on pivot (preferring uv)...
407 Installation Complete (via uv)!
408 ```
409
410 ```bash
411 echo 'from scipy.stats import zscore; print(zscore([1.2, 1.5, 1.1, 10.4, 1.3]))' | uv run colab --auth=adc repl -s pivot
412 ```
413
414 ```output
415 [-0.52020639 -0.43806854 -0.54758568 1.99868773 -0.49282711]
416 ```
417
418 ```bash
419 uv run colab --auth=adc upload -s pivot /tmp/raw_data.csv /content/raw_data.csv
420 ```
421
422 ```output
423 [colab] Uploaded '/tmp/raw_data.csv' to '/content/raw_data.csv'
424 ```
425
426 ```bash
427 echo 'print(open("/content/raw_data.csv").read())' | uv run colab --auth=adc exec -s pivot
428 ```
429
430 ```output
431 id,name,score
432 1,alice,0.92
433 2,bob,0.74
434 3,carol,0.88
435 4,dave,0.61
436 5,eve,0.95
437
438 ```
439
440 ```bash
441 uv run colab --auth=adc log -s pivot -o /tmp/pivot_discovery.ipynb && ls -la /tmp/pivot_discovery.ipynb
442 ```
443
444 ```output
445 [colab] Exported history to '/tmp/pivot_discovery.ipynb'.
446 -rw-r----- 1 rtp primarygroup 2854 May 7 23:21 /tmp/pivot_discovery.ipynb
447 ```
448
449 ```bash
450 uv run colab --auth=adc stop -s pivot
451 ```
452
453 ```output
454 [colab] Stopping session 'pivot'...
455 [colab] Session terminated.
456 ```
457
458 ## Demo 8: Local + cloud hybrid
459
460 Run a local script against the remote VM and pull a result back. The full-fidelity version of this demo also calls `colab drivemount` to make Google Drive available at `/content/drive` on the VM (so the script can read shared data); `drivemount` is interactive and skipped here. The kept portion — `colab exec -f local_script.py` running a script that lives on your laptop against a kernel that lives in Colab — is the workflow worth highlighting.
461
462 ```bash
463 uv run colab --auth=adc new -s hybrid
464 ```
465
466 ```output
467 [colab] Creating session 'hybrid'...
468 [colab] Session READY.
469 ```
470
471 ```bash
472 uv run colab --auth=adc exec -s hybrid -f /tmp/local_analysis.py
473 ```
474
475 ```output
476 Running on: Linux-6.6.113+-x86_64-with-glibc2.35
477 Hostname: 699413ff1767
478 Python: 3.12.13
479 This script lives on my laptop but ran on the Colab VM.
480 ```
481
482 ```bash
483 uv run colab --auth=adc stop -s hybrid
484 ```
485
486 ```output
487 [colab] Stopping session 'hybrid'...
488 [colab] Session terminated.
489 ```
490
491 ## Demo 9: Multi-session orchestration
492
493 Run multiple sessions concurrently, list them, inspect one, stop one. The two sessions here are both CPU; in practice you'd more likely have a mix of accelerator types (e.g. one TPU for training, one GPU for evaluation).
494
495 ```bash
496 uv run colab --auth=adc new -s tpu-cluster && uv run colab --auth=adc new -s gpu-eval
497 ```
498
499 ```output
500 [colab] Creating session 'tpu-cluster'...
501 [colab] Session READY.
502 [colab] Creating session 'gpu-eval'...
503 [colab] Session READY.
504 ```
505
506 ```bash
507 uv run colab --auth=adc sessions
508 ```
509
510 ```output
511 [gpu-eval] m-s-kkb-usc1c0-3cickkby8ivx5 | Hardware: CPU | Variant: DEFAULT
512 [tpu-cluster] m-s-kkb-use1b1-3b5xes33630p3 | Hardware: CPU | Variant: DEFAULT
513 ```
514
515 ```bash
516 uv run colab --auth=adc status -s gpu-eval
517 ```
518
519 ```output
520 [gpu-eval] m-s-kkb-usc1c0-3cickkby8ivx5 | Hardware: CPU | Variant: DEFAULT | Status: IDLE
521 ```
522
523 ```bash
524 uv run colab --auth=adc stop -s gpu-eval && uv run colab --auth=adc stop -s tpu-cluster
525 ```
526
527 ```output
528 [colab] Stopping session 'gpu-eval'...
529 [colab] Session terminated.
530 [colab] Stopping session 'tpu-cluster'...
531 [colab] Session terminated.
532 ```
533
534 ## Demo 10: One-shot pipeline
535
536 Chain several commands with `&&` so any failure aborts. The script here is a tiny stand-in (writes a JSON result to `/content`) so the chain runs in a few seconds on CPU; the typical real version would be `--gpu A100` plus a heavier dependency like `flash-attn`.
537
538 ```bash
539 uv run colab --auth=adc new -s pipeline \
540 && uv run colab --auth=adc install -s pipeline requests 2>&1 | tail -2 \
541 && uv run colab --auth=adc exec -s pipeline -f /tmp/local_pipeline.py \
542 && uv run colab --auth=adc download -s pipeline /content/results.json /tmp/results.json \
543 && uv run colab --auth=adc stop -s pipeline
544 ```
545
546 ```output
547 [colab] Creating session 'pipeline'...
548 [colab] Session READY.
549 [colab] Installing packages on pipeline (preferring uv)...
550 Installation Complete (via uv)!
551 Wrote results.json: {'status': 'ok', 'computed_at': '2026-05-07T23:22:34.924350Z', 'value': 42}
552 /tmp/ipykernel_38852/1782062088.py:5: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
553 "computed_at": datetime.datetime.utcnow().isoformat() + "Z",
554 [colab] Downloaded '/content/results.json' to '/tmp/results.json'
555 [colab] Stopping session 'pipeline'...
556 [colab] Session terminated.
557 ```
558
559 ## Demo 11: Reproducible environment
560
561 Upload a `requirements.txt` to the VM, install via `-r`, then verify the version on the VM matches what we asked for.
562
563 ```bash
564 uv run colab --auth=adc new -s env-test
565 ```
566
567 ```output
568 [colab] Creating session 'env-test'...
569 [colab] Session READY.
570 ```
571
572 ```bash
573 uv run colab --auth=adc upload -s env-test /tmp/requirements.txt /content/requirements.txt
574 ```
575
576 ```output
577 [colab] Uploaded '/tmp/requirements.txt' to '/content/requirements.txt'
578 ```
579
580 ```bash
581 uv run colab --auth=adc install -s env-test -r /tmp/requirements.txt 2>&1 | tail -3
582 ```
583
584 ```output
585 [colab] Installing packages on env-test (preferring uv)...
586 Installation Complete (via uv)!
587 ```
588
589 ```bash
590 echo 'import requests; print("requests:", requests.__version__)' | uv run colab --auth=adc exec -s env-test
591 ```
592
593 ```output
594 requests: 2.31.0
595 ```
596
597 ## Bridging back to the browser
598
599 `colab url -s <name>` prints a URL that, when opened in a browser, makes the Colab frontend connect to the existing colab-cli session instead of provisioning a fresh VM. By default it just prints the URL (pipeable, e.g. `colab url -s s1 | xclip`); `--open` would open it directly in the system browser.
600
601 ```bash
602 uv run colab --auth=adc url -s env-test
603 ```
604
605 ```output
606 https://colab.research.google.com/notebooks/empty.ipynb?dbu=%2Ftun%2Fm%2Fm-s-kkb-usc1b1-3tpcjymikv7t3
607 ```
608
609 ```bash
610 uv run colab --auth=adc stop -s env-test
611 ```
612
613 ```output
614 [colab] Stopping session 'env-test'...
615 [colab] Session terminated.
616 ```
617
618 ```bash
619 uv run colab --auth=adc sessions
620 ```
621
622 ```output
623 [colab] Pruned 1 stale local session(s).
624 [colab] No active sessions found on server.
625 ```