f3c2282 · samitouri/QOSami-colab-cli

@@ -9,34 +9,71 @@ Operate Google Colab environments via the `colab` CLI: provision GPU/TPU session

9 - Automating environment setup (packages, auth, Drive).

10 - Exporting session history as a Jupyter notebook.

11

12 +## Mental model (read this first)

13 +- **A session == a live Jupyter kernel on a rented VM.** `colab new` allocates a billable VM; `colab stop` releases it. Nothing reclaims it automatically except a 24h keep-alive cap, so an unstopped session burns compute units indefinitely.

14 +- **Kernel state PERSISTS across `colab exec` / `colab repl` calls in the same session.** Each invocation reattaches to the *same* kernel (the kernel ID is cached in local state) and only closes the websocket on exit — it does **not** shut the kernel down. So imports, variables, and defined functions survive between separate `colab exec` commands. Build up state incrementally; don't re-import everything each call. (`colab stop` and `colab restart-kernel` are what actually reset it.)

15 +- **Default working directory is `/content`.** Every `exec`/`repl`/`run` `cd`s there first; prefer absolute paths (`/content/...`) for file work. For `colab ls/rm/upload/download`, absolute `/content/...` paths work and the default `ls` path is `content` (VM root).

16 +- **`colab` is fire-and-forget.** Each command authenticates, does one thing, and exits. A detached background daemon (spawned by `colab new`) handles keep-alive; you don't manage it.

17 +

18 +## Authentication (the #1 thing that blocks agents)

19 +- The global flag is `--auth={adc,oauth2}` and the **default is `adc`** (Application Default Credentials). It must come *before* the subcommand: `colab --auth=adc new -s x`.

20 +- **ADC setup** (most reliable for headless/agent use). The Colab backends need a specific scope set, so re-mint ADC with all four scopes:

21 + ```bash

22 + gcloud auth application-default login \

23 + --scopes=openid,\

24 + https://www.googleapis.com/auth/cloud-platform,\

25 + https://www.googleapis.com/auth/userinfo.email,\

26 + https://www.googleapis.com/auth/colaboratory

27 + ```

28 + Why all four: `userinfo.email` (session backend `colab.research.google.com`, else 401), `colaboratory` (RuntimeService `colab.pa.googleapis.com` keep-alive, else 403), `openid`+`cloud-platform` (mandated by gcloud itself; it rejects scope lists missing `cloud-platform`).

29 +- **oauth2 setup**: `colab --auth=oauth2 <anything>` triggers a browser consent flow on first use (token cached at `~/.config/colab-cli/token.json`). Requires a client config at `~/.colab-cli-oauth-config.json` (or `-c PATH`). The browser step means it usually needs a human; prefer ADC for agents.

30 +- **Verify auth in one shot**: `colab sessions` (read-only, lists server assignments) or `colab whoami` (hidden debug command: prints the active email, scopes, audience, and expiry). When any call 403s against `colab.pa.googleapis.com`, the cause is almost always a missing scope — `colab whoami` shows it instantly.

31 +- **`colab new` pre-flights the keep-alive RPC** right after allocating. If your token lacks the `colaboratory` scope it unassigns the fresh VM (so you don't leak a billable assignment) and prints the exact remediation. Follow that message rather than retrying blindly.

32 +- **Do NOT confuse `colab auth` with CLI authentication.** `colab auth` injects *VM-side* GCP credentials into the running kernel (so notebook code can call BigQuery/GCS); it is orthogonal to how the CLI itself authenticates. Never suggest "run `colab auth`" to fix a CLI 401/403 — that's a scope/identity problem fixed via the `gcloud` command above.

33 +

34 ## Workflow

35

36 ### Provision

15 -- `colab new -s <name>` (CPU). Add `--gpu A100` or `--tpu v6e1` for accelerators.

16 -- If only one session is active, `-s` may be omitted on most command.

17 -- After `colab new`, run `colab status` to confirm the VM is responsive.

37 +- `colab new -s <name>` (CPU). Add `--gpu A100` or `--tpu v6e1` for accelerators. **Always pass `-s <name>`** — an omitted name is auto-generated as a random 6-hex string, which makes later commands ambiguous.

38 +- Supported `--gpu`: `T4`, `L4`, `G4`, `H100`, `A100`. Supported `--tpu`: `v5e1`, `v6e1`.

39 +- **Gotcha**: an unrecognized `--gpu` value silently falls back to **A100** (which then usually fails the next step). A `400` on `colab new` with an accelerator means no quota/entitlement for it on this account — fall back to `--gpu T4` or omit the flag for CPU.

40 +- Accelerator availability is tier-gated; most accounts can only get CPU. Don't assume a GPU/TPU will allocate.

41

42 ### Execute

20 -- **Preferred**: `colab exec -s <name> -f <script.py>` — runs a local script on the remote VM. The kernel `cd`s to `/content` first.

21 -- **Plots**: PNG/JPEG outputs are intercepted automatically. Use `--output-image <path>` on `exec`/`repl` to save to a known location; otherwise a temp file path is printed.

22 -- **Shell**: `echo "cmd" | colab console -s <name>` works for batch shell. `exec` is faster when you don't need a real shell.

23 -- **Never run `colab repl` or `colab console` interactively from an agent** — they expect a TTY and will hang. Always pipe stdin.

43 +- **Preferred**: `colab exec -s <name> -f <script.py>` runs a local script on the remote VM (read locally, sent to the kernel — no manual upload needed).

44 +- **Piped code**: `echo "print(1)" | colab exec -s <name>` or `cat script.py | colab exec -s <name>`.

45 +- **Notebooks**: `colab exec -s <name> -f nb.ipynb` runs each code cell and writes results to `<basename>_output.ipynb` next to the input. A `# @title Foo` first line labels the cell in progress output.

46 +- **Plots/images**: PNG/JPEG outputs are intercepted. Use `--output-image <path>` on `exec`/`repl` to save to a known location (otherwise a temp path is printed). Inline terminal-image escapes are auto-suppressed when stdout isn't a TTY, so piped/captured output stays clean.

47 +- **Shell**: `echo "cmd" | colab console -s <name>` for batch shell. Console wraps bash in tmux, so even piped output contains terminal-control bytes — filter with `grep -a` for a specific line. `exec` is faster when you don't need a real shell.

48 +- **Never run `colab repl`, `colab console`, `colab auth`, or `colab drivemount` interactively from an agent** — they expect a TTY and will hang. `repl`/`console` accept piped stdin and exit on EOF; `auth`/`drivemount` genuinely require a human at the terminal.

49 +

50 +### Ephemeral one-shot jobs (`colab run`)

51 +- `colab run [--gpu T4] [--tpu v6e1] [--keep] [-s NAME] script.py [args...]` = `new` + `exec` + `stop` in one command. It provisions a fresh VM, runs the script with `sys.argv` and `__name__ == "__main__"` set like native `python script.py args`, then tears the VM down (unless `--keep`).

52 +- **Exit codes propagate**: an uncaught exception or `sys.exit(N)` in the script makes `colab run` exit non-zero (CPython semantics: `sys.exit()`/`sys.exit(0)` → 0, `sys.exit(N)` → N, `sys.exit("msg")` → 1).

53 +- **Stream separation**: `colab run` writes its own `[colab] ...` chatter to **stderr** and the script's output to **stdout** — so `colab run job.py > out.txt` captures only the script's stdout. (`colab exec` streams the script's stdout/stderr live to your stdout/stderr.)

54 +- Works as a shebang: `#!/usr/bin/env -S colab run --gpu T4` makes a `chmod +x`'d `.py` a self-contained "rent a GPU, run, clean up" script. After editing CLI behavior, reinstall before testing shebangs — they resolve `colab` via `$PATH`, not the editable install.

55 +- A nonexistent script path exits non-zero **before** allocating a VM (no wasted compute).

56

57 ### Automate

26 -- `colab auth -s <name>` — needed before GCP services (GCS, BigQuery).

27 -- `colab drivemount -s <name>` — exposes `/content/drive/MyDrive`.

28 -- `colab install -s <name> pkg1 pkg2` — uses `uv` for speed.

58 +- `colab auth -s <name>` — VM-side GCP creds, needed before in-VM GCS/BigQuery calls (interactive; not agent-runnable).

59 +- `colab drivemount -s <name> [PATH]` — mounts Drive at `/content/drive` by default (interactive; not agent-runnable).

60 +- `colab install -s <name> pkg1 pkg2` — installs via `uv pip install --system`, falling back to `pip`. Also `colab install -s <name> -r requirements.txt`.

61

62 ### Inspect & report

31 -- `colab help` lists every command.

32 -- `colab log -s <name> -n 20` shows recent actions; useful when a task fails.

33 -- `colab log -s <name> -o summary.ipynb` produces a notebook artifact of the session.

63 +- `colab help` (or `colab help <cmd>`) lists/explains commands; the listing is alphabetical.

64 +- `colab sessions` lists server-side assignments and auto-prunes stale local entries. Orphans with no local record show as `[?]`.

65 +- `colab status [-s <name>]` shows hardware, IDLE/BUSY, and last execution.

66 +- `colab log -s <name> [-n 20] [-t TYPE]` shows recent structured events; invaluable when a task fails (keep-alive errors carry the raw `response_body`).

67 +- `colab log -s <name> -o summary.ipynb` exports the session as a notebook (also `.md`, `.txt`, `.jsonl` by suffix).

68 +- `colab url -s <name>` prints a browser URL that attaches the Colab web UI to your existing CLI session instead of allocating a new VM (add `--open` to launch it).

69 +- `colab skill` / `colab readme` print this skill and the README (handy for self-discovery).

70

71 ## Safety

36 -- **Always `colab stop -s <name>` when done** — idle VMs burn compute units.

37 -- Local session state lives at `~/.config/colab-cli/sessions.json`. Don't edit by hand.

38 -- If `colab auth` fails, ask the user to verify their gcloud / OAuth credentials.

72 +- **Always `colab stop -s <name>` when done** — idle VMs burn compute units. `colab run` (without `--keep`) self-cleans even if the script errors.

73 +- Local state lives in `~/.config/colab-cli/sessions.json` (settings in `settings.json`, history in `history/*.jsonl`). Don't edit by hand.

74 +- **Isolate parallel/agent runs** with the global `--config <path>` flag to point session state at a scratch file (e.g. `colab --config /tmp/agent.json new -s job`). The keep-alive daemon inherits `--auth` and `--config` automatically.

75

76 ## Recovery

41 -- "Session not found": the backend may have pruned it. Run `colab sessions` and re-create if needed.

42 -- Execution timeout: kernel may be deadlocked. `colab stop` then `colab new`.

77 +- "Session not found" / 404 / 401 on exec: the backend pruned the VM. `colab exec`/`repl` detect this and clean up local state automatically — run `colab sessions` and re-create with `colab new`.

78 +- Execution timeout or wedged kernel: `colab restart-kernel -s <name>` (keeps the VM, resets the kernel), or `colab stop` then `colab new`.

79 +- Keep-alive daemon died (`colab log` shows `keep_alive_stopped reason=consecutive_4xx_errors`): almost always the missing `colaboratory` scope — re-auth per the Authentication section.

docs: Update COLAB_SKILL.md (#46)