@samitouri / QOSami-colab-cli
Code Commits Issues Pulls Sessions CI/CD
main
main
md 327 lines 22 KB
Rendered Raw
1 ---
2 log:
3 2026-06-11: Replaced the `oauth2` provider's `run_local_server()` (localhost redirect) with a remote copy-paste flow (`_run_remote_flow` in `auth.py`). The CLI now prints an authorization URL built with `redirect_uri=https://sdk.cloud.google.com/applicationdefaultauthcode.html` and `token_usage=remote`, then reads the pasted authorization code via `input()` and exchanges it with `flow.fetch_token(code=...)`. This is the same flow `gcloud auth application-default login` uses and works identically in local and remote/headless/container environments, removing the heuristic of whether to auto-open a browser. Confirmed server-side acceptance with a live GET-only check against the bundled cloud-SDK client (`764086051850-...`); the OOB redirect and a non-bundled client id were both verified to be rejected (`OOB flow has been blocked` / `redirect_uri_mismatch`). Unit tests in `tests/test_auth.py` assert no localhost server is started, the redirect URI + `token_usage=remote` are set, and the pasted code is exchanged.
4 2026-06-01: Enabled `colab update --install` self-update on macOS in addition to Linux. Refactored platform check logic to keep the implementation DRY and updated both tests and documentation. Also, on these platforms, an additional message is shown recommending `colab update --install` to upgrade in place, positioned above the standard `pip`/`uv` installation command.
5 2026-05-29: Added default OAuth2 client config (`oauth_config.json`) as a bundled package resource and restored fallback loading logic in `get_credentials()`. The CLI now falls back to using these default credentials when no explicit local config is found. Added `integration/repro_bundled_oauth` integration test.
6 2026-05-27: Refactored `colab README` and `colab AGENT` to bundle `README.md` and `AGENTS.md` via Hatchling's `force-include` and read them using `importlib.resources` instead of `importlib.metadata`. `colab AGENT` now correctly prints `AGENTS.md`.
7 2026-05-27: Extended `colab update --install` to detect if the CLI was installed via `uv tool install` (by checking if `sys.executable` contains `/uv/`) and if so, use `uv tool install -U google-colab-cli` to upgrade.
8 2026-05-27: Updated auto-update upgrade hint to recommend `pip install --upgrade google-colab-cli` instead of `colab`, aligning with the PyPI package name.
9 2026-05-27: `colab url` now emits BOTH the `?dbu=<urlencoded path>` query parameter (existing) AND a new `#datalabBackendUrl=<full URL>` hash fragment (new). Format: `https://<host>/notebooks/empty.ipynb?dbu=%2Ftun%2Fm%2F<endpoint>#datalabBackendUrl=<host>/tun/m/<endpoint>`. Why both: some Colab frontend code paths consult the hash fragment first and ignore `dbu` entirely, so the previously-emitted query-only form failed silently for those users (the frontend fell through to allocating a fresh VM via `/tun/m/assign`). The fragment value is a FULL URL with scheme + host (NOT just the path) and is emitted RAW (no URL encoding) because browsers don't decode the fragment before passing `location.hash` to page JS — Colab's parser calls `new URL(rawString)` directly. The fragment host always matches `--host` so Colab's same-origin enforcement on embedded backend URLs doesn't block the connection, and sandbox/dev users (`--host https://colab.sandbox.google.com`) get a sandbox fragment automatically. Three new test cases in `tests/test_url.py` cover the raw-encoding requirement (`%3A`/`%2F` must NOT appear in the fragment), the both-signals-present invariant, and `--open` propagating the fragment to `webbrowser.open()`. Integration-verified live against synthetic session state with three host shapes (default, sandbox, trailing-slash); all produced correctly-shaped URLs with no `//` artifacts.
10 2026-05-07: Added a developer-only `colab whoami` subcommand (hidden from `colab --help`). Mints an access token via the same `auth.get_credentials(...)` path the rest of the CLI uses (honoring the global `--auth=...` flag), refreshes the credentials, then queries `https://oauth2.googleapis.com/tokeninfo` to print the email, scopes, audience, and expiry of whatever the CLI is about to send. Built specifically to short-circuit the "why is my call to colab.pa.googleapis.com 403-ing" debugging loop — the answer is almost always "missing scope" or "wrong identity", both of which `whoami` makes immediately visible. Hidden via `app.command(hidden=True)`; reachable via `colab whoami` or `colab whoami --help`. Suppressed from the daily-update banner check (added to `_AUTO_UPDATE_SUPPRESSED` in `cli.py`) so the banner doesn't obscure the auth output.
11 2026-05-11: Removed the local-file update source (`update_file_path` setting and `_fetch_local` helper); `colab update` now consults PyPI only. Switched the default `update_url` to the canonical PyPI JSON API (`https://pypi.org/pypi/google-colab-cli/json`), which already exposes the `info.version` schema the auto-update subsystem expects. Re-added `colab update --install` as a public self-install path that runs `pip install -U google-colab-cli` against the current `sys.executable`; Linux-only (other platforms exit non-zero with an explanatory message), and a silent no-op when the cached `latest_version` is already at or below the current install.
12 2026-05-12: Added an optional `timeout=` parameter to `ColabRuntime.execute_code` that flows through to both the `execute()` and `execute_interactive()` branches. `colab auth` and `colab drivemount` now pass `timeout=600` (10 min) via a shared `INTERACTIVE_AUTOMATION_TIMEOUT_SEC` constant in `commands/automation.py`. Background: `jupyter_kernel_client` defaults to a 10s wall-clock timeout that is consumed even when the kernel is idle waiting on `input_request`. With the drivefs hook intercepting that request and prompting the user to OAuth in their browser, any user that takes >10s to click through (essentially everyone) hit `TimeoutError` and saw "drivemount failed" even though the mount had actually succeeded server-side. The fix is scoped narrowly to the two human-in-the-loop subcommands; non-interactive paths (`colab exec`, `colab run`, `colab install`, `colab repl --pipe`, `colab console --pipe`) keep the upstream default since they receive continuous iopub traffic that resets the practical inactivity ceiling.
13 ---
14
15 # Design: Automation and Utility (`auth`, `install`, `log`, `pay`, `version`, `update`, `whoami`)
16
17 ## Overview
18
19 These subcommands are implemented by executing Python code on the Colab VM,
20 managing local state, or inspecting the environment.
21
22 ## Authentication Strategies (CLI Backend)
23
24 The CLI supports two authentication strategies for talking to the Colab
25 backend, selected via the global `--auth=<provider>` flag:
26
27 1. **`oauth2`** (default): Public `InstalledAppFlow` via
28 `google-auth-oauthlib`, but run with a **remote copy-paste flow** rather
29 than a localhost server. The CLI prints an authorization URL (with
30 `token_usage=remote`) using the registered HTTPS landing page
31 `https://sdk.cloud.google.com/applicationdefaultauthcode.html`; the user
32 signs in, copies the code Google displays, and pastes it back at the
33 prompt. The refresh token is cached at `~/.config/colab-cli/token.json`.
34 This is the same mechanism `gcloud auth application-default login` uses,
35 and it behaves identically on local, remote, headless, and container
36 hosts (no auto-opened browser, no bound port). We deliberately do **not**
37 use `run_local_server()` (environment-dependent) or the out-of-band (OOB)
38 redirect `urn:ietf:wg:oauth:2.0:oob` (blocked by Google in 2022 — see
39 `_run_remote_flow` / `REMOTE_REDIRECT_URI` in `auth.py`). The
40 `sdk.cloud.google.com` redirect is registered to the cloud-SDK OAuth
41 client (`764086051850-...`), which is also the client shipped in the
42 bundled `oauth_config.json`; reusing it with any other client id yields
43 `redirect_uri_mismatch`. If no local config is provided via
44 `-c/--client-oauth-config` or found at `~/.colab-cli-oauth-config.json`,
45 it falls back to that bundled `oauth_config.json`.
46 2. **`adc`**: Application Default Credentials via `google.auth.default()`.
47 Honors the standard ADC discovery chain
48 (`GOOGLE_APPLICATION_CREDENTIALS`, `gcloud auth application-default
49 login`, GCE/GKE metadata server). Useful when running the CLI from
50 environments that already have ambient Google credentials.
51
52 The choices are encoded as the `AuthProvider` string-enum in `auth.py`. The
53 `get_credentials(config_path, provider)` entry point dispatches on this enum,
54 allowing the core `Client` to remain authentication-agnostic — it only sees a
55 `requests.AuthorizedSession`.
56
57 ### Required Scopes
58
59 The CLI talks to the Colab session backend at `colab.research.google.com`
60 for assignment, unassignment, the contents API, **and keep-alive** (the TFE
61 tunnel ping — see `01_session_management.md`). The `userinfo.email` scope is
62 sufficient for this host.
63
64 > Historical note: keep-alive previously used the `RuntimeService`
65 > (`KeepAliveAssignment`) at `colab.pa.googleapis.com`, which required the
66 > `https://www.googleapis.com/auth/colaboratory` scope **and** the caller to
67 > be a `serviceusage` consumer of Colab's internal project `1014160490159`.
68 > The latter is impossible for ordinary user accounts, which made keep-alive
69 > fail with HTTP 403 `USER_PROJECT_DENIED` for all external users (issue #14).
70 > Keep-alive no longer touches `colab.pa.googleapis.com`.
71
72 How each provider supplies the scope:
73
74 - **`oauth2`**: `PUBLIC_SCOPES` already includes `colaboratory`, so the
75 InstalledAppFlow consent screen lists it. Existing cached tokens at
76 `~/.config/colab-cli/token.json` that were minted before this change must
77 be deleted to trigger a fresh consent flow.
78 - **`adc`**: `google.auth.default(scopes=PUBLIC_SCOPES)` is called, and for
79 credential subclasses that support `with_scopes` (service accounts,
80 GCE/GKE metadata, impersonated) we re-apply via `creds.with_scopes(...)`.
81 User credentials from `gcloud auth application-default login` ignore the
82 `scopes=` kwarg AND raise `NotImplementedError` on `with_scopes`; those
83 users must explicitly re-authenticate:
84
85 ```
86 gcloud auth application-default login \
87 --scopes=openid,\
88 https://www.googleapis.com/auth/cloud-platform,\
89 https://www.googleapis.com/auth/userinfo.email,\
90 https://www.googleapis.com/auth/colaboratory
91 ```
92
93 `userinfo.email` is required for the session backend at
94 `colab.research.google.com` (otherwise assign/unassign/sessions/keep-alive
95 return HTTP 401); `colaboratory` is retained for forward compatibility and
96 other Colab features; `openid` and `cloud-platform` are mandated by
97 `gcloud` itself (`gcloud auth application-default login` rejects scope
98 lists that omit `cloud-platform` with `Invalid value for [--scopes]`).
99
100 `colab new` performs a one-shot keep-alive pre-flight after `assign`
101 succeeds so missing-scope failures surface immediately (with per-provider
102 remediation guidance) rather than silently after ~1 minute via the daemon.
103
104 ## Approach
105
106 ### 1. Authentication (`colab auth`)
107
108 - **Action**: Execute code on the VM to trigger user-interactive
109 authentication using the classic Gcloud fallback.
110 - **Code**: `python import os os.environ['USE_AUTH_EPHEM'] = '0' from
111 google.colab import auth auth.authenticate_user()`
112 - **Handling**: Setting `USE_AUTH_EPHEM` to `'0'` forces the kernel to print a
113 standard `gcloud` verification URL and trigger an `input_request` message on
114 the `iopub` channel. The CLI intercepts this via a `stdin_hook` and prompts
115 the user locally, returning the code to unlock the kernel.
116
117 ### 2. Package Installation (`colab install`)
118
119 - **Action**: Execute `pip` on the VM.
120 - **Code**: `python import sys, subprocess
121 subprocess.check_call([sys.executable, "-m", "pip", "install", "..."])`
122 - **Requirements File**: Upload `requirements.txt` if provided with `-r` and
123 then run `pip install -r`.
124
125 ### 3. Drive Mounting (`colab drivemount`)
126
127 - **Action**: Execute `drive.mount()` and transparently proxy Colab's
128 proprietary credential propagation flow.
129 - **Code**: `python from google.colab import drive
130 drive.mount('/content/drive')`
131 - **Handling**: Because `drivefs` enforces the ephemeral side-channel
132 propagation (`colab_request` over websocket), the CLI intercepts these
133 messages using `ColabRuntime.colab_request_hook`. When intercepted, the CLI
134 automatically interacts with the Colab backend
135 (`/tun/m/credentials-propagation/`), prompts the user with the Google OAuth
136 consent URL if needed, and dispatches the required `colab_reply` message to
137 the `stdin` channel to unlock the kernel thread.
138 - **Timeout**: The kernel is silent (no iopub traffic) the entire time the
139 user is OAuthing in their browser. To avoid the upstream 10s
140 `jupyter_kernel_client` default raising `TimeoutError` mid-flow, this
141 subcommand passes `timeout=INTERACTIVE_AUTOMATION_TIMEOUT_SEC` (600s) to
142 `ColabRuntime.execute_code`. Same applies to `colab auth`.
143
144 ### 4. Logging and Notebook Capture (`colab log`)
145
146 - **Action**: Capture the session's command history and outputs.
147 - **Storage**: Maintain a local JSON-L file of all major operations,
148 executions, and stdin interactions in
149 `~/.config/colab-cli/history/<session_name>.jsonl`.
150 - **Viewing**: `colab log list` and `colab log show <session>`.
151 - **Conversion (Planned)**: Future expansion to convert history logs to
152 `.ipynb` or `.html`.
153
154 ### 5. Subscription Management (`colab pay`)
155
156 - **Action**: Open the Colab signup page in the user's browser.
157 - **Implementation**: Uses
158 `webbrowser.open("https://colab.research.google.com/signup")`.
159
160 ### 6. Version Information (`colab version`)
161
162 - **Action**: Show the current version of the Colab CLI.
163 - **Implementation**:
164 - Attempts to retrieve the version using
165 `importlib.metadata.version("colab")`.
166 - If not installed (e.g., running from source), it falls back to the short
167 Git commit hash using `git rev-parse --short HEAD`.
168 - Dynamic versioning is supported in the build system via `hatch-vcs`.
169
170 ### 7. Auto-Update (`colab update`)
171
172 - **Action**: Check if a new version of the Colab CLI is available.
173 - **Auto-check**: The CLI automatically checks for updates once every 24 hours
174 during the execution of any command. Independently, the cached
175 `latest_version` (see below) is consulted on **every** invocation so the
176 upgrade banner remains visible between fetches without requiring a network
177 round-trip.
178 - **Suppressed subcommands**: To keep machine-parseable output clean, the
179 daily fetch and the cached banner are suppressed for `update` (which
180 runs its own check), `version`, `log`, `pay`, `url`, `help`, and
181 `whoami`. The list lives as `_AUTO_UPDATE_SUPPRESSED` in the global
182 Typer callback in `cli.py`.
183 - **Manual-check**: `colab update` forces a check and prints the status.
184 - **Implementation**:
185 - Fetches a PyPI-style JSON document from a configurable `update_url`
186 (default: `https://pypi.org/pypi/google-colab-cli/json`) and reads
187 `info.version`.
188 - Compares the fetched version with the current CLI version using
189 PEP 440 / semantic versioning, falling back to string equality when a
190 version is unparseable.
191 - Persists the following fields in `~/.config/colab-cli/settings.json`:
192 - `update_url`: source configuration.
193 - `last_check`: timestamp of the last fetch (drives the daily
194 throttle).
195 - `enable_update_check`: master switch for both the daily fetch and
196 the cached banner.
197 - `latest_version`: highest version observed during the most
198 recent successful check. Updated whenever a strictly-newer
199 version is observed (never downgraded), and preserved verbatim
200 across failed checks so transient network issues do not erase
201 the cache.
202 - **Notification**: If a new version is found, a non-intrusive message is
203 printed to the console with a `Run 'pip install --upgrade google-colab-cli' to
204 update.` hint. On Linux and macOS platforms where `--install` self-update is supported,
205 an additional hint `You can run 'colab update --install' to upgrade in place.`
206 is displayed above the pip/uv install command. The cached banner shown between
207 fetches uses the generic `Run 'colab update' to update.` hint.
208 - **Self-install (`--install`)**: An opt-in `--install` flag (default
209 `False`) makes `colab update` upgrade the CLI in place (**Linux and macOS**).
210 It detects how the CLI was installed:
211 - If `sys.executable` contains `/uv/tools` (indicating it was installed via
212 `uv tool install`), it runs `uv tool install -U google-colab-cli`.
213 - Otherwise, runs `pip install -U google-colab-cli` using `sys.executable`
214 to ensure the upgrade lands in the same interpreter.
215 On other platforms, the command exits non-zero with an explanatory
216 message. When the cached `latest_version` is already at or below the
217 current install, the flag is a silent no-op so it is safe to wire into
218 automation. If the upgrade command exits non-zero, `colab update --install`
219 propagates the same exit code.
220
221 ### 8. Identity Inspection (`colab whoami`) [developer-only]
222
223 - **Action**: Resolve the active credentials, mint an access token, and
224 print the email, audience, scopes, and expiry of that token.
225 - **Visibility**: Registered with `hidden=True` so it does not appear in
226 `colab --help`. Discoverable via source code, `colab whoami --help`, or
227 word-of-mouth. The intent is to keep the public surface focused on
228 end-user commands while still giving developers a one-shot debugging
229 aid.
230 - **Implementation**:
231 - Calls `auth.get_credentials(state.client_oauth_config,
232 provider=state.auth_provider)` — the exact same code path the
233 `Client` uses — so the token reflects what the rest of the CLI
234 would actually send.
235 - Always calls `creds.refresh(Request())` before reading
236 `creds.token`. Service-account, GCE/GKE-metadata, and some
237 impersonated credentials lazy-mint the token; without an explicit
238 refresh `creds.token` is `None` even for valid credentials.
239 - Hits `https://oauth2.googleapis.com/tokeninfo?access_token=<token>`
240 via stdlib `urllib.request` rather than the already-authorized
241 `requests.AuthorizedSession`. The tokeninfo endpoint accepts the
242 token as a query parameter and does NOT want a `Bearer` header
243 alongside it.
244 - Renders `expires_in` (seconds) as minutes for readability.
245 - On HTTP 4xx from tokeninfo (typical for revoked/expired tokens),
246 the JSON error body is surfaced verbatim rather than being
247 swallowed; the developer needs to see *why* the token was
248 rejected.
249 - **Output shape**:
250 ```
251 Auth provider: adc
252 Email: user@example.com
253 Audience: 764086051850-...apps.googleusercontent.com
254 Expires in: 47m
255 Scopes:
256 - email
257 - https://www.googleapis.com/auth/cloud-platform
258 - https://www.googleapis.com/auth/colaboratory
259 - https://www.googleapis.com/auth/userinfo.email
260 - openid
261 ```
262
263 ### 9. README and AGENT (`colab README`, `colab AGENT`)
264
265 - **Action**: Print the bundled `README.md` or `AGENTS.md` file.
266 - **Implementation**:
267 - Uses `importlib.resources.files("colab_cli").joinpath(...)` to read the
268 bundled `README.md` (for `colab README`) or `AGENTS.md` (for `colab AGENT`)
269 from the package resources.
270 - The files are bundled into the package via Hatchling's `force-include`
271 configuration in `pyproject.toml`.
272 - If reading from resources fails (e.g. during development when not
273 installed), it falls back to reading the files from the project root.
274 - Prints the content to stdout.
275
276 ## Implementation Details
277
278 - **Code Injection**: Use a standard `run_code(session, code)` helper via
279 `ColabRuntime`.
280 - **History Management**: Use `HistoryLogger` class to append structured
281 events to session-specific `.jsonl` files.
282 - **Interactive Prompts**: Instrumented `stdin_hook` and `colab_request_hook`
283 to record interactive user input and proprietary backend requests.
284
285 ## Testing Strategy
286
287 TDD is mandatory for all automation features.
288
289 ### 1. Mock Kernel Injection
290
291 - **Test Case**: Verify `colab auth` correctly injects `from google.colab
292 import auth; auth.authenticate_user()`.
293 - **Test Case**: Verify `colab install` correctly injects `pip install` or `uv
294 install` commands to the remote VM kernel.
295 - **Test Case**: Verify `colab drivemount` correctly injects `drive.mount()`
296 commands and registers the `colab_request_hook` to intercept credential
297 propagation events.
298
299 ### 2. History Capture
300
301 - **Test Case**: Verify all code sent via `exec` is correctly appended to the
302 JSON-L history file for that session.
303 - **Test Case**: Verify `colab log` correctly generates an `.ipynb` from a
304 populated history file.
305
306 ### 3. `whoami` Identity Resolution
307
308 - **Test Case**: Mock the credentials + `urllib.request.urlopen` to return a
309 fake tokeninfo payload; verify the printed output contains the email, the
310 active auth provider name, the scopes (one per line), and a human-readable
311 expires-in (minutes, not raw seconds).
312 - **Test Case**: When `urlopen` raises `HTTPError(400)` (revoked/expired
313 token), `whoami` exits non-zero with a message identifying the failure
314 rather than emitting an unhandled traceback.
315 - **Test Case**: `colab --help` does NOT mention `whoami` (regression
316 against accidental un-hiding) but `colab whoami --help` still shows the
317 command's own help text.
318 - **Test Case**: `creds.refresh()` is called before `creds.token` is read
319 (regression against silently-`None` tokens for service-account /
320 GCE-metadata creds).
321
322 ### 4. `README` and `AGENT` Commands
323
324 - **Test Case**: Verify `colab README` prints the expected content when package metadata is available.
325 - **Test Case**: Verify `colab AGENT` prints the same content.
326 - **Test Case**: Verify fallback to local `README.md` file when metadata is not available.
327 - **Test Case**: Verify error exit when both metadata and local file are unavailable.