d99b8202 · samitouri/QOSAMI-X402Tunnel-1

@@ -32,7 +32,7 @@ UDP client

32 ### Transport and Routing

33

34 - Raw TCP reverse-connect is the canonical stream transport.

35 -- Do not introduce websocket or legacy compatibility paths unless a new ADR supersedes ADR-0002.

35 +- Do not introduce websocket or legacy compatibility paths by default.

36 - Derive lease hostnames from the full normalized `PORTAL_URL` host, not from apex extraction.

37 - Preserve explicit root-host fallback through SNI no-route handling to the admin/API listener.

38 - Stream ingress is TLS-only. UDP exposure, when enabled, is raw UDP.

@@ -100,75 +100,12 @@ Portal has three distinct network roles:

100

101 That distinction matters because `/sdk/connect` stops being ordinary HTTP once hijacked, while the UDP backhaul is a separate internal QUIC carrier.

102

103 -## Core Components

104 -

105 -### Relay Server (`cmd/relay-server`)

106 -

107 -- Admin/API TLS listener on `--api-port` (default `:4017`)

108 -- SNI listener on `--sni-port` (default `:443`)

109 -- Public frontend routes under `/`, `/app`, `/assets/*`

110 -- Minimal admin surface at `/admin`, `/admin/snapshot`, and admin action/auth routes under `/admin/*`

111 -- Tunnel bootstrap routes at `/install.sh`, `/install.ps1`, and `/install/bin/*`

112 -- Keyless signer endpoint at `/v1/sign`

113 -

114 -### Relay Core (`portal/`)

115 -

116 -- `Server`: owns listeners, lease registry, API handlers, discovery, and shutdown lifecycle

117 -- `routeTable`: exact + single-label wildcard hostname lookup

118 -- `transport.RelayStream`: per-lease ready queue for reverse stream sessions

119 -- `transport.RelayTCPPort`: per-lease TCP listener on an allocated port; bridges incoming connections to reverse sessions using raw TCP (no TLS)

120 -- `transport.RelayDatagram`: per-lease raw UDP socket plus QUIC DATAGRAM bridge runtime

121 -- `transport.PortAllocator`: range-based port allocator with sticky name-based reservation and grace period

122 -- UDP and raw TCP allocate independently from the same inclusive `MIN_PORT-MAX_PORT` range, so the same numeric port may exist on both protocols

123 -- `transport.datagramSession`: internal QUIC DATAGRAM bind/send/receive primitive shared by relay and SDK datagram runtimes

124 -- `acme`: Cloudflare/Google Cloud DNS/Route53-backed root/wildcard A-record sync + certificate provisioning/renewal for the relay root host and wildcard

125 -- `keyless`: admin/API TLS attach helpers and tenant-side signer integration

126 -- `auth`: SIWE register challenge creation/verification plus lease access token issue/verify

127 -- `discovery`: relay descriptor publication over relay HTTPS plus relay-set synchronization

128 -- `wireguard`: optional relay overlay network used to reach peer relay APIs over internal overlay IPs and keep relay peer state synchronized

129 -- `Server` additionally owns `quicTunnel` (QUIC listener, ALPN `portal-tunnel`) when UDP transport is enabled

130 -

131 -### SDK (`sdk/`)

132 -

133 -- `ExposeConfig.Discovery`: when true, `Expose` fetches the default Portal relay registry, merges it with explicit relay inputs, normalizes the result, and runs the relay discovery loop

134 -- Entry points can opt out of registry defaults and call `utils.NormalizeRelayURLs` directly when they need explicit relay inputs only

135 -- `Listener`: validates one relay URL locally, then starts relay compatibility checks, SIWE-based lease registration, reverse session maintenance, and lease renewal in the background until ready

136 -- `Listener` owns a `transport.ClientStream` and, when UDP is enabled, a `transport.ClientDatagram`

137 -- `api_client.go`: internal relay client for register challenge, register, renew, unregister, reverse session dialing, and QUIC tunnel setup

138 -- `mitm.go`: tenant-side TLS passthrough self-probe. The SDK opens a probe connection to its own public URL, compares TLS exporter values on both SDK-controlled ends, and logs suspected relay-side TLS termination on mismatch; strict callers can opt into relay banning instead

139 -- `ListenerConfig.RetryCount <= 0` means retry forever; positive values close the listener after the retry budget is exhausted

140 -- `NewListener` callers provide explicit normalized relay URLs

141 -- Default exposure flow is `Expose{Discovery: true} -> PublicURLs -> http.Server.Serve(exposure)`, with an opt-out path for explicit relay inputs only

142 -- `expose.go`: optional `RunHTTP` helper for serving one handler on both a local HTTP port and the relay listener

143 -- `Expose` keeps one listener per configured relay URL. Relay startup and reconnect failures are retried independently per relay, and successful relays remain available while failed relays keep retrying in the background

144 -- `Exposure.RelayURLs()` returns the configured normalized relay URLs, while `Exposure.PublicURLs()` returns only relays that are currently registered and ready

145 -- Relay-aware entry inspection is reserved for advanced callers such as `portal-tunnel`

146 -- Tenant TLS is created automatically through the relay keyless signer; callers do not provide a local self-signed fallback path

147 -- MITM self-probes are traffic-triggered, not periodic. A listener triggers at most one asynchronous probe per 30-second cooldown, and only after real tenant traffic performs I/O on an accepted connection

148 -- Probe identification does not use a dedicated ALPN or fixed plaintext marker. The first encrypted probe payload is `nonce + random padding`, and inbound probe matching is only attempted while a probe is in flight

149 -- `Listener.AcceptDatagram()` / `SendDatagram()`: read/write datagram frames via the client datagram runtime

150 -- `Listener.DatagramReady()`: reports the published `udp_addr` plus whether the QUIC datagram plane is currently connected

151 -- `Exposure.AcceptDatagram()`: receives datagrams from all backing relay listeners with relay context populated on `DatagramFrame`

152 -- `Exposure.SendDatagram()`: sends a datagram frame back through the owning relay listener

153 -- `Exposure.WaitDatagramReady()`: blocks until at least one relay listener has both a published `udp_addr` and a connected datagram plane

154 -

155 -### Tunnel (`cmd/portal-tunnel`)

156 -

157 -- Builds the `portal` CLI and exposes subcommands such as `portal expose` and `portal list`

158 -- Loads or creates the local signing identity from `--identity-path` before starting the SDK exposure

159 -- Creates one SDK listener per relay through the SDK and consumes one aggregate listener

160 -- Accepts claimed tenant connections from the relay

161 -- Proxies raw TCP to a local target passed to `portal expose`

162 -- Optionally requests a dedicated TCP port on the relay for raw TCP services when `--tcp` is enabled

163 -- Optionally proxies raw UDP to a separate local UDP target when `--udp` is enabled

164 -- Returns an HTTP 503 response when the local target is unavailable

165 -- `--tcp` flag (bool, default `false`): requests a dedicated TCP port on the relay for non-TLS services (e.g., Minecraft, game servers)

166 -- `--udp` flag (bool, default `false`): enables UDP relay in addition to TCP

167 -- `--udp-addr` flag (string): local UDP target address (`host:port` or port only); required when `--udp` is enabled

168 -- `--ban-mitm` flag (bool, default `false`): when enabled, TLS self-probe mismatches ban the relay for the current exposure instead of only logging

169 -- `runUDPBestEffort`: waits for datagram readiness, then calls `proxyExposureDatagrams`

170 -- `proxyExposureDatagrams` (`relays.go`): per-flow UDP sockets to local target with idle cleanup; uses `Exposure.SendDatagram()` for the return path

171 -- Best-effort UDP failures are logged but do not terminate the TCP tunnel

103 +## Package Layout

104 +

105 +The relay runtime lives in `portal/` (server, route table, transport runtimes, ACME, keyless, auth, discovery, WireGuard overlay, policy).

106 +The SDK client library lives in `sdk/` (listener, exposure, relay API client, MITM self-probe, transport clients).

107 +CLI entry points live in `cmd/relay-server` and `cmd/portal-tunnel`; they import `portal/` and `sdk/` respectively but never each other.

108 +Shared wire types, API envelope, error codes, path constants, and transport frame codec live in `types/`.

109

110 ## Transport Model

111

@@ -199,132 +136,63 @@ Result: this is a detect-only signal by default. It raises the cost of adaptive

136

137 ### TCP Port Transport (non-TLS)

138

202 -1. SDK/tunnel requests a register challenge with `tcp_enabled=true`, signs the returned SIWE message, and then completes `POST /sdk/register`.

203 -2. Relay validates that the TCP port plane is enabled (server has `TCP_ENABLED=true`, a valid `MIN_PORT/MAX_PORT` range, and admin has enabled TCP port), allocates a TCP port via `PortAllocator`, and creates a `transport.RelayTCPPort` for the lease.

204 -3. Registration response includes `tcp_addr` (public TCP endpoint, e.g., `hostname:40001`).

205 -4. `RelayTCPPort` starts a TCP listener on the allocated port.

206 -5. An external TCP client connects to `tcp_addr`.

207 -6. `RelayTCPPort.acceptLoop` accepts the connection and claims a reverse session from the lease `RelayStream` using `ClaimRaw` (writes `0x01` marker instead of `0x02`).

208 -7. SDK-side `ClientStream.runSession` receives `0x01`, calls `activateRaw` which passes the raw connection directly without TLS handshake.

209 -8. `RelayTCPPort.bridgeConns` copies data bidirectionally between the external client and the reverse session.

210 -

211 -```text

212 -Client --TCP--> [:MIN_PORT-MAX_PORT Relay] --raw TCP--> [RelayTCPPort] --ClaimRaw--> [reverse session] --0x01--> [ClientStream] --> Local Service

213 - <--bidirectional bridge--

214 -```

139 +1. SDK/tunnel requests a register challenge with `tcp_enabled=true`, signs the returned SIWE message, and completes registration.

140 +2. Relay validates that the TCP port plane is enabled, allocates a TCP port, and creates a per-lease TCP listener.

141 +3. Registration response includes `tcp_addr` (public TCP endpoint).

142 +4. An external TCP client connects to `tcp_addr`.

143 +5. The relay accepts the connection, claims a reverse session from the lease stream queue, and writes `0x01` (raw TCP activation marker).

144 +6. SDK-side receives `0x01` and passes the raw connection directly without TLS handshake.

145 +7. Data is copied bidirectionally between the external client and the reverse session.

146

147 Result: the relay allocates a dedicated TCP port per lease and bridges raw TCP without TLS. This is ideal for non-TLS protocols like Minecraft, game servers, or any raw TCP service.

148

149 ### UDP/QUIC Datagram Transport

150

220 -1. SDK/tunnel requests a register challenge with `udp_enabled=true`, signs the returned SIWE message, and then completes `POST /sdk/register`.

221 -2. Relay validates that the datagram plane is enabled (server has `UDP_ENABLED=true`, a valid `MIN_PORT/MAX_PORT` range, and admin has enabled UDP), allocates a UDP port via `PortAllocator`, and creates a `transport.RelayDatagram` for the lease.

222 -3. Registration response includes `udp_addr` (public UDP endpoint), `access_token`, and `sni_port`. The SDK dials QUIC to the relay URL host on that `sni_port`.

223 -4. SDK `transport.ClientDatagram` opens a QUIC connection with ALPN `portal-tunnel` and QUIC DATAGRAM support enabled.

224 -5. Authentication: SDK sends `{access_token}` JSON on the first QUIC stream; the relay validates that lease access token before calling `RelayDatagram.Register(conn)`.

225 -6. External UDP client sends a packet to `udp_addr` -> `RelayDatagram.readLoop` -> `TouchFlow` (assigns flow ID) -> `SendDatagram` -> QUIC DATAGRAM frame.

226 -7. SDK-side `datagramSession.receiveLoop` decodes frames -> `Listener.AcceptDatagram()` -> `Exposure.AcceptDatagram()` -> `proxyExposureDatagrams` -> local UDP target.

227 -8. Return path: local response -> `Exposure.SendDatagram()` -> `Listener.SendDatagram()` -> `ClientDatagram.Send()` -> QUIC DATAGRAM -> `RelayDatagram.dispatch()` -> `conn.WriteToUDP` to the original client.

151 +1. SDK/tunnel requests a register challenge with `udp_enabled=true`, signs the returned SIWE message, and completes registration.

152 +2. Relay validates that the datagram plane is enabled, allocates a UDP port, and creates a per-lease datagram runtime.

153 +3. Registration response includes `udp_addr`, `access_token`, and `sni_port`. The SDK dials QUIC to the relay on `sni_port`.

154 +4. SDK opens a QUIC connection with ALPN `portal-tunnel` and DATAGRAM support enabled.

155 +5. Authentication: SDK sends `{access_token}` JSON on the first QUIC stream; relay validates before accepting the tunnel.

156 +6. External UDP client sends a packet to `udp_addr` -> relay assigns a flow ID -> QUIC DATAGRAM frame to SDK.

157 +7. SDK-side decodes frames and delivers to local UDP target.

158 +8. Return path: local response -> SDK -> QUIC DATAGRAM -> relay -> `WriteToUDP` to the original client.

159

229 -```text

230 -Client --UDP--> [:MIN_PORT-MAX_PORT Relay] --DATAGRAM--> [RelayDatagram] --QUIC--> [ClientDatagram] --UDP--> Local Service

231 - <--QUIC DATAGRAM return path--

232 -```

233 -

234 -Wire format (`types/transport.go`): `[flowID uvarint][payload bytes]`

160 +Result: raw public UDP exposure with an internal QUIC datagram backhaul. UDP and TCP port allocations are independent from the same `MIN_PORT-MAX_PORT` range.

161

162 ## WireGuard Overlay and Discovery

163

238 -- Discovery starts from bootstrap relay URLs over normal public HTTPS.

239 -- Discovery descriptors are currently transport-authenticated by the queried relay endpoint, not by embedded descriptor signatures.

240 -- Current discovery validation covers protocol version, descriptor normalization, required fields, expiry, target URL/identity matching, and overlay field sanity only.

241 -- Descriptor `identity.address` is a relay claim inside discovery. Independent `domain -> address` verification comes from optional ENS/DNSSEC evidence, not from the discovery payload itself.

242 -- Each relay publishes a descriptor over relay HTTPS that may advertise:

243 - - `wireguard_public_key`

244 - - `wireguard_endpoint`

245 - - `overlay_ipv4`

246 - - optional `overlay_cidrs`

247 -- When discovery is enabled and the relay has a WireGuard private key, the relay creates an internal overlay interface and derives:

248 - - a relay WireGuard public key

249 - - an overlay IPv4 identity

250 - - a peer API listener bound on the overlay address

164 +- Discovery bootstraps from public HTTPS relay URLs, then optionally synchronizes over WireGuard overlay.

165 +- Discovery descriptors are transport-authenticated by the queried relay endpoint, not by embedded signatures. Independent `domain -> address` verification comes from optional ENS/DNSSEC evidence, not from the discovery payload itself.

166 - The overlay peer API is plain HTTP on the WireGuard network, not public Internet HTTP. It serves the same discovery payload shape used by public `/discovery`.

252 -- Bootstrap relays are discovered first over public HTTPS. Non-bootstrap relays that advertise overlay support become sync candidates and are polled again over the WireGuard overlay.

253 -- Relay-set snapshots are translated into WireGuard peers with:

254 - - peer public key

255 - - endpoint

256 - - allowed IPs = peer overlay `/32` plus advertised overlay CIDRs

257 -- Overlay failure affects inter-relay discovery and mesh synchronization only. Tenant stream routing, keyless TLS, register/renew/connect, and public UDP ingress do not depend on the WireGuard transport path directly.

167 +- Overlay failure affects inter-relay discovery and mesh synchronization only. Tenant stream routing, keyless TLS, register/renew/connect, and public UDP ingress do not depend on the WireGuard transport path.

168

169 ## Control Plane Flow

170

171 ### 1. Register

172

263 -- `POST /sdk/register/challenge`

264 -- `POST /sdk/register`

265 -- JSON envelope response

266 -- Challenge request fields:

267 - - `identity`

268 - - `metadata`

269 - - `ttl`

270 - - `udp_enabled`

271 - - `tcp_enabled`

272 -- Challenge response fields:

273 - - `challenge_id`

274 - - `expires_at`

275 - - `siwe_message`

276 -- Caller signs the returned SIWE message with the identity Ethereum private key (`personal_sign`) and then submits:

277 - - `challenge_id`

278 - - `siwe_message`

279 - - `siwe_signature`

280 -- `name` must be a valid single DNS label and relay publishes the lease at `<name>.<root host>`

281 -- Registration reserves the hostname and publishes the route immediately; if no reverse session is ready yet, inbound SNI claims wait up to `ClaimTimeout`

282 -- When registration succeeds, the response includes:

283 - - `identity`

284 - - `hostname`

285 - - `expires_at`

286 - - `access_token`

287 - - optional `udp_addr`

288 - - optional `tcp_addr`

289 -- `access_token` is a relay-issued ES256K JWT signed by the relay identity key and validated with:

290 - - `iss = PORTAL_URL`

291 - - `aud = portal-sdk`

292 - - `sub = identity.key()`

293 - - `identity`

294 - - `iat`, `nbf`, `exp`

295 - - `jti`

296 -- UDP registration requires three conditions: server must have `UDP_ENABLED=true`, a valid `MIN_PORT/MAX_PORT` range, and admin must enable UDP in the admin panel

297 -- `APIErrorCodeUDPDisabled` (HTTP 403) when UDP is disabled by admin policy

298 -- `APIErrorCodeUDPCapacityExceeded` (HTTP 503) when the admin-configured max UDP lease limit is reached

299 -- `APIErrorCodeUDPPortExhausted` (HTTP 503) when the UDP port pool is exhausted

300 -- TCP port registration requires three conditions: server must have `TCP_ENABLED=true`, a valid `MIN_PORT/MAX_PORT` range, and admin must enable TCP port in the admin panel

301 -- `APIErrorCodeTCPPortDisabled` (HTTP 403) when TCP port is disabled by admin policy

302 -- `APIErrorCodeTCPPortCapacityExceeded` (HTTP 503) when the admin-configured max TCP port lease limit is reached

303 -- `APIErrorCodeTCPPortExhausted` (HTTP 503) when the TCP port pool is exhausted

304 -- `PORTAL_URL` is normalized to its host component only; path/query segments are ignored for routing

173 +- `POST /sdk/register/challenge` then `POST /sdk/register`.

174 +- Caller signs the returned SIWE message with the identity secp256k1 key (`personal_sign`).

175 +- `name` must be a valid single DNS label; the relay publishes the lease at `<name>.<root host>`.

176 +- Registration reserves the hostname and publishes the route immediately; if no reverse session is ready yet, inbound SNI claims wait up to `ClaimTimeout`.

177 +- On success, the relay issues a lease-scoped ES256K JWT access token signed by the relay identity key, used for the rest of the lease lifecycle.

178 +- UDP registration requires server `UDP_ENABLED=true`, a valid `MIN_PORT/MAX_PORT` range, and admin enablement. Failures: `udp_disabled` (403), `udp_capacity_exceeded` (503), `udp_port_exhausted` (503).

179 +- TCP port registration has equivalent three-condition gating. Failures: `tcp_port_disabled` (403), `tcp_port_capacity_exceeded` (503), `tcp_port_exhausted` (503).

180 +- `PORTAL_URL` is normalized to its host component only; path/query segments are ignored for routing.

181

182 ### 2. Reverse Connect

183

308 -- `GET /sdk/connect`

309 -- Requires HTTP/1.1

310 -- Requires `X-Portal-Access-Token` header with the lease access token

311 -- Relay validates:

312 - - lease exists and is not expired

313 - - the lease access token signature, issuer, audience, identity, and expiry are valid

314 -- After claim, relay writes `0x02` before switching the session into tenant TLS passthrough

315 -- After hijack, the connection becomes a broker-managed reverse session

184 +- `GET /sdk/connect` (HTTP/1.1 only, `X-Portal-Access-Token` header).

185 +- Relay validates: lease exists and is not expired; access token signature, issuer, audience, identity, and expiry are all valid.

186 +- After claim, relay writes `0x02` before switching the session into tenant TLS passthrough.

187 +- After hijack, the connection becomes a broker-managed reverse session.

188

189 ### 3. Renew

190

319 -- `POST /sdk/renew`

320 -- Requires `access_token`

321 -- Extends lease TTL and returns a refreshed `access_token`

191 +- `POST /sdk/renew` with `access_token`. Extends lease TTL and returns a refreshed token.

192

193 ### 4. Unregister

194

325 -- `POST /sdk/unregister`

326 -- Requires `access_token`

327 -- Removes the lease, routes, and ready reverse sessions

195 +- `POST /sdk/unregister` with `access_token`. Removes the lease, routes, and ready reverse sessions.

196

197 ## Routing Behavior

198

@@ -342,61 +210,11 @@ Notes:

210

211 ## Admin and Frontend Surface

212

345 -Current relay-served public routes:

346 -

347 -- `/`

348 -- `/app`

349 -- `/app/*`

350 -- `/assets/*`

351 -- `/admin`

352 -- `/admin/snapshot`

353 -- `/admin/leases/*`

354 -- `/install.sh`

355 -- `/install.ps1`

356 -- `/install/bin/*`

357 -- `/healthz`

358 -- `/v1/sign`

359 -- `/sdk/*`

360 -

361 -The admin surface is intentionally small in the current Go runtime: an HTML index, one JSON snapshot endpoint, and a small set of admin action/auth routes.

362 -

363 -## Shared Contract Surface

364 -

365 -Cross-package public contract lives in:

366 -

367 -- `types/api.go`

368 - - API envelope

369 - - shared request/response DTOs

370 - - lease metadata

371 -- `types/error.go`

372 - - shared API error codes (including TCP port: `tcp_port_disabled`, `tcp_port_exhausted`, `tcp_port_capacity_exceeded`)

373 - - shared MITM self-probe reason codes

374 -- `types/types.go`

375 - - shared headers

376 - - reverse marker constants

377 -- `types/paths.go`

378 - - shared `/sdk/*`, admin, health, install, and signer paths

379 -- `types/transport.go`

380 - - `ErrDatagramTooSmall`

381 - - `DatagramFrame` wire frame plus SDK relay context

382 - - `EncodeDatagram` / `DecodeDatagram`

383 -

384 -Relay-local frontend asset filenames stay in `cmd/relay-server`, not `types/`.

385 -

386 -## Keyless and Certificates

387 -

388 -- Relay admin/API TLS uses the certificate in `KEYLESS_DIR`

389 - - `fullchain.pem`

390 - - `privatekey.pem`

391 -- For non-localhost deployments, Portal can either use those files directly or manage them through ACME.

392 -- When ACME is enabled, DNS-01 currently supports `cloudflare`, `gcloud`, and `route53`, and keeps:

393 - - root host A record

394 - - wildcard host A record

395 - - relay certificate renewal

396 -- When manual certificate files are present and valid, Portal uses them instead of provisioning a new certificate, but can still use `ACME_DNS_PROVIDER` for ENS gasless DNSSEC/TXT automation.

397 -- SDK/tunnel fetches the relay certificate chain from the relay root host, verifies that the leaf covers tenant hostnames, and builds a tenant-side `tls.Config` with a remote signer backed by `/v1/sign`

398 -- During tenant TLS handshake, the SDK/tunnel endpoint acts as the TLS server and derives tenant session keys locally; the relay only signs handshake digests and does not receive tenant TLS traffic secrets

399 -- Relay control-plane TLS and reverse-session setup still terminate on the relay's admin/API listener and are not protected by the tenant keyless TLS path

213 +The admin surface is intentionally small: an HTML index, one JSON snapshot endpoint, and a small set of admin action/auth routes. Route paths are enumerated in `types/paths.go` and `cmd/relay-server`.

214 +

215 +## Keyless TLS Trust Model

216 +

217 +The relay signs handshake digests via `/v1/sign` but never receives tenant TLS traffic secrets. The SDK/tunnel endpoint runs the full TLS server handshake and derives session keys locally. Relay control-plane TLS and reverse-session setup terminate on the relay's admin/API listener and are not protected by the tenant keyless path.

218

219 ## Design Properties

220

@@ -413,7 +231,3 @@ Relay-local frontend asset filenames stay in `cmd/relay-server`, not `types/`.

231 - Optional QUIC/UDP datagram transport coexisting with TCP on the same lease

232 - Per-lease UDP and TCP port allocation with sticky name-based reservation

233 - QUIC tunnel authentication via control stream (`access_token`)

416 -

417 -## ADRs

418 -

419 -- Decision records: [docs/adr/README.md](./adr/README.md)

refact: streamline architecture documentation for clarity and organization