main
md 369 lines 19.7 KB
Rendered Raw
1 ---
2 title: Architecture
3 description: Portal system architecture, transport model, and protocol details.
4 priority: P1
5 ---
6
7 <script>
8 import Mermaid from '$lib/components/Mermaid.svelte'
9
10 const overviewDiagram = `flowchart TD
11 CB[Client Browser]
12 ETC[External TCP Client]
13 EUC[External UDP Client]
14
15 subgraph Relay["Relay Server"]
16 SNI["SNI Router :443"]
17 API["API Server / Control Plane"]
18 TCP["TCP Port Listener"]
19 QUIC["QUIC Datagram Listener"]
20 end
21
22 subgraph SDK["SDK / portal-tunnel"]
23 RC["Reverse Connect Handler"]
24 TLS["Tenant TLS Terminator"]
25 SDKUDP["UDP Forwarder"]
26 end
27
28 LS["Local Service"]
29 LU["Local UDP Service"]
30
31 CB -- TLS ClientHello --> SNI
32 SNI -- SNI route + 0x02 marker --> RC
33 RC --> TLS --> LS
34
35 ETC -- raw TCP --> TCP
36 TCP -- 0x01 marker + bridge --> RC
37
38 EUC -- UDP packet --> QUIC
39 QUIC -- QUIC DATAGRAM frame --> SDKUDP --> LU
40
41 SDK -- POST /sdk/register, GET /sdk/connect --> API`
42
43 const tlsStreamDiagram = `sequenceDiagram
44 participant SDK as SDK / portal-tunnel
45 participant Relay as Relay Server
46 participant Client as Client Browser
47
48 SDK->>Relay: POST /sdk/register/challenge
49 Relay->>SDK: SIWE challenge message
50 SDK->>Relay: POST /sdk/register (signed)
51 Relay->>SDK: access_token + lease info
52
53 SDK->>Relay: GET /sdk/connect (HTTP/1.1 hijack)
54 Relay->>SDK: connection hijacked, 0x00 keepalives
55 Note over Relay: Session queued in per-lease stream ready queue
56
57 Client->>Relay: TLS ClientHello (SNI: name.relay.host)
58 Note over Relay: SNI peek, resolve lease, claim reverse session
59 Relay->>SDK: write 0x02 (TLS activation marker)
60 Note over SDK: Starts tenant TLS handshake locally via keyless signer
61 Relay->>Client: bridges raw encrypted bytes bidirectionally
62 Note over Client,SDK: End-to-end TLS, relay never sees plaintext`
63
64 const tcpPortDiagram = `sequenceDiagram
65 participant SDK as SDK / portal-tunnel
66 participant Relay as Relay Server
67 participant ExtClient as External TCP Client
68
69 SDK->>Relay: POST /sdk/register (tcp_enabled=true, signed SIWE)
70 Note over Relay: Validates TCP plane enabled, allocates port from MIN_PORT-MAX_PORT
71 Relay->>SDK: tcp_addr + access_token
72
73 SDK->>Relay: GET /sdk/connect (reverse session, HTTP/1.1 hijack)
74 Note over Relay: Session queued in per-lease stream ready queue
75
76 ExtClient->>Relay: TCP connect to tcp_addr
77 Note over Relay: Accepts connection, claims reverse session
78 Relay->>SDK: write 0x01 (raw TCP activation marker)
79 Note over SDK: Receives 0x01, passes raw connection without TLS handshake
80 Relay->>ExtClient: bridges data bidirectionally (raw TCP)
81 Note over ExtClient,SDK: No TLS, pure raw TCP passthrough`
82
83 const udpQuicDiagram = `sequenceDiagram
84 participant SDK as SDK / portal-tunnel
85 participant Relay as Relay Server
86 participant UDPClient as External UDP Client
87
88 SDK->>Relay: POST /sdk/register (udp_enabled=true, signed SIWE)
89 Note over Relay: Allocates UDP port from MIN_PORT-MAX_PORT
90 Relay->>SDK: udp_addr + access_token + sni_port
91
92 SDK->>Relay: QUIC connect to sni_port (ALPN: portal-tunnel, DATAGRAM enabled)
93 SDK->>Relay: Send access_token on first QUIC stream
94 Note over Relay: Validates token, registers QUIC tunnel for lease
95
96 UDPClient->>Relay: UDP packet to udp_addr
97 Note over Relay: Assigns flow ID, wraps in QUIC DATAGRAM frame
98 Relay->>SDK: QUIC DATAGRAM frame (flowID + payload)
99 Note over SDK: Decodes frame, forwards to local UDP service
100
101 SDK->>Relay: QUIC DATAGRAM frame (flowID + response)
102 Relay->>UDPClient: WriteToUDP back to original client address`
103
104 const registrationDiagram = `sequenceDiagram
105 participant SDK as SDK / portal-tunnel
106 participant Relay as Relay Server
107
108 SDK->>Relay: POST /sdk/register/challenge (address)
109 Relay->>SDK: SIWE challenge message
110
111 Note over SDK: Signs SIWE message with secp256k1 identity key (personal_sign)
112
113 SDK->>Relay: POST /sdk/register (message, signature, name, tcp_enabled?, udp_enabled?)
114 Note over Relay: Validates SIWE signature, checks name availability
115 Note over Relay: Creates lease, publishes route at name.relay-host
116 Note over Relay: Allocates TCP/UDP ports if requested
117 Relay->>SDK: access_token (ES256K JWT) + lease info (tcp_addr?, udp_addr?, sni_port?)`
118 </script>
119
120 <div class="not-prose mb-8 rounded-lg border border-blue-200 bg-blue-50 px-4 py-3 text-sm text-blue-800 dark:border-blue-800 dark:bg-blue-950/30 dark:text-blue-300">
121 <strong>Advanced Documentation</strong> — This page covers internal architecture details for contributors and advanced users.
122 </div>
123
124 # Architecture
125
126 ## Overview
127
128 Portal publishes local services on public subdomains, optional dedicated TCP ports, and optional UDP ports through a relay.
129 Backends connect outward to the relay. Stream traffic is routed by SNI, and tenant TLS remains end-to-end between the client and the SDK or tunnel endpoint for the stream path.
130
131 High-level path:
132
133 ```text
134 Stream client
135 -> Relay SNI listener (:443 by default)
136 -> Claimed reverse session
137 -> SDK / portal-tunnel
138 -> Local service
139
140 TCP port client
141 -> Relay lease TCP port (within configured MIN_PORT-MAX_PORT)
142 -> Claimed reverse session (raw TCP, no TLS)
143 -> SDK / portal-tunnel
144 -> Local service
145
146 UDP client
147 -> Relay lease UDP port (within configured MIN_PORT-MAX_PORT)
148 -> Internal QUIC tunnel
149 -> SDK / portal-tunnel
150 -> Local UDP service
151 ```
152
153 <Mermaid code={overviewDiagram} />
154
155 ## Architecture Invariants
156
157 ### Transport and Routing
158
159 - Raw TCP reverse-connect is the canonical stream transport.
160 - Do not introduce websocket or legacy compatibility paths by default.
161 - Derive lease hostnames from the full normalized `PORTAL_URL` host, not from apex extraction.
162 - Preserve explicit root-host fallback through SNI no-route handling to the admin/API listener.
163 - Stream ingress is TLS-only. UDP exposure, when enabled, is raw UDP.
164
165 ### TLS and Identity
166
167 - Relay terminates admin/API TLS on the root host and exposes `/v1/sign` for tenant-side keyless signing.
168 - Control-plane HTTP (`/sdk/*`), reverse-session establishment (`/sdk/connect`), and tenant TLS are separate connections with different trust boundaries.
169 - Relay API TLS, SDK relay-client TLS, SDK tenant-server TLS, and QUIC tunnel TLS are distinct configs even when they reuse the same relay certificate material.
170 - Relay does not terminate tenant TLS. It peeks ClientHello for SNI and bridges raw encrypted bytes after routing.
171 - SDK/tunnel endpoints terminate tenant TLS locally with a keyless-backed signer that calls the relay.
172 - In keyless TLS, the relay performs certificate private-key signing through `/v1/sign`, but the SDK/tunnel endpoint still runs the TLS server handshake and derives tenant TLS session keys locally.
173 - `/sdk/connect`, `/sdk/renew`, and `/sdk/unregister` are authorized by lease existence plus a relay-issued lease access token.
174 - `/sdk/register` is authenticated by a SIWE challenge/response flow using the SDK identity secp256k1 key. On success, the relay issues a lease-scoped ES256K JWT access token signed by the relay identity key and used for the rest of the lease lifecycle.
175 - Relay URLs must use `https://`.
176 - HTTP/2 stays disabled on the admin/API TLS listener because `/sdk/connect` depends on HTTP/1.1 hijacking semantics.
177 - WireGuard, when enabled, is relay-to-relay overlay transport only. It carries multi-hop relay forwarding and overlay discovery, but it is not used for direct tenant TLS termination, public UDP ingress, or `/sdk/*` control-plane traffic.
178
179 ### Reverse Session Protocol
180
181 - SNI wildcard matching is one level only. `*.parent.example.com` matches `foo.parent.example.com`, not deeper labels.
182 - Reverse TCP marker bytes remain protocol state:
183 - `0x00` = idle keepalive
184 - `0x01` = raw TCP activation (non-TLS port routing)
185 - `0x02` = TLS passthrough activation
186 - `/sdk/connect` remains HTTP/1.1 only.
187
188 ### JSON and Shared Contract
189
190 - All JSON control-plane responses use `APIEnvelope`: `{ ok, data?, error? }`.
191 - JSON handlers should write responses through the shared API helpers.
192 - `types/` is reserved for shared wire/public types and cross-package constants only.
193 - Shared control-plane and public route constants belong in `types/paths.go`.
194 - Relay-local frontend asset filenames stay local to `cmd/relay-server`.
195 - Do not import `portal` from `cmd/*` or `sdk` just to reach shared DTOs or constants.
196
197 ### Operational Constraints
198
199 - For non-localhost deployments, relay TLS can run from manual certificate files in the relay `IDENTITY_PATH` directory or from managed ACME.
200 - When managed ACME is enabled, supported DNS providers are `cloudflare`, `gcloud`, `hetzner`, `njalla`, `route53`, and `vultr`.
201 - ENS gasless automation reuses `ACME_DNS_PROVIDER` for DNSSEC and ENS TXT sync when the selected provider supports DNSSEC.
202 - Relay stores its state under `IDENTITY_PATH`, including `identity.json`, `policy.json`, and certificate material. Tunnel and demo-app identities still use `IDENTITY_PATH` / `--identity-path` as a direct JSON file path.
203 - Managed non-localhost ACME keeps both root and wildcard DNS A records in sync.
204 - Relay certificate material lives under `IDENTITY_PATH` as `fullchain.pem` and `privatekey.pem`.
205 - Localhost uses the development certificate path instead of public managed/manual certificate setup.
206
207 ## Connection Model
208
209 Portal has three distinct network roles:
210
211 - **Control-plane HTTP requests**
212 - `POST /sdk/register/challenge`
213 - `POST /sdk/register`
214 - `POST /sdk/renew`
215 - `POST /sdk/unregister`
216 - `GET /sdk/domain`
217 - **Reverse session connection**
218 - `GET /sdk/connect`
219 - HTTP/1.1 only
220 - hijacked into a long-lived raw TCP session
221 - starts idle in the per-lease stream ready queue, then becomes the tenant data path when claimed
222 - **Internal datagram tunnel**
223 - QUIC to the relay URL host plus the relay-advertised `sni_port` from `POST /sdk/register` with ALPN `portal-tunnel`
224 - authenticated by a first-stream control message carrying `access_token`
225 - carries relay-to-SDK/tunnel datagram traffic only
226
227 That distinction matters because `/sdk/connect` stops being ordinary HTTP once hijacked, while the UDP backhaul is a separate internal QUIC carrier.
228
229 ## Package Layout
230
231 The relay runtime lives in `portal/` (server, route table, transport runtimes, ACME, keyless, auth, discovery, WireGuard overlay, policy).
232 The SDK client library lives in `sdk/` (listener, exposure, relay API client, MITM self-probe, transport clients).
233 CLI entry points live in `cmd/relay-server` and `cmd/portal-tunnel`; they import `portal/` and `sdk/` respectively but never each other.
234 Shared wire types, API envelope, error codes, path constants, and transport frame codec live in `types/`.
235
236 ## Transport Model
237
238 ### Raw reverse transport (TLS only)
239
240 1. SDK/tunnel registers one lease per relay through `POST /sdk/register/challenge` followed by `POST /sdk/register`.
241 2. SDK opens one or more reverse sessions per registered lease with `GET /sdk/connect`.
242 3. Each relay hijacks `/sdk/connect` requests and places the connection in the per-lease stream ready queue.
243 4. While idle, the relay writes `0x00` keepalive markers.
244 5. A stream client connects to the relay SNI listener.
245 6. Relay extracts SNI from ClientHello, resolves a lease, and waits up to `ClaimTimeout` for one reverse session from that lease stream queue.
246 7. Relay writes `0x02` to activate the claimed session.
247 8. SDK/tunnel receives `0x02`, starts tenant TLS locally using the relay-backed keyless signer, and the relay bridges raw encrypted bytes end-to-end.
248
249 Result: the relay decides routing, but tenant TLS termination still happens at the SDK/tunnel side.
250
251 <Mermaid code={tlsStreamDiagram} />
252
253 ### Tenant TLS Self-Probe Detection
254
255 1. After a real tenant connection begins I/O, the SDK may start one asynchronous self-probe for that listener if no probe is in flight and the 30-second cooldown has expired.
256 2. The SDK opens a new TLS connection to its own public URL using the same tenant-facing TLS characteristics as normal traffic.
257 3. The probe client exports TLS keying material (`ExportKeyingMaterial`) from that probe connection and stores it under a random nonce.
258 4. The first encrypted probe payload is `16-byte nonce + random padding`; there is no fixed probe magic or dedicated ALPN.
259 5. When the probe connection comes back through the relay and reaches the SDK-side tenant TLS terminator, the SDK peeks only the first 16 encrypted application bytes while a probe is pending.
260 6. If those bytes match a pending nonce, the SDK exports keying material on the server side and compares it with the client-side exporter value.
261 7. Matching exporter values mean the probe observed passthrough for that connection. A mismatch is logged as suspected relay-side TLS termination. A timeout is logged as probe failure, not proof of MITM.
262
263 Result: this is a detect-only signal by default. It raises the cost of adaptive relay-side TLS termination, but it does not prove passthrough for every user connection. Callers that need stricter behavior can opt into relay banning.
264
265 ### TCP Port Transport (non-TLS)
266
267 1. SDK/tunnel requests a register challenge with `tcp_enabled=true`, signs the returned SIWE message, and completes registration.
268 2. Relay validates that the TCP port plane is enabled, allocates a TCP port, and creates a per-lease TCP listener.
269 3. Registration response includes `tcp_addr` (public TCP endpoint).
270 4. An external TCP client connects to `tcp_addr`.
271 5. The relay accepts the connection, claims a reverse session from the lease stream queue, and writes `0x01` (raw TCP activation marker).
272 6. SDK-side receives `0x01` and passes the raw connection directly without TLS handshake.
273 7. Data is copied bidirectionally between the external client and the reverse session.
274
275 Result: the relay allocates a dedicated TCP port per lease and bridges raw TCP without TLS. This is ideal for non-TLS protocols like Minecraft, game servers, or any raw TCP service.
276
277 <Mermaid code={tcpPortDiagram} />
278
279 ### UDP/QUIC Datagram Transport
280
281 1. SDK/tunnel requests a register challenge with `udp_enabled=true`, signs the returned SIWE message, and completes registration.
282 2. Relay validates that the datagram plane is enabled, allocates a UDP port, and creates a per-lease datagram runtime.
283 3. Registration response includes `udp_addr`, `access_token`, and `sni_port`. The SDK dials QUIC to the relay on `sni_port`.
284 4. SDK opens a QUIC connection with ALPN `portal-tunnel` and DATAGRAM support enabled.
285 5. Authentication: SDK sends `{access_token}` JSON on the first QUIC stream; relay validates before accepting the tunnel.
286 6. External UDP client sends a packet to `udp_addr` -> relay assigns a flow ID -> QUIC DATAGRAM frame to SDK.
287 7. SDK-side decodes frames and delivers to local UDP target.
288 8. Return path: local response -> SDK -> QUIC DATAGRAM -> relay -> `WriteToUDP` to the original client.
289
290 Result: raw public UDP exposure with an internal QUIC datagram backhaul. UDP and TCP port allocations are independent from the same `MIN_PORT-MAX_PORT` range.
291
292 <Mermaid code={udpQuicDiagram} />
293
294 ## WireGuard Overlay and Discovery
295
296 - Discovery bootstraps from public HTTPS relay URLs, then expands through relay-to-relay `/discovery` polling and periodic self-announces to bootstrap relays through `/discovery/announce`.
297 - SDK exposures consume relay discovery results to choose relays, but they do not announce themselves and do not serve `/discovery`.
298 - Discovery descriptors are signed relay self-descriptions. They bind relay routing metadata such as `api_https_addr`, `supports_overlay`, `wireguard_public_key`, and `wireguard_port` to the relay identity. Lease access tokens remain separate and authorize tenant lease operations only.
299 - `/discovery/announce` accepts only signed relay descriptors. Loopback or localhost relay descriptors are rejected because they cannot join the public discovery mesh.
300 - The overlay peer API is plain HTTP on the WireGuard network, not public Internet HTTP. It serves the same discovery payload shape used by public `/discovery`.
301 - Overlay failure affects inter-relay discovery, mesh synchronization, and multi-hop relay forwarding. Direct tenant TLS routing, keyless TLS, register/renew/connect, and public UDP ingress do not depend on the WireGuard transport path.
302
303 ## Control Plane Flow
304
305 ### 1. Register
306
307 - `POST /sdk/register/challenge` then `POST /sdk/register`.
308 - Caller signs the returned SIWE message with the identity secp256k1 key (`personal_sign`).
309 - `name` must be a valid single DNS label; the relay publishes the lease at `<name>.<root host>`.
310 - Registration reserves the hostname and publishes the route immediately; if no reverse session is ready yet, inbound SNI claims wait up to `ClaimTimeout`.
311 - On success, the relay issues a lease-scoped ES256K JWT access token signed by the relay identity key, used for the rest of the lease lifecycle.
312 - UDP registration requires server `UDP_ENABLED=true`, a valid `MIN_PORT/MAX_PORT` range, and admin enablement. Failures: `udp_disabled` (403), `udp_capacity_exceeded` (503), `udp_port_exhausted` (503).
313 - TCP port registration has equivalent three-condition gating. Failures: `tcp_port_disabled` (403), `tcp_port_capacity_exceeded` (503), `tcp_port_exhausted` (503).
314 - `PORTAL_URL` is normalized to its host component only; path/query segments are ignored for routing.
315
316 <Mermaid code={registrationDiagram} />
317
318 ### 2. Reverse Connect
319
320 - `GET /sdk/connect` (HTTP/1.1 only, `X-Portal-Access-Token` header).
321 - Relay validates: lease exists and is not expired; access token signature, issuer, audience, identity, and expiry are all valid.
322 - After claim, relay writes `0x02` before switching the session into tenant TLS passthrough.
323 - After hijack, the connection becomes a broker-managed reverse session.
324
325 ### 3. Renew
326
327 - `POST /sdk/renew` with `access_token`. Extends lease TTL and returns a refreshed token.
328
329 ### 4. Unregister
330
331 - `POST /sdk/unregister` with `access_token`. Removes the lease, routes, and ready reverse sessions.
332
333 ## Routing Behavior
334
335 Route lookup order:
336
337 1. Exact hostname match
338 2. Single-label wildcard match (`*.example.com`)
339 3. Root-host fallback to the admin/API listener
340
341 Notes:
342
343 - Wildcards are one level only.
344 - The exact root host is never served by the wildcard route.
345 - For non-apex `PORTAL_URL` values such as `https://portal.example.com:8443/admin`, a lease named `demo` is published at `demo.portal.example.com`.
346
347 ## Admin API Surface
348
349 The relay server is intentionally API-only: public state endpoints, relay policy endpoints, public status endpoints, installer endpoints, and a small set of admin auth routes. Route paths are enumerated in `types/paths.go` and `cmd/relay-server`.
350
351 ## Keyless TLS Trust Model
352
353 The relay signs handshake digests via `/v1/sign` but never receives tenant TLS traffic secrets. The SDK/tunnel endpoint runs the full TLS server handshake and derives session keys locally. Relay control-plane TLS and reverse-session setup terminate on the relay's admin/API listener and are not protected by the tenant keyless path.
354
355 ## Design Properties
356
357 - Reverse-only backend connectivity
358 - One canonical raw TCP reverse transport
359 - Dedicated TCP port allocation for non-TLS services with raw TCP bridging
360 - Raw public UDP exposure with an internal QUIC datagram backhaul
361 - Optional WireGuard relay overlay for relay discovery, peer synchronization, and multi-hop relay forwarding
362 - SNI-based routing with root-host fallback
363 - End-to-end tenant TLS with relay-backed keyless signing
364 - Traffic-triggered detect-only MITM self-probing for probable relay-side TLS termination
365 - SIWE identity proof for registration plus relay-issued ES256K JWT access tokens for the lease lifecycle
366 - Lease-local stream and datagram ownership through per-lease transport runtimes
367 - Optional QUIC/UDP datagram transport coexisting with TCP on the same lease
368 - Per-lease UDP and TCP port allocation with sticky name-based reservation
369 - QUIC tunnel authentication via control stream (`access_token`)