@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 06:00 UTC

main
md 106 lines 4.24 KB
Rendered Raw
1 # AWS SageMaker HyperPod Example
2
3 SageMaker HyperPod clusters in TypeScript, covering both orchestrators and
4 every workload tier — from `sbatch` on a Slurm login node to effectful
5 TypeScript jobs arbitrated by task governance.
6
7 HyperPod is a persistent, resilient fleet of ML compute. How work lands on
8 it depends on the orchestrator you pick at creation:
9
10 | | Slurm (default) | EKS (`orchestrator: { Eks }`) |
11 |---|---|---|
12 | Provision | [`alchemy.run.ts`](./alchemy.run.ts) | [`eks.run.ts`](./eks.run.ts) |
13 | Low-level workloads | `ssm start-session` → `sbatch` | `Kubernetes.Manifest` (or `kubectl`) |
14 | High-level workloads | — (no submission API) | `Kubernetes.Job` / `Kubernetes.Deployment` via HyperPod attributes |
15 | Governance | Slurm accounting | `ClusterSchedulerConfig` + `ComputeQuota` (Kueue) |
16
17 ## The Slurm stack (`alchemy.run.ts`)
18
19 - [`src/infra.ts`](./src/infra.ts) — the lifecycle-script bucket and the
20 instance execution role (inline grant built with `Output.interpolate`).
21 - [`src/lifecycle.ts`](./src/lifecycle.ts) — a deploy-time `Alchemy.Action`
22 that uploads `on_create.sh` with the Effect-native distilled SDK
23 (`s3.putObject`). Bucket → script → cluster ordering is inferred from the
24 data flow.
25 - Slurm requires a `LifeCycleConfig` per instance group — that's where real
26 clusters install the scheduler, mount FSx, and wire observability.
27
28 ```sh
29 bun run --filter aws-hyperpod-example deploy # ~5 minutes at this size
30 bun run --filter aws-hyperpod-example destroy
31 ```
32
33 Workloads are submitted **on the cluster** — each node is an SSM target:
34
35 ```sh
36 aws sagemaker list-cluster-nodes --cluster-name <clusterName output>
37 aws ssm start-session \
38 --target sagemaker-cluster:<cluster-id>_controller-<instance-id>
39 # then, on the node:
40 sbatch --nodes=1 train.sbatch
41 ```
42
43 ## The EKS stack (`eks.run.ts`)
44
45 - [`src/eks-infra.ts`](./src/eks-infra.ts) — network (private subnets +
46 NAT), a plain EKS control plane (HyperPod supplies the nodes), the
47 HyperPod instance role (managed policy + the EKS networking/ECR/pod
48 identity grants), the HyperPod cluster attached via
49 `orchestrator: { Eks: { ClusterArn } }`, the
50 `amazon-sagemaker-hyperpod-taskgovernance` add-on, a scheduler policy,
51 and the research team's compute quota. `LifeCycleConfig` is required
52 here too — the API enforces it for EKS-orchestrated instance groups.
53 - **Low level** (`eks.run.ts`) — a raw batch/v1 Job applied with
54 `Kubernetes.Manifest`, pinned to HyperPod nodes with the well-known labels
55 and submitted through governance with the Kueue labels:
56
57 ```typescript
58 nodeSelector: hyperpod.instanceGroups.workers.nodeSelector,
59 labels: {
60 [AWS.SageMaker.KUEUE_QUEUE_NAME_LABEL]: researchQuota.queueName,
61 [AWS.SageMaker.KUEUE_PRIORITY_CLASS_LABEL]: "training-priority",
62 },
63 ```
64
65 - **High level** ([`src/TrainJob.ts`](./src/TrainJob.ts)) — an effectful
66 `Kubernetes.Job` bundled from TypeScript, written in plain Kubernetes
67 vocabulary — the HyperPod resources expose the derived values as
68 **attributes referenced through the graph**: the instance-group keys
69 carry through to the cluster's attributes as types (a typo'd name is a
70 compile error), and the quota materializes the governed namespace and
71 Kueue queue:
72
73 ```typescript
74 yield* Kubernetes.Job("TrainJob", {
75 cluster: eks,
76 main: import.meta.url,
77 namespace: researchQuota.namespace, // hyperpod-ns-research
78 labels: {
79 [AWS.SageMaker.KUEUE_QUEUE_NAME_LABEL]: researchQuota.queueName,
80 [AWS.SageMaker.KUEUE_PRIORITY_CLASS_LABEL]: "training-priority",
81 },
82 podTemplate: {
83 spec: {
84 // health-checked nodes of the `workers` group (key-typed)
85 nodeSelector: hyperpod.instanceGroups.workers.nodeSelector,
86 },
87 },
88 });
89 ```
90
91 Bindings resolve in init and land IAM on the pod-identity role, exactly
92 like any other Kubernetes Job or Deployment on EKS.
93
94 ```sh
95 bun alchemy deploy --config ./eks.run.ts # EKS ~10-15 min + HyperPod ~10-20 min
96 bun alchemy destroy --config ./eks.run.ts
97 ```
98
99 ## Inspection
100
101 ```sh
102 aws sagemaker describe-cluster --cluster-name <name>
103 aws eks update-kubeconfig --name <eksClusterName output>
104 kubectl get nodes -l sagemaker.amazonaws.com/node-health-status=Schedulable
105 kubectl get workloads -n hyperpod-ns-research # Kueue admission
106 ```