wslc: idle-terminate inactive per-user session VMs (#41077)

* wslc: idle-terminate per-user container VMs and lazily restart Per-user WSLC container session VMs now idle-terminate when no container is in a non-terminal (Created/Running) state, freeing host memory, and lazily restart on the next operation that needs the VM. - Centralize VM lifecycle in WSLCSession via TearDownVmLockHeld / StartVmLockHeld and an atomic VmExitDisposition (Active / StopRequested / ExitClaimed) to arbitrate expected stops vs. spontaneous VM exits without a polling thread. - Gate VM-requiring entrypoints behind AcquireVmLease(), which brings the VM up on demand and keeps it alive for the operation's duration. - Add IWSLCSession::BeginContainerOperation so a CLI command can hold the VM alive across resolve + operate + streamed output. - Preserve the session WarningCallback for the lifetime of the session so warnings emitted by the lazy VM start (e.g. resource recovery) are still delivered to the CLI invocation. - Remove the dtor lock in HcsVirtualMachine; OnExit/OnCrash are lock-free. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: don't pin the session VM for merely-created containers Only Running containers now hold an activity reference that keeps the per-user session VM alive. Previously a container in either Created or Running state held the reference, so a `create`d-but-never-started container pinned the VM indefinitely and defeated idle termination. A created container's metadata persists on the containerd VHD across VM teardown and is rebuilt by RecoverExistingContainers on the next VM-requiring operation, so create -> idle-terminate -> start later works; the 30s grace period covers the common create-then-start gap. Also fix m_stateChangedAt recovery for created containers: docker inspect reports FinishedAt as the zero date ("0001-01-01T00:00:00Z") for a never-started container, which parsed to year 1 and rendered as "created 2026 years ago". Use the container's Created time for the Created state. This recovery path was previously unreachable, since created containers never got torn down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: fix stale Created/Running comments on the activity hold Addresses review feedback: WSLCContainer.h still described the activity hold as held while Created/Running, but it now only pins the VM while Running. Update the two header comments to match the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: address idle-termination review feedback - Make the VM idle grace period configurable via settings.yaml (session.idleTimeout, default 30s) instead of a hardcoded constant. - Assert UserSid is non-null in PersistSettings rather than tolerating a null SID. - Drop the session warning-callback GIT fallback; warnings emitted outside a callback-bearing operation are logged and event-logged only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: update recovery warning tests for log-only behavior Recovery warnings emitted during lazy VM start run outside the user's current command, so they are now logged (and written to the event log) instead of being routed back to the session-creation warning callback. Update the three WarningCallback*Recovery unit tests and the e2e test to assert the warning is no longer delivered to the session callback / printed on stderr. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: address review feedback on lazy VM lifecycle - BeginContainerOperation: reject new operations once the session is terminating/terminated, mirroring EnsureVmRunning's gate, so a started operation cannot pin a VM that is being torn down. - TearDownVmLockHeld: reset m_storageMounted after unmounting so the flag does not stay stale across an idle teardown. - CLI Session model: stop retaining the IWarningCallback for the session lifetime. Recovery warnings from lazy VM start are logged rather than delivered to the session callback, so the stashed callback (and its now-misleading comments) is dead state. The callback is still passed to CreateSession, where it is consumed during initialization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: use non-throwing filesystem::exists for storage VHD probe Probe the storage VHD once with the std::error_code overload so an access-denied/transient I/O error surfaces as a clear Win32 error via WIL instead of a generic filesystem_error-to-HRESULT conversion, and avoid the duplicate exists() check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: extract WSLCSessionRuntime from WSLCSession Move the VM instance and its lifecycle state (VM, IO relay, docker client, event tracker, volumes, containerd/dockerd processes, exit-disposition and vm-state atomics, session lock, idle state, published-port bookkeeping, and ephemeral swap/storage state) out of WSLCSession into a new WSLCSessionRuntime. WSLCSession retains identity and orchestration and drives the runtime through BringUp/RecoverState/TearDownSessionState/OnSpontaneousExit/OnCrashDump hooks plus a SessionContext carrying the lifecycle primitives the runtime observes, so the runtime no longer reaches into WSLCSession private members. VM state is reached through the runtime, with Acquire()/LockedRuntime as the ergonomic lease accessor. Concurrency invariants are preserved: the exit-disposition claim protocol, the relay-thread teardown guard, lock ordering, GIT re-fetch per VM creation, and release-lock-before-Disarm in shutdown. The VM-exit monitor is armed between bring-up and state recovery to match the pre-extraction ordering. This is an isolated refactor commit on top of the idle-terminate work so it can be reverted independently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix formatting * wslc: skip containerd VHD unmount when storage was never mounted Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: publish termination reason before signaling the terminated event Copy the termination reason/details into the session-visible fields before SetEvent() so a waiter woken by the terminated event cannot observe the default Unknown reason. Restores the pre-refactor ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: add idle-termination test hook and pass runtime into container factories Add a test-only TriggerIdleTermination COM method to force VM idle teardown on demand, with three TAEF tests that hammer the VM idle-termination race paths. Refactor WSLCContainer construction to take WSLCSessionRuntime& instead of five individual runtime members. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: force-delete recovery test container on all exit paths The TriggerIdleTerminationRecoversRunningContainer test left the persisted container in the shared default session on early-exit or if the recovered handle's best-effort delete failed, making later ListContainers-based tests order-dependent. Add a scope_exit that force-deletes the container by name regardless of exit path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: list WSLCIdleState.h in wslcsession HEADERS Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: clang-format Open() call site Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: correct idle grace-period comment to reference IdleTimeoutSec Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: avoid reserved identifier in Fork structured binding Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WSLC VM start/stop plugin hooks Session-level WSLC plugin hooks (OnSessionCreated/OnSessionStopping) assumed session == running VM. With idle termination (PR #40781) the VM is created lazily, torn down when idle, and transparently recreated while the session persists, so plugins had no notification of actual VM lifecycle. Add VM-level hooks OnWslcVmStarted/OnWslcVmStopping that fire on every VM (re)start and teardown, in addition to the once-per-session hooks. Both are best-effort (errors logged and ignored). OnWslcVmStarted fires after releasing the runtime exclusive lock so a plugin may reentrantly call back into the session (e.g. WSLCCreateProcess) without deadlocking; OnWslcVmStopping fires under the lock during teardown, gated on a fire-once flag so a failed bring-up emits no spurious stopping. Adds a PluginTests::WslcVmRestart case that drives first start, forced idle teardown (TriggerIdleTermination), and lazy restart, asserting the hook sequence and proving reentrancy is deadlock-free. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f1c36ca0-19d0-46a7-82b3-10f1e1ee3e4c * Pair OnVmStopping only when OnVmStarted fired Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * wslc: address PR feedback and harden idle-terminate VM lifecycle - Session-scope DockerEventTracker; rebind per VM start, preserve subscriptions - Fire OnVmStopping with m_lock dropped to avoid plugin-reentrancy deadlock - Keep containers alive on idle teardown; clear only on permanent shutdown - Redesign OnIdleTimer to commit teardown unconditionally after notifying, matching the test-trigger path; removes phantom OnVmStopping->OnVmStarted and the concurrent-Terminate is_signaled crash - Dedup containerd storage mount point into WSLCSessionDefaults.h - Mark WslcVmStarted/WslcVmStopping hooks as introduced in 2.9.5 * wslc: fix reentrant deadlock in VM start/stop notifications NotifyVmStarted/NotifyVmStopping invoked the plugin hook while holding m_notifyLock. During idle teardown a plugin may reentrantly restart the VM (WSLCCreateProcess), re-entering these notifications and self-deadlocking on the non-recursive mutex. Flip the pairing state under the lock, copy the hook, then invoke it after releasing the lock. Also clarify Shutdown's lock parameter (runtimeLock is the exclusive hold on m_lock) per review feedback. * test: exercise reentrant mount from the WSLC OnVmStarted plugin hook The VM-restart plugin test already validates a reentrant process launch and mount+unmount from OnVmStopping. Extend OnVmStarted to also mount+unmount so both lifecycle notifications validate reentrant mount management does not deadlock, and assert the new log lines. * wslc: hold a VmLease across container archive upload/download UploadArchive and DownloadArchive streamed data without holding a VmLease, so idle teardown could race and tear down the docker client mid-transfer. Acquire a VmLease and wrap in try/CATCH_RETURN to match Export/Logs and the other VM-dependent container operations. * test: fail fast instead of hanging on idle-termination deadlock detection TriggerIdleTerminationConcurrentWithOperations bounded the worker join with a std::async future, but on timeout the failed VERIFY would unwind and block in the future destructor until the (deadlocked) task completed -- hanging the test host. Fail fast with a dump on timeout so a real deadlock fails cleanly. * wslc: fix idle-teardown activity race and stopping-only hook pairing OnIdleTimer dropped the runtime lock to fire OnVmStopping, then unconditionally tore the VM down after reacquiring. A lease that started in that window found the VM still Running, so it did not restart and could leave a long-lived activity token (e.g. a process keep-alive), and the teardown would then kill it. Re-check the activity count after reacquiring the lock and, if non-zero, abandon the stop and re-pair the notification with a fresh OnVmStarted. Also set m_vmStartNotified whenever the VM starts rather than only when an OnVmStarted hook is installed, so a hooks user that sets only OnVmStopping still receives paired stop notifications. * wslc: skip guest volume unmount when the VM has already exited ReleaseRuntimeResources unconditionally called UnmountWindowsFolder via Vm() during container teardown. After an unexpected VM exit the guest is already gone, so each call only blocks on the RPC timeout and emits a spurious unmount-failed warning. A dead VM has already dropped every guest mount, so mark the mounts inactive locally instead. Add WSLCSessionRuntime::VmExited() (mirrors TearDownVmLockHeld's VM-dead check) so the container can detect this via the runtime it already references. * wslc: track VM-exited with an atomic to avoid a lock-free event race VmExited() read m_vmExitedEvent without the runtime lock, but StartVmLockHeld and TearDownVmLockHeld reset/replace that event under the lock, so a container teardown running on another thread could observe the handle mid-reset and fail-fast. Mirror the state in a std::atomic<bool>: cleared when a VM instance starts and latched when the guest is observed dead at the top of teardown (before session-state cleanup, so ReleaseRuntimeResources sees it). VmExited() now returns the atomic. * wslc: close two VM-lifecycle races found in multi-model review OnIdleTimer's abort path re-paired OnVmStarted without checking whether the VM died while m_lock was dropped for OnVmStopping. OnVmExited() declines that exit (the expected-stop claim is held) and the exit handle is one-shot, so the session was left with a dead VM marked Running until a later idle cycle. Tear the VM down in that case so a waiting lease restarts a fresh instance, matching the commit path. EnsureVmRunning's running fast path returned before the terminating gate, so a reentrant plugin lease during Shutdown's OnVmStopping (lock dropped) could run work against a VM being permanently torn down. Enforce the gate before the fast path, as its comment already documents. * wslc: narrow the WSLCSession/WSLCSessionRuntime seam The runtime extraction exposed mutable internals directly, letting WSLCSession drive them from outside. Replace those handouts with intent-revealing methods so the runtime owns its invariants: - Remove dead accessors StateAtomic() and VmExitedEvent() (no callers). - Replace ExitDispositionAtomic().load() call sites with the existing ExitDisposition() value getter and drop the atomic-ref accessor. - Replace the raw srwlock handout Lock() with TryLockExclusive(), used only by the Terminate/Shutdown handoff. - Replace DockerdReadyEvent() with Reset/Is/SignalDockerdReady(); replace the ContainerdProcess()/DockerdProcess() ref assignments with setters; replace SwapVhdPath() with SetSwapVhdPath(). No behavior change. Idle, VM restart, VM-kill, and plugin tests pass. * wslc: mark per-operation activity leases [[maybe_unused]] The BeginContainerOperation() lease is held only for its scope lifetime to keep the session VM alive during the operation. Annotate it so its intent is explicit and a future edit doesn't drop it to a discarded temporary. * wslc: fix idle-timer crash race and notification ordering race - OnIdleTimer: release the expected-stop disposition claim before the second lock drop for NotifyVmStarted(), so a VM crash in that window is claimable by OnVmExited() as a normal spontaneous exit instead of being silently dropped while m_vmState stays Running. - NotifyVmStarted/NotifyVmStopping: hold m_notifyLock (now recursive) across both the pairing-state flip and the hook invocation, so a start and a stop racing on different threads cannot interleave and deliver a stale OnVmStarted after its OnVmStopping. * wslc: address idle runtime review feedback Copilot-Session: 65bff365-20af-4701-8edd-09641ba75747 * wslc: report never-started sessions as idle Copilot-Session: 65bff365-20af-4701-8edd-09641ba75747 * wslc: fix session host crash releasing containers after idle teardown A graceful idle teardown releases the VM without the exit event ever being signaled, leaving the runtime in a state where VmExited() is false and there is no VM object. ReleaseRuntimeResources() only checked VmExited(), so the unmount path called Vm() and hit the empty optional. Terminating a session in that state clears m_containers, so this ran from ~WSLCContainerImpl and took down the per-user session host (STATUS_ASSERTION_FAILURE in debug, std::terminate in release) on a mainline flow: run a container, idle out, stop the session. Treat "no VM" like "VM exited", and wrap the container destructor body in CATCH_LOG so no future VM-dependent call can terminate the host from a destructor. An inner try is used rather than a function-try-block, which would implicitly rethrow. Also move Shutdown()'s idle-timer disarm into a scope_exit so it runs on every exit path. IdleState outlives the runtime via activity token copies, so an exception escaping the teardown left a late 1->0 transition able to re-arm a timer bound to a destroyed runtime. Adds SessionTerminationAfterIdleTerminationWithContainer, which fails without the fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: gate VM leases across the OnVmStopping notification window Idle teardown drops the runtime lock to fire OnVmStopping so a plugin handler can take a VM lease without deadlocking. That window also let an unrelated thread lease the VM, which forced the teardown to be abandoned and a second OnVmStarted to be re-paired onto the very same VM instance -- plugins observed a stop/start pair the VM never actually performed. Publish a stop-pending state before dropping the lock instead. EnsureVmRunning holds off every other thread until the teardown completes, so each waiter then starts, and is notified about, a genuinely new VM. The notifying thread is exempt, since its reentrant leases are why the lock is dropped at all. This makes the notification unconditional and removes the cancel/re-pair path from both OnIdleTimer and the test hook. Also drop the SKIP_TEST_SERVER calls added by this change: the WSLC tests are disabled on server either way. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * Revert "wslc: gate VM leases across the OnVmStopping notification window" This reverts the runtime portion of 2ebee52a. The gate cannot work: it exempts the notifying thread by OS thread id, but a plugin OnVmStopping handler re-enters the session host over COM (wslservice.exe -> IWSLCPluginNotifier -> WSLCCreateProcess), so the reentrant lease lands on an RPC worker thread and never matches the exemption. It then waits on m_vmStopComplete, which is only signalled after NotifyVmStopping returns -- a hard cycle that hangs the session host, the plugin and the service permanently. This is a documented, tested contract: WslPluginApi.h states a callback needing the VM transparently restarts it during idle teardown, and PluginTests::WslcVmRestart expects the reentrant WSLCCreateProcess to succeed. That test hangs with the gate and passes without it. The gate was also not airtight: AddActivity takes no runtime lock, so a lease could pass the check before the stop was published, then acquire the shared lock as soon as it was dropped for the notification. With the post-notification activity re-check removed, teardown could destroy a VM under a freshly created keep-alive token. Keeps the SKIP_TEST_SERVER removal, which was unrelated and correct. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: commit to an announced VM stop before notifying plugins OnWslcVmStopping has to fire while the VM is still alive -- its purpose is to warn a plugin that the VM it is using is going away, while it can still act on it. Delivering it means dropping the runtime lock, because a plugin handler may call back into the session, and that opens a window in which new work can arrive. Previously that work could keep the VM alive, so an announced stop could silently not happen and the plugin was left holding a warning that never came true. Decide the stop under the exclusive lock and publish it before announcing it, so the notification states a fact rather than a forecast: - The last silent back-out is the ActivityCount() check immediately before BeginVmStopLockHeld(). From there the teardown runs unconditionally. - A lease arriving during the window releases its shared lock, waits on m_vmStopCompleteEvent and retries, so it is served by a fresh VM rather than keeping a promised-dead one alive. Releasing the lock before waiting is required: the teardown must be able to reacquire it exclusively. - EndVmStop() is a scope_exit declared after the lock guard, so reverse-order destruction runs it while the lock is still held and a waiter can only wake into a fully torn-down, published state. - Calls the stopping handler makes itself must not wait for the teardown they are holding up. wslservice marks the thread running the callout and tags those calls with FromVmLifecycleCallback, so the session serves them from the VM that is going away. A plugin calling from a thread of its own is untagged and simply waits for the next VM. - NotifyVmStopping retires the notified generation as it announces the stop, so a Terminate() racing an idle teardown cannot deliver OnWslcVmStopping twice. Work a plugin leaves running when its handler returns now dies with the VM, which is exactly what the handler was just told would happen. Documented in WslPluginApi.h along with the rule that the handler must not block waiting on another thread calling into the session. Also publish VM presence in an atomic, so ~WSLCContainerImpl -- which deliberately holds no VM lease -- can query it without racing the teardown's reset(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: fix spurious OnVmStopping for a session that never started a VM NotifyVmStopping retires the notified generation with a compare-exchange against the generation being stopped. Both m_vmGeneration and m_notifiedGeneration start at 0, so a session torn down without ever starting a VM -- bring-up is lazy, a VM is only created on the first operation that needs one -- reached Shutdown with generation 0 and the CAS matched, delivering an OnWslcVmStopping that no OnWslcVmStarted ever paired with. Treat 0 as the 'nothing announced' sentinel it already is everywhere else. Covered by a new WslcVmNeverStarted plugin test. Also from review: - Don't attach a process keep-alive activity token for a call made from the OnVmStopping handler. That VM is committed to stopping, so the token cannot do its job, but it would keep counting activity for as long as the plugin held the proxy and block idle termination of every later VM in the session. - Deliver the notification under CATCH_LOG at all three sites. The stop is already published and its generation retired by then, so a throwing handler must not be able to skip the teardown and leave waiters to be served by the VM they were promised would die. - Correct three comments: TearDownVmLockHeld's claim that the notify paths never run mid-teardown, WslPluginApi.h's claim that blocked callers are always served by a next VM (there is none when the session itself is terminating), and the IDL's 'must be set only by the service' for FromVmLifecycleCallback, which the session cannot enforce across the process boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: close the exit event handle and tighten the recovery-warning assert IWSLCProcess::GetExitEvent marshals its handle as [out, system_handle(sh_event)], so the caller owns the duplicate. The stop-window thread waited on it and dropped it -- a leak, and the test plugin is reference code plugin authors copy. WSLCE2E_Warning_ContainerRecoveryNotPrintedOnStderr guarded its only assertion on Stderr.has_value(), so the test would pass vacuously if stderr capture broke -- exactly when it would stop proving anything. No stderr at all already satisfies 'the warning was not printed', so assert against an empty string instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: assert stderr capture in the recovery-warning test and close the leaked exit event RunWslc always populates Stderr, so guarding the search on has_value() -- or searching value_or({}) -- lets the test pass without proving anything if capture ever breaks. Assert has_value() first, matching the other wslc e2e tests. Also close the duplicated exit event at session teardown when the stop-window thread did not get far enough to claim it. The leaked process wrapper is left alone deliberately: it holds a COM proxy marshalled to the OnWslcVmStopping callback's thread, so releasing it from the session-stopping thread risks the same RPC_E_WRONG_THREAD hazard that forced the exit event to be cached as a plain handle, and it is one wrapper for the lifetime of a test process. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: serialize test plugin logging and roll back port reservations on failure Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: reject plugin calls when the session has no running VM Plugin-originated mount/exec calls used to acquire a VM lease like any other client, so a plugin could bring a VM up as a side effect of a session-level callback, or block on a teardown it was itself holding up. Rename the IWSLCSession flag from FromVmLifecycleCallback to AcquireVmLease, which names what it does rather than when the service sets it, and invert its meaning: the service now passes FALSE for every plugin call. Such a call is served by whatever VM is already running, including one committed to stopping, and fails with the new WSLC_E_VM_NOT_RUNNING when there is none. The thread-local tagging that used to distinguish a call made from inside OnWslcVmStopping is no longer needed and is removed. Also drop the WSLCContainerImpl accessors that only forwarded to the runtime, and use wil::unique_handle for the test plugin's duplicated exit event. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: surface WSLC_E_VM_NOT_RUNNING in the plugin header and refresh stale comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: hold a container operation token across copy to/from container Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 * wslc: include <set> in WSLCVirtualMachine.h Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33 --------- Co-authored-by: Ben Hillis <benhill@ntdev.microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f1c36ca0-19d0-46a7-82b3-10f1e1ee3e4c Copilot-Session: 65bff365-20af-4701-8edd-09641ba75747 Copilot-Session: e6fa3a33-6af7-4323-be66-2d59dcca8d33