feat(mxc): add Windows ETW-to-OCSF audit trail by araza008 · Pull Request #3015 · NVIDIA/OpenShell

araza008 · 2026-08-28T20:16:34Z

Summary

Adds a Windows MXC audit trail that consumes Sandboxing ETW events, attributes them to OpenShell sandboxes, and emits structured OCSF events to a durable JSONL log. This provides Windows audit coverage and output parity with the Linux OCSF path.

Changes

Add a real-time Windows ETW consumer for the Sandboxing provider.
Map all supported MXC ETW events to OCSF lifecycle, configuration, process, and finding classes.
Attribute events to sandbox IDs using PID and strong correlation keys.
Buffer and safely replay events that arrive before sandbox attribution is registered.
Guard attribution against deletion races, duplicate command lines, and Windows PID recycling.
Report ETW startup and unexpected ProcessTrace failures accurately.
Add OS-aware OCSF device and hostname metadata.
Write gateway OCSF events to a synchronous, daily-rotated JSONL audit log.
Add configuration controls for enabling ETW auditing and selecting the OCSF log directory.
Document the privacy implications of recording raw process command lines.
Add an example PowerShell audit workflow with coverage reporting and client-safe output handling.

Testing

Ran mise run --skip-tools windows:ci successfully on an x64 Windows host.
Passed x64 and ARM64 workspace checks.
Passed x64 and ARM64 release builds for openshell-gateway.exe and openshell.exe.
Passed native x64 workspace tests: 3,195 passed, 0 failed, 25 ignored.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable)

Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019, process 1007, finding 2004). cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux: - openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge thread-local AND stamps sandbox_id+message in one dispatch) plus public set/clear_current_event; OS-aware device (Device::windows/for_current_os) so device.os.name reflects the host instead of a hardcoded Linux stub. - etw_consumer: emit via the routed emit (previously fired a bare info! that never populated the bridge, so the structured event was dropped). - openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via %PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR). - device.hostname now resolves to the real gateway machine name. Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid OCSF JSON, per-sandbox attribution intact, disabled state writes nothing. Signed-off-by: Akber Raza <akberr@nvidia.com>

Close the last three ETW->OCSF gaps so the audit trail covers the full set of events the Sandboxing provider emits (12/12): - ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start; carries the real processId/threadId, the twin of CreateProcessInSandbox which only has the request + command line). - SandboxProxyConfigured -> Device Config State Change [5019] (the one network-plane setup event; surfaces proxyPort, "no proxy" when 0). - SandboxConsoleReferencePlumbed -> Device Config State Change [5019] (console-handle plumbing). map_config_state now handles the full config/hardening/setup family and carries proxyPort/hasConsoleReference/creationFlags as unmapped fields. Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy (SandboxProxyConfigured requires proxy config to fire). Signed-off-by: Akber Raza <akberr@nvidia.com>

Address CodeRabbit review on !31: - Prevent stale ETW attribution on a delete/launch race: register the wxc-exec pid while holding the registry lock, and bail if the sandbox entry is already gone. Previously the attribution key could be seeded after `delete` had removed the sandbox, leaving a stale key that could misroute later Sandboxing ETW events to a dead sandbox_id. Lock order (registry -> attribution) matches the delete path, so no deadlock. - Add unit tests for the new Device::windows and Device::for_current_os constructors to harden Windows/Linux OCSF device parity. Signed-off-by: Akber Raza <akberr@nvidia.com>

Addresses two ETW->OCSF attribution review items (Shailendra NVIDIA#1, NVIDIA#2). NVIDIA#2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved(). NVIDIA#1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute. Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0). Signed-off-by: Akber Raza <akberr@nvidia.com>

Review item NVIDIA#3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path). Signed-off-by: Akber Raza <akberr@nvidia.com>

…s real status Review item NVIDIA#4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage). Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace. Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit). Signed-off-by: Akber Raza <akberr@nvidia.com>

CodeRabbit flagged that drain_resolved() re-resolved buffered events against the live by_pid map, so if Windows recycled a wxc-exec PID within PENDING_TTL a stale event from the dead sandbox could be emitted under the new owner. Stamp each by_pid registration with its Instant and add resolve_replay(), used only on the buffered/replay path. It (a) never falls back to the recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID match only when the registration is not newer than the buffered event by more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well outside that window, so the stale event ages out instead of misattributing. The legitimate NVIDIA#2 seed race (registration lands ~immediately) still replays. Adds unit tests for the recycle-refusal, in-grace acceptance, and weak-fallback exclusion. Signed-off-by: Akber Raza <akberr@nvidia.com>

…DIA#4) start_session already returns Err on OpenTraceW failure (runs on the caller thread since e41a770), closing the first half of Shailendra's NVIDIA#4. This closes the second half: ProcessTrace's result was discarded, so if capture died mid-run the backend had no way to know. Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR and, when the pump returns without a deliberate stop, logs at ERROR that MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping` before teardown so a normal shutdown isn't misreported, and EtwSession::is_capture_alive() exposes the state for status/diagnostics. Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows, JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure). Signed-off-by: Akber Raza <akberr@nvidia.com>

…figured message Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1, mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit trail across all four classes (6002/5019/1007/2004). Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's presence already indicates proxy configuration. Verified on-box: 26 events, all mapped ETW event types present. Signed-off-by: Akber Raza <akberr@nvidia.com>

Improve the ETW to OCSF audit-trail example output and make it safe to ship. Report: - Add an event-type coverage count ("N of M expected event types fired"); the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy). - Split the checklist into expected event types vs anomaly findings (ActivityError/FallbackError), which are reported separately and not counted toward coverage (a clean run may emit none). - Verdict is now coverage-based (all expected types must fire) instead of the looser "at least 3 OCSF classes". - Call out the absolute path to the durable OCSF JSONL log prominently. Client-safety: - Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to opt in. Removes a hardcoded internal share path from a published example. - Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:") in favor of neutral "Results bundle:". - Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior. Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8 event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer fallback) -> reduced set as expected, clean output. Signed-off-by: Akber Raza <akberr@nvidia.com>

copy-pr-bot · 2026-08-28T20:16:38Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

araza008 · 2026-08-28T20:17:18Z

I have read the DCO document and I hereby sign the DCO.

drew · 2026-08-31T04:54:07Z

/ok to test 3e46489

drew · 2026-08-31T05:49:32Z

gator-agent

Follow-Up Needed

I cannot validate this submission yet because it is a large, cross-cutting Windows audit feature (17 files across the MXC driver, OCSF, and server configuration) without a linked validated issue, roadmap item, reviewed RFC, or maintainer confirmation of the scope. I checked PR #2959 as well; it was closed because it targeted the wrong branch, so this PR appears to be the intended continuation rather than a competing duplicate.

@araza008, please link the issue, roadmap item, or RFC that authorizes this work, or ask a maintainer to confirm in this thread that this scope should proceed. If the original submitter or a maintainer does not respond within 48 business hours, this may be closed as not planned. Weekend hours do not count toward the TTL.

Gator metadata

Head SHA: 3e4648932dbe1e823ac190560a3e507c7dd97ad2
Gator payload: 7
Next state: gator:follow-up-needed

drew · 2026-08-31T05:54:04Z

Maintainer confirmation: this scope should proceed. Please continue the Gator code review on the current head SHA. This is an explicit same-SHA review request; post the resulting review disposition if warranted.

drew

gator-agent

PR Review Status

Thanks @drew, I verified your maintainer confirmation that this cross-cutting Windows audit scope should proceed and completed the requested initial review of the current head. The review found five blocking correctness, security, and audit-integrity issues.

Action required: @araza008, address the five inline findings and push an updated head for a focused follow-up review.

Blocking findings:

GATOR-3e464893-01: the shipped example uses unsupported MXC configuration and cannot complete its advertised workflow.
GATOR-3e464893-02: raw command arguments are copied into durable and routed audit logs.
GATOR-3e464893-03: a second gateway stops the first gateway's shared ETW session.
GATOR-3e464893-04: system-wide ETW input feeds an unbounded in-memory queue.
GATOR-3e464893-05: weak and stale correlation keys can misattribute host ETW activity to a sandbox.

Carried findings:

None.

Non-blocking suggestions:

GATOR-3e464893-06: document etw_audit, OPENSHELL_OCSF_JSON, and OPENSHELL_OCSF_LOG_DIR in the canonical gateway and observability docs.

Gator metadata

Validation: Project-valid because verified maintainer @drew explicitly confirmed this cross-cutting scope should proceed.
Docs: Missing canonical Fern coverage for the new gateway configuration and audit-output controls.
Checks: Current-head OpenShell / Branch Checks is failing in Rust lint jobs; Helm Lint and existing E2E gate statuses are green.
E2E: test:e2e will be required for MXC lifecycle and gateway audit behavior after review feedback is resolved; it has not been dispatched by Gator yet.
Head SHA: 3e4648932dbe1e823ac190560a3e507c7dd97ad2
Base SHA: 74654ac30dfe7115b6b9dfdf9480342b34290607
Merge base SHA: 65745a06ef7b4aafc00600fd7f2d16ef7da49c26
Patch ID: 776036d10cddf58d3776c3d2c356ba322336966b
Gator payload: 7
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

drew · 2026-08-31T06:06:24Z

+backend = "process_container"
+
+default_configuration_id = "composable"
+


gator-agent

Warning — GATOR-3e464893-01 · The shipped audit example cannot start

Summary: An operator following this example reaches gateway configuration with share_dir, agent_cwd, agent_command, and egress-proxy keys that MxcComputeConfig rejects. Removing those keys still leaves sandbox create without the required per-sandbox mxc command payload, while the PASS criteria require proxy behavior the current MXC driver does not support. The advertised workflow therefore cannot produce its claimed audit coverage.

Fix: Keep only supported gateway fields here, pass command and cwd through --driver-config-json, derive a disposable policy when ShareDir changes, and remove proxy configuration and coverage unless governed MXC egress is separately implemented.

Verify: Deserialize this TOML successfully, then run the documented create loop and confirm it creates a sandbox and evaluates only supported ETW events.

Agent context

Location: crates/openshell-driver-mxc/examples/mxc-ocsf-audit.toml:26

Sibling sites: run-ocsf-audit.ps1:204, run-ocsf-audit.ps1:275

drew · 2026-08-31T06:06:25Z

+        .actor_process(Process::new("wxc-exec", i64::from(ev.process_id)))
+        .message(format!(
+            "MXC sandbox launched process: {}{cwd_suffix}",
+            truncate(cmd_line, 160)


gator-agent

Warning — GATOR-3e464893-02 · Raw command secrets enter durable audit logs

Summary: A user or agent can legitimately put a token, password, signed URL, or PII in an MXC workload argument. This path copies the full ETW command line into process.cmd_line and embeds its first 160 characters in the message, exposing it to gateway logs, sandbox log consumers, retained JSONL, SIEM export, and optional result bundles. Documenting the exposure does not establish a safe logging boundary.

Fix: Emit only safe executable identity by default, or apply a defined redactor to both process.cmd_line and the message. If raw command auditing is required, make it a separate explicit opt-in with secure-storage guidance.

Verify: Map tool --token secret-value; neither serialized OCSF nor shorthand output should contain secret-value by default.

Agent context

Location: crates/openshell-driver-mxc/src/etw_consumer.rs:1505

Ownership: This PR introduces the ETW capture and durable gateway sinks.

drew · 2026-08-31T06:06:25Z

+///
+/// Returns an [`EtwSession`] that must be kept alive; dropping it stops capture.
+pub(crate) fn start_session(index: Arc<Mutex<AttributionIndex>>) -> Result<EtwSession, String> {
+    cleanup_stale_session();


gator-agent

Warning — GATOR-3e464893-03 · Starting a gateway stops another gateway's audit capture

Summary: Two supported gateway processes can run on distinct ports on one Windows host. Every instance uses the fixed OpenShell-MXC-ETW session name, and startup unconditionally stops that name before creating its own session. Starting the second gateway therefore terminates the first gateway's ProcessTrace, leaving the first instance without its promised audit coverage.

Fix: Derive a collision-resistant session name from stable gateway identity plus process identity and clean up only sessions proven stale. If one shared session is intentional, coordinate its ownership explicitly instead of issuing an unconditional stop.

Verify: Start two ETW-enabled gateways on separate ports. After the second starts, both sessions must remain capture-alive and continue receiving their own sandbox events.

Agent context

Location: crates/openshell-driver-mxc/src/etw_consumer.rs:309

Execution path: start_session → cleanup_stale_session → named ControlTraceW(...STOP).

drew · 2026-08-31T06:06:25Z

+    let handle = start_trace_session()?;
+    enable_provider(handle)?;
+
+    let (tx, rx) = mpsc::channel::<RawEtwEvent>();


gator-agent

Warning — GATOR-3e464893-04 · ETW overload can exhaust gateway memory

Summary: The process-wide Sandboxing provider can deliver events faster than this consumer decodes and synchronously writes them. The callback deep-copies every record into an unbounded channel, so a sustained provider burst or stalled disk grows queued memory without limit. That turns audit overload into gateway memory exhaustion while giving operators no signal that coverage is falling behind.

Fix: Use a bounded, non-blocking queue sized for the documented creation burst. Count rejected events and emit a rate-limited warning or audit-health signal identifying the coverage gap; never block the ETW callback.

Verify: Stall the consumer and submit more than the chosen capacity. Queued bytes must remain bounded, the callback must remain non-blocking, and an overload counter or finding must be observable.

Agent context

Location: crates/openshell-driver-mxc/src/etw_consumer.rs:314

Sink: event_record_callback deep-copies and sends every event at line 632.

drew · 2026-08-31T06:06:25Z

+                    .and_then(|a| self.by_activity.get(a).cloned())
+            })
+            .or_else(|| cv.as_ref().and_then(|c| self.by_cv.get(c).cloned()))
+            .or_else(|| cmd.as_ref().and_then(|c| self.by_cmd.get(c).cloned()))


gator-agent

Warning — GATOR-3e464893-05 · Weak keys can corrupt sandbox audit attribution

Summary: Another local Sandboxing-provider process can emit a command line matching a live sandbox, and Windows can recycle a completed wxc-exec PID before sandbox deletion. The live resolver treats command text and persistent PID hints as sufficient ownership evidence, then routes the event under that sandbox ID. Unrelated host activity can consequently corrupt a sandbox's audit trail and disclose another process's command data to its log consumers.

Fix: Remove command-line and last_pid_sid authority fallbacks. Retire the PID anchor when monitor_exec observes exit, while retaining established identity/activity/CV correlations only long enough for legitimate late events.

Verify: An unknown PID with a matching command must remain unresolved, and a PID-only event after child completion must not resolve before sandbox deletion.

Agent context

Location: crates/openshell-driver-mxc/src/etw_consumer.rs:1120

Sibling sites: etw_consumer.rs:1106, etw_consumer.rs:1121, driver.rs:821

jamieknvidia added 10 commits August 28, 2026 11:51

araza008 requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners August 28, 2026 20:16

drew added the gator:follow-up-needed Gator needs submitter or maintainer follow-up label Aug 31, 2026

drew reviewed Aug 31, 2026

View reviewed changes

drew added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:follow-up-needed Gator needs submitter or maintainer follow-up labels Aug 31, 2026

Navigation Menu

Uh oh!

feat(mxc): add Windows ETW-to-OCSF audit trail - #3015

feat(mxc): add Windows ETW-to-OCSF audit trail#3015
araza008 wants to merge 10 commits into
NVIDIA:mainfrom
araza008:akberr/etw-to-ocsf-mapping

araza008 commented Aug 28, 2026

Uh oh!

copy-pr-bot Bot commented Aug 28, 2026

Uh oh!

araza008 commented Aug 28, 2026

Uh oh!

drew commented Aug 31, 2026

Uh oh!

drew commented Aug 31, 2026

Uh oh!

drew commented Aug 31, 2026

Uh oh!

drew left a comment

Uh oh!

drew Aug 31, 2026

Uh oh!

drew Aug 31, 2026

Uh oh!

drew Aug 31, 2026

Uh oh!

drew Aug 31, 2026

Uh oh!

drew Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		backend = "process_container"

		default_configuration_id = "composable"

Uh oh!

Conversation

araza008 commented Aug 28, 2026

Summary

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Aug 28, 2026

Uh oh!

araza008 commented Aug 28, 2026

Uh oh!

drew commented Aug 31, 2026

Uh oh!

drew commented Aug 31, 2026

Follow-Up Needed

Uh oh!

drew commented Aug 31, 2026

Uh oh!

drew left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

drew Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

drew Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

drew Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

drew Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

drew Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants