Skip to content

Add timeout and cooldown to raw IOCTL calls - #21

Closed
Zafer-Liu wants to merge 1 commit into
Blacktempel:masterfrom
Zafer-Liu:fix/ioctl-hang-timeout
Closed

Zafer-Liu wants to merge 1 commit into
Blacktempel:masterfrom
Zafer-Liu:fix/ioctl-hang-timeout

Conversation

@Zafer-Liu

Copy link
Copy Markdown

Problem

SendRawIoControl performs a synchronous DeviceIoControl with no timeout and no cancellation. When a storage device is wedged — for example a failing USB bridge whose driver never completes the IRP — the call blocks forever, and so does every caller above it.

This is not theoretical. FanControl V281 embeds DiskInfoToolkit through LibreHardwareMonitor, and a real-world failure mode is the app freezing at startup with the main window never appearing. A dotnet-dump of the hung process shows the WPF UI thread parked inside the IOCTL indefinitely (trigger: a failing Realtek USB card reader, USB\VID_0BDA&PID_9210, behind which a portable SSD stopped answering):

DiskInfoToolkit.Native.Kernel32Native.DeviceIoControl(...)
DiskInfoToolkit.Core.WindowsStorageIoControl.SendRawIoControl(...)
DiskInfoToolkit.Core.WindowsStorageIoControl.TryGetDriveGeometryEx(...)
DiskInfoToolkit.Core.StorageDetectionEngine.AttachStandardStorageProperties(...)
DiskInfoToolkit.Core.StorageDetectionEngine.GetDisks()
DiskInfoToolkit.Storage.EnumerateRawDisks()
DiskInfoToolkit.Storage.GetDisks()
LibreHardwareMonitor.Hardware.Storage.StorageGroup.AddHardware(...)
LibreHardwareMonitor.Hardware.Computer.Open()
FanControl.Domain.BackendProviders.LHM.LHMBackendProvider.Open()
FanControl.Domain.ComputerAccessLayer.Initialize()
FanControl.Bootstrap.<EnterFullAppMode>d__31.MoveNext()
System.Windows.Threading.Dispatcher.ProcessQueue()   <-- UI thread

The stuck thread sits in kernel, which also makes the hung process unkillable. The same holds for LHM's periodic storage refresh in long-running instances.

OpenDevice already protects itself against exactly this class of device with a worker thread + timeout + per-path cooldown. Raw IOCTLs deserve the same treatment.

Solution

Mirror the existing OpenDeviceWorker / OpenDeviceRequest machinery for IOCTLs:

  • SendRawIoControl now runs the blocking DeviceIoControl on an IoControlWorker thread and waits up to IoControlTimeoutMilliseconds (default 30 s).
  • On timeout the worker is abandoned and replaced (identical to the open path), the device enters the IoControlTimeouts cooldown for IoControlTimeoutCooldownMilliseconds (default 5 min), and the call returns false with LastIoControlError = ERROR_TIMEOUT (1460). Every existing TryGet* / pass-through failure path then simply skips the device — no API or caller changes needed.
  • The cooldown key is the device path, recovered from a ConditionalWeakTable<SafeFileHandle, string> populated by OpenDevice, so the quarantine survives across enumeration rounds and WindowsStorageIoControl instances (a re-opened device is skipped immediately instead of hanging again).
  • IoControlTimeoutMilliseconds <= 0 restores the previous direct synchronous call, following the same convention as OpenDeviceTimeoutMilliseconds.
  • Open and IOCTL cooldown bookkeeping share two small private helpers (IsPathInTimeoutCooldown / RememberPathTimeout); behavior on the open side is unchanged.

New public knobs: WindowsStorageIoControl.IoControlTimeoutMilliseconds and WindowsStorageIoControl.IoControlTimeoutCooldownMilliseconds.

Trade-offs (the same ones the open path already accepts)

  • All IOCTLs are serialized through one background worker. A wedged device delays other devices' IOCTLs by at most one timeout, after which the worker is abandoned/replaced and the wedged device is quarantined in cooldown.
  • The abandoned worker thread remains blocked in the driver until the IRP completes (or never); its late result is discarded. Handles stay owned by the caller.
  • If a timed-out IRP completes late, the marshaled output buffer of that request may be written by the completing call. Current callers discard output buffers after a false return, so this is benign — same reasoning as the abandoned open path.

Testing

  • Full library type-checked against netstandard2.0 reference assemblies (all 242 sources, Roslyn 4.14): zero diagnostics on the changed files.
  • A full six-TFM dotnet build has not been run yet (no SDK on the affected machine); happy to iterate on any CI findings.
  • Runtime verification against the failing reader that produced the trace above is planned — the repro is deterministic on that machine.

Related: LibreHardwareMonitor calls Storage.GetDisks() synchronously from StorageGroup.AddHardware, and FanControl runs that whole chain on its UI thread at startup. This PR fixes the library level; the caller-side threading is tracked separately with the FanControl project.

A DeviceIoControl sent to a wedged storage device can block forever,
which hangs any thread doing storage enumeration. OpenDevice already
runs behind a worker with a timeout and a per-path cooldown; raw IOCTL
calls did not and would stall indefinitely.

Route SendRawIoControl through the same pattern: the blocking
DeviceIoControl runs on an IoControlWorker thread, the caller waits
with IoControlTimeoutMilliseconds (default 30 s). On timeout the
worker is abandoned and replaced, the device path (tracked from the
handle that opened it) enters IoControlTimeouts for
IoControlTimeoutCooldownMilliseconds (default 5 min), and the call
returns false with LastIoControlError = ERROR_TIMEOUT so existing
failure paths skip the device. Setting IoControlTimeoutMilliseconds
to 0 or below restores the direct synchronous call.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The change adds cross-thread native-IOCTL handling with abandoned-worker and late-completion buffer semantics on a UI-critical path, and the author notes the full multi-TFM build and runtime repro have not yet been verified, so human review is warranted.

Review effort: Balanced
Findings: None

What changed in this PR

This PR hardens WindowsStorageIoControl.SendRawIoControl against wedged storage devices whose driver never completes a DeviceIoControl IRP. Previously such a device blocked the calling thread (and, via LibreHardwareMonitor/FanControl, the WPF UI thread) indefinitely. The change mirrors the already-existing OpenDevice protection: the blocking IOCTL now runs on a dedicated background worker thread with a timeout, and a per-device-path cooldown quarantines a device that times out so subsequent calls skip it immediately.

Changes:

  • Added IoControlWorker and IoControlRequest (structural copies of the existing OpenDeviceWorker/OpenDeviceRequest) to run DeviceIoControl off-thread with abandon-on-timeout semantics.
  • Reworked SendRawIoControl to check a cooldown, run the core call directly when IoControlTimeoutMilliseconds <= 0, or dispatch through the worker with a timeout; extracted SendRawIoControlCore.
  • Generalized the open-path cooldown helpers into shared IsPathInTimeoutCooldown/RememberPathTimeout, added a ConditionalWeakTable<SafeFileHandle,string> to recover a handle's device path, and exposed IoControlTimeoutMilliseconds / IoControlTimeoutCooldownMilliseconds knobs.
File Description
DiskInfoToolkit/​Core/​WindowsStorageIoControl.cs Adds IOCTL timeout/cooldown fields, knobs, worker dispatch, handle→path tracking, and shared cooldown helpers; extracts SendRawIoControlCore.
DiskInfoToolkit/​Core/​Windows/​IoControlWorker.cs New background worker thread that executes the blocking IOCTL and supports abandonment on timeout.
DiskInfoToolkit/​Core/​Windows/​IoControlRequest.cs New request object carrying IOCTL parameters/results with completion, timeout, and disposal synchronization.

I reviewed the concurrency model (global lock held across Wait, Complete/Dispose ordering, memory visibility via the completion event), the cooldown key handling (null-safe TrimStorageString, OrdinalIgnoreCase, weak-table path recovery), handle uniqueness for ConditionalWeakTable.Add, and LastIoControl* state on every return path. The implementation faithfully mirrors the existing, working OpenDevice machinery, and I found no high-confidence defects that differ from that established pattern. Note that the timeout/worker machinery (both the pre-existing open path and this new IOCTL path) has no direct unit tests — tests exercise a mock IStorageIoControl — so no test-coverage comment is warranted here.


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@Zafer-Liu

Copy link
Copy Markdown
Author

Follow-up on the verification status: the library now compiles cleanly against all six target frameworks — net472, net481, netstandard2.0, net8.0, net9.0, net10.0 — all 242 sources, zero errors, only the pre-existing CS0649 warnings on the interop structs.

Method note: the affected machine has no .NET SDK, so this was done with Roslyn 4.14 (csc) directly against the official reference assemblies per TFM (Microsoft.NETFramework.ReferenceAssemblies for net472/481, Microsoft.NETCore.App.Ref for net8–10, NETStandard.Library for netstandard2.0) with the same DEBUG/implicit-usings setup the SDK build uses. It is a type-level compile check of every source file rather than a full dotnet build (pack/resource generation steps not exercised) — happy to re-run the regular build on a dev box or CI if anything still looks off.

Remaining open item is the runtime verification against the failing reader that produced the trace in the description.

@Blacktempel

Copy link
Copy Markdown
Owner

While I don't think this is ready to be merged, it does explain a possible issue.

Realtek USB card reader, USB\VID_0BDA&PID_9210

Could you please state which hardware exactly this is ?
Exact model name and ideally a link to where this hardware can be purchased / where you purchased it.

@Zafer-Liu

Copy link
Copy Markdown
Author

Sure — here is everything identifiable about the hardware:

  • Bridge: Realtek RTL9210B NVMe→USB bridge (USB\VID_0BDA&PID_9210), the chip used in many M.2 NVMe enclosures. The bridge firmware only exposes generic strings ("USB Mass Storage Device", placeholder serial 012345679017), so Windows cannot see any enclosure brand from the software side. The enclosure itself is an unbranded/generic one and I no longer have the purchase link, sorry — I can't name the commercial product beyond the chip.
  • Drive: Samsung MZ9L4256HCJQ-00BMV (PM991a family, 256 GB OEM M.2 NVMe module) mounted in that enclosure; enumerates as "MZ9L4256 HCJQ-00BMV-SAMSU USB Device", drive H:, \Device\Harddisk3\DR3.

Failure signature: the device storms Event Log with disk event 11 (controller error) / Ntfs 140, after which IOCTLs sent to it never return (IOCTL_DISK_GET_DRIVE_GEOMETRY_EX in the trace). After a reboot the device re-enumerates and works normally — until it flakes out again. That intermittency is also why I flagged the runtime verification as still pending: the trigger is not always present, but it is reproducible on this machine across reboots.

@Blacktempel

Copy link
Copy Markdown
Owner

I can't name the commercial product beyond the chip.

Unfortunately that does not help for further analysis.

It is quite weird that it is stuck in the TryGetDriveGeometryEx IOCTL.
I have seen issues with SD card readers but not with a NVME enclosure, if I remember correctly.
Even my dirt cheap enclosure works fine and that one is also VID_0BDA&PID_9210 with the RTL chip/controller.

I'm going to try something else, perhaps that works properly.

Could you please verify the same issue persists when you execute DiskInfoViewer (take from latest release or compile yourself; click Refresh button once after startup) ?

@Zafer-Liu

Copy link
Copy Markdown
Author

Baseline verification done, as requested (reporting back from this thread).

The release asset download (v2.2.0.7z) fails from this network (TLS resets on the GitHub asset CDN), so I built from the 2.2.0 tag (Roslyn csc + NuGet ref packs, net8.0) and ran the repo's own ConsoleOutputTest unmodified — it performs the same Storage.GetDisks() + per-disk Storage.Refresh() cycle that DiskInfoViewer's startup + Refresh click triggers, with per-stage timing.

Result while the device is healthy: no hang, fast

The machine rebooted since the incident; the RTL9210B enclosure re-enumerated and is currently in a good phase (PnP status OK on both bridge and disk, zero disk 11 / Ntfs 140 / 153 events since boot, H: mounted). Output (full log kept locally):

Starting DiskInfoToolkit v2.2.0 (git 2.2.0 tag)
'ZHITAI TiPlus7100 2TB': ...
'KINGSTON SNV3S2000G': ...
'KINGSTON SA400S37960G': ...
'MZ9L4256HCJQ-00BMV-SAMSUNG': ...
### Detecting all disks took 1116.42 ms. ###      <- 4 disks, incl. the enclosure
### Detecting 0 Microsoft Storage Spaces pools took 7.18 ms. ###
### Update of all disks took 56.76 ms. ###         <- Refresh equivalent, all 4 disks

Process exited cleanly (code 0), ~11.7 s wall clock (10 s of that is the test's deliberate device-change wait). Every stage — device opens, geometry/descriptor/scsi-address ioctls, USB probe strategy, refresh — completed normally against the problem device.

Two observations from the healthy dump that may be relevant:

  • The enclosure shows BusType: Usb, TransportKind: Nvme, ProbeStrategy: UsbProbe.
  • It reports SupportsSmart: False / SmartVersionRaw: 0 / Temperature: null even when healthy — the bridge does not pass SMART/SMART-version ioctls through. So on this hardware the toolkit always walks the full ioctl chain (open → geometry → descriptor → smart version → ...) with several of them failing or returning nothing, and the wedge happens somewhere on that same chain when the device flips into its bad state.

What this means for the original issue

The hang does not reproduce while the device is in its healthy phase — consistent with it being intermittent across reboots (it had wedged twice before the config workaround). The wedged state can't be triggered on demand right now (it arises on its own, typically after the device has been powered a while / after standby); catching it would mean leaving an enumeration loop or the viewer running until it flips, which I can do if useful. The unprotected synchronous SendRawIoControl is still the path in 2.2.0 (the Storage Spaces changes don't touch it), so the original stack from the issue still applies verbatim whenever the device does flip.

@Blacktempel

Copy link
Copy Markdown
Owner

Unfortunately all of this is useless as it is not what I requested.
I assume you will have to run the software for a while longer to see if the issue persists.

Please also check yourself what your AI is about to reply and if it makes sense or if it is just bloating stuff up.
I'm not here to tell another AI what to do and what not.
I simply have no time for this.
Thanks.

@Blacktempel

Copy link
Copy Markdown
Owner

Please try current master with DiskInfoViewer if you are able to reproduce the issue as previously described.

@Zafer-Liu Zafer-Liu closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants