[Merlin0220]

Summary

Optimize large volume directory listings by fetching per-entry metadata concurrently, with concurrency scaled by the number of entries and bounded by a configurable limit.

This targets network storage such as NFS, where serial metadata lookups can accumulate significant latency. Small directories continue to use serial lookups to avoid concurrency overhead.

Changes

Adaptive concurrency

Introduce ReadDirScaling to determine concurrency for each directory listing. By default:

• Fewer than 5,000 entries: process metadata lookups serially.
• At 5,000 entries: start with 4 concurrent lookups.
• Add 1 concurrent lookup for every additional 10,000 entries.
• Cap concurrency at 16.

Use errgroup.SetLimit to bound concurrent metadata lookups.

Configurable scaling

Add Orchestrator configuration and pass it through Builder to each Chrooted filesystem instance.

Environment variable | Default | Description -- | -- | -- VOLUME_READ_DIR_CONCURRENCY_THRESHOLD | 5000 | Entry count at which concurrent lookups begin VOLUME_READ_DIR_CONCURRENCY_BASE | 4 | Initial concurrency at the threshold VOLUME_READ_DIR_CONCURRENCY_STEP | 10000 | Additional entries required to increase concurrency by one VOLUME_READ_DIR_CONCURRENCY_MAX | 16 | Maximum concurrency per listing

Reject invalid configuration during parsing: the threshold must be nonnegative, the base and step must be positive, and the maximum must be at least the base.

Preserve listing behavior

Write metadata results into their corresponding sorted entry positions, preserving name-sorted output regardless of lookup completion order.

Existing behavior is preserved:

• Return metadata for symbolic links themselves.
• Skip entries removed during the listing.
• Return other metadata lookup errors to the caller.
• Wait for all concurrent lookups before releasing directory resources.

Tests

Add unit tests covering:

• Configuration defaults, environment overrides, and invalid values.
• Concurrency thresholds, scaling steps, and upper limits.
• Zero-value and degenerate scaling parameters.
• Consistent directory contents and ordering between serial and concurrent execution.

Tests and performance benchmarks were not run while preparing this description.