[cryptosven76]

Draft fix for RED-51747 (write-back data corruption on a 4 KiB page shared by two writers).

Problem

iomap_writeback_handle_eof() zeroes the folio straddling i_size past that size, then calls ->writeback_range() with end_pos = i_size. fuse_iomap_writeback_range() ignores end_pos and queues the whole block, relying on the crop in fuse_send_writepage(). That crop reads i_size again later, when the request is sent.

Under DLM with the writeback cache, i_size is only this node's view. Writers hold the inode lock shared and each claims its own extension under fi->lock. A partial EOF folio can hold another node's bytes, read in by the read-modify-write read under the write grant. If a concurrent local writer extends i_size between the zeroing and the crop, the zeroed tail is written over the other node's acknowledged data.

Evidence

The reproducer is in red/red PR 29256 (test/client/brlock_repro): three nodes writing interleaved 47008-byte blocks of one file. It fails about once every 2000 iterations. A redfsd build with crc64 tracing on every read reply and writeback payload showed, at iteration 1820:

• the lower writer's read-modify-write read of the shared page returned the correct bytes;
• the lower writer's kernel flushed the page as "own block + zeros", with the zeros starting exactly at its local i_size (the end of its own block);
• a higher-offset local write had extended i_size 40 µs before the flush reached redfsd.

Full analysis and traces are in RED-51747.

Fix

Send only up to end_pos, but still report the whole range as handled, so the bytes iomap zeroed never leave the node. The 6.12 tree is not affected: its writeback crops at i_size and does not zero.

Status

Draft. A 3-hour test with this change (as DKMS 20261016.clamp1 on Ubuntu 6.17.0-42) is running on tyan-h3-u24. The result will be added here and in RED-51747. redfs-ubuntu-resolute-7.0.0-14.14 has the same fuse_iomap_writeback_range() and needs the same change.

Open question for review: the zeroed tail stays in the clean, uptodate folio. After a revoke it is invalidated, but after a flusher writeback a later local i_size extension could expose those zeros to local readers. Should the EOF folio be dropped or kept non-uptodate past end_pos under DLM?

🤖 Generated with Claude Code

[achhenderson]

Approve for the failure this targets, with one follow-up to track (also summarised on RED-51747, comment 10490229). The nits at the end are non-blocking.

What I checked

I read the diff against redfs-ubuntu-hwe-6.17.0-16.16-24.04.1 together with fs/iomap/buffered-io.c on the same branch, and matched it against the traces on RED-51747 (iteration 1820 with crc tracing, iteration 720 with kernel ftrace). Both traces show the same window: iomap_writeback_handle_eof() zeroes the folio at the node-local i_size, a concurrent local write then extends i_size before fuse_flush_writepages() reads it as crop, and fuse_send_writepage() sends the whole page. Sending end_pos - pos closes that window because end_pos is the exact size the zeroing used, so whatever i_size does afterwards cannot widen the send.

Correctness of the diff

• send_len cannot be zero or underflow. iomap_writeback_folio() asserts end_pos > pos and bounds the dirty-range walk by round_up(end_pos, blocksize), so every range passed to ->writeback_range starts below end_pos.
• Returning len rather than send_len is required: iomap_writeback_range() treats 0 or > rlen as an error and advances by the return value.
• A short descriptor cannot misplace a following folio. fuse_writepage_need_send() checks byte contiguity, so the next folio opens a new request. In practice the short descriptor is always last anyway, since the next folio is entirely past end_pos and skipped by handle_eof.
• fuse_send_writepage() sums descs[i].length, so the send-time crop runs on the already clamped size and the shrinking-i_size branch is unchanged.
• Writeback completion counts folios (iomap_start_folio_write(..., 1) / iomap_finish_folio_write(..., 1)), not bytes, so the shorter length does not affect write_bytes_pending.
• nr_bytes and the alignment_pages arithmetic use send_len, which is consistent.
• The clamp is not gated on fc->dlm. I think that is right: bytes past i_size at zeroing time are never meaningful to the server, and single-client behaviour is unchanged in effect.

The commit message matches the code and the traces. The statement that 6.12 crops at i_size without zeroing is consistent with the 6.12 runs on smc-a9 not reproducing.

What this does not close (follow-up, not a blocker)

After the clamp, the folio on the lower writer still holds zeros where the other node's bytes were, and it is clean and uptodate. Concretely, with the reproducer's per-block fsync:

1. U writes the head of block b+1 into page P; the server stores it. L takes the grant, read-modify-writes P (now holding U's bytes), writes the tail of block b. L's i_size is the end of block b.
2. L fsyncs while still holding the grant. handle_eof zeroes U's bytes in L's copy of P. The clamp sends only L's bytes, which is correct on the wire. P stays cached on L with zeros.
3. L's next write (block b+3) raises i_size past P.
4. A buffered read on L of P now returns zeros from cache with no server round trip, for as long as L holds the grant. If L ever re-dirties its own part of P, the whole block is dirty, end_pos is past P, and the zeros go to the server.

The revoke path is safe: fuse_reverse_inval_inode() writes back and then drops the range, and a clean folio drops without -EBUSY. So the exposure is between a flusher/fsync writeback and the next revoke. Whether the reproducer's read-back hits step 4 depends on whether the writer mount reads before another mount's grant revokes it; a buffered read on the writer mount immediately after its fsync would test it directly.

The fix for that is the other proposal already on the ticket: under fc->dlm && writeback_cache, raise i_size to max(i_size, pos + nob) under fi->lock when a read reply arrives, in both fuse_do_readfolio() and fuse_readpages_end(), bumping attr_version and honouring FUSE_I_SIZE_UNSTABLE as fuse_read_update_size() does. That stops handle_eof from zeroing other nodes' bytes at all and keeps this clamp as the second guard. It needs a server-side confirmation first: a remote shrink must revoke grants on the range before the new size is visible, otherwise a read reply could raise i_size over a truncated-away region. I'd keep that as a separate PR.

Nits (non-blocking)

1. Add a WARN_ON_ONCE(send_len == 0) (or equivalent) next to the clamp. It is unreachable today, but the invariant lives in iomap, not here.
2. Add one sentence to the commit message noting that the zeroed tail remains in the cached folio and that the read-reply i_size update is the follow-up, so the next reader does not take this as the complete fix.
3. Please post the result of the 3-hour 20261016.clamp1 run on the ticket or here before merge, so the PR carries its test evidence.

[cryptosven76]

Test result on tyan-h3-u24 (Ubuntu 6.17.0-42, this change built as DKMS 20261016.clamp1 plus a rate-limited dmesg line for the case it guards):

• 3 hours, 15,108 iterations of the RED-51747 reproducer with 4 mounts and 3 writers: 0 corrupt, 0 metadata mismatches. All mounts are verified every iteration, buffered and O_DIRECT. Unpatched, the same setup failed about once every 2000 to 2300 iterations.
• The clamp stepped in 11 times with i_size already past end_pos at send time, which is the case that used to send zeros. Every end_pos was an exact 47008-byte block boundary, the zeroed tails were 896 to 4032 bytes, and one event (end_pos 1316224, 2,688-byte tail) has exactly the offset and length of the original field corruption.

The open question about the zeroed tail staying in the uptodate folio still stands; no local buffered-read mismatch showed up in this run.

🤖 Generated with Claude Code

[yongzech]

Tested on 6 client hosts (tyan-h4-u7..u12, Ubuntu 6.17.0-40-generic) with one write-back redfs mount each, using a multi-host version of the RED-51747 reproducer. Three hosts write interleaved 47008-byte blocks and three only verify.

The module is the installed red-client-fs-dkms 2.5.1~alpha.1 source plus this change, as Sven's DKMS patch, which also adds a rate-limited "clamped writeback past EOF" dmesg line.

Module	Result
Stock (srcversion A904329957AB094C49E96FD)	Corrupted at iteration 2596, and at 86 in a traced run
This PR (srcversion 18F0CC903CD1F955282A0D4)	17,990 iterations in 1 hour, no corruption

The clamp fired 34 times during the run. Each time i_size had already moved past end_pos when the folio was queued, so the stock code would have sent the zeroed tail. A bpftrace run on the stock module caught the failing case directly: end_pos = i_size = 423072 at writeback, a local write claimed i_size about 10 µs later, and the FUSE_WRITE went out with the full page. Details are in RED-51747.

The open review point still applies: the zeroed tail stays in the clean, uptodate folio after a writeback that is not a revoke. A follow-up test will add raising i_size on read replies under DLM.

[yongzech]

Companion change: #267 raises i_size over the bytes a DLM read reply returns, so iomap_writeback_handle_eof() no longer zeroes another node's bytes in a shared page in the common case. It is independent of this PR, but the raise is skipped for short reads, for a whole-file invalidate that races the read, and for pages that never came from a read reply, and in those cases only this clamp keeps the zeroed tail off the server. With both applied, tyan-h4-u7..u12 ran 1 h of brlock_mh.py (17,782 iterations) and 1 h of brlock_mh.py --midsync (12,036 iterations) with 0 corruption, and the clamp hit 16 and 2 times, against 34 for this PR alone. Please merge both.

[yongzech]

LGTM

[cryptosven76]

Post-merge check on tyan-h3-u24: the deployed redfs.ko (DKMS red-client-fs-dkms 20261016.092513, kernel 6.17.0-42) contains this change together with #267 and #268. On 4 write-back mounts running the RED-51747 reproducer, one writer's data crc trace (red/red #29285) showed 3,293 of 73,349 writebacks ending mid-page at a block boundary after about 6,000 iterations, for example off=798720 len=47424 ending at 846144 = 18 × 47008. Those are the clamped sends. The reproducer had 0 corruptions in 6,110 iterations at that point; a 3-hour run is in progress, with results on RED-51747.

🤖 Generated with Claude Code