Skip to content

Act on listed identities in expansion, lookups, invalidation and fan-outs (GH-1402) - #1414

Merged
laughingman7743 merged 7 commits into
masterfrom
fix/1402-listed-identity-fan-outs
Nov 16, 2026
Merged

laughingman7743 merged 7 commits into
masterfrom
fix/1402-listed-identity-fan-outs

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

WHAT

rm(), chmod(), cat() and du() now act on the keys that a listing returns, also when a key ends in what looks like a version ID query. The paths that the caller passes still work as path strings, as before.

Behavior before this change (6c37452)

The examples use the keys a/x and a/x?versionId=v1. Each fan-out parsed the listed name bucket/a/x?versionId=v1 as version v1 of a/x, on both filesystems:

  • rm("bucket/a", recursive=True) deleted a/x and version v1 of a/x, and left the key a/x?versionId=v1.
  • chmod("bucket/a", "private", recursive=True) sent PutObjectAcl for a/x twice, once with VersionId=v1, and never for the literal key.
  • cat("bucket/a/*") returned version v1 of a/x under the name of the literal key.
  • du("bucket/a") looked up version v1 of a/x instead of the literal key.

expand_path(..., recursive=True) of a glob expanded below each match by its name. A match such as bucket/a/dir?versionId=v1 was looked up as a version, and the keys below it were left out.

The lookup of an explicit version was cached under the string bucket/key?versionId=v, which is also the name of a key that ends in that query.

_invalidate_path() split a version off the string of a deleted path. So deleting the key a/dir?versionId=v1/z left the listing of bucket/a/dir?versionId=v1 in the cache.

Changes

  • Expansion. _expand_paths() expands as expand_path() does, and returns each name with the path it names:
    • a path that the caller passes as the string it was given;
    • a listed entry as its bucket and key (_entry_path()).
      A recursive glob expands below each match by the match's key prefix (_find_under()), with the caller's prefix. As in fsspec, a recursive glob that matches nothing raises FileNotFoundError, and a caller's detail option is accepted. _find_paths() does the same for find(), and names a version path by the path as given. expand_path() returns the same names.
  • Fan-outs. They parse a string where they use it, as before, and act on a listed path by its bucket and key:
    • rm() and the async _rm(), through _delete_paths(), _delete_objects() and _delete_batches();
    • chmod();
    • cat() and the async _cat(), which read a string with cat_file() and a listed path with _cat_object(). A name that names two paths, such as an explicit version and a listed key with the same query, is read once, as the path that the caller passed, because the contents are returned by name;
    • du() and the async _du(), which look up the path itself with info() and each listed entry by its bucket and key (_target_info()). Each entry is still looked up once, and the async _du() awaits each lookup on its own, as fsspec does. Summing the listing sizes would be a separate optimization.
  • Lookups. The lookup of an explicit version is cached under (name, "versionId", version_id), which no name equals (_lookup_cache_key()).
    • _head_object(), _ls_from_cache() and the typed core of info() (_path_info()) take their cache key from an S3Path.
    • Unchanged:
      • an unversioned lookup is cached under its key, also when version_aware pins a version;
      • an explicit null version is not cached;
      • lookups with different authorization parameters stay apart.
  • Invalidation. _invalidate() evicts a typed path, its version, and the directories above it as they are listed.
    • Delete batches and the file writers, sync and async, pass the paths they know.
    • invalidate_cache() keeps its behavior for strings: it walks the names as they are spelled. A version-qualified string evicts the typed lookup of the version, the path without the version, and the names that end in its query in each spelling (versionId, versionID, versionid, version_id), as before.

This is step 2 of 4 of #1381 (plan in this comment). The remaining steps:

WHY

Closes #1402.

TEST

Tested commit: 99c1b79 for TestS3FileSystem, test_s3_async.py, TestS3File and TestCompressedBuffer. The rest of the filesystem suite was run on f5706ca; later commits only inline the sizes of du() and change invalidate_cache().

  • just lint passes.
  • New tests against the offline fake S3 store, each comparing the result and the operations, keys and version IDs sent with explicit expected values:
    • test_fan_outs_act_on_listed_keys: 5 sync cases (rm, chmod, cat, du, du of a version path) and 3 async cases (_rm, _cat, _du);
    • test_expand_path_expands_below_listed_matches;
    • test_lookups_keep_versions_apart_from_listed_keys, in both directions;
    • test_deletion_invalidates_listed_directories.
  • All 11 pass. With the source at 6c37452, 10 of them fail; the du-of-a-version-path case pins behavior that 6c37452 already had.
  • Tests for the repairs from the independent review, which pass, and of which 6 of 7 fail with the source at 733d4a0:
    • test_rm_of_unmatched_recursive_glob_deletes_nothing;
    • test_expand_path_passes_options_as_fsspec (prefix below the matches, detail);
    • test_invalidate_cache_evicts_the_name_as_given (bucket/?versionId=v1, bucket/, a directory named by another spelling of the query, and the key a/x?versionId=v1, which 733d4a0 already evicted);
    • test_du_cancellation_sends_no_further_lookup (async).
  • Each of these mutations fails only its own test:
    • caching a version under its string;
    • invalidating a deleted path by parsing it;
    • expanding glob matches by name;
    • naming a found version by its entry.
  • The tests below assert the old string cache keys, or patch methods that the fan-outs no longer call. They now use the typed keys or the private expansion, and keep their intent:
    • test_listing_of_version_path_looks_up_the_version;
    • test_invalidate_cache_version_drops_object_path;
    • test_copy_skipped_directory_does_not_conflict;
    • test_cat_interrupted_while_assembling_keeps_no_contents;
    • two chmod tests that stop running requests.
  • The full tests/pyathena/filesystem/ suite against AWS, run locally in parts:
    • the test files other than test_s3.py, with the fsspec conformance tests: 1023 passed, 3 skipped;
    • TestS3FileSystem: 703 passed, 3 skipped;
    • test_s3_async.py, TestS3File and TestCompressedBuffer: 328 passed, 3 skipped.
      Two background runs of the whole suite were stopped by the session while the machine was low on memory.

🤖 Generated with Claude Code

laughingman7743 and others added 4 commits November 16, 2026 02:24
…outs (GH-1402)

rm(), chmod(), cat() and du() parsed the names of the entries that their
expansion listed, so a key that ends in what looks like a version ID
query, such as "a/x?versionId=v1", was taken as that version of "a/x":
rm() deleted the version and left the key, chmod() updated the version
twice over, cat() read the version, and du() looked the version up.

- _expand_paths() expands as expand_path() does and returns, with each
  name, the path that it names: a path that the caller passes is parsed
  once, and a listed entry keeps its bucket and key (_entry_path). A
  recursive glob expands below each match by its key prefix
  (_find_under) instead of finding below its name. expand_path() returns
  the same names.
- rm() deletes, chmod() updates, cat() reads, and du() looks up those
  paths, on both filesystems; the asynchronous _cat() and _du() run the
  typed reads and lookups.
- The lookup of an explicit version is cached under
  (name, "versionId", version_id), so that it no longer shares the
  string key of a key whose name ends in the same query. info(),
  exists(), _head_object() and _ls_from_cache() take the cache key from
  the typed path (_lookup_cache_key).
- _invalidate() invalidates a typed path and the directories above it,
  named as they are listed. Delete batches and the file writers pass the
  paths they know. invalidate_cache() of a version-qualified path also
  invalidates the key that ends in its query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…H-1402)

Self-review repairs:

- du() of a path with a version looked up the current object, because
  the entry of the version names its key alone. _find_paths() returns
  the requested path for the version and the bucket and key of the
  other entries, which du(), chmod() and the expansion use.
- cat_file() looks up the size of a negative offset or an empty range
  with info() of the path that it was given again; only the reads of
  listed paths use the typed lookup.
- The typed lookup cores take the version in the path
  (_path_info(), the typed core of info() renamed from _info() so that
  it is not taken for an fsspec coroutine, and _head_object()).
- expand_path() returns the names of the sorted expansion once without
  sorting them again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repairs. The expansion and find() typed every path, also those
that the caller passed, which changed how they are read and looked up:
cat() no longer called cat_file() and exists() and find() no longer
called info() for them, an invalid path such as "s3://" raised before
anything ran, and cat() read both an explicit version and a listed key
of the same name but returned one of them.

- _expand_paths(), _find_paths() and _find_under() name the paths that
  the caller passed by the strings as given, which cat() reads with
  cat_file(), and rm() and chmod() parse, as before; only the entries
  that a listing returns name their bucket and key. find() and du()
  look the path itself up with info() again.
- cat() and the async _cat() read each name once, as the first path
  that it names, since the contents are returned by name.
- The tests that cache the lookup of a version, invalidate it, or patch
  the lookups and the expansion of cat() and chmod() follow the typed
  cache keys and the private expansion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne (GH-1402)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread pyathena/filesystem/s3.py
)
)

def _expand_paths(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior): FINDINGS, repaired in 868d961, f2a04cb and 733d4a0

Scope: the full diff 6c37452b13b8f732ac4f064fd05118d5fc26545a..572f5fb1. It covers:

  • pyathena/filesystem/s3.py and s3_async.py;
  • tests/pyathena/util.py;
  • the new tests.

Passes:

  • /code-review. It ran both the PR head and master against the fake store and reproduced its findings.
  • A cleanup pass for reuse, simplification, efficiency and the depth of the fix.
  • My own trace of every caller of the expansion and of the lookups.

Findings and repairs:

  1. Typing the paths that the caller passes changed how they are read and looked up. This broke about 37 existing offline tests. Specifically:
    • cat() no longer called cat_file(), and exists() and find() no longer called info(). Tests that patch them failed or hung: the batch, interrupt and on_error tests of cat, and the cancellation tests of the async _cat.
    • expand_path("s3://") raised ValueError instead of returning [''].
    • One invalid path made all of cat(on_error="omit") raise.
      Repair (f2a04cb): _expand_paths(), _find_paths() and _find_under() keep a path that the caller passed as the string it was given. Only the entries that a listing returns carry their bucket and key. So cat() reads a caller's path with cat_file(), rm() and chmod() parse it as before, and find(), du() and exists() look the path itself up with info().
  2. The tests that assert the old string cache key of a version lookup, or mock _head_object with strings, failed:
    • test_listing_of_version_path_looks_up_the_version;
    • test_invalidate_cache_version_drops_object_path;
    • test_copy_skipped_directory_does_not_conflict.
      They now use the (name, "versionId", id) keys and S3Path arguments. The cases for each query spelling and trailing slash are kept.
  3. A regression that I found myself: du() of a version path looked up the current object, because the entry of a version names its key alone. Repair (868d961): _find_paths() names the requested path for a version. Tested by the du-version case.
  4. When an explicit version and a listed key share a name, cat() read both objects but returned one of them. Repair: _first_targets() reads each name once, as the first path that it names, which is the path that the caller passed.
  5. The cat, chmod and interrupt tests patched expand_path() or find(), which the fan-outs no longer call. They now patch _expand_paths() or _find_paths(), and keep their intent: an interrupt after both reads, and the stopped requests.
  6. Cleanup:
    • The sync typed core of info() is renamed _path_info(), so that it is not taken for an fsspec coroutine.
    • The unused version_id parameters of _head_object() and _path_info() are removed.
    • expand_path() no longer sorts its names twice.
    • Docstrings now describe the typed cache keys.
    • s3_request_targets() sorts a missing version ID.

Deferred, with reasons:

Validation:

  • just lint passes.
  • The 11 new tests pass. With the source at 6c37452, 10 fail; the du-version case is new.
  • Each of these mutations fails only its test:
    • caching a version under its string;
    • invalidating a deleted path by parsing it;
    • expanding glob matches by name;
    • naming a found version by its entry;
    • not clearing the contents of an interrupted cat().
  • The full tests/pyathena/filesystem/ suite against AWS was run in parts, because two background runs were stopped while the machine was low on memory:
    • the test files other than test_s3.py: 750 passed, 3 skipped;
    • TestS3File, TestCompressedBuffer and the fsspec conformance tests: 370 passed;
    • TestS3FileSystem: 696 passed, 3 skipped.

Comment thread pyathena/filesystem/s3.py
return file

@staticmethod
def _lookup_cache_key(path: S3Path) -> str | tuple[str, str, str]:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, callers, evidence): CLEAN after the PR body was rewritten for the repairs

Scope:

  • the full diff 6c37452b13b8f732ac4f064fd05118d5fc26545a..733d4a0c1265c7409ee50682726e2ef458293daf;
  • the PR body, the commit messages and the docstrings.

Claims checked:

  1. The before-state of rm, chmod, cat and du. With the source at 6c37452, the fan-out cases fail on the operations, keys and version IDs that the PR body lists.
  2. "expand_path() returns the same names." It now returns list(dict.fromkeys(...)) of the expansion, which is sorted by name. That equals the previous sorted(set(...)). A name that names two paths still appears once.
  3. "The paths that the caller passes still work as path strings." cat() reads them with cat_file(). rm() and chmod() parse them at their use. find(), du() and exists() look the path itself up with info(). The existing batch, interrupt and on_error tests of cat and chmod pass through those seams again, after their patches moved to the private expansion.
  4. "invalidate_cache() keeps its behavior for strings."
    • For a version string, the lookup of that version is evicted under its typed key. Before, it was evicted under the four string spellings.
    • The object path and the parents are evicted as before.
    • The key that ends in the query is evicted as well.
    • The rewritten test_invalidate_cache_version_drops_object_path covers every spelling and trailing-slash case of the old test.
  5. The docstrings of info(), _get_cached_lookup() and _cache_lookup() described the old string keys. They were corrected in 733d4a0.

Existing callers:

  • expand_path(), find(), info(), exists(), cat_file() and invalidate_cache() keep their signatures and outputs.
  • _head_object(), _ls_from_cache(), _delete_*() and _chmod_object() are private.
  • The benchmark harness calls AioS3FileSystem._info(), the fsspec coroutine, which is unchanged. That is why the sync core is named _path_info().

AWS operator:

  • The fan-outs send the same requests as before, except that they now target the listed keys.
  • du() looks up each entry once, as before, through the same listing caches.
  • A recursive glob lists the same prefixes below its matches.
  • No request is added.

Evidence:

  • Local runs only; CI on AWS has not run yet (Draft).
  • The full filesystem suite was run in parts on f2a04cb (counts in the round-one record).
  • 733d4a0 changes only docstrings and a test helper's sort.

Independent review repairs:

- A recursive glob that matches nothing raises FileNotFoundError again,
  as fsspec's expansion of no paths does, so that rm() deletes nothing.
- The expansion below the matches of a recursive glob receives the
  prefix, and a caller's detail option is accepted.
- invalidate_cache() invalidates the name as it is spelled, as before,
  so that a bucket path with a trailing slash evicts the listing of the
  common prefix "/", and a key that ends in a version ID query and has
  no key before it is evicted; a version also evicts its typed lookup.
- The async _du() looks up the found entries one at a time again, so a
  cancellation sends no further lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread pyathena/filesystem/s3.py
return sorted(out)
return sorted(out, key=lambda item: item[0])

def _expand_unversioned(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed result, Codex): FINDINGS (6), all verified and repaired in f5706ca

Reviewer and run:

  • Reviewer: Codex CLI codex exec, model gpt-6-astra, reasoning effort max, read-only sandbox. Session 01a1e50b-0ac4-70d3-8cf4-f5d211288ece.
  • Scope: a detached snapshot of 733d4a0, reviewing 6c37452b13b8f732ac4f064fd05118d5fc26545a..733d4a0c.
  • The prompt gave the intended contract of Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402, but not the PR, the commit messages or earlier findings. The snapshot and the PR worktree were unchanged afterwards.
  • This was a static review: the reviewer ran no tests.

Coverage, in the reviewer's words: "sync/async fan-outs; fsspec 2026.9.0 expansion semantics; version-aware and parameterized cache lookups; deletion/writer invalidation; batching, errors and cancellation; changed docstrings and tests."

Findings (all P2), with the repairs:

  1. Unmatched recursive globs no longer abort deletion. With an existing bucket/a, rm(["bucket/a", "bucket/missing*"], recursive=True) deleted a, whereas fsspec raises FileNotFoundError before deleting.
    Repair: _expand_unversioned() raises as fsspec's expansion of no paths does. As on master, the expansion of the paths with a version ID still continues when another path matched.
  2. Async du kept issuing requests after cancellation. The whole traversal ran in one thread call, so cancelling during the first HeadObject still looked up every remaining entry.
    Repair: the async _du() again awaits each lookup on its own, in a thread of the filesystem.
  3. Query-only literal keys escaped public invalidation. invalidate_cache("bucket/?versionId=v1") left the lookup of the key ?versionId=v1 cached, because the parsed path has no key.
    Repair: see 4.
  4. A trailing bucket slash no longer invalidated its listed spelling. invalidate_cache("bucket/") parsed both spellings to the bucket and left the listing of the common prefix / (("bucket/", "/")) cached.
    Repair, for 3 and 4: invalidate_cache() walks the names as they are spelled again, as master does (_invalidate_name()). A version also evicts its typed lookup (_evict_lookup()), and then the path without the version.
  5. The recursive glob expansion dropped prefix. expand_path("bucket/a/*", recursive=True, prefix="keep") included a/keepdir/drop.txt.
    Repair: _find_under() receives the prefix below the matches.
  6. Forwarding detail raised TypeError. expand_path("bucket/a/*", detail=False) failed.
    Repair: the expansion consumes the caller's detail.

Tests (offline fake store):

  • test_rm_of_unmatched_recursive_glob_deletes_nothing (1)
  • test_du_cancellation_sends_no_further_lookup in test_s3_async.py (2)
  • test_invalidate_cache_evicts_the_name_as_given: the bucket/?versionId=v1 case (3) and the bucket/ case (4)
  • test_expand_path_passes_options_as_fsspec: the prefix-below-matches case (5) and the detail case (6)

With the source at 733d4a0, 6 of these 7 tests fail. The a/x?versionId=v1 case of the invalidation test passes there. It stays, because it is the only test that the key ending in the query of a version path is evicted.

Validation at f5706ca:

  • just lint passes.
  • The tests/pyathena/filesystem/ suite against AWS, run locally in parts:
    • the files other than test_s3.py: 1023 passed, 3 skipped;
    • TestS3FileSystem: 702 passed, 3 skipped;
    • TestS3File and TestCompressedBuffer: 98 passed.

An independent follow-up review of the repair follows.

…GH-1402)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread pyathena/filesystem/s3.py
self._evict_lookup(self._lookup_cache_key(path))
self._invalidate_name(path.name)

def _invalidate_name(self, name: str) -> None:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the independent-review repair (rounds one and two): one simplification applied in d145334; otherwise CLEAN

Scope: 733d4a0c1265c7409ee50682726e2ef458293daf..d145334b1e86e0e62d89d2f49ad473ce6d80749a, the PR body, and the commit messages.

Round one (behavior and implementation):

  • Expansion. As on master, the paths with a version ID are expanded first. The rest is expanded with fsspec's semantics:
    • An unmatched recursive glob raises. With maxdepth=1 it does not, as in fsspec.
    • When only paths without a version ID were given and nothing matches, FileNotFoundError names those paths, as fsspec's expansion did on master.
    • detail and assume_literal are consumed from a copy of the keyword arguments.
    • prefix and refresh are the only options that find() reads besides maxdepth, withdirs and detail, and both reach _find_under().
  • Invalidation. _invalidate_name() evicts, per name, the same keys as master's walk: the lookup, the lookup with parameters, and both listings.
    • For a version, it additionally evicts the typed lookup and walks the parents of both the path without the version and the path as spelled.
    • The second walk only adds evictions, such as those of the directories of a key like a/dir?versionId=v1/z.
    • _invalidate() for typed paths is unchanged in what it evicts.
  • Async du. _du() follows fsspec's _du(): one find, then one awaited lookup per entry. Each lookup runs through _thread_pool.run(), which the other async methods use for cancellation.
  • Simplification: _du_sizes() was left with one caller, so it is inlined into du() (d145334). The du tests pass.

Round two (claims, callers, AWS operator):

  • The PR body now describes:
    • the expansion options;
    • the raise for an unmatched recursive glob;
    • the per-entry awaits of the async _du();
    • the string walk of invalidate_cache();
    • the new tests and the test counts.
      The invalidate_cache() docstring was already accurate.
  • Callers:
    • rm([... "missing*"], recursive=True) raises again before deleting, as on master.
    • expand_path(..., detail=...) and expand_path(..., prefix=...) behave as on master.
    • No public signature changed.
  • AWS operator: the repairs only remove requests. An unmatched recursive glob deletes nothing, and a cancelled async du() sends no further HeadObject. The prefix narrows the listing below the matches as on master.
  • Evidence: local runs against AWS only; CI has not run yet (Draft). An independent follow-up review of this repair is next.

Independent review repair: as on master, invalidate_cache() of a
version-qualified path also evicts the entries of the keys that end in
the query in each spelling that names the version, such as the listing
of the directory bucket/a/dir?versionID=v1 for
bucket/a/dir?versionId=v1. The async _du() docstring says that a
cancellation stops the lookups of the entries, not the traversal of
find().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread pyathena/filesystem/s3.py
# keys that end in the query in each spelling that names the
# version, and the object path without the version.
self._evict_lookup((base, _VERSION_CACHE_KEY, version_id))
for query in ("versionId", "versionID", "versionid", "version_id"):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review (relayed result, Codex): FINDINGS (2). 1 was verified and repaired in 99c1b79; 2 is pre-existing

Reviewer and run:

  • Reviewer: Codex CLI codex exec, model gpt-6-astra, reasoning effort max, read-only sandbox. Session 01a1e523-01a5-7a53-a45d-2df1722bfd9d.
  • Scope: a detached snapshot of d145334, reviewing 733d4a0c1265c7409ee50682726e2ef458293daf..d145334b, with the surrounding code and master (6c37452) as the reference behavior.
  • The prompt stated the intended behavior, but not the PR or the earlier findings. The snapshot and the PR worktree were unchanged afterwards.
  • This was a static review: the reviewer ran no tests.

Coverage, in the reviewer's words: "expansion options/errors and version ordering, affected callers, master cache evictions, sync/async du, docstrings, and new-test contracts."

Findings:

  1. P2: alternate query spellings retained stale listings. With ("bucket/a/dir?versionID=v1", "/") cached, invalidate_cache("s3://bucket/a/dir?versionId=v1") left that listing in place. Master evicts the entries of all four accepted spellings of the query.
    Verified: a regression, already present before the first repair, because 733d4a0 invalidated only the spelling given.
    Repair: for a version, invalidate_cache() invalidates the names that end in the query in each spelling (versionId, versionID, versionid and version_id), as master does, and then the path without the version. The docstring says so.
    Test: the ("bucket/a/dir?versionID=v1", "/") case of test_invalidate_cache_evicts_the_name_as_given. It fails with the source at d145334.
  2. P2: a cancellation still permits requests during discovery. If _du("s3://bucket/missing") is cancelled while its listing is pending, _find_paths() still looks up the path itself afterwards.
    Pre-existing, not changed by this PR. On master, the async _find() runs the whole sync _find() in one _thread_pool.run() call, which fsspec's _du() awaited. So a cancellation during discovery did not stop the lookup that the traversal makes after its listing there either.
    This PR only changes the lookups of the found entries, which again stop at a cancellation. The broader claim was mine, in the review prompt.
    Correction: the _du() docstring now says that a cancellation sends no further lookup of an entry. Making the traversal of find() stop between its own requests would be a change to the async _find(), outside Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402.

Validation at 99c1b79:

  • just lint passes.
  • TestS3FileSystem: 703 passed, 3 skipped.
  • test_s3_async.py, TestS3File and TestCompressedBuffer: 328 passed, 3 skipped.
  • The rest of the filesystem suite is unaffected by this repair, since nothing there depends on invalidate_cache(). It passed at f5706ca: 1023 passed, 3 skipped.

Self-review of this repair:

  • Behavior: only invalidate_cache() of a version-qualified string changed. It evicts at least what master evicts; walking the parents of the alternate spellings evicts nothing extra unless the version ID contains /.
  • Claims: the PR body's description of invalidate_cache() is updated to name each spelling.

A further independent follow-up of this repair follows.

Comment thread pyathena/filesystem/s3.py
# keys that end in the query in each spelling that names the
# version, and the object path without the version.
self._evict_lookup((base, _VERSION_CACHE_KEY, version_id))
for query in ("versionId", "versionID", "versionid", "version_id"):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review of the query-spelling repair (relayed result, Codex): CLEAN

Reviewer and run:

  • Reviewer: Codex CLI codex exec, model gpt-6-astra, reasoning effort max, read-only sandbox. Session 01a1e52f-8635-7361-af1d-64275cd3be42.
  • Scope: a detached snapshot of 99c1b79, reviewing d145334b1e86e0e62d89d2f49ad473ce6d80749a..99c1b79a4968b0101780b78a5551c096b0f159df, with master's _invalidate_path() (6c37452) as the reference.
  • The snapshot and the PR worktree were unchanged afterwards.
  • This was a static review: the reviewer ran no tests.

Coverage, in the reviewer's words: "Cache eviction versus master: roots, buckets, trailing slashes, empty segments, all four version-query spellings, literal query-suffixed names, ancestors, listing caches, and parameterized/typed version lookups."

The reviewer also checked:

  • The _du() cancellation wording: it "accurately describes stopping subsequent entry lookups; an already-running worker may finish".
  • The new test case: it "protects alternate-spelling directory-listing invalidation, a distinct contract", and by inspection fails without the fix. The local run confirmed that failure.

The independent review is complete. The PR moves to Ready for CI.

@laughingman7743
laughingman7743 marked this pull request as ready for review November 16, 2026 16:30
@laughingman7743
laughingman7743 merged commit b720199 into master Nov 16, 2026
12 checks passed
@laughingman7743
laughingman7743 deleted the fix/1402-listed-identity-fan-outs branch November 16, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2)

1 participant