Skip to content

Act on listed identities in pairing and copy/mv/get (GH-1403) - #1421

Open
laughingman7743 wants to merge 2 commits into
masterfrom
fix/1403-listed-identity-pairing
Open

laughingman7743 wants to merge 2 commits into
masterfrom
fix/1403-listed-identity-pairing

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

WHAT

copy(), mv() and get() now act on the keys that a listing returns by their buckets and keys, on both filesystems. This includes a key that ends in what looks like a version ID query. The paths that the caller passes work as path strings, as before.

Behavior before this change (b720199)

The examples use the literal keys a/x and a/x?versionId=null in "bucket". Every listed name was parsed again, and bucket/a/x?versionId=null names the null version of a/x:

Call Sync Async
copy("bucket/a", "bucket/b", recursive=True) raised Cannot copy several paths to the same destination. copied a/x, then raised Cannot copy to a versioned file.
copy("bucket/a/*", "bucket/c/") the same; its directory check looked up a/x version null the same
mv("bucket/a", "bucket/b", recursive=True) raised Cannot move several paths to the same destination. the same
get("bucket/a", "out", recursive=True) raised Cannot get several paths to the same destination. wrote a/x and its null version to out/x

With only the key a/x?versionId=v1, a recursive get raised FileNotFoundError, because get_file() opened the path, and S3FileTarget.resolve() parsed it as version v1 of a/x.

Changes

  • Pairing. S3PathPairing also accepts S3Path sources. Strings are paired, compared and returned as before.
    • An S3Path source is named after its bucket and key as they are, so copy_pairs() gives it a destination such as out/x?versionId=null.
    • New s3_copy_pairs(): when path2 is a string, the S3 destination of an S3Path source is also an S3Path. It is the parsed path2 plus the literal suffix of the source. If path2 names a version, it raises ValueError before anything is copied. The other destinations are the strings of copy_pairs().
    • Moves and copies compare their paths as S3Path objects, so the key a/x?versionId=null is never the null version of a/x.
    • conflict_candidates() returns the candidates of S3Path sources as S3Path objects, and missing accepts them.
  • Filesystems.
  • Lookups and files.
  • Docs. The "Path pairing" section of docs/filesystem.md describes S3Path sources and s3_copy_pairs().

This is step 3 of 4 of #1381 (plan in this comment). The string-addressing limits and the destinations of put() follow in #1404, including the path string of a file opened by a typed path (comment).

WHY

Closes #1403.

TEST

Tested commit: fa1d263.

  • just lint, just docs lint and just benchmark test (363 passed, 1 skipped) pass.
  • New tests against the offline fake S3 store compare the stored keys, the CopyObject sources and destinations, the deletions, the downloads and the callback branches with explicit expected values:
    • test_copies_act_on_listed_keys: sync and async, cases copy, copy-glob and mv;
    • test_get_downloads_listed_keys: sync and async.
  • New pairing unit tests:
    • test_copy_pairs_names_path_sources_by_key;
    • test_s3_copy_pairs (a directory root, a bucket root with a trailing slash, a list of sources);
    • test_s3_copy_pairs_versioned_destination;
    • test_move_pairs_keeps_listed_keys_apart_from_versions;
    • test_move_pairs_path_conflict_candidates.
  • All 15 new tests pass; with the source at b720199, all 15 fail.
  • Tests for the repairs of the self-review, which fail with the source at e3bfb0c:
    • test_expansion_acts_on_an_object_once (copy, rm);
    • test_info_looks_up_path_as_it_is;
    • TestS3Path::test_of.
  • Mutations, each of which fails the tests that protect it:
    • naming a callback branch by the S3Path instead of its string (sync and async);
    • parsing the pairs from their strings.
  • Three existing tests patched the public expand_path(), which the pairing no longer calls; with the real expansion against a mock client they hung. They now patch _expand_paths() and keep their intent:
    • test_mv_nothing_within_maxdepth;
    • test_mv_conflicting_destinations_of_listed_sources;
    • test_get_put_pass_kwargs_to_expansion.
  • The full tests/pyathena/filesystem/ suite against AWS, run locally in parts with a per-test timeout:
    • the files other than test_s3.py, including the fsspec conformance tests: 1033 passed, 3 skipped. Three tests failed when refreshing the login token hit a 429 ("Rate exceeded" on CreateOAuth2Token), and passed when rerun;
    • TestS3FileSystem: 711 passed, 3 skipped;
    • TestS3File and TestCompressedBuffer: 98 passed.

🤖 Generated with Claude Code

laughingman7743 and others added 2 commits November 17, 2026 03:05
copy(), mv() and get(), on both filesystems, now act on the keys that
a listing returns by their buckets and keys. A key such as
a/x?versionId=null was parsed again as the null version of a/x, so a
recursive copy, move or download raised that two paths had the same
destination, copied the version, or wrote both objects to one file.

- S3PathPairing accepts S3Path sources, names them after their bucket
  and key as they are, and compares the paths of moves and copies as
  S3Path objects; strings are paired, compared and returned as before.
  s3_copy_pairs() returns the S3 destination of an S3Path source as the
  parsed destination root plus the literal suffix of the source, and
  rejects a destination root that names a version.
- The filesystems pair the expansion of _expand_paths(), whose listed
  entries are S3Path objects, and look the directories and conflict
  candidates up by them.
- cp_file(), get_file(), S3File and S3FileTarget.resolve() accept an
  S3Path, which they look up and read by its bucket, key and version.
- The async _copy() pairs every copy as the sync copy() does, and the
  async _get() runs the downloads as fsspec does for lists of paths,
  with the branches of the callback named after the sources as strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repairs:

- The expansion names an object once: a path that the caller passes and
  a listed key that name the same object are returned once, as the
  first of them, as expand_path() names them once. copy(), mv() and
  get() raised that two paths had the same destination for, e.g.,
  ["s3://bucket/a/x", "s3://bucket/a/*"], and rm() sent the key twice.
- info() and cat_file(), and the async _info() and _cat_file(), accept
  an S3Path, which they look up and read by its bucket, key and
  version, so that the copies, the downloads, the files and the
  fan-outs call them for both forms instead of choosing a typed helper
  at each call; _target_info(), _target_isdir(), _cat_target() and the
  async _cat_object() are removed.
- S3Path.of() returns a path string parsed or an S3Path as it is, in
  place of a helper on the filesystem and one on the pairing.
- cp_file() and _cp_file() keep an S3Path from stringify_path(), a copy
  invalidates its destination by its bucket and key, and the conflict
  candidates of the pairing are its named pairs.
- The async _copy() pairs lists without a thread, passes fsspec no
  maxdepth that it would not use, and says that its pairs are not
  checked for conflicts (GH-1398); _get_pairs() returns nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread pyathena/filesystem/s3.py
if not out:
raise FileNotFoundError(path)
return sorted(out, key=lambda item: item[0])
# The first name and path of each object, by what the path names.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior): FINDINGS, repaired in fa1d263

Scope: the full diff b7201990..e3bfb0cb, covering pyathena/filesystem/, the docs and the tests.

Passes:

  • /code-review at high effort.
  • /simplify, with four cleanup agents: reuse, simplification, efficiency and altitude.

Findings and repairs:

  1. Duplicate sources (regression; also Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402's rm()). _expand_paths() returned a path that the caller passed and a listed key that named the same object both, where expand_path() named them once.
    • With only a/x, copy(["s3://bucket/a/x", "s3://bucket/a/*"], "s3://bucket/b/") raised Cannot copy several paths to the same destination. The same happened for mv() and get() with such lists.
    • rm() of those paths sent a/x twice.
    • The async copy and download had no conflict check, so they would have run twice against one destination.
    • Repair: the expansion keeps the first name and path of each object, by what its path names (S3Path.of()). Two objects of one name, such as an explicit version and a listed key ending in its query, both stay; cat() still reads one per name.
    • Test: test_expansion_acts_on_an_object_once (copy, rm).
  2. The same lookup dispatch, written again at about ten sites. All four cleanup agents and /code-review flagged it. The sites were _copy_file (sync and async), _object_exists, S3FileTarget.resolve() (read, append, append data, exclusive create), get_file, _target_isdir, and Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402's _target_info/_cat_target.
    • A new consumer that forgot the branch would parse a listed key again.
    • Repair: info() and cat_file() (and the async _info()/_cat_file()) accept an S3Path, and every site calls them. The helpers _target_info(), _target_isdir(), _cat_target() and the async _cat_object() are gone.
    • String paths still go through info(), isdir(), cat_file() and open() as before, so the existing mocks of those methods keep working. One test that had moved to _target_isdir is back on fs.isdir.
    • Tests: test_info_looks_up_path_as_it_is; cat_file() of an S3Path is covered by the cat cases of Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402.
  3. The same conversion written four times. _pair_path() on the filesystem, _as_path() on the pairing, and two inline copies, all for str→S3Path.
    • Repair: S3Path.of(). Test: TestS3Path::test_of.
  4. stringify_path() passed an S3Path through only by accident: S3Path has neither __fspath__ nor path.
    • Repair: cp_file() and _cp_file() check for an S3Path before stringifying.
  5. Smaller cleanups:
    • A copy invalidates its destination by its bucket and key, so the branch to invalidate_cache() is gone; it evicts the same entries.
    • _moves() returns its candidates as _NamedPairs.
    • _get_pairs() returns nothing, and its inner coroutine matches its annotation.
    • The async _copy() pairs lists without a thread and no longer passes maxdepth, which fsspec ignores for lists.
    • _key_exists() passes its S3Path instead of its URI.

Deferred, with reasons:

  • The async _copy() runs no conflict checks. Pre-existing, tracked in AioS3FileSystem batch operations do not reject conflicting pairs #1398. The docstring now says so.
  • The path string of a file opened by a typed path. sign(), metadata(), getxattr(), setxattr() and the pickled reopen all use it. Only get_file() opens typed paths here, in "rb", and calls none of them; carried to Type generated put destinations and document path addressing limits (#1381 step 4) #1404 (comment), along with tests of the typed write modes.
  • _get_pairs() copies fsspec's list loop for _get(). fsspec's branch_coro passes the source on as the callback's branch label, so no simpler form keeps string labels for typed sources. The altitude agent agreed.
  • The upfront os.makedirs() in _get_pairs(). Kept, since fsspec's _get() did the same before.
  • One lookup per source in the directory filter. Pre-existing. Carrying the listed entries' types through the expansion would be a separate optimization.
  • s3_copy_pairs() derives the suffix by slicing the destination. It relies on other_paths() naming each destination after path2; test_s3_copy_pairs would fail if fsspec changed that.
  • The sync and async tests repeat their tables. Kept explicit per file.

Validation:

  • just lint, just docs lint and just benchmark test pass.
  • The filesystem suite against AWS, in parts: 1033 + 711 + 98 passed, 6 skipped. Three conformance tests hit a token-refresh 429 and passed when rerun.

destinations = other_paths(names, path2, exists=exists, flatten=not source_is_str)
return list(zip(sources, destinations, strict=True))

def s3_copy_pairs(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, callers, AWS operator, evidence): CLEAN

Scope:

  • the full diff b7201990..fa1d263e58004cc72647564f01b8724dcfb1c83a;
  • the PR body, the commit messages, the docstrings and docs/filesystem.md.

Claims checked:

  1. The before-state table. It was reproduced at b720199 against the fake store. The async get row changed since Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402: both objects were now written to out/x, which the body states.
  2. "Strings work as before," for every widened method:
    • info(): the root path and the version merge are unchanged; only a local variable was renamed.
    • cat_file(str) still passes name=path. _cat_object()'s size lookup info(name or path, version_id=...) equals the former _path_info(path.with_version_id(...)), because info() merges version_id only into a path without one.
    • get_file(str) still calls isdir(rpath) and open(rpath).
    • S3FileTarget.resolve(str) calls fs.info(path, version_id=...) with the version-suffixed path, as the existing assertions of fs.info expect. It also calls fs.info(path, refresh=True), fs.cat_file(path, ...) and _object_exists(path) with the same arguments as before.
    • cp_file(str) still stringifies.
  3. Invalidation of a copy's destination. _invalidate(S3Path.of(path2)) evicts at least what invalidate_cache(path2) evicted. A destination string's trailing slashes are kept in the key, and its names are walked with rpartition.
    • A destination with a version raises before the copy.
    • One difference: a subclass's override of invalidate_cache() is no longer called for string copies. Typed copies and S3File already bypassed it.
  4. S3PathPairing for string callers. copy_pairs, move_pairs, check_copy_pairs and conflict_candidates return the same values. Candidates are str(target) in bucket/key form, as before.
    • The comparisons moved from strings to S3Path equality. For parsed strings that is the same relation, because S3Path.parse() normalizes exactly what the strings differed in.
    • The documented example in docs/filesystem.md is unchanged.
  5. The expansion's deduplication against expand_path(). expand_path() already returned each name once, so its output is unchanged. rm(), chmod(), du() and the copies now act on each object once.
  6. The async _copy() through the pairing, against fsspec's _copy(). fsspec used dest_is_dir only where looks_up_destination is true, so the pairs are the same, and a lookup is skipped where it could not change them. on_error and batch_size are fsspec's.

Existing callers:

  • No public signature was narrowed. info, cat_file, cp_file, get_file, S3File, S3FileTarget.resolve and the async _info, _cat_file, _cp_file, _get_file only accept more.
  • S3Path.of() and S3PathPairing.s3_copy_pairs() are new.
  • The removed helpers were private and unreleased (Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402 merged after v4.3.0): _target_info, _cat_target, the async _cat_object, _pair_path and _target_isdir.
  • The benchmark tooling tests, which patch cp_file/_cp_file/get_file, pass.

AWS operator:

  • The same requests, or fewer:
    • each object once;
    • no destination lookup that cannot change the pairing;
    • no HeadObject with a VersionId for a listed key that ends in a query.
  • No new permission.

Evidence: the local runs in the round-one record. CI against AWS has not run yet (Draft).

destinations = other_paths(names, path2, exists=exists, flatten=not source_is_str)
return list(zip(sources, destinations, strict=True))

def s3_copy_pairs(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed result, Codex): CLEAN

Reviewer and run:

  • Reviewer: Codex CLI codex exec, model gpt-6-astra, reasoning effort max, read-only sandbox. Session 01a1e5a6-3290-77d3-8b56-a830e5334fd4.
  • Scope: a detached snapshot of fa1d263, reviewing b7201990..fa1d263e58004cc72647564f01b8724dcfb1c83a.
  • The prompt stated the agreed contract of Act on listed identities in pairing and copy/mv/get (#1381 step 3) #1403 and its out-of-scope items (put() destinations, the path string of a typed file, the conflict checks of the async _copy()), but not the PR, the commit messages or the self-review findings.
  • The snapshot and the PR worktree were unchanged afterwards.
  • This was a static review: the reviewer ran no tests.

Coverage, in the reviewer's words:

  • "Sync and async copy(), mv(), and get(): globs, recursion, maxdepth, lists, trailing slashes, bucket roots, and versions."
  • "Typed/string pairing, destination construction, conflict lookups, null-version comparisons, and expansion deduplication."
  • "info(), cat_file(), cp_file(), get_file(), and file-target resolution across read, write, append, and exclusive-create modes."
  • "Async scheduling, batch_size, on_error, cancellation tracking, and string callback labels against fsspec 2026.9.0."
  • "Public string compatibility, documentation, and changed tests for distinct contracts and regression sensitivity."

Result: "CLEAN — no actionable findings."

The independent review is complete. The PR moves to Ready for CI.

@laughingman7743
laughingman7743 marked this pull request as ready for review November 16, 2026 18:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Act on listed identities in pairing and copy/mv/get (#1381 step 3)

1 participant