Repository navigation
Act on listed identities in pairing and copy/mv/get (GH-1403) - #1421
laughingman7743 wants to merge 2 commits into
Conversation
copy(), mv() and get(), on both filesystems, now act on the keys that a listing returns by their buckets and keys. A key such as a/x?versionId=null was parsed again as the null version of a/x, so a recursive copy, move or download raised that two paths had the same destination, copied the version, or wrote both objects to one file. - S3PathPairing accepts S3Path sources, names them after their bucket and key as they are, and compares the paths of moves and copies as S3Path objects; strings are paired, compared and returned as before. s3_copy_pairs() returns the S3 destination of an S3Path source as the parsed destination root plus the literal suffix of the source, and rejects a destination root that names a version. - The filesystems pair the expansion of _expand_paths(), whose listed entries are S3Path objects, and look the directories and conflict candidates up by them. - cp_file(), get_file(), S3File and S3FileTarget.resolve() accept an S3Path, which they look up and read by its bucket, key and version. - The async _copy() pairs every copy as the sync copy() does, and the async _get() runs the downloads as fsspec does for lists of paths, with the branches of the callback named after the sources as strings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repairs: - The expansion names an object once: a path that the caller passes and a listed key that name the same object are returned once, as the first of them, as expand_path() names them once. copy(), mv() and get() raised that two paths had the same destination for, e.g., ["s3://bucket/a/x", "s3://bucket/a/*"], and rm() sent the key twice. - info() and cat_file(), and the async _info() and _cat_file(), accept an S3Path, which they look up and read by its bucket, key and version, so that the copies, the downloads, the files and the fan-outs call them for both forms instead of choosing a typed helper at each call; _target_info(), _target_isdir(), _cat_target() and the async _cat_object() are removed. - S3Path.of() returns a path string parsed or an S3Path as it is, in place of a helper on the filesystem and one on the pairing. - cp_file() and _cp_file() keep an S3Path from stringify_path(), a copy invalidates its destination by its bucket and key, and the conflict candidates of the pairing are its named pairs. - The async _copy() pairs lists without a thread, passes fsspec no maxdepth that it would not use, and says that its pairs are not checked for conflicts (GH-1398); _get_pairs() returns nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if not out: | ||
| raise FileNotFoundError(path) | ||
| return sorted(out, key=lambda item: item[0]) | ||
| # The first name and path of each object, by what the path names. |
There was a problem hiding this comment.
Self-review round one (implementation behavior): FINDINGS, repaired in fa1d263
Scope: the full diff b7201990..e3bfb0cb, covering pyathena/filesystem/, the docs and the tests.
Passes:
/code-reviewat high effort./simplify, with four cleanup agents: reuse, simplification, efficiency and altitude.
Findings and repairs:
- Duplicate sources (regression; also Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402's
rm())._expand_paths()returned a path that the caller passed and a listed key that named the same object both, whereexpand_path()named them once.- With only
a/x,copy(["s3://bucket/a/x", "s3://bucket/a/*"], "s3://bucket/b/")raisedCannot copy several paths to the same destination.The same happened formv()andget()with such lists. rm()of those paths senta/xtwice.- The async copy and download had no conflict check, so they would have run twice against one destination.
- Repair: the expansion keeps the first name and path of each object, by what its path names (
S3Path.of()). Two objects of one name, such as an explicit version and a listed key ending in its query, both stay;cat()still reads one per name. - Test:
test_expansion_acts_on_an_object_once(copy,rm).
- With only
- The same lookup dispatch, written again at about ten sites. All four cleanup agents and
/code-reviewflagged it. The sites were_copy_file(sync and async),_object_exists,S3FileTarget.resolve()(read, append, append data, exclusive create),get_file,_target_isdir, and Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402's_target_info/_cat_target.- A new consumer that forgot the branch would parse a listed key again.
- Repair:
info()andcat_file()(and the async_info()/_cat_file()) accept anS3Path, and every site calls them. The helpers_target_info(),_target_isdir(),_cat_target()and the async_cat_object()are gone. - String paths still go through
info(),isdir(),cat_file()andopen()as before, so the existing mocks of those methods keep working. One test that had moved to_target_isdiris back onfs.isdir. - Tests:
test_info_looks_up_path_as_it_is;cat_file()of anS3Pathis covered by thecatcases of Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402.
- The same conversion written four times.
_pair_path()on the filesystem,_as_path()on the pairing, and two inline copies, all for str→S3Path.- Repair:
S3Path.of(). Test:TestS3Path::test_of.
- Repair:
stringify_path()passed anS3Paththrough only by accident:S3Pathhas neither__fspath__norpath.- Repair:
cp_file()and_cp_file()check for anS3Pathbefore stringifying.
- Repair:
- Smaller cleanups:
- A copy invalidates its destination by its bucket and key, so the branch to
invalidate_cache()is gone; it evicts the same entries. _moves()returns its candidates as_NamedPairs._get_pairs()returns nothing, and its inner coroutine matches its annotation.- The async
_copy()pairs lists without a thread and no longer passesmaxdepth, which fsspec ignores for lists. _key_exists()passes itsS3Pathinstead of its URI.
- A copy invalidates its destination by its bucket and key, so the branch to
Deferred, with reasons:
- The async
_copy()runs no conflict checks. Pre-existing, tracked in AioS3FileSystem batch operations do not reject conflicting pairs #1398. The docstring now says so. - The path string of a file opened by a typed path.
sign(),metadata(),getxattr(),setxattr()and the pickled reopen all use it. Onlyget_file()opens typed paths here, in"rb", and calls none of them; carried to Type generated put destinations and document path addressing limits (#1381 step 4) #1404 (comment), along with tests of the typed write modes. _get_pairs()copies fsspec's list loop for_get(). fsspec'sbranch_coropasses the source on as the callback's branch label, so no simpler form keeps string labels for typed sources. The altitude agent agreed.- The upfront
os.makedirs()in_get_pairs(). Kept, since fsspec's_get()did the same before. - One lookup per source in the directory filter. Pre-existing. Carrying the listed entries' types through the expansion would be a separate optimization.
s3_copy_pairs()derives the suffix by slicing the destination. It relies onother_paths()naming each destination afterpath2;test_s3_copy_pairswould fail if fsspec changed that.- The sync and async tests repeat their tables. Kept explicit per file.
Validation:
just lint,just docs lintandjust benchmark testpass.- The filesystem suite against AWS, in parts: 1033 + 711 + 98 passed, 6 skipped. Three conformance tests hit a token-refresh 429 and passed when rerun.
| destinations = other_paths(names, path2, exists=exists, flatten=not source_is_str) | ||
| return list(zip(sources, destinations, strict=True)) | ||
|
|
||
| def s3_copy_pairs( |
There was a problem hiding this comment.
Self-review round two (claims, callers, AWS operator, evidence): CLEAN
Scope:
- the full diff
b7201990..fa1d263e58004cc72647564f01b8724dcfb1c83a; - the PR body, the commit messages, the docstrings and
docs/filesystem.md.
Claims checked:
- The before-state table. It was reproduced at b720199 against the fake store. The async
getrow changed since Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402: both objects were now written toout/x, which the body states. - "Strings work as before," for every widened method:
info(): the root path and the version merge are unchanged; only a local variable was renamed.cat_file(str)still passesname=path._cat_object()'s size lookupinfo(name or path, version_id=...)equals the former_path_info(path.with_version_id(...)), becauseinfo()mergesversion_idonly into a path without one.get_file(str)still callsisdir(rpath)andopen(rpath).S3FileTarget.resolve(str)callsfs.info(path, version_id=...)with the version-suffixed path, as the existing assertions offs.infoexpect. It also callsfs.info(path, refresh=True),fs.cat_file(path, ...)and_object_exists(path)with the same arguments as before.cp_file(str)still stringifies.
- Invalidation of a copy's destination.
_invalidate(S3Path.of(path2))evicts at least whatinvalidate_cache(path2)evicted. A destination string's trailing slashes are kept in the key, and its names are walked withrpartition.- A destination with a version raises before the copy.
- One difference: a subclass's override of
invalidate_cache()is no longer called for string copies. Typed copies andS3Filealready bypassed it.
S3PathPairingfor string callers.copy_pairs,move_pairs,check_copy_pairsandconflict_candidatesreturn the same values. Candidates arestr(target)inbucket/keyform, as before.- The comparisons moved from strings to
S3Pathequality. For parsed strings that is the same relation, becauseS3Path.parse()normalizes exactly what the strings differed in. - The documented example in
docs/filesystem.mdis unchanged.
- The comparisons moved from strings to
- The expansion's deduplication against
expand_path().expand_path()already returned each name once, so its output is unchanged.rm(),chmod(),du()and the copies now act on each object once. - The async
_copy()through the pairing, against fsspec's_copy(). fsspec useddest_is_dironly wherelooks_up_destinationis true, so the pairs are the same, and a lookup is skipped where it could not change them.on_errorandbatch_sizeare fsspec's.
Existing callers:
- No public signature was narrowed.
info,cat_file,cp_file,get_file,S3File,S3FileTarget.resolveand the async_info,_cat_file,_cp_file,_get_fileonly accept more. S3Path.of()andS3PathPairing.s3_copy_pairs()are new.- The removed helpers were private and unreleased (Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402 merged after v4.3.0):
_target_info,_cat_target, the async_cat_object,_pair_pathand_target_isdir. - The benchmark tooling tests, which patch
cp_file/_cp_file/get_file, pass.
AWS operator:
- The same requests, or fewer:
- each object once;
- no destination lookup that cannot change the pairing;
- no HeadObject with a
VersionIdfor a listed key that ends in a query.
- No new permission.
Evidence: the local runs in the round-one record. CI against AWS has not run yet (Draft).
| destinations = other_paths(names, path2, exists=exists, flatten=not source_is_str) | ||
| return list(zip(sources, destinations, strict=True)) | ||
|
|
||
| def s3_copy_pairs( |
There was a problem hiding this comment.
Independent review (relayed result, Codex): CLEAN
Reviewer and run:
- Reviewer: Codex CLI
codex exec, modelgpt-6-astra, reasoning effortmax, read-only sandbox. Session01a1e5a6-3290-77d3-8b56-a830e5334fd4. - Scope: a detached snapshot of fa1d263, reviewing
b7201990..fa1d263e58004cc72647564f01b8724dcfb1c83a. - The prompt stated the agreed contract of Act on listed identities in pairing and copy/mv/get (#1381 step 3) #1403 and its out-of-scope items (
put()destinations, the path string of a typed file, the conflict checks of the async_copy()), but not the PR, the commit messages or the self-review findings. - The snapshot and the PR worktree were unchanged afterwards.
- This was a static review: the reviewer ran no tests.
Coverage, in the reviewer's words:
- "Sync and async
copy(),mv(), andget(): globs, recursion, maxdepth, lists, trailing slashes, bucket roots, and versions." - "Typed/string pairing, destination construction, conflict lookups, null-version comparisons, and expansion deduplication."
- "
info(),cat_file(),cp_file(),get_file(), and file-target resolution across read, write, append, and exclusive-create modes." - "Async scheduling,
batch_size,on_error, cancellation tracking, and string callback labels against fsspec 2026.9.0." - "Public string compatibility, documentation, and changed tests for distinct contracts and regression sensitivity."
Result: "CLEAN — no actionable findings."
The independent review is complete. The PR moves to Ready for CI.
WHAT
copy(),mv()andget()now act on the keys that a listing returns by their buckets and keys, on both filesystems. This includes a key that ends in what looks like a version ID query. The paths that the caller passes work as path strings, as before.Behavior before this change (b720199)
The examples use the literal keys
a/xanda/x?versionId=nullin "bucket". Every listed name was parsed again, andbucket/a/x?versionId=nullnames thenullversion ofa/x:copy("bucket/a", "bucket/b", recursive=True)Cannot copy several paths to the same destination.a/x, then raisedCannot copy to a versioned file.copy("bucket/a/*", "bucket/c/")a/xversionnullmv("bucket/a", "bucket/b", recursive=True)Cannot move several paths to the same destination.get("bucket/a", "out", recursive=True)Cannot get several paths to the same destination.a/xand itsnullversion toout/xWith only the key
a/x?versionId=v1, a recursivegetraisedFileNotFoundError, becauseget_file()opened the path, andS3FileTarget.resolve()parsed it as versionv1ofa/x.Changes
S3PathPairingalso acceptsS3Pathsources. Strings are paired, compared and returned as before.S3Pathsource is named after its bucket and key as they are, socopy_pairs()gives it a destination such asout/x?versionId=null.s3_copy_pairs(): whenpath2is a string, the S3 destination of anS3Pathsource is also anS3Path. It is the parsedpath2plus the literal suffix of the source. Ifpath2names a version, it raisesValueErrorbefore anything is copied. The other destinations are the strings ofcopy_pairs().S3Pathobjects, so the keya/x?versionId=nullis never thenullversion ofa/x.conflict_candidates()returns the candidates ofS3Pathsources asS3Pathobjects, andmissingaccepts them.copy(),mv()andget()pair the expansion of_expand_paths()(Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402). Its listed entries areS3Pathobjects, and its other entries are the caller's strings.copy(["s3://bucket/a/x", "s3://bucket/a/*"], ...)raised that two paths had the same destination, andrm()of the same paths sent the key twice. That was a regression of Act on listed identities in expansion, lookups, invalidation and rm/chmod/cat/du (#1381 step 2) #1402._copy()now pairs every copy as the synccopy()does, and fsspec copies the lists of the pairs, with itsbatch_sizeandon_errorhandling. As before, its pairs are not checked for conflicts (AioS3FileSystem batch operations do not reject conflicting pairs #1398)._get()runs the downloads as fsspec's_get()does for lists of paths: the same batching, local parent directories and progress callback. Each branch of the callback is named after its source as a string, as before.info(),cat_file(),cp_file(),get_file(),S3FileandS3FileTarget.resolve(), and the async_info(),_cat_file(),_cp_file()and_get_file(), also accept anS3Path. They look it up and read it by its bucket, key and version. Strings work as before._target_info(),_cat_target(), and the async_cat_object().S3Path.of()returns a path string parsed, or anS3Pathas it is.docs/filesystem.mddescribesS3Pathsources ands3_copy_pairs().This is step 3 of 4 of #1381 (plan in this comment). The string-addressing limits and the destinations of
put()follow in #1404, including the path string of a file opened by a typed path (comment).WHY
Closes #1403.
TEST
Tested commit: fa1d263.
just lint,just docs lintandjust benchmark test(363 passed, 1 skipped) pass.test_copies_act_on_listed_keys: sync and async, casescopy,copy-globandmv;test_get_downloads_listed_keys: sync and async.test_copy_pairs_names_path_sources_by_key;test_s3_copy_pairs(a directory root, a bucket root with a trailing slash, a list of sources);test_s3_copy_pairs_versioned_destination;test_move_pairs_keeps_listed_keys_apart_from_versions;test_move_pairs_path_conflict_candidates.test_expansion_acts_on_an_object_once(copy,rm);test_info_looks_up_path_as_it_is;TestS3Path::test_of.S3Pathinstead of its string (sync and async);expand_path(), which the pairing no longer calls; with the real expansion against a mock client they hung. They now patch_expand_paths()and keep their intent:test_mv_nothing_within_maxdepth;test_mv_conflicting_destinations_of_listed_sources;test_get_put_pass_kwargs_to_expansion.tests/pyathena/filesystem/suite against AWS, run locally in parts with a per-test timeout:test_s3.py, including the fsspec conformance tests: 1033 passed, 3 skipped. Three tests failed when refreshing the login token hit a 429 ("Rate exceeded" on CreateOAuth2Token), and passed when rerun;TestS3FileSystem: 711 passed, 3 skipped;TestS3FileandTestCompressedBuffer: 98 passed.🤖 Generated with Claude Code