[BUG] HTTP error responses from Zenodo bypass the "use local copy" fallback #190

Describe the bug

ZenodoRecord._fetch_uncached() contains a fallback that uses the already-downloaded local
copy when Zenodo cannot be reached:

if not zenodo_response:
    logger.warning(f"Zenodo unreachable. Using latest downloaded {self.label}.")
    return self.target_dir / latest_local

This branch is unreachable for HTTP error responses. In _get_metadata_and_archive_url(),
the non-200 case raises ZenodoException from inside the try block, while the only
except clause catches requests.exceptions.RequestException. ZenodoException does not
inherit from it, so the exception propagates out of fetch() instead of being converted
into the None that triggers the fallback.

Consequently None is only ever returned for connection-level failures (DNS, connection
reset, read timeout). Any HTTP error status — 502, 504, 407, 429 — aborts the call even
though a valid, complete atlas folder is present on disk.

In practice this made a long batch job fail on ~30 of 201 subjects during transient Zenodo
gateway errors (502/504), each time with the atlases sitting unused in
registration/atlases/15236131_v2.0.0.

To Reproduce

Steps to reproduce the behavior:

1. Make sure the atlases have been downloaded once, so a valid local copy exists:

   ls "$(python -c 'import brainles_preprocessing, os; print(os.path.dirname(brainles_preprocessing.__file__))')/registration/atlases"
   # 15236131_v2.0.0

2. Install brainles-preprocessing
3. Simulate any non-200 response from the Zenodo API (equivalent to the 502/504 we hit):

   import brainles_preprocessing.utils.zenodo as zen
   
   zen.ZenodoRecord.BASE_URL = "https://zenodo.org/api/records-this-path-does-not-exist"
   print(zen.fetch_atlases())

Result:

ERROR | Cannot find record '15236131' on Zenodo (response.status_code=404).
brainles_preprocessing.utils.zenodo.ZenodoException: Cannot find record '15236131' on Zenodo (response.status_code=404).

Expected behavior

The local copy is used and a path is returned, i.e. the same behaviour as for a
connection-level failure:

WARNING | Zenodo returned 404 for record 15236131
INFO    | Found local atlases: 15236131_v2.0.0
WARNING | Zenodo unreachable. Using latest downloaded atlases.
.../registration/atlases/15236131_v2.0.0

A ZenodoException should only be raised when Zenodo is unusable and there is no valid
local copy to fall back to.

Screenshots

n/a

Environment

operating system and version?

Fedora release 44

NVIDIA drivers and GPUs

Not relevant to this bug (it fails before any GPU work)

Python environment and version?

Conda environment with Python 3.12.

version of brainles_preprocessing ?

0.6.10

I will try to reproduce the bug on the latest 0.6.13 version, and see if parts or all bugs are fixed already.

Additional context

Real log lines from the affected run (the job continues to the next subject, so the same
failure repeats whenever Zenodo has a bad minute):

2026-09-02 17:13:47.761 | ERROR | brainles_preprocessing.utils.zenodo:_get_metadata_and_archive_url:144 - Cannot find record '15236131' on Zenodo (response.status_code=502).
2026-09-02 17:14:18.286 | ERROR | brainles_preprocessing.utils.zenodo:_get_metadata_and_archive_url:144 - Cannot find record '15236131' on Zenodo (response.status_code=504).

Three smaller issues in the same file, happy to fold them into the same PR or split them out:

1. Neither requests.get() call passes a timeout, so a stalled connection (e.g. behind an
   authenticating HTTP proxy) can hang the process indefinitely.
2. _get_latest_version_folder_name() inspects only the newest matching folder and returns
   None if it is empty. Since _download() does folder.mkdir(...) before fetching the
   archive, a failed download leaves an empty <record_id>_v<newer> directory that then
   shadows an intact older copy — turning a recoverable state into
   "... not found locally and Zenodo could not be reached." Staging the download in a temp
   directory and moving it into place only on success would avoid creating the empty folder
   at all.
3. The "new version available" path calls shutil.rmtree() on the local copy before
   downloading the replacement, so a failure mid-upgrade leaves no usable copy.

The error text Cannot find record '...' on Zenodo (response.status_code=502) is also
misleading — 502/504 are gateway errors and say nothing about whether the record exists.