[ML] Prefetch Docker image with retry before running aarch64 tests - #3176
Merged
edsavage merged 2 commits intoSep 3, 2026
Merged
Conversation
Reuses the existing prefetch_docker_image function from dev-tools/docker/prefetch_docker_image.sh (up to 5 retries) before the docker run in the Linux aarch64 test step. Without this, a transient 503 from docker.elastic.co causes the test step to fail immediately, requiring a full manual retry including re-downloading the 458MB test bundle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
edsavage
added
auto-backport
|
Pinging @elastic/ml-core (Team:ML) |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There’s a small but concrete scripting convention inconsistency (source vs .) in the newly added lines that should be aligned with existing CI scripts.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR improves CI robustness for Linux aarch64 by prefetching the Docker image used for running tests, leveraging the existing retrying prefetch_docker_image helper to avoid transient registry failures causing immediate job failures.
Changes:
- Source
dev-tools/docker/prefetch_docker_image.shin the aarch64 Docker test path. - Prefetch the aarch64 base image before
docker runto apply pull retry logic.
File summaries
| File | Description |
|---|---|
| .buildkite/scripts/steps/run_tests.sh | Prefetches the aarch64 Docker base image (with retries) prior to running tests in the container. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
elastic-vault-github-plugin-prod Bot
added a commit
that referenced
this pull request
Sep 4, 2026
…3176) (#3178) Reuses the existing prefetch_docker_image function from dev-tools/docker/prefetch_docker_image.sh (up to 5 retries) before the docker run in the Linux aarch64 test step. Without this, a transient 503 from docker.elastic.co causes the test step to fail immediately, requiring a full manual retry including re-downloading the 458MB test bundle. (cherry picked from commit 70e830c) Co-authored-by: Ed Savage <ed.savage@elastic.co>
elastic-vault-github-plugin-prod Bot
added a commit
that referenced
this pull request
Sep 4, 2026
…3176) (#3179) Reuses the existing prefetch_docker_image function from dev-tools/docker/prefetch_docker_image.sh (up to 5 retries) before the docker run in the Linux aarch64 test step. Without this, a transient 503 from docker.elastic.co causes the test step to fail immediately, requiring a full manual retry including re-downloading the 458MB test bundle. (cherry picked from commit 70e830c) Co-authored-by: Ed Savage <ed.savage@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Linux aarch64 test step in CI pulls
docker.elastic.co/ml-dev/ml-linux-aarch64-native-buildimplicitly viadocker run. A transient 503 from the Docker registry causes the step to fail immediately with exit code 125, requiring a full manual retry — including re-downloading the 458MB test bundle.This PR reuses the existing
prefetch_docker_imagefunction fromdev-tools/docker/prefetch_docker_image.sh, which retriesdocker pullup to 5 times before giving up. The build step (build.sh) already uses this function (viaprefetch_docker_base_image) — this brings the test step into line.All other CI scripts that use
docker runwith registry-pulled images already callprefetch_docker_image(viadev-tools/docker_build.sh,docker_test.sh,docker_check_style.sh).run_tests.shwas the only gap.Test plan
Relates to: https://buildkite.com/elastic/ml-cpp-snapshot-builds/builds/7014#01a05d5a-cf70-4c7a-b556-ae172e0a2bd7