Repository navigation
feat(readfallback): allow disabling FindMissing replication - #386
Merged
EdSchouten merged 3 commits intoOct 9, 2026
Merged
EdSchouten merged 3 commits into
EdSchouten merged 3 commits into
Conversation
joeljeske
force-pushed
the
joeljeske/readfallback-disable-findmissing-replication
branch
from
October 7, 2026 17:37
083611a to
bca7048
Compare
EdSchouten
reviewed
Oct 8, 2026
| @@ -650,6 +650,15 @@ message ReadFallbackBlobAccessConfiguration { | |||
| // the secondary backend to the primary backend. If unset, objects | |||
| // will not be copied. | |||
| BlobReplicatorConfiguration replicator = 3; | |||
Member
There was a problem hiding this comment.
Why not change this field into two separate ones:
- get_replicator
- find_missing_replicator
And then you can set the latter to { noop: {} }.
Contributor
Author
There was a problem hiding this comment.
Hey @EdSchouten I can make that change but I didn't for 2 reasons:
- Users that want to replicate both (like today), will now need to configure 2 replicators, which means it will duplicate the concurrency limits between the replicators , and it will prevent deduplication across the replicators from functioning. For some setups using a remote replicator, this may be fine, but others using a local replicator (e.g. with bb_clientd) it can cause duplicate replication.
- Its not backwards compatible, and users must update their configuration
What do you think? Can we keep it this way, or would you prefer to have separate replicators despite the potential performance impact?
Member
There was a problem hiding this comment.
Fair point. Then let's aim for a solution in between:
- Let the BlobAccess take a pair of replicators.
- Let the configuration files use the structure as is.
Let ReadFallbackBlobAccess take a pair of replicators instead of a bool. The configuration is unchanged: by default the same replicator is used for both, so concurrency limits and deduplication remain shared. When disable_find_missing_replication is set, a no-op replicator is used for FindMissing().
EdSchouten
approved these changes
Oct 9, 2026
Member
|
LGTM! Will merge once I've confirmed CI is happy. |
EdSchouten
approved these changes
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
disableFindMissingReplicationto the sharedreadFallbackconfiguration. When true,FindMissingBlobs()still checks the primary backend and then queries the secondary for digests missing from the primary, but does not replicate blobs found only in the secondary. Replication duringGet()andGetFromComposite()is unchanged.This lets existence checks avoid pulling blobs into the primary before they are actually read. The option is available wherever a
readFallbackbackend is configured, including nested backends.NewReadFallbackBlobAccess()now takes a pair of replicators: one forGet()/GetFromComposite()and one forFindMissing(). The configuration layer passes the same replicator instance for both by default, so concurrency limits and deduplication remain shared. WhendisableFindMissingReplicationis set, it passes a no-op replicator forFindMissing().Compatibility
bool, matching existing configuration fields such asdisable_http2; presence tracking is unnecessary.falsevalues retain the existing replication behavior; existing configuration files need no changes.NewReadFallbackBlobAccess()gains a second replicator argument. All in-repository callers are updated.Testing
bazel test //pkg/blobstore/...— all 12 test targets passed.Get(),GetFromComposite()andFindMissing()each use their own replicator when they differ.bazel run //:gazelle— no BUILD changes.Generated with Toshi