Objects treated as missing despite being present, due to race with geometric repacking

[newren]

Changes since v2:

• Ripped out the old 3/4 dealing with QUICK readers; QUICK readers are left alone
• Insert a new 3/4 fixing git mktree --batch to stop passing QUICK (with new testcase in t1010)
• undo bad paragraph comment change
• renamed fill_midx_entry() -> midx_fill_entry(), so that we catch any other new callers and appropriately check their return value (caught one in test-read-midx.c)

Changes since v1:

• Rebased on top of ps/odb-generic-corrupt-objects, and conflicts with it resolved
• Removed useless test_grep line spotted by Junio in PATCH 1
• Switched fill_midx_entry() to a tri-state to avoid duplicate bsearch_midx(), as suggested by Peff
• Only do the re-read on SECOND_READ, as suggested by Peff
• Handle multiple objects shared across multiple packs correctly (issue caught & corrected & new testcase by deeper AI review)
• Inserted two new patches:
  • 2/4: Fix a leak in git mktree --batch since I use it in new testcases and don't want the *-leaks jobs failing
  • 3/4: Demonstrate and fix QUICK reader problems, while keeping expected QUICK performance for normal cases (we've already been discussing this patch in this thread a bunch anyway, and it's logically related)

Cover letter addendum/update:

We also fix git mktree --batch to no longer erroneously pass QUICK.

Note here that Stolee's suggestion to defer pack deletion via git multi-pack-index expire seems like a good complementary mitigation; it would reduce how often we fall into recovery, while this series tries to fix recovery to work more robustly.

Original cover letter (focused on the final patch):

When an object is found in multiple packs that are in a multi-pack-index, and a subsequent geometric repacking creates a new multi-pack-index and removes the pack that was considered the owner of the object in the old multi-pack-index, then an already-running process that had opened the old multi-pack-index and hadn't yet opened the removed packfile will not be able to access the object -- lookups will return it as missing. Additionally, replay has a separate bug where a missing object causes a SIGSEGV rather than an error message.

This appears to affect a very small percentage of git operations in production since it is a tiny window, but I've found evidence of it occurring in at least eight distinct server-side operations, covering seven different git commands:

git operation                        symptom
-----------------------------------  -----------------------------
git replay (server-side rebase)      SIGSEGV (this series, 1/2)
git merge-tree                       spurious read-miss failure
git diff (raw and tree-vs-tree)      spurious read-miss failure
git rev-list --count                 spurious read-miss failure
git merge-base                       spurious read-miss failure
object/rev resolution (rev-parse,    spurious read-miss failure
  cat-file)
repository repair (fsck/repack)      spurious read-miss failure

There are also commands that could be changing behavior without throwing an error -- e.g. object negotiation thinking an object doesn't exist and instead negotiating based on an older common commit, or cat-file --batch reporting that some objects don't exist.

This series fixes the replay bug first, since it's simpler; investigating it, together with my other recent repacking work, is what led me to the underlying multi-pack-index issue that 2/2 addresses.

cc: Patrick Steinhardt ps@pks.im
cc: Elijah Newren newren@gmail.com
cc: Jeff King peff@peff.net
cc: Derrick Stolee stolee@gmail.com

[newren]

/submit

[gitgitgadget]

Submitted as pull.2207.git.1787092446.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2207/newren/midx-removed-pack-recovery-v1

To fetch this version to local tag pr-2207/newren/midx-removed-pack-recovery-v1:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2207/newren/midx-removed-pack-recovery-v1

[gitgitgadget]

User Patrick Steinhardt <ps@pks.im> has been added to the cc: list.

[gitgitgadget]

This branch is now known as en/midx-missing-pack-fallback.

[gitgitgadget]

This patch series was integrated into seen via git@8ef1c4d.

[gitgitgadget]

There was a status update in the "New Topics" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Waiting for response.
cf. <xmqqfr0augls.fsf@gitster.g>
cf. <aoayppoxHAkcFTBN@pks.im>
source: <pull.2207.git.1787092446.gitgitgadget@gmail.com>

[gitgitgadget]

User Elijah Newren <newren@gmail.com> has been added to the cc: list.

[gitgitgadget]

This patch series is no longer integrated into seen.

[gitgitgadget]

User Jeff King <peff@peff.net> has been added to the cc: list.

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Waiting for response.
cf. <xmqqfr0augls.fsf@gitster.g>
cf. <aoayppoxHAkcFTBN@pks.im>
source: <pull.2207.git.1787092446.gitgitgadget@gmail.com>

[gitgitgadget]

User Derrick Stolee <stolee@gmail.com> has been added to the cc: list.

[gitgitgadget]

This patch series was integrated into seen via git@e4a2bf3.

[newren]

/submit

[gitgitgadget]

Submitted as pull.2207.v2.git.1787684429.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2207/newren/midx-removed-pack-recovery-v2

To fetch this version to local tag pr-2207/newren/midx-removed-pack-recovery-v2:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2207/newren/midx-removed-pack-recovery-v2

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Needs review.
source: <pull.2207.v2.git.1787684429.gitgitgadget@gmail.com>

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Expecting a reroll.
cf. <CABPp-BEmReAR-f-aweM=f=5QhRPxG1K-KLTsbyRt2aDQD_QnVA@mail.gmail.com>
source: <pull.2207.v2.git.1787684429.gitgitgadget@gmail.com>

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Expecting a reroll.
cf. <CABPp-BEmReAR-f-aweM=f=5QhRPxG1K-KLTsbyRt2aDQD_QnVA@mail.gmail.com>
source: <pull.2207.v2.git.1787684429.gitgitgadget@gmail.com>

[newren]

/submit

[gitgitgadget]

Submitted as pull.2207.v3.git.1787986831.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2207/newren/midx-removed-pack-recovery-v3

To fetch this version to local tag pr-2207/newren/midx-removed-pack-recovery-v3:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2207/newren/midx-removed-pack-recovery-v3

[gitgitgadget]

Jeff King wrote on the Git mailing list (how to reply to this email):

On Sat, Aug 29, 2026 at 07:00:30AM +0000, Elijah Newren via GitGitGadget wrote:

> mktree_line() checks each referenced object's type with
> odb_read_object_info_extended() under OBJECT_INFO_QUICK.  QUICK skips the
> reprepare-and-retry that reloads the on-disk pack set, so a resident
> "git mktree --batch" reader reports an object that a concurrent repack
> just relocated into a new pack as missing, and rejects the entry.
> 
> QUICK entered this lookup in 817b0f602710 (mktree: do not check type of
> remote objects, 2022-06-21) only to avoid lazily fetching promisor
> objects; OBJECT_INFO_SKIP_FETCH_OBJECT already provides that.  Drop
> OBJECT_INFO_QUICK and keep OBJECT_INFO_SKIP_FETCH_OBJECT, so mktree still
> avoids a promisor fetch but recovers an object that was merely repacked.

I think this line of reasoning is fine.

We probably _could_ use QUICK when the caller specified --missing, which
would optimize out the SECOND_READ effort if the caller told us they
expect (or at least allow) some items to be missing. But:

  1. It's not clear how people use --missing. If you are just trying to
     be gentle with an occasional missing entry, then the optimization
     is not that interesting. If you run mktree all the time to make
     synthetic trees full of objects you don't have, then maybe you do
     care about the optimization. But if you are doing that then you
     probably are better off with an option that avoids the lookup
     entirely (i.e., we should just trust the type found in the input).

     So there's maybe room for a --yolo argument to mktree, though I
     guess in practice you could just use "hash-object" for that. But
     either way that is way out of scope for this patch.

  2. Prior to 817b0f602710 we were not QUICK either! And that commit was
     only trying to trigger SKIP_FETCH_OBJECT. So whether there is an
     argument for linking --missing and QUICK or not, it should be made
     separately. This patch is just fixing the extra flag that probably
     should not have been added by 817b0f602710.

> +test_expect_success PIPE 'mktree --batch survives a concurrent repack retiring a pack' '

OK. I was hoping we could test this without all of the PIPE complexity,
but I don't think we can. We really need a case where the first lookup
fails but SECOND_READ succeeds, which is inherently a race. Feeding one
entry at a time lets us implement that in a deterministic way, and I
think is the simplest we can get.

So the patch looks good to me overall.

-Peff

[gitgitgadget]

Jeff King wrote on the Git mailing list (how to reply to this email):

On Sat, Aug 29, 2026 at 07:00:31AM +0000, Elijah Newren via GitGitGadget wrote:

> +	/*
> +	 * Recovery for a concurrent-repack race: a stale MIDX may still name a
> +	 * vanished owning pack even though the object survives in another pack
> +	 * the same MIDX covers.  The regular fallback above skips MIDX-covered
> +	 * packs, and repreparing the on-disk pack set does not reload the
> +	 * borrowed, cached MIDX, so scan its packs directly for the survivor.
> +	 *
> +	 * Do this only on the second read, by which point repreparing packs has
> +	 * already had a chance to find an object merely relocated into a new,
> +	 * uncovered pack; only a genuine hidden duplicate reaches here.
> +	 */
> +	if (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&
> +	    (flags & OBJECT_INFO_SECOND_READ)) {
> +		struct multi_pack_index *m = store->midx;
> +		uint32_t i;
> +
> +		for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
> +			struct packed_git *p;
> +
> +			if (prepare_midx_pack(m, i))
> +				continue;
> +			p = nth_midxed_pack(m, i);
> +			if (p && packfile_fill_entry(p, oid, e, bad_pack))
> +				return 1;
> +		}
> +	}

So I think this workaround is fine to do (as long as we are not going to
actually refresh the midx on SECOND_READ, which I agree is probably a
bigger change).

I always get confused about m->num_packs and m->num_packs_in_base, and
whether we are looking at the packs in a midx slice versus the whole
thing. I _think_ what you have here is correct, because we are iterating
from 0 up to the total number of packs, and prepare_midx_pack() etc will
look back through the incremental slices as necessary.

But I wonder if it would be simpler to just iterate over the actual pack
list in the usual way, since we already do that in this function. I
_thought_ this would work:

diff --git a/odb/source-packed.c b/odb/source-packed.c
index 90d88c0a12..86e6a80d2f 100644
--- a/odb/source-packed.c
+++ b/odb/source-packed.c
@@ -33,40 +33,19 @@ static int find_pack_entry(struct odb_source_packed *store,
 	for (l = store->packs.head; l; l = l->next) {
 		struct packed_git *p = l->pack;
 
-		if (!p->multi_pack_index && packfile_fill_entry(p, oid, e, bad_pack)) {
+		/* ...explain tricky race case here... */
+		if (p->multi_pack_index &&
+		    (midx_result != MIDX_FILL_OWNER_UNAVAILABLE ||
+		     !(flags & OBJECT_INFO_SECOND_READ)))
+			continue;
+
+		if (packfile_fill_entry(p, oid, e, bad_pack)) {
 			if (!store->skip_mru_updates)
 				packfile_list_prepend(&store->packs, p);
 			return 1;
 		}
 	}
 
-	/*
-	 * Recovery for a concurrent-repack race: a stale MIDX may still name a
-	 * vanished owning pack even though the object survives in another pack
-	 * the same MIDX covers.  The regular fallback above skips MIDX-covered
-	 * packs, and repreparing the on-disk pack set does not reload the
-	 * borrowed, cached MIDX, so scan its packs directly for the survivor.
-	 *
-	 * Do this only on the second read, by which point repreparing packs has
-	 * already had a chance to find an object merely relocated into a new,
-	 * uncovered pack; only a genuine hidden duplicate reaches here.
-	 */
-	if (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&
-	    (flags & OBJECT_INFO_SECOND_READ)) {
-		struct multi_pack_index *m = store->midx;
-		uint32_t i;
-
-		for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
-			struct packed_git *p;
-
-			if (prepare_midx_pack(m, i))
-				continue;
-			p = nth_midxed_pack(m, i);
-			if (p && packfile_fill_entry(p, oid, e, bad_pack))
-				return 1;
-		}
-	}
-
 	return 0;
 }
 

but it doesn't because we don't always load the midx'd packs into the
pack list (we do it on-demand as they become useful to us). So I think
you'd essentially end up needing to do a loop like the one you have
anyway to prepare_midx_pack() on them all.

And we want to avoid doing that if we can find it outside the midx
(since that was the whole point of waiting for SECOND_READ). Which would
happen...in that loop. So we really do want to have our own
midx-specific loop like you have here.

Sorry, I know that was a lot of text to end up at "you have already
written it the best way", but it took me a while to reason through it.

The patch looks good to me. ;)

-Peff

[gitgitgadget]

Junio C Hamano wrote on the Git mailing list (how to reply to this email):

Jeff King <peff@peff.net> writes:

> ...
> Sorry, I know that was a lot of text to end up at "you have already
> written it the best way", but it took me a while to reason through it.
>
> The patch looks good to me. ;)

Thanks for a very informative and well reasoned write-up in support
of the series.

Shall we mark it for 'next' then?

[gitgitgadget]

Patrick Steinhardt wrote on the Git mailing list (how to reply to this email):

On Sun, Aug 30, 2026 at 01:53:51PM -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
> 
> > ...
> > Sorry, I know that was a lot of text to end up at "you have already
> > written it the best way", but it took me a while to reason through it.
> >
> > The patch looks good to me. ;)
> 
> Thanks for a very informative and well reasoned write-up in support
> of the series.
> 
> Shall we mark it for 'next' then?

Here's my a lot less well reasoned +1, for what it's worth. Thanks!

Patrick

[gitgitgadget]

Jeff King wrote on the Git mailing list (how to reply to this email):

On Sun, Aug 30, 2026 at 01:53:51PM -0700, Junio C Hamano wrote:

> Thanks for a very informative and well reasoned write-up in support
> of the series.
> 
> Shall we mark it for 'next' then?

Yeah, that sounds good to me.

-Peff

[gitgitgadget]

Derrick Stolee wrote on the Git mailing list (how to reply to this email):

On 8/30/2026 4:53 PM, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
> 
>> ...
>> Sorry, I know that was a lot of text to end up at "you have already
>> written it the best way", but it took me a while to reason through it.
>>
>> The patch looks good to me. ;)
> 
> Thanks for a very informative and well reasoned write-up in support
> of the series.
> 
> Shall we mark it for 'next' then?

I'm late in responding, but I support the series, too!

thanks,
-Stolee

[gitgitgadget]

Junio C Hamano wrote on the Git mailing list (how to reply to this email):

Derrick Stolee <stolee@gmail.com> writes:

> On 8/30/2026 4:53 PM, Junio C Hamano wrote:
>> Jeff King <peff@peff.net> writes:
>> 
>>> ...
>>> Sorry, I know that was a lot of text to end up at "you have already
>>> written it the best way", but it took me a while to reason through it.
>>>
>>> The patch looks good to me. ;)
>> 
>> Thanks for a very informative and well reasoned write-up in support
>> of the series.
>> 
>> Shall we mark it for 'next' then?
>
> I'm late in responding, but I support the series, too!
>
> thanks,
> -Stolee

Thanks, all.

[gitgitgadget]

This patch series is no longer integrated into seen.

[gitgitgadget]

This patch series was integrated into seen via git@8dcd61c.

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Will merge to 'next'?
cf. <apVa72AU-cD4IO46@pks.im>
cf. <20260829120721.GF40814@coredump.intra.peff.net>
cf. <20260829114620.GE40814@coredump.intra.peff.net>
source: <pull.2207.v3.git.1787986831.gitgitgadget@gmail.com>

[gitgitgadget]

Derrick Stolee wrote on the Git mailing list (how to reply to this email):

On 8/29/2026 3:00 AM, Elijah Newren via GitGitGadget wrote:
> From: Elijah Newren <newren@gmail.com>

I'm late in reviewing this patch, so forgive me responding inline as
I discover how it works.

tl;dr: Good patch. LGTM.

> Teach find_pack_entry() to recover.  The MIDX lookup now returns a
> tri-state, distinguishing an object absent from the MIDX from one it owns
> via a pack that can no longer be opened; in the latter case, once the
> regular fallback has also missed, scan the MIDX's packs directly for a
> surviving copy.  Because the return value is no longer a boolean, rename
> fill_midx_entry() to midx_fill_entry() so callers must reckon with the
> new enum rather than silently treat MIDX_FILL_OWNER_UNAVAILABLE as a hit.

This tri-state is valuable!
 
> Do the scan only on the second read (OBJECT_INFO_SECOND_READ): by then
> the cheaper on-disk reload has run, so an object merely relocated into a
> new (uncovered) pack has already been found by the regular fallback, and
> only a genuine hidden duplicate reaches the rescan.  A QUICK caller that
> skips the second read simply accepts the false negative, as QUICK is
> designed to.
> 
> Reloading the stale MIDX would be a more complete fix but is much more
> involved (the borrowers above need proper invalidation), so leave that
> for later.

> -		if (m && fill_midx_entry(m, oid, &e, NULL)) {
> +		if (m && midx_fill_entry(m, oid, &e, NULL) == MIDX_FILL_HIT) {

One major benefit to the rename is that we can guarantee that
all callers are updated to reflect the new tri-state response.

It also has a better naming convention, overall.

(reordered header file diff up)
> +/*
> + * Result of looking an object up in a multi-pack-index.  MIDX_FILL_HIT means
> + * "e was filled in"; the two miss variants distinguish an object the midx does
> + * not know about (MIDX_FILL_MISS) from one it does know about but whose owning
> + * pack we can no longer open (MIDX_FILL_OWNER_UNAVAILABLE -- the signature of a
> + * concurrent repack having removed that pack).  A known-bad (corrupt) object
> + * reports MIDX_FILL_MISS but also sets *bad_pack, if provided, to the owning
> + * pack so the caller can tell "corrupt" apart from "absent".
> + */
> +enum midx_fill_result {
> +	MIDX_FILL_MISS = 0,
> +	MIDX_FILL_HIT,
> +	MIDX_FILL_OWNER_UNAVAILABLE,
> +};
> +
> +enum midx_fill_result midx_fill_entry(struct multi_pack_index *m,
> +				      const struct object_id *oid,
> +				      struct pack_entry *e,
> +				      struct packed_git **bad_pack);

This is good documentation that will help future uses know how to
react to the different modes.

> -int fill_midx_entry(struct multi_pack_index *m,
> -		    const struct object_id *oid,
> -		    struct pack_entry *e,
> -		    struct packed_git **bad_pack)
> +enum midx_fill_result midx_fill_entry(struct multi_pack_index *m,
> +				      const struct object_id *oid,
> +				      struct pack_entry *e,
> +				      struct packed_git **bad_pack)
>  {
>  	uint32_t pos;
>  	uint32_t pack_int_id;
>  	struct packed_git *p;
>  
>  	if (!bsearch_midx(oid, m, &pos))
> -		return 0;
> +		return MIDX_FILL_MISS;

Obviously correct: this OID isn't in the sorted list.

>  	midx_for_object(&m, pos);
>  	pack_int_id = nth_midxed_pack_int_id(m, pos);
>  
>  	if (prepare_midx_pack(m, pack_int_id))
> -		return 0;
> +		return MIDX_FILL_OWNER_UNAVAILABLE;

Obviously correct: we tried to open the pack index but failed.

>  	p = m->packs[pack_int_id - m->num_packs_in_base];
>  
>  	/*
> @@ -616,19 +616,19 @@ int fill_midx_entry(struct multi_pack_index *m,
>  	* loaded!
>  	*/
>  	if (!is_pack_valid(p))
> -		return 0;
> +		return MIDX_FILL_OWNER_UNAVAILABLE;

Same: Pack is invalid somehow, likely that the .pack disappeared.

>  	if (oidset_size(&p->bad_objects) &&
>  	    oidset_contains(&p->bad_objects, oid)) {
>  		if (bad_pack && !*bad_pack)
>  			*bad_pack = p;
> -		return 0;
> +		return MIDX_FILL_MISS;

This one is tricky, but makes sense: we have marked this as a
"bad" object so we should act like it doesn't exist. Good.

>  	}
>  
>  	e->offset = nth_midxed_offset(m, pos);
>  	e->p = p;
>  
> -	return 1;
> +	return MIDX_FILL_HIT;

finally: success!>  }


>  static int find_pack_entry(struct odb_source_packed *store,
>  			   const struct object_id *oid,
>  			   struct pack_entry *e,
> +			   enum object_info_flags flags,
>  			   struct packed_git **bad_pack)
>  {
>  	struct packfile_list_entry *l;
> +	enum midx_fill_result midx_result = MIDX_FILL_MISS;
>  
>  	odb_source_prepare(&store->base, 0);
> -	if (store->midx && fill_midx_entry(store->midx, oid, e, bad_pack))
> -		return 1;
> +	if (store->midx) {
> +		midx_result = midx_fill_entry(store->midx, oid, e, bad_pack);
> +		if (midx_result == MIDX_FILL_HIT)
> +			return 1;
> +	}

This looks good. On a hit, we return. Act like a MIDX-miss if we
don't have a midx.

Outside of the patch context is the "reprepare packfiles" to pick
up a copy from a packfile that doesn't exist within the current
(stale) midx.
> +	/*
> +	 * Recovery for a concurrent-repack race: a stale MIDX may still name a
> +	 * vanished owning pack even though the object survives in another pack
> +	 * the same MIDX covers.  The regular fallback above skips MIDX-covered
> +	 * packs, and repreparing the on-disk pack set does not reload the
> +	 * borrowed, cached MIDX, so scan its packs directly for the survivor.
> +	 *
> +	 * Do this only on the second read, by which point repreparing packs has
> +	 * already had a chance to find an object merely relocated into a new,
> +	 * uncovered pack; only a genuine hidden duplicate reaches here.
> +	 */

This comment does a lot of important context-setting to show
that we are in a very narrow case: the stale MIDX has multiple
packs that contain the requested object, but the "newer" one
was deleted without creating a new packfile, so we need to
look at each contained pack for the object from its pack-index.

> +	if (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&
> +	    (flags & OBJECT_INFO_SECOND_READ)) {
> +		struct multi_pack_index *m = store->midx;
> +		uint32_t i;
> +
> +		for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
> +			struct packed_git *p;
> +
> +			if (prepare_midx_pack(m, i))
> +				continue;
> +			p = nth_midxed_pack(m, i);
> +			if (p && packfile_fill_entry(p, oid, e, bad_pack))
> +				return 1;
> +		}
> +	}
> +

This is hopefully a very rare case, but it's good to have
this "fall back to O(num packs)" situation.

> +test_expect_success 'lookup recovers object whose midx-owning pack was removed' '
> +	test_when_finished "rm -fr repo" &&
> +	git init repo &&
> +	(
> +		cd repo &&
> +
> +		# "keep" ends up only in the big pack; "dup" is deliberately
> +		# placed in two packs so the midx has to choose an owner.
> +		test_commit keep &&
> +		echo duplicated-content >dup &&
> +		git add dup &&
> +		git commit -m dup &&
> +		dup_oid=$(git rev-parse HEAD:dup) &&
> +
> +		# Roll every object, including dup, into a single big pack.
> +		git repack -adq &&
> +
> +		# Build a second, "moderate" pack that also contains dup, so dup
> +		# now lives in two packs that the midx will cover.
> +		moderate=$(echo "$dup_oid" |
> +			git pack-objects --quiet $objdir/pack/pack) &&
> +
> +		# Attribute dup to the moderate pack in the midx.
> +		git multi-pack-index write \
> +			--preferred-pack="pack-$moderate.idx" &&

This use of preferred pack is a good way of getting around mtimes
that could be equal. We could also consider updating mtimes, but
this works so don't change it.

> +		# Simulate a concurrent "git repack" retiring the moderate pack:
> +		# its files disappear, but the now-stale midx still names it as
> +		# the owner of dup.  A valid copy of dup survives in the big pack.
> +		rm -f $objdir/pack/pack-$moderate.* &&
> +
> +		# The midx routes the lookup to the deleted pack, and the regular
> +		# pack fallback skips midx-covered packs, so without recovery dup
> +		# would appear missing even though it is physically present.
> +		echo blob >expect &&
> +		git cat-file -t "$dup_oid" >actual &&
> +		test_cmp expect actual
> +	)
> +'
Thanks for adding this test so we can keep this narrow case
working in perpetuity.

Thanks,
-Stolee

[gitgitgadget]

Elijah Newren wrote on the Git mailing list (how to reply to this email):

On Tue, Sep 1, 2026 at 8:26 AM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/29/2026 3:00 AM, Elijah Newren via GitGitGadget wrote:
> > From: Elijah Newren <newren@gmail.com>
>
> I'm late in reviewing this patch, so forgive me responding inline as
> I discover how it works.
>
> tl;dr: Good patch. LGTM.

Thanks for taking a look; I wanted to point out two minor clarifications...

> > +     /*
> > +      * Recovery for a concurrent-repack race: a stale MIDX may still name a
> > +      * vanished owning pack even though the object survives in another pack
> > +      * the same MIDX covers.  The regular fallback above skips MIDX-covered
> > +      * packs, and repreparing the on-disk pack set does not reload the
> > +      * borrowed, cached MIDX, so scan its packs directly for the survivor.
> > +      *
> > +      * Do this only on the second read, by which point repreparing packs has
> > +      * already had a chance to find an object merely relocated into a new,
> > +      * uncovered pack; only a genuine hidden duplicate reaches here.
> > +      */
>
> This comment does a lot of important context-setting to show
> that we are in a very narrow case: the stale MIDX has multiple
> packs that contain the requested object, but the "newer" one
> was deleted without creating a new packfile, so we need to
> look at each contained pack for the object from its pack-index.

Actually, a new packfile is typically created, it just doesn't have
the object in question -- and doesn't need to, because a pre-existing
(also midx-covered) pack already has it.

> > +     if (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&
> > +         (flags & OBJECT_INFO_SECOND_READ)) {
> > +             struct multi_pack_index *m = store->midx;
> > +             uint32_t i;
> > +
> > +             for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
> > +                     struct packed_git *p;
> > +
> > +                     if (prepare_midx_pack(m, i))
> > +                             continue;
> > +                     p = nth_midxed_pack(m, i);
> > +                     if (p && packfile_fill_entry(p, oid, e, bad_pack))
> > +                             return 1;
> > +             }
> > +     }
> > +
>
> This is hopefully a very rare case, but it's good to have
> this "fall back to O(num packs)" situation.

It's actually a fall back to O(num_packs_in_the_midx); on developer
laptops that's probably about the same as O(num_packs), but on busy
servers constantly receiving pushes, the total number of packs often
dwarfs the number of packs in the midx.

[gitgitgadget]

Derrick Stolee wrote on the Git mailing list (how to reply to this email):

On 9/1/2026 12:47 PM, Elijah Newren wrote:
> On Tue, Sep 1, 2026 at 8:26 AM Derrick Stolee <stolee@gmail.com> wrote:
>>
>> On 8/29/2026 3:00 AM, Elijah Newren via GitGitGadget wrote:
>>> From: Elijah Newren <newren@gmail.com>
>>
>> I'm late in reviewing this patch, so forgive me responding inline as
>> I discover how it works.
>>
>> tl;dr: Good patch. LGTM.
> 
> Thanks for taking a look; I wanted to point out two minor clarifications...
> 
>>> +     /*
>>> +      * Recovery for a concurrent-repack race: a stale MIDX may still name a
>>> +      * vanished owning pack even though the object survives in another pack
>>> +      * the same MIDX covers.  The regular fallback above skips MIDX-covered
>>> +      * packs, and repreparing the on-disk pack set does not reload the
>>> +      * borrowed, cached MIDX, so scan its packs directly for the survivor.
>>> +      *
>>> +      * Do this only on the second read, by which point repreparing packs has
>>> +      * already had a chance to find an object merely relocated into a new,
>>> +      * uncovered pack; only a genuine hidden duplicate reaches here.
>>> +      */
>>
>> This comment does a lot of important context-setting to show
>> that we are in a very narrow case: the stale MIDX has multiple
>> packs that contain the requested object, but the "newer" one
>> was deleted without creating a new packfile, so we need to
>> look at each contained pack for the object from its pack-index.
> 
> Actually, a new packfile is typically created, it just doesn't have
> the object in question -- and doesn't need to, because a pre-existing
> (also midx-covered) pack already has it.

Thanks. That helps me understand why this can occur regularly
enough to be triggered in the wild.

>>> +     if (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&
>>> +         (flags & OBJECT_INFO_SECOND_READ)) {
>>> +             struct multi_pack_index *m = store->midx;
>>> +             uint32_t i;
>>> +
>>> +             for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
>>> +                     struct packed_git *p;
>>> +
>>> +                     if (prepare_midx_pack(m, i))
>>> +                             continue;
>>> +                     p = nth_midxed_pack(m, i);
>>> +                     if (p && packfile_fill_entry(p, oid, e, bad_pack))
>>> +                             return 1;
>>> +             }
>>> +     }
>>> +
>>
>> This is hopefully a very rare case, but it's good to have
>> this "fall back to O(num packs)" situation.
> 
> It's actually a fall back to O(num_packs_in_the_midx); on developer
> laptops that's probably about the same as O(num_packs), but on busy
> servers constantly receiving pushes, the total number of packs often
> dwarfs the number of packs in the midx.

Thanks. You're absolutely right that I was not specific enough and
in server situations this loop will be very short.

Thanks,
-Stolee

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Will merge to 'next'?
cf. <20260831231005.GA973618@coredump.intra.peff.net>
cf. <374bffe1-47ff-4cb6-9d69-f4b7da7292da@gmail.com>
source: <pull.2207.v3.git.1787986831.gitgitgadget@gmail.com>

[gitgitgadget]

This patch series was integrated into next via git@73a4cd7.

[gitgitgadget]

There was a status update in the "Cooking" section about the branch en/midx-missing-pack-fallback on the Git mailing list:

The object lookup machinery has been taught to gracefully recover
when a multi-pack-index points to an owning pack that was removed
during a concurrent geometric repack, and 'git replay' has been
fixed to not segfault when reading such missing objects.

Will merge to 'master'.
cf. <20260831231005.GA973618@coredump.intra.peff.net>
cf. <374bffe1-47ff-4cb6-9d69-f4b7da7292da@gmail.com>
source: <pull.2207.v3.git.1787986831.gitgitgadget@gmail.com>