Schema subclasses share one ABCMeta cache on Python <=3.10, so isinstance/issubclass return whichever answer was asked first

[JarryShaw]

On Python ≤ 3.10, every Schema subclass shares Schema's _abc_impl instead of getting its own. One ABCMeta subclass-check cache therefore serves the entire schema family, and because it is keyed only on the class being tested and not on the class it is tested against, the first answer for a given class is cached and returned for every later question about it.

The result: issubclass and isinstance against schema classes return whichever answer was asked for first.

Reproduction

Two fresh 3.10 interpreters, same tree, differing only in the order of the two questions:

import sys; sys.path.insert(0, '<tree>')
from pcapkit.protocols.schema.schema import Schema
from pcapkit.protocols.schema.internet.mh import (
    ANIGeoLocationSuboption as G, ANINetworkIdentifierSuboption as N)

order asked	first answer	second answer
issubclass(G, Schema) then issubclass(G, N)	True ✔	True ✘ (siblings)
issubclass(G, N) then issubclass(G, Schema)	False ✔	False ✘ (it is a Schema)

Python 3.10.20:  G._abc_impl is Schema._abc_impl  ->  True
Python 3.14.7 :  G._abc_impl is Schema._abc_impl  ->  False   (and both answers correct)

The first question is always right; the second always inherits the first's answer. No instances are involved, so this is not about __eq__, __hash__ or value equality — it is a shared cache object. SchemaMeta derives from abc.ABCMeta (pcapkit/protocols/schema/schema.py:136) and every schema descends from collections.abc.Mapping.

Verified on 3.10.20 and 3.14.7. CPython reworked the _abc caches across the 3.11/3.12 line, which is why 3.14 is unaffected — but 3.10 is in the supported matrix (pyproject.toml claims >=3.6, and CI runs 3.10 through 3.15).

Why this is worth fixing rather than avoiding

It surfaced through #437, where eight new Mobility Header helpers dispatched sub-options through a chain of isinstance(schema, Schema_X) tests. On 3.10 a false positive sent a sub-option down a sibling's branch (three AttributeErrors), and a cached negative made ListField.pack's isinstance(item, Schema) false, producing FieldValueError and zero-filled reconstructions — six CI failures on 3.10 while 3.14 was entirely green. Those helpers have been changed to dispatch on the wire type code, which is what the rest of that module does, so #437 no longer depends on this. The underlying defect is untouched.

grep finds 39 isinstance(…, Schema…) sites under pcapkit/ on main. Most ask a single question and so cannot see the bug. The ones that ask several in sequence can, and the clearest candidate is HTTP/2 frame dispatch:

pcapkit/protocols/application/httpv2.py:316   if isinstance(frame, Schema_UnassignedFrame):
pcapkit/protocols/application/httpv2.py:318   if isinstance(frame, Schema_DataFrame):
pcapkit/protocols/application/httpv2.py:321   if isinstance(frame, Schema_HeadersFrame):
pcapkit/protocols/application/httpv2.py:325   if isinstance(frame, Schema_PriorityFrame):
pcapkit/protocols/application/httpv2.py:327   if isinstance(frame, Schema_RSTStreamFrame):

That is the same shape that failed in MH. Whether it actually misbehaves on 3.10 depends on which question the process asked first, which makes it order- and import-dependent — the worst kind of latent defect, since it can pass CI and fail for a user. Others worth checking in the same pass: corekit/fields/collections.py:111, corekit/fields/misc.py:326, protocols/protocol.py:467, protocols/internet/ipv4.py:1195, protocols/internet/hopopt.py:1266, protocols/internet/esp.py:1321.

Two ways to fix, and they are not equivalent

1. Make each schema class get its own _abc_impl — the root fix. It restores correct isinstance/issubclass everywhere at once, but it touches SchemaMeta, which every protocol in the tree depends on, so it needs its own change with a full byte-identical sweep behind it.
2. Stop dispatching on isinstance where a type code is available — what protocols: complete the Mobility Header registry -- all 24 message types, 70 of 71 options, all 4 CGA extensions #437 did. Narrower, and arguably better design regardless, since dispatching on the wire code is what the registry already keys on. It does not fix the 37 other sites.

Both are probably wanted: (2) removes the dependency wherever a code exists, (1) stops the trap being re-laid by the next person who writes an isinstance chain.

Deliberately not folded into #437 — the blast radius belongs in its own change rather than in a Mobility Header PR.

[JarryShaw]

Second confirmed victim: PCAP-NG name resolution — and a correction to my own earlier claim

I previously stated that #440's seven Python 3.10 failures were not this issue. That was wrong. They are this issue, and the way I got it wrong is worth recording because it will mislead the next person the same way.

I ran two checks and both were individually correct while both missed the actual site:

• I measured EndRecord._abc_impl is Info._abc_impl on the data models in pcapkit.protocols.data.misc.pcapng — False on both interpreters. True, and irrelevant: Info subclasses really are unaffected.
• I grepped pcapkit/protocols/misc/pcapng.py, the protocol module, for an isinstance-against-schema chain — none there. Also true.

The chain is in the schema module, and schema classes are exactly what shares the cache:

pcapkit/protocols/schema/misc/pcapng.py:1249
    for record in self.records:
        if isinstance(record, (IPv4Record, IPv6Record)):
            for name in record.names:

Measured on the same tree, same script, main at c28ffc287:

	3.10.20	3.14.7
schema EndRecord._abc_impl is IPv4Record._abc_impl	True	False
schema EndRecord._abc_impl is Schema._abc_impl	True	False
data EndRecord._abc_impl is Info._abc_impl	False	False

So on 3.10 an EndRecord answers True to isinstance(record, (IPv4Record, IPv6Record)), NameResolutionBlock.post_process then reads record.names, and it dies with AttributeError: 'EndRecord' object has no attribute 'names'. Every Name Resolution Block carries a terminating EndRecord, which is why the block, its three ns_dns* options and its three records all fail together — seven cases from one wrong answer.

A detail that matters for anyone trying to reproduce this: two synthetic reproductions did not show the wrong answer. The cache is order-sensitive, so whether it is poisoned depends on which question the process asked first — only driving the real packing path produced the failure. That is very likely why an earlier attempt at ruling this in or out came back clean, and it is the property that makes this defect dangerous: it can pass a targeted test and fail in situ.

Where that leaves the issue

The prediction in the original report was that httpv2.py:316-327 was the likeliest next victim, since it has the same chain shape. That remains unverified — nobody has driven it on 3.10. PCAP-NG name resolution is now a confirmed second site, found by accident rather than by the sweep, which argues the other 37 isinstance(…, Schema…) sites deserve the same treatment: exercised on 3.10 through their real path, not probed synthetically.

Note also that fixing this turns #440's Python 3.10 job red, by design — that PR records the seven as interpreter-dependent gaps asserted to fail in exactly the recorded way, so the fix is what flips them.

[JarryShaw]

Root cause found, reproduced in five lines, and a candidate fix verified. This has resisted synthetic reproduction because the missing ingredient was never the cache — it was the question order, and specifically that Mapping has to be asked before Schema.

The cause is an explicit version-guarded workaround

pcapkit/protocols/schema/schema.py:238-241:

# NOTE: for unknown reason, the following code will cause an error
# for duplicated keyword arguments in class definition.
if sys.version_info < (3, 11):
    return type.__new__(cls, name, bases, attrs, **kwargs)
return super().__new__(cls, name, bases, attrs, **kwargs)

type.__new__ bypasses abc.ABCMeta.__new__, and ABCMeta.__new__ is what calls _abc_init(cls) to give each new class its own _abc_impl — the per-class registry, negative cache and positive cache. So on Python < 3.11 every Schema subclass never gets one and inherits Schema's, and a single ABCMeta cache serves the entire family. That is the whole defect, and the sys.version_info < (3, 11) guard is exactly why it is 3.10-and-below only.

Why dict is the thing that leaks

Schema inherits Mapping — class Schema(Mapping[str, _VT], Generic[_VT], metaclass=SchemaMeta) at :245 — and dict is a registered virtual subclass of collections.abc.Mapping. So a True cached from the Mapping question is served straight back for Schema and for every subclass.

Reproduction on 3.10.20, against 94a93e721

from collections.abc import Mapping
from pcapkit.protocols.schema.schema import Schema
from pcapkit.protocols.schema.misc.pcapng import EndRecord

isinstance({}, Mapping)     # True  -- asked first, caches True
isinstance({}, Schema)      # True  -- WRONG
isinstance({}, EndRecord)   # True  -- WRONG

EndRecord._abc_impl is Schema._abc_impl     : True
IPv4Record._abc_impl is Schema._abc_impl    : True
CommentOption._abc_impl is Schema._abc_impl : True

Asked cold, without the Mapping question first, isinstance({}, Schema) correctly returns False. That ordering dependence is why every synthetic probe in this programme has failed to show it.

Where it bites in real code

pcapkit/protocols/schema/schema.py:519-520:

if isinstance(value, Schema):
    dict_[out_key] = value.to_dict()

A plain dict tests True, so to_dict() is called on it — AttributeError: 'dict' object has no attribute 'to_dict' at :520. This currently turns Python 3.10 and Integration Python 3.10 red on PR #457, in tests/protocols/misc/test_pcapng_unit.py::PCAPNGUnitTests::test_pcapng_remaining_constructor_branches_and_custom_dispatch. That PR's ChainMap change does not introduce a defect — it changes the order in which schema classes are first touched and so flips this latent case into an active one. Confirmed by its reviewer: the same test passes on origin/main under the identical 3.10.20 interpreter and fails on the PR head.

Candidate fix, verified

Giving each class its own _abc_impl restores correct answers without touching anything else. Measured on 3.10.20:

import abc
for klass in (Schema, EndRecord):
    abc._abc_init(klass)

EndRecord._abc_impl is Schema._abc_impl : False
isinstance({}, Mapping)                 : True
isinstance({}, Schema)                  : False   <- correct
isinstance({}, EndRecord)               : False   <- correct
isinstance(EndRecord.__new__(EndRecord), EndRecord) : True   <- still correct

So the fix direction is to stop bypassing ABCMeta.__new__ on < 3.11. Three ways, roughly in order of preference:

1. Solve the original problem instead of working around it. The comment says super().__new__ "will cause an error for duplicated keyword arguments in class definition" — that is the code=-style class keyword this codebase uses widely (class CommentOption(Option, code=…)). On < 3.11 ABCMeta.__new__ forwards **kwargs in a way that collides with __init_subclass__ receiving them too. Consuming those kwargs before delegating, rather than skipping ABCMeta.__new__, keeps the ABC machinery intact.
2. Keep type.__new__ but restore the invariant explicitly — call abc._abc_init(new_cls) on the result. Minimal and verified above, but _abc_init is private API.
3. Stop inheriting Mapping, which removes the dict-shaped leak specifically. This does not fix the shared cache in general — any other virtual-subclass relationship would still leak — so it treats the symptom.

Whichever is taken, the regression test has to ask Mapping before Schema, or it will pass on the broken code. That is the property that makes this bug invisible.

Consequence for sequencing

Earlier notes in this programme had #439 recorded as unblocked by #446/#445 landing. That is backwards: #439 blocks PR #457, which cannot go green on 3.10 until this is fixed or that one test is guarded. And the INTERPRETER_GAPS table in tests/protocols/test_option_roundtrip_unit.py is not an available route here — it serves the option round-trip harness, whereas this failure is a plain unit test.

[JarryShaw]

The duplicated keyword arguments NOTE is explained

The code comment at pcapkit/protocols/schema/schema.py:238-239 says:

for unknown reason, the following code will cause an error for duplicated
keyword arguments in class definition.

Measured, and it is not mysterious: ABCMeta.__new__'s parameters became
positional-only in Python 3.11, and seven schema classes are declared with a
class keyword argument called namespace, which collides with one of them on
3.10 and below.

The signature change

python 3.10.21  ABCMeta.__new__ signature: (mcls, name, bases, namespace, **kwargs)
python 3.11.15  ABCMeta.__new__ signature: (mcls, name, bases, namespace, /, **kwargs)
python 3.12.13  ABCMeta.__new__ signature: (mcls, name, bases, namespace, /, **kwargs)
python 3.14.7   ABCMeta.__new__ signature: (mcls, name, bases, namespace, /, **kwargs)

The guard's sys.version_info < (3, 11) boundary is exactly where the / was
added. That is the whole coincidence.

The collision, reproduced

With the guard removed, so SchemaMeta.__new__ delegates via
super().__new__(cls, name, bases, attrs, **kwargs):

python 3.10.21
  class kwarg 'code'       -> OK
  class kwarg 'name'       -> TypeError: SchemaMeta.__new__() got multiple values for argument 'name'
  class kwarg 'bases'      -> TypeError: SchemaMeta.__new__() got multiple values for argument 'bases'
  class kwarg 'namespace'  -> TypeError: ABCMeta.__new__() got multiple values for argument 'namespace'

python 3.12.13
  class kwarg 'code'       -> OK
  class kwarg 'name'       -> TypeError: SchemaMeta.__new__() got multiple values for argument 'name'
  class kwarg 'bases'      -> TypeError: SchemaMeta.__new__() got multiple values for argument 'bases'
  class kwarg 'namespace'  -> OK

namespace is the one that changes behaviour between the two, and it is the one
this codebase actually uses. attrs is passed positionally into ABCMeta's
namespace slot while namespace='...' also arrives in **kwargs, so on 3.10
the same parameter is bound twice.

The seven declarations that trigger it

pcapkit/protocols/schema/misc/pcapng.py:475   class _OPT_Option(Option, namespace='opt')
pcapkit/protocols/schema/misc/pcapng.py:626   class _IF_Option(Option, namespace='if')
pcapkit/protocols/schema/misc/pcapng.py:895   class _EPB_Option(Option, namespace='epb')
pcapkit/protocols/schema/misc/pcapng.py:1162  class _NS_Option(Option, namespace='ns')
pcapkit/protocols/schema/misc/pcapng.py:1269  class _ISB_Option(Option, namespace='isb')
pcapkit/protocols/schema/misc/pcapng.py:1573  class _DSB_Option(Option, namespace='dsb')
pcapkit/protocols/schema/misc/pcapng.py:1646  class _PACK_Option(Option, namespace='pack')

The consumer is Option.__init_subclass__ at
pcapkit/protocols/schema/misc/pcapng.py:401-402, whose signature is
(cls, /, code=None, namespace=None, *args, **kwargs); it assigns
cls.__namespace__ = namespace at :432.

Why this matters for the fix

The version guard was a workaround for a name collision, not for anything
about ABCMeta's behaviour — so it can be removed outright once the collision is
gone, rather than being replaced by an abc._abc_init call that undoes the
damage of having bypassed the public path.

Two directions, in preference order:

1. Rename the class keyword so it no longer collides — namespace= to
   something like ns= or opt_namespace=. That is seven class declarations
   plus the __init_subclass__ signature and its docstring, all in one file,
   and it lets SchemaMeta.__new__ become an unconditional
   super().__new__(...). Note that code= is unaffected and needs no change.
   Worth checking whether name/bases are used as class keywords anywhere
   too — I found none under pcapkit/, but those would collide on every
   version, not just 3.10.
2. Keep the guard and call abc._abc_init(klass) before returning. This
   works — see the measurements in the earlier comment — but it relies on a
   private CPython API to repair a problem that direction 1 removes.

For reference, the bug the guard causes is that no Schema class ever gets its
own _abc_impl, so Schema._abc_impl is collections.abc.Mapping._abc_impl is
True and the corruption runs both ways: asking Mapping first makes
isinstance({}, Schema) return True, and asking Schema first makes
isinstance({}, Mapping) return False.