[jmandel]

Core interfaces for database-backed providers

This PR adds a few small hooks to FHIRsmith's code-system and value-set provider interfaces. A provider that keeps its content in a database (such as the SQLite-backed engine we have proposed) needs them to answer some questions without loading everything into memory.

Every hook defaults to today's behaviour. Main's providers don't override any of them, so they take exactly the same paths as before. The PR is interfaces only: there is no new provider and no new configuration.

Each change below is one commit with its own tests. For each one we say what we need, what the PR does, and the other options we would also be happy with. If you'd rather have one of those, tell us and we'll redo that commit in that shape.

Related PRs:

• 2026 11 gg big bug hunt #331 already covers the request-language need: getPrepContext(iterate, languages), with the languages passed from expand.js and validate.js. Nothing here duplicates it. This branch applies cleanly on top of 2026 11 gg big bug hunt #331 and its tests pass there.
• VSAC: read value sets from the database when asked for, instead of holding them all in memory #333 (draft) carries scanAllValueSets() for the usage scan and the paged-search loadPage. They belong with change 4 below, but they are not repeated here.
• Small fixes: requests no longer wait behind batches, one hung reply, regex alternation, and a few speed-ups #332 (draft fixes bundle) contains commit 9ab694a ("a code not in an enumerated include is located once, not once per listed code"). This PR's first commit is that same commit, unchanged, because change 3 builds on it. Whichever PR merges second, a rebase drops the duplicate. We checked that this branch replays cleanly onto main + Small fixes: requests no longer wait behind batches, one hung reply, regex alternation, and a few speed-ups #332.

---

1. Hand a whole compose to the provider

What we need. When a value set selects from one code system, the provider should get the whole compose: includes, excludes, offset and count. It should return one page with an exact total. For example, $expand?url=http://snomed.info/sct?fhir_vs=isa/404684003&offset=50000&count=10 gets a 422 "too many codes to produce" from tx-dev.fhir.org. A provider that pages by itself answers with 10 codes and a total of 129,013 in about a third of a second.

handlesSelecting() and processSelection() are already declared. expand.js sets vsInfo.handleByCS, but processCodes, the method it calls next, is an empty stub, so no provider is ever handed a compose.

What this PR does. It fills in processCodes:

• When handleByCS is set, it calls cs.processSelection(params, includes, excludes, excludeInactive, offset, count, context).
• context is a new optional last argument: { valueSet: { url, version } }, the value set whose compose this is. A provider that refuses can name the value set without reading the request's state. Providers that ignore it behave as before.
• If the provider returns null, the compose takes the usual include/exclude path, unchanged.
• Some composes are never handed over: one with a text filter (the provider isn't told about it), one with an enumerated concept list (its displays and extensions come from the value set), and one with a filter that has no value (the usual path reports the error).

The return contract is in cs-api.js and cs-api.md:

• The provider returns { sets, total, notClosed }, or null to decline.
• It has already applied offset and count, so sets holds just the page.
• total is the size of the whole selection, or -1 if the provider hasn't counted it.

The old comment said the method returns FilterConceptSet[]. We wrapped the sets in an object because a page needs a total that the sets themselves can't carry. Nothing implemented the old shape.

A provider may also refuse as too costly. cs-api.js and cs-api.md say it may throw the standard too-costly Issue, and show the exact form $expand uses when it refuses a value set itself:

throw new Issue('error', 'too-costly', null, 'VALUESET_TOO_COSTLY',
  this.opContext.i18n.translate('VALUESET_TOO_COSTLY', params.httpLanguages, [vurl, '>' + limit]),
  null, 422).withDiagnostics(this.opContext.diagnostics());

Here vurl is url|version from context.valueSet.

Three small fixes were needed to make this path live:

• scanValueSet now records the compose's system and version.
• isSimpleSelect checked the misspelled inc.valueset, so a compose that imports a value set counted as "simple". It now checks inc.valueSet.
• The up-front "too costly" check in checkSource is skipped when the provider will page the compose itself. processCodes applies it, with the same message, once the provider has reported the total. Only the page size counts against the limit: the usual path needs offset + count to fit because it builds every code before the page, but a provider that pages itself does not, so a deep page (offset 200,000, count 10) is answered as it is for filtered selections. A request with no count, or a count over the limit, is refused as before. Without it, an over-limit whole-system include was refused later by includeCode, whose message names the plain url instead of url|version (tx-ecosystem loinc-expand-all checks this).

Tests (tests/tx/expand-process-selection.test.js) use a fake provider that handles selection. They cover:

• a handled compose, paged and unpaged;
• count=0;
• a provider that reports no total;
• a declined compose, which must give the same result as the usual path;
• composes that are never handed over;
• a whole-system selection over the limit, which must be refused with the same message as on the usual path, and a page that fits inside the limit, which must not be;
• the context the provider is told, and a provider that throws the too-costly Issue.

Other options we'd be happy with:

• B. Paged requests only. Call processSelection only when count or offset is given. Unpaged expansions of big sets would still take the slow path.
• C. No core change. The provider keeps its own sets behind the filter API, and expand.js still slices. Totals and deep pages would stay as they are today.
• Return shape. We could keep FilterConceptSet[] and read the total from filterSize(), with the rule that the provider returns only the page. That makes filterSize mean two different things, which is why we didn't choose it.

Possible later widenings. This PR doesn't change these, but they are where a provider still can't take a compose, or can't serve the response:

• Composes that import value sets.
• Set handles for imports. Today a compose that imports value sets goes through the worker's nested expansion, which expands each import into a full list and then combines the lists; the nested limits and too-costly refusals come from there. The worker could stay in charge of resolving imports (versions, pins, supplements, used-valueset) but ask the provider for an opaque set handle for each simple selection, and, when the pieces come from the same provider, ask it to combine them (union, intersect, exclude) and page the result. The rules stay in one place, and the work happens on sets.
• Excludes of another version of the same code system.
• The status-property declaration in the expansion.
• Responses above main's external limit (3,000 by default). A provider that pages could serve them, but includeCode refuses at the limit. The smallest change would be to skip includeCode's limit check when the provider pages (vsInfo.csDoOffset), and cap at a limit the provider reports instead.

2. Look up a list of codes together

What we need. When a value set lists codes (compose.include.concept), a database-backed provider should be able to read them in a few statements instead of one at a time. For example, expanding us-core-documentreference-type (about 3,000 listed LOINC codes) took about 3 statements per code: 9,114 statements and 2.4 s from a cold start. With a hint before the loop, it takes 12 statements.

What this PR does. It adds async prefetchCodes(codes, withDesignations) to CodeSystemProvider. The default does nothing. expand.js calls it once, with the whole list, before the loops that locate each listed code of an include or exclude. validate.js calls it before the loop that compares each listed code with a code given in another form. A provider may read the codes together and answer the locate() calls that follow from that read, for this request only. A test checks that the provider gets the list once, before the first locate.

Other options we'd be happy with:

• B. A batch lookup. locateMany(codes) would return the contexts, and the workers would use them directly. It is cleaner, but it changes the loops in expand.js and validate.js.
• C. Only through change 1. A provider that takes whole composes can read listed codes inside its own processSelection. That covers simple expansions, but not validate-code or composes that use several systems.

3. Ask a value set about one code

What we need. $validate-code against a value set that lists thousands of codes should ask "is this code listed?" without reading the whole list. Today validate.js walks include.concept as an array, so a provider that stores lists as rows must build that array on every request. For a code that is not in the list:

value set	listed codes	main	asking the stored list
VSAC Medical Cohort	57,017	32 ms	1.1 ms
VSAC Medication Brand Name	33,341	215 ms	0.9 ms

What this PR does. An include may carry a stored-list handle, { count, lookup(code) }, under a well-known symbol (tx/library/stored-concepts.js). Where it has one:

• validate.js asks the handle for the entries with that code, in determineSystem and checkConceptSet.
• The renderer's include summary in the operation log shows the count instead of listing every code.
• The whole list is read only when the code might be listed in another form (ICD-11's short code and entity id). That case is asked once per include, by 9ab694a's hasOtherForm.

A plain array behaves exactly as before. The include's concept property must still give the whole list when read, so code that needs every entry keeps working.

Tests use a stored list and compare its answers with a plain array:

• a listed code;
• a code that is not listed;
• an inferred system;
• a code listed in another form.

They also check that the list is not read when it doesn't need to be.

Overlap with #332. The "does this code have another form?" question is asked once per include (9ab694a, also in #332). This PR builds on that same logic, as described at the top.

Other options we'd be happy with:

• B. A provider method. containsCode(valueSet, includeIndex, system, code) on the value-set provider. It is more explicit, but the worker has to know which provider a value set came from.
• C. No change. Build the array on each request, as VSAC: read value sets from the database when asked for, instead of holding them all in memory #333 does. That costs 10–50 ms on the largest lists.

4. Ask value-set providers instead of reading their maps

What we need. Core code that reads a provider's in-memory valueSetMap should ask the provider instead, because a database-backed provider has no such map. applyTHOPrecedence in provider.js does this twice: it finds THO's value sets with instanceof PackageValueSetProvider and a walk over valueSetMap, and it finds the core package the same way.

What this PR does.

• Two new methods on the value-set provider API.

  • thoVersions(map) adds the url → versions of a THO package's value sets.
  • yieldToTHO(map) keeps only the pinned references for a core package.

  Both default to doing nothing. PackageValueSetProvider implements them with the old loop and the old core-package check, so the result is the same, and provider.js just asks every provider.

• Shared THO check. isTHOPackage moves to tho-precedence.js so the two places that need it share it.

• Startup log. tx/library.js now uses the existing vsCount() instead of valueSetMap.size. The "VS" column now shows the number of value sets. It used to show the number of map keys (id, url and url|version for each value set), so the figures are lower.

Tests check that thoVersions gives exactly what the old loop gave, and that yieldToTHO affects core packages only. They also check that a provider with no map can take part.

The concept-map side of applyTHOPrecedence has the same pattern. We left it alone because no current need touches it, but the same two methods would fit there.

The other two reach-ins are the usage scan (scanAllValueSets()) and the ValueSet? search page (loadPage). Both are in #333. On 40,267 VSAC value sets, the scan drops from 10 s to 1.9 s, and a page of search results from 680 ms (+200 MB) to under 125 ms.

Other option: a shared base class that owns the maps. We don't recommend it, because it would force every provider back into memory.

5. Let a stored CodeSystem choose its provider

What we need. A package CodeSystem kept in a database should get a provider that reads it there. Main's version, THO-precedence and supplement rules must still apply. FhirCodeSystemProvider.create picks a subclass from a registry keyed by url (providerClassFor, which holds ICF today). That can't pick a provider by where the content is kept.

What this PR does.

• FhirCodeSystemProvider.create first asks the CodeSystem object's optional providerFactory(opContext, supplements, cls). Here cls is the class the url registry would use, so a factory can leave registered subclasses such as ICF alone.

• A non-null result is used. null, or no method, leaves the choice to the registry as before.

• No CodeSystem in main has the method.

• providerClassFor is now exported from cs-cs.js. Code that stores package CodeSystems when the server loads has to know which ones need a specialised class (ICF today), so it can leave those to the registry.

Tests cover the case without a factory (including ICF), a factory that supplies the provider, and a factory that returns null for ICF. A test also checks what providerClassFor returns.

Other options we'd be happy with:

• B. Extend the registry. providerClassFor would also take the CodeSystem object, so an entry can match on its kind as well as its url.
• C. An instanceof check in provider.js for the stored CodeSystem class. It is the smallest change, but it ties core to our class.

---

Testing

• Jest, the suites this PR touches or adds (22 suites, run one at a time), on this branch and on 2026 11 gg big bug hunt #331 + this branch: all pass.
• HL7 tx-ecosystem test cases (tests/tx/test-cases.test.js, validator 7.0.3). The current test package was refreshed on 2026-11-02, and on it main fails 66 of 4,565 tests. This branch fails exactly the same 66, so it adds no failures. On the package before the refresh, this branch (before its last three commits) passed 4,565 / 4,565, the same as main.
• On 2026 11 gg big bug hunt #331 + this branch, before the refresh the result was the same as on 2026 11 gg big bug hunt #331 alone in the same environment: 4,520 / 4,565, with the same 45 tests failing in both runs. The commits added since apply cleanly there.

🤖 Generated with Claude Code