fix: fail closed and bound semantic search execution

.github/dependabot.yml

@@ -2,18 +2,21 @@ version: 2
 updates:
     - package-ecosystem: npm
       directory: /
+      target-branch: dev
       schedule:
           interval: weekly
       open-pull-requests-limit: 5
 
     - package-ecosystem: docker
       directory: /
+      target-branch: dev
       schedule:
           interval: weekly
       open-pull-requests-limit: 5
 
     - package-ecosystem: github-actions
       directory: /
+      target-branch: dev
       schedule:
           interval: weekly
       open-pull-requests-limit: 5

README.md

@@ -132,11 +132,13 @@ npm start
 
 The Search API will be served at `localhost:3000`.
 
+Successful port binding emits a sanitized `runtime.lifecycle` event with the `started` outcome. Synchronous configuration failures and asynchronous listener failures such as an unavailable or unauthorized port emit `startup_failed`, set a nonzero process exit code, and do not emit `started`. Startup failure logs contain only a stable error code and, for configuration errors, the relevant configuration path; exception messages, stacks, addresses, and port details are omitted.
+
 ## Health Checks
 
 `GET /api/health-check` is a process liveness check. It returns HTTP 200 when the Node.js process is running and deliberately does not contact OpenSearch. Use it where a dependency outage should not restart otherwise healthy API tasks.
 
-`GET /api/ready-check` sends a minimal zero-result search to the configured CVE index. This verifies connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 only when search requests can still be served after the dependency check completes. Configuration errors, connection failures, timeouts, incompatible `cve-core` SearchReader contracts, a missing index, and an in-progress process shutdown return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. The OpenSearch request is limited to five seconds by default and can be changed with `SearchReadinessTimeoutMs`. If shutdown is already active, readiness fails immediately without contacting OpenSearch. If shutdown begins while the dependency check is pending, the completed check is not reported as ready.
+`GET /api/ready-check` sends a minimal zero-result search to the configured CVE index. This verifies connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 only when search requests can still be served after the dependency check completes and OpenSearch explicitly reports that the request did not time out and no shards failed. Configuration errors, connection failures, timeouts, incomplete response metadata, failed shards, incompatible `cve-core` SearchReader contracts, a missing index, and an in-progress process shutdown return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. The OpenSearch request is limited to five seconds by default and can be changed with `SearchReadinessTimeoutMs`. If shutdown is already active, readiness fails immediately without contacting OpenSearch. If shutdown begins while the dependency check is pending, the completed check is not reported as ready.
 
 An AWS load balancer can use `/api/ready-check` to stop routing search traffic to an unready task while the container orchestrator continues using `/api/health-check` for process liveness. Coordinate that probe selection with the infrastructure team before changing deployed health-check behavior.
 
@@ -339,7 +341,7 @@ The validation tiers have separate purposes:
 - The optional OpenSearch contract check verifies that the installed client can communicate with a non-empty externally managed index whose representative searchable fields satisfy the application contract.
 - Load and performance testing remain in the separate automation performance-testing repository.
 
-Dependabot checks npm packages, Docker base-image digests, and GitHub Actions weekly. Both Dockerfiles pin the Node build image and Distroless runtime image to immutable multi-architecture digests; Dependabot updates those references through reviewable pull requests.
+Dependabot checks npm packages, Docker base-image digests, and GitHub Actions weekly. Version-update pull requests target the `dev` integration branch; GitHub security updates continue to follow the repository default-branch behavior. Both Dockerfiles pin the Node build image and Distroless runtime image to immutable multi-architecture digests; Dependabot updates those references through reviewable pull requests.
 
 ### Operational Logging
 
@@ -349,7 +351,9 @@ Access logs omit query strings and request bodies. By default, the API retains 1
 
 The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, shared-operation waiter saturation, queue wait timeout, search timeout, client disconnect, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, cancellation reason, and bounded queue or waiter state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included.
 
-Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, shared waiter counts, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, response bytes, JSON serialization time, process heap usage and per-request heap delta, cache entry and cached CVE ID counts, process-wide event-loop delay, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable.
+Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, shared waiter counts, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, response bytes, JSON serialization time, process heap usage and per-request heap delta, cache entry and cached CVE ID counts, recent process-wide event-loop delay, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable.
+
+Event-loop delay uses one low-overhead process monitor sampled in rolling 10-second windows. Request events report the most recently completed window through `processEventLoopDelayMeanMs`, `processEventLoopDelayP99Ms`, `processEventLoopDelayMaxMs`, `processEventLoopDelayWindowMs`, and `processEventLoopDelaySampleAgeMs`. These values describe process-wide event-loop saturation near the request, not delay caused by that individual request. They are omitted until the first complete window is available, and concurrent requests read the same completed snapshot without resetting one another's measurements.
 
 Heap deltas and event-loop delay are process-level diagnostics rather than isolated per-request measurements because concurrent requests share the same Node.js process. Use them to correlate latency with memory pressure or event-loop stalls across a test window, not to attribute resource use precisely to one request.
 
@@ -369,7 +373,7 @@ Searches that use `cpeName`, `virtualMatchString`, or `virtualMatchString` with
 
 The post-filter cache stores only the filtered CVE ID list and candidate count for that expensive candidate query. It does not cache full CVE records. On a cache hit, the API skips the 10,000-candidate OpenSearch query and semantic post-filtering, slices the cached CVE IDs for the requested page, and fetches that page of full CVE records by ID.
 
-Candidate retrieval asks OpenSearch for exact-version or wildcard-version criteria and leaves range applicability decisions to the semantic post-filter. The full-record page fetch uses an exact CVE ID set without total-hit counting or OpenSearch sorting; the API restores the requested CVE ID order locally. Standard searches retain exact total counting and deterministic sorting.
+Candidate retrieval asks OpenSearch for exact-version or wildcard-version criteria and leaves range applicability decisions to the semantic post-filter. The full-record page fetch uses an exact CVE ID set without total-hit counting or OpenSearch sorting; the API requires one unique full record for every requested CVE ID and restores the requested CVE ID order locally. Missing, duplicate, malformed, or unexpected page records fail closed with a sanitized HTTP 502 response and evict the associated semantic candidate cache entry so the next request must repeat candidate discovery. Standard searches retain exact total counting and deterministic sorting.
 
 The cache is intentionally small and short-lived:
 
@@ -382,7 +386,7 @@ The cache is intentionally small and short-lived:
 - Cold-search queue depth: up to 500 unique cold searches can wait per process behind the concurrency limit
 - Cold-search queue wait: up to 10 seconds before a waiting search returns HTTP 429
 - Shared operation waiters: up to 500 callers can wait on one candidate or page-fetch operation per process
-- Post-filter evaluation chunk: 250 candidates between cancellation checks and event-loop yields
+- Post-filter evaluation chunk: 250 CPE Match Criteria entries between cancellation checks and event-loop yields
 - OpenSearch request timeout: 20 seconds for each operation
 - Overall `/search` deadline: 30 seconds
 - Eviction: oldest entries are removed when the cache exceeds its max size
@@ -391,7 +395,7 @@ The admission queue applies only to cold CPE candidate searches. Standard `/sear
 
 When a cold search waits in the queue for more than 10 seconds, it fails closed with HTTP 429 and a `SEARCH_QUEUE_TIMEOUT` response. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed 20 seconds. Standard searches use the shorter of that timeout and the request's remaining overall deadline. Shared candidate and page-fetch operations retain the independent OpenSearch timeout so one caller's remaining time does not shorten work still needed by other callers. Every caller retains its own 30-second overall deadline.
 
-Semantic filtering runs in bounded chunks and yields to the Node.js event loop between chunks. This lets disconnect and deadline cancellation signals interrupt a large candidate evaluation instead of waiting for the full set to finish synchronously.
+Semantic filtering validates and evaluates each direct CPE Match Criteria entry in one pass. Once a candidate has satisfied every requested post-filter, later entries are still validated before that candidate can be returned or cached, but redundant match comparisons are skipped. Work is bounded by CPE Match Criteria entry count rather than CVE record count, and filtering yields to the Node.js event loop between chunks. This lets disconnect and deadline cancellation signals interrupt a single record containing many entries instead of waiting for the full record to finish synchronously.
 
 Healthy dev timing logs showed the impact for repeated identical `virtualMatchString` range searches:
 

config/default.jsonc

@@ -29,7 +29,7 @@
             "postFilterColdSearchMaxWaitMs": "10000",
             // Limits callers sharing one candidate or page-fetch operation per Node.js process.
             "inFlightMaxWaitersPerOperation": "500",
-            // Allows cancellation checks between batches of semantic CPE candidate evaluations.
+            // Allows cancellation checks between batches of semantic CPE Match Criteria entry evaluations.
             "postFilterEvaluationChunkSize": "250",
             // Limits each request from the API process to OpenSearch.
             "openSearchRequestTimeoutMs": "20000",

controllers/searchController.js

@@ -8,6 +8,7 @@ const searchInFlightRegistry = require('../utils/searchInFlightRegistry');
 const searchPostFilterWorkQueue = require('../utils/searchPostFilterWorkQueue');
 const searchProvider = require('../utils/searchProvider');
 const searchQueryBuilder = require('../utils/searchQueryBuilder');
+const { OPEN_SEARCH_RESPONSE_FAILURE_TYPES, OpenSearchResponseError } = require('../utils/searchResponseValidator');
 const constants = require('../utils/constants');
 const error = new errors.errors();
 const openSearchUnavailableErrorNames = new Set(['ConnectionError', 'NoLivingConnectionsError']);
@@ -135,10 +136,24 @@ function getHitCveIds(hits) {
     return hits.map(getHitCveId).filter(Boolean);
 }
 
-function orderHitsByCveIds(hits, cveIds) {
-    const hitsByCveId = new Map(hits.map((hit) => [getHitCveId(hit), hit]));
+function validateAndOrderPageHits(hits, cveIds) {
+    const requestedCveIds = new Set(cveIds);
+    const hitsByCveId = new Map();
 
-    return cveIds.map((cveId) => hitsByCveId.get(cveId)).filter(Boolean);
+    if (requestedCveIds.size !== cveIds.length || hits.length !== cveIds.length) {
+        throw new OpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.INCOMPLETE_PAGE_FETCH);
+    }
+
+    for (const hit of hits) {
+        const hitCveId = hit?._source?.cveMetadata?.cveId;
+        if (typeof hitCveId !== 'string' || !requestedCveIds.has(hitCveId) || hitsByCveId.has(hitCveId)) {
+            throw new OpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.INCOMPLETE_PAGE_FETCH);
+        }
+
+        hitsByCveId.set(hitCveId, hit);
+    }
+
+    return cveIds.map((cveId) => hitsByCveId.get(cveId));
 }
 
 function combineTook(...values) {
@@ -193,7 +208,7 @@ async function fetchFullHitsByCveIds(cveIds, metrics, signal) {
             metrics.pageFetchElapsedMs = searchPerformance.elapsedMs(pageFetchStart);
             metrics.pageFetchTookMs = pageData?.took;
 
-            return orderHitsByCveIds(pageData?.hits?.hits ?? [], cveIds);
+            return validateAndOrderPageHits(pageData.hits.hits, cveIds);
         },
         {
             signal,
@@ -303,7 +318,15 @@ async function executeOptimizedPostFilterSearch(
     metrics.filteredCount = candidateResult.filteredCount ?? candidateResult.filteredCveIds.length;
     metrics.returnedCount = pageCveIds.length;
 
-    const fullPageHits = await fetchFullHitsByCveIds(pageCveIds, metrics, signal);
+    let fullPageHits;
+    try {
+        fullPageHits = await fetchFullHitsByCveIds(pageCveIds, metrics, signal);
+    } catch (err) {
+        if (err?.failureType === OPEN_SEARCH_RESPONSE_FAILURE_TYPES.INCOMPLETE_PAGE_FETCH) {
+            searchPostFilterCache.remove(cacheKey);
+        }
+        throw err;
+    }
     const response = {
         status: 'ok',
         data: {

controllers/searchController.test.js

@@ -176,6 +176,17 @@ function cpeHit(id, cpeMatch) {
     };
 }
 
+function fullCveHit(id) {
+    return {
+        _id: id,
+        _source: {
+            cveMetadata: {
+                cveId: id
+            }
+        }
+    };
+}
+
 describe('Search controller SEARCH handler', () => {
     let res, next;
 
@@ -920,6 +931,108 @@ describe('Search controller SEARCH handler', () => {
         });
     });
 
+    test.each([
+        ['missing', ['CVE-2024-0001', 'CVE-2024-0002'], [fullCveHit('CVE-2024-0001')]],
+        ['duplicate', ['CVE-2024-0001', 'CVE-2024-0002'], [fullCveHit('CVE-2024-0001'), fullCveHit('CVE-2024-0001')]],
+        ['malformed', ['CVE-2024-0001'], [{ _id: 'CVE-2024-0001', _source: { cveMetadata: {} } }]],
+        ['unexpected', ['CVE-2024-0001'], [fullCveHit('CVE-2024-9999')]]
+    ])('SEARCH fails closed for a %s full-record page response', async (_caseName, candidateIds, pageHits) => {
+        const candidates = candidateIds.map((id) =>
+            cpeHit(id, {
+                criteria: 'cpe:2.3:o:linux:linux_kernel:2.5:*:*:*:*:*:*:*'
+            })
+        );
+        mockSearchRaw.mockResolvedValue(rawResponse(candidates.length, candidates));
+        mockApiSearch.mockResolvedValue(advancedResponse(pageHits.length, pageHits));
+
+        await controller.SEARCH(
+            {
+                requestId: `${_caseName}-page-fetch-request`,
+                searchParams: {
+                    virtualMatchString: 'cpe:2.3:o:linux:linux_kernel',
+                    resultsPerPage: candidateIds.length,
+                    pageNumber: 1
+                }
+            },
+            res,
+            next
+        );
+
+        expect(res.status).toHaveBeenCalledWith(502);
+        expect(res.json).toHaveBeenCalledWith({
+            error: 'UPSTREAM_SEARCH_ERROR',
+            message: 'Search backend returned an error. Please try again later.'
+        });
+        expect(searchPostFilterCache.getState()).toStrictEqual({
+            entryCount: 0,
+            cachedCveIdCount: 0
+        });
+        expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({
+            event: 'search.performance',
+            requestId: `${_caseName}-page-fetch-request`,
+            outcome: 'upstream_invalid_response',
+            statusCode: 502,
+            upstreamFailureType: 'incomplete_page_fetch'
+        });
+        expect(next).not.toHaveBeenCalled();
+    });
+
+    test.each([
+        ['cpeName', { cpeName: 'not-a-cpe-name' }],
+        ['virtualMatchString', { virtualMatchString: 'not-a-cpe-match-string' }],
+        ['virtualMatchString range', { versionStart: '1.0', versionStartType: 'including' }]
+    ])(
+        'SEARCH does not contact OpenSearch when invalid %s input bypasses middleware',
+        async (_caseName, searchParams) => {
+            await controller.SEARCH(
+                {
+                    searchParams
+                },
+                res,
+                next
+            );
+
+            expect(mockSearchOpenSearch).not.toHaveBeenCalled();
+            expect(res.status).not.toHaveBeenCalled();
+            expect(res.json).not.toHaveBeenCalled();
+            expect(next).toHaveBeenCalledWith(expect.any(TypeError));
+        }
+    );
+
+    test('SEARCH re-runs candidate discovery after an incomplete page fetch evicts the cache entry', async () => {
+        const candidate = cpeHit('CVE-2024-0001', {
+            criteria: 'cpe:2.3:o:linux:linux_kernel:2.5:*:*:*:*:*:*:*'
+        });
+        const fullHit = fullCveHit('CVE-2024-0001');
+        const req = {
+            searchParams: {
+                virtualMatchString: 'cpe:2.3:o:linux:linux_kernel',
+                resultsPerPage: 1,
+                pageNumber: 1
+            }
+        };
+        mockSearchRaw.mockResolvedValue(rawResponse(1, [candidate]));
+        mockApiSearch
+            .mockResolvedValueOnce(advancedResponse(0, []))
+            .mockResolvedValueOnce(advancedResponse(1, [fullHit]));
+
+        await controller.SEARCH(req, res, next);
+        const secondRes = createResponse();
+        await controller.SEARCH(req, secondRes, next);
+
+        expect(mockSearchRaw).toHaveBeenCalledTimes(2);
+        expect(mockApiSearch).toHaveBeenCalledTimes(2);
+        expect(res.status).toHaveBeenCalledWith(502);
+        expect(secondRes.status).toHaveBeenCalledWith(200);
+        expect(secondRes.json).toHaveBeenCalledWith(
+            expect.objectContaining({
+                page: 1,
+                pageSize: 1,
+                status: 'ok'
+            })
+        );
+    });
+
     test('SEARCH coalesces concurrent identical cold semantic post-filter searches', async () => {
         const candidateSearch = deferred();
         const applicable = cpeHit('CVE-2024-0001', {