diff --git a/.env.example b/.env.example index 0b81c9b..a90f4b1 100644 --- a/.env.example +++ b/.env.example @@ -10,6 +10,7 @@ SearchPostFilterColdSearchMaxQueueDepth=500 SearchPostFilterColdSearchMaxWaitMs=10000 SearchOpenSearchRequestTimeoutMs=20000 SearchRequestDeadlineMs=30000 +SearchReadinessTimeoutMs=5000 RequestLogSuccessSampleRate=0.1 RequestLogHealthChecks=false WebSearchEnabled=true diff --git a/README.md b/README.md index bb6300a..21080c4 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ Rename `.env.example` to `.env` and replace or check the following values. - `SearchPostFilterColdSearchMaxWaitMs` - optional maximum cold CPE candidate queue wait, default `10000` - `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000` - `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000` +- `SearchReadinessTimeoutMs` - optional timeout for the OpenSearch index readiness check, default `5000` - `RequestLogSuccessSampleRate` - optional fraction of successful non-health-check access logs to retain, default `0.1` - `RequestLogHealthChecks` - optional flag to retain successful health-check access logs, default `false` - `WebSearchEnabled` - optional deprecated `/webSearch` feature flag, default `true` @@ -84,9 +85,32 @@ npm start The Search API will be served at `localhost:3000`. +## Health Checks + +`GET /api/health-check` is a process liveness check. It returns HTTP 200 when the Node.js process is running and deliberately does not contact OpenSearch. Use it where a dependency outage should not restart otherwise healthy API tasks. + +`GET /api/ready-check` sends a minimal zero-result search to the configured CVE index. This verifies connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 when search requests can be served. Configuration errors, connection failures, timeouts, incompatible `cve-core` SearchReader contracts, and a missing index return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. The OpenSearch request is limited to five seconds by default and can be changed with `SearchReadinessTimeoutMs`. + +An AWS load balancer can use `/api/ready-check` to stop routing search traffic to an unready task while the container orchestrator continues using `/api/health-check` for process liveness. Coordinate that probe selection with the infrastructure team before changing deployed health-check behavior. + ## API Input Validation -`POST /search` validates supported JSON request body fields, boolean parameters, pagination values, CVSS severity values, required parameter combinations, CPE Name requirements, and date-time formats. +`POST /search` accepts an optional JSON object request body and validates supported fields, boolean parameters, pagination values, CVSS severity values, required parameter combinations, CPE Name requirements, and date-time formats. A non-empty body must use the `application/json` media type with UTF-8 encoding; unsupported media types or encodings return HTTP 415, and malformed JSON returns HTTP 400. If OpenSearch is unavailable, the endpoint returns HTTP 503 with a `Retry-After` header. + +String inputs are bounded before query construction: + +| Field | Limit | +| ---------------------------- | ---------------------------------------------------------------------: | +| `keywordSearch` | 512 characters and 64 whitespace-delimited terms | +| `cveIds` | 4,096 characters overall, 100 IDs, and 64 characters per ID | +| `cweId` | 32 characters | +| `sourceIdentifier` | 512 characters | +| `cpeName` | 2,048 characters | +| `virtualMatchString` | 2,048 characters | +| `versionStart`, `versionEnd` | 256 characters each | +| CVSS metric fields | 256 characters, 32 slash-delimited tokens, and 32 characters per token | + +Requests that exceed a limit return HTTP 400 before contacting OpenSearch. These bounds limit input size and query complexity without changing NVD-style CPE matching semantics. Use `cveIds` to request up to 100 CVE records by comma-separated CVE ID. @@ -128,6 +152,8 @@ curl -X POST "http://localhost:3000/search" \ The valid query can match stored CPE Match Criteria that use either a concrete version or a wildcard version plus range fields such as `versionStartIncluding`, `versionStartExcluding`, `versionEndIncluding`, or `versionEndExcluding`. Use `virtualMatchString` instead of `cpeName` when searching for partial CPE Match String values such as all versions of a product. +Set `isVulnerable` to `true` with `cpeName` to require the matching CPE Match Criteria entry itself to be marked vulnerable. A vulnerable flag on an unrelated CPE entry does not satisfy the request. Omitting `isVulnerable`, or setting it to `false`, does not apply a vulnerability restriction. + ### Virtual Match String Search Use `virtualMatchString` when a user needs NVD-style CPE Match Criteria compatibility matching instead of concrete asset matching. Omitted trailing components and `*` components act as broad matches, matching NVD's broad CPE Match String behavior. @@ -150,7 +176,7 @@ curl -X POST "http://localhost:3000/search" \ ### Virtual Match String Version Ranges -Use `versionStart`, `versionStartType`, `versionEnd`, and `versionEndType` with `virtualMatchString` to search CPE Match String ranges. Each provided version bound must include its matching type, and the `virtualMatchString` must be a partial CPE 2.3 match string without a version component. +Use `versionStart`, `versionStartType`, `versionEnd`, and `versionEndType` with `virtualMatchString` to search CPE Match String ranges. Each provided version bound must include its matching type, and the `virtualMatchString` must be a partial CPE 2.3 match string without a version component. When both bounds are provided, `versionStart` must be less than `versionEnd`; equal values are valid only when both bounds are `including`. ```bash curl -X POST "http://localhost:3000/search" \ @@ -214,11 +240,13 @@ npm run quality Every request receives an `X-Request-ID` response header. A caller-provided `X-Request-ID` is preserved when it contains only letters, numbers, `.`, `_`, `:`, or `-` and is no longer than 128 characters; otherwise the API generates a UUID. The same value appears as `request-id` in access logs and `requestId` in structured `/search` events. -Access logs omit query strings and request bodies. By default, the API retains 10% of successful non-health-check access logs, suppresses successful `/api/health-check` access logs, and retains every HTTP 4xx or 5xx access log. Set `RequestLogSuccessSampleRate` from `0` through `1` to change successful-request sampling. Set `RequestLogHealthChecks=true` to retain successful health-check logs. +Access logs omit query strings and request bodies. By default, the API retains 10% of successful non-health-check access logs, suppresses successful `/api/health-check` and `/api/ready-check` access logs, and retains every HTTP 4xx or 5xx access log. Set `RequestLogSuccessSampleRate` from `0` through `1` to change successful-request sampling. Set `RequestLogHealthChecks=true` to retain successful health-check logs. + +The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, queue wait timeout, search timeout, client disconnect, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, cancellation reason, and queue state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included. -The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, queue wait timeout, search timeout, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, and queue state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included. +Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable. -Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable. +Each `/search` request owns a lifecycle cancellation signal. When its overall deadline expires, the API aborts the active OpenSearch transport request before returning HTTP 504. When the caller disconnects, the API aborts work that is needed only by that caller and does not attempt to write a response. This prevents timed-out or abandoned standard searches from continuing until the transport timeout. ### Application Rate Limiting @@ -240,7 +268,7 @@ The cache is intentionally small and short-lived: - Max entries: 100 - Scope: one Node.js process - Stored value: filtered CVE IDs and candidate count -- In-flight behavior: concurrent identical cold searches share one candidate query per process +- In-flight behavior: concurrent identical cold searches share one reference-counted candidate query per process - Cold-search concurrency: cold CPE candidate queries are limited to 10 per process before reaching OpenSearch - Cold-search queue depth: up to 500 unique cold searches can wait per process behind the concurrency limit - Cold-search queue wait: up to 10 seconds before a waiting search returns HTTP 429 @@ -250,7 +278,7 @@ The cache is intentionally small and short-lived: When the cold-search queue is full, new unique CPE post-filter searches fail closed with HTTP 429 and a `SEARCH_QUEUE_FULL` response. Identical requests for a search that is already in flight still wait on the shared in-flight result and do not consume additional queue slots. -When a cold search waits in the queue for more than 10 seconds, it fails closed with HTTP 429 and a `SEARCH_QUEUE_TIMEOUT` response. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed 20 seconds. The same response is returned when a `/search` request exceeds the 30-second overall processing deadline. +When a cold search waits in the queue for more than 10 seconds, it fails closed with HTTP 429 and a `SEARCH_QUEUE_TIMEOUT` response. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed 20 seconds. The same response is returned when a `/search` request exceeds the 30-second overall processing deadline, and the active OpenSearch request is aborted. Healthy dev timing logs showed the impact for repeated identical `virtualMatchString` range searches: @@ -265,9 +293,9 @@ Healthy dev timing logs showed the impact for repeated identical `virtualMatchSt | Avg candidates scanned | 10000 | cached from prior result | | Avg filtered CVEs | 4 | 4 | -This helps most when multiple users submit the same expensive CPE search within the TTL. The first request is a cold cache miss; later identical requests on the same running task are warm cache hits. If identical cold requests arrive concurrently before the first one finishes, the later requests wait on the same candidate query instead of starting duplicate 10,000-candidate OpenSearch requests. Each request still fetches its requested page of full CVE records after the shared candidate result is available. +This helps most when multiple users submit the same expensive CPE search within the TTL. The first request is a cold cache miss; later identical requests on the same running task are warm cache hits. If identical cold requests arrive concurrently before the first one finishes, the later requests wait on the same candidate query instead of starting duplicate 10,000-candidate OpenSearch requests. Disconnecting one waiter does not cancel work still needed by another waiter; the shared operation is aborted when its final waiter disconnects or times out. Each active request still fetches its requested page of full CVE records after the shared candidate result is available. -The cache does not share entries across ECS tasks. With multiple tasks, each task warms independently depending on load balancer routing. Steady repeated traffic can warm every active task, but unique one-off searches still pay the cold-cache cost. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, OpenSearch timeout, and overall request deadline can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchOpenSearchRequestTimeoutMs`, and `SearchRequestDeadlineMs`. Future scale improvements could add a shared cache such as Redis/ElastiCache. +The cache does not share entries across ECS tasks. With multiple tasks, each task warms independently depending on load balancer routing. Steady repeated traffic can warm every active task, but unique one-off searches still pay the cold-cache cost. The request lifecycle and in-flight waiter registry are separate from this result cache, so the cache can later be replaced with Redis or ElastiCache without changing cancellation behavior. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, OpenSearch timeout, and overall request deadline can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchOpenSearchRequestTimeoutMs`, and `SearchRequestDeadlineMs`. Docker runtime smoke checks can be run after building the image. The default `Dockerfile` avoids BuildKit `RUN --mount` syntax for AWS builders that do not support it: diff --git a/api-docs/openapi.json b/api-docs/openapi.json index eea8e4d..a860ab1 100644 --- a/api-docs/openapi.json +++ b/api-docs/openapi.json @@ -36,11 +36,14 @@ } }, "400": { - "description": "Bad Request" + "description": "Malformed JSON, invalid request body, parameter combination, or version range" }, "404": { "description": "Invalid cpeName value" }, + "415": { + "description": "A non-empty request body did not use the application/json media type with UTF-8 encoding" + }, "429": { "description": "Application rate limit exceeded, or the cold CPE search queue is full or exceeded its maximum wait", "headers": { @@ -64,6 +67,17 @@ } } }, + "503": { + "description": "OpenSearch is unavailable", + "headers": { + "Retry-After": { + "description": "Seconds before retrying an unavailable OpenSearch dependency", + "schema": { + "type": "integer" + } + } + } + }, "504": { "description": "Search exceeded its OpenSearch or overall processing deadline" } @@ -143,8 +157,8 @@ }, "/api/health-check": { "get": { - "summary": "Health check for AWS monitoring", - "description": "Health check for AWS monitoring", + "summary": "Process liveness check", + "description": "Returns 200 when the API process is running. This endpoint does not check OpenSearch.", "operationId": "healthCheck", "responses": { "200": { @@ -161,6 +175,43 @@ } } }, + "/api/ready-check": { + "get": { + "summary": "OpenSearch readiness check", + "description": "Sends a minimal zero-result search to the configured CVE index. Returns 200 when the index is available with search permission, or a sanitized 503 response when configuration, connectivity, timeout, permission, or index availability checks fail.", + "operationId": "readinessCheck", + "responses": { + "200": { + "description": "API is ready to serve search requests", + "headers": { + "X-Request-ID": { + "description": "Request correlation ID", + "schema": { + "type": "string" + } + } + } + }, + "503": { + "description": "Search service is not ready", + "headers": { + "Retry-After": { + "description": "Seconds before retrying the readiness check", + "schema": { + "type": "integer" + } + }, + "X-Request-ID": { + "description": "Request correlation ID", + "schema": { + "type": "string" + } + } + } + } + } + } + }, "/webSearch": { "post": { "tags": [ @@ -253,8 +304,9 @@ "properties": { "keywordSearch": { "type": "string", + "maxLength": 512, "example": "remote code execution", - "description": "Returns CVE records where a word or phrase is included in the record's description. Multiple keywords operate like an AND statement, returning records whose descriptions include every keyword." + "description": "Returns CVE records where a word or phrase is included in the record's description. Multiple keywords operate like an AND statement, returning records whose descriptions include every keyword. Limited to 512 characters and 64 whitespace-delimited terms." }, "keywordExactMatch": { "type": "boolean", @@ -263,8 +315,9 @@ }, "cveIds": { "type": "string", + "maxLength": 4096, "example": "CVE-2024-1234,CVE-2024-5678", - "description": "Returns one or more CVE records identified by a comma-separated list of CVE IDs. Limited to 100 CVE IDs." + "description": "Returns one or more CVE records identified by a comma-separated list of CVE IDs. Limited to 100 CVE IDs, 64 characters per ID, and 4096 characters overall." }, "noRejected": { "type": "boolean", @@ -273,6 +326,7 @@ }, "cweId": { "type": "string", + "maxLength": 32, "example": "CWE-79", "description": "Returns CVE records that include the provided Common Weakness Enumeration ID." }, @@ -357,18 +411,20 @@ }, "sourceIdentifier": { "type": "string", + "maxLength": 512, "example": "security@example.org", "description": "Returns CVE records that contain the provided source identifier." }, "cpeName": { "type": "string", + "maxLength": 2048, "example": "cpe:2.3:a:freetype:freetype:2.8.1:*:*:*:*:*:*:*", "description": "Returns CVE records whose CPE Match Criteria apply to the provided CPE 2.3 asset name. The part, vendor, product, and version components must contain values other than *. The concrete version can match stored criteria with the same version or criteria with a wildcard version and version range fields. Invalid cpeName values return HTTP 404. Use virtualMatchString for partial CPE Match String searches." }, "isVulnerable": { "type": "boolean", "default": false, - "description": "When true, returns CVE records whose CPE Match Criteria match cpeName and are marked vulnerable. cpeName is required when isVulnerable is true. virtualMatchString cannot be provided with isVulnerable." + "description": "When true, returns CVE records where the same CPE Match Criteria entry both matches cpeName and is marked vulnerable. False or omitted does not apply a vulnerability restriction. cpeName is required when isVulnerable is true. virtualMatchString cannot be provided with isVulnerable." }, "lastModStartDate": { "type": "string", @@ -411,8 +467,9 @@ }, "versionStart": { "type": "string", + "maxLength": 256, "example": "2.2", - "description": "Sets the starting version bound for a CPE Match String range search. versionStartType and virtualMatchString are required when versionStart is provided. virtualMatchString must not include a version component." + "description": "Sets the starting version bound for a CPE Match String range search. versionStartType and virtualMatchString are required when versionStart is provided. When both bounds are provided, versionStart must be less than versionEnd; equal values require both bounds to be including. virtualMatchString must not include a version component." }, "versionStartType": { "type": "string", @@ -424,8 +481,9 @@ }, "versionEnd": { "type": "string", + "maxLength": 256, "example": "2.6", - "description": "Sets the ending version bound for a CPE Match String range search. versionEndType and virtualMatchString are required when versionEnd is provided. virtualMatchString must not include a version component." + "description": "Sets the ending version bound for a CPE Match String range search. versionEndType and virtualMatchString are required when versionEnd is provided. When both bounds are provided, versionStart must be less than versionEnd; equal values require both bounds to be including. virtualMatchString must not include a version component." }, "versionEndType": { "type": "string", @@ -437,6 +495,7 @@ }, "virtualMatchString": { "type": "string", + "maxLength": 2048, "example": "cpe:2.3:o:microsoft:windows_10_1809", "description": "Returns CVE records whose CPE Match Criteria are compatible with the provided CPE 2.3 match string. Omitted trailing components and * components act as broad matches. When used with versionStart or versionEnd, provide a partial CPE 2.3 match string without a version component." } diff --git a/api-docs/openapi.test.js b/api-docs/openapi.test.js index 1083eb7..3b205f1 100644 --- a/api-docs/openapi.test.js +++ b/api-docs/openapi.test.js @@ -20,6 +20,7 @@ describe('OpenAPI contract', () => { test('documents the supported route surface', () => { expect(openapi.paths['/search'].post.operationId).toBe('search'); expect(openapi.paths['/api/health-check'].get.operationId).toBe('healthCheck'); + expect(openapi.paths['/api/ready-check'].get.operationId).toBe('readinessCheck'); expect(openapi.paths['/webSearch'].post.operationId).toBe('webSearch'); }); @@ -60,13 +61,23 @@ describe('OpenAPI contract', () => { expect(schema.properties.cveIds.type).toBe('string'); expect(schema.properties.cveIds.description).toContain('comma-separated list of CVE IDs'); expect(schema.properties.cveIds.description).toContain(`${constants.MAX_CVE_IDS} CVE IDs`); + expect(schema.properties.cveIds.description).toContain(`${constants.MAX_CVE_ID_LENGTH} characters per ID`); + expect(schema.properties.keywordSearch.description).toContain( + `${constants.MAX_KEYWORD_SEARCH_TOKENS} whitespace-delimited terms` + ); + Object.entries(constants.SEARCH_STRING_LENGTH_LIMITS).forEach(([field, maxLength]) => { + expect(schema.properties[field].maxLength).toBe(maxLength); + }); expect(schema.properties.noRejected.type).toBe('boolean'); expect(schema.properties.noRejected.description).toContain('When true'); + expect(schema.properties.isVulnerable.description).toContain('same CPE Match Criteria entry'); + expect(schema.properties.isVulnerable.description).toContain('False or omitted'); expect(schema.properties.versionStartType.enum).toStrictEqual(constants.CPE_VERSION_RANGE_TYPES); expect(schema.properties.versionEndType.enum).toStrictEqual(constants.CPE_VERSION_RANGE_TYPES); expect(schema.properties.resultsPerPage.maximum).toBe(constants.MAX_RESULTS_LIMIT); expect(schema.properties.cveId).toBeUndefined(); expect(schema.properties.versionStart.description).toContain('versionStartType'); + expect(schema.properties.versionStart.description).toContain('less than versionEnd'); expect(schema.properties.versionEnd.description).toContain('versionEndType'); expect(schema.properties.virtualMatchString.description).toContain('CPE Match Criteria'); expect(schema.properties.virtualMatchString.description).toContain('compatible'); @@ -111,6 +122,20 @@ describe('OpenAPI contract', () => { expect(openapi.paths['/search'].post.responses['429'].description).toContain('rate limit'); }); + test('documents fail-closed request and dependency responses for POST /search', () => { + const responses = openapi.paths['/search'].post.responses; + + expect(responses['400'].description).toContain('Malformed JSON'); + expect(responses['415'].description).toContain('application/json'); + expect(responses['503'].description).toContain('OpenSearch'); + expect(responses['503'].headers['Retry-After']).toStrictEqual({ + description: 'Seconds before retrying an unavailable OpenSearch dependency', + schema: { + type: 'integer' + } + }); + }); + test('documents request correlation headers', () => { expect(openapi.paths['/search'].post.responses['200'].headers['X-Request-ID']).toStrictEqual({ description: 'Request correlation ID', @@ -119,6 +144,8 @@ describe('OpenAPI contract', () => { } }); expect(openapi.paths['/api/health-check'].get.responses['200'].headers['X-Request-ID']).toBeDefined(); + expect(openapi.paths['/api/ready-check'].get.responses['200'].headers['X-Request-ID']).toBeDefined(); + expect(openapi.paths['/api/ready-check'].get.responses['503'].headers['Retry-After']).toBeDefined(); expect(openapi.paths['/webSearch'].post.responses['200'].headers['X-Request-ID']).toBeDefined(); }); diff --git a/config/custom-environment-variables.jsonc b/config/custom-environment-variables.jsonc index a194540..3470a97 100644 --- a/config/custom-environment-variables.jsonc +++ b/config/custom-environment-variables.jsonc @@ -18,7 +18,8 @@ "postFilterColdSearchMaxQueueDepth": "SearchPostFilterColdSearchMaxQueueDepth", "postFilterColdSearchMaxWaitMs": "SearchPostFilterColdSearchMaxWaitMs", "openSearchRequestTimeoutMs": "SearchOpenSearchRequestTimeoutMs", - "requestDeadlineMs": "SearchRequestDeadlineMs" + "requestDeadlineMs": "SearchRequestDeadlineMs", + "readinessTimeoutMs": "SearchReadinessTimeoutMs" }, "requestLogging": { "successSampleRate": "RequestLogSuccessSampleRate", diff --git a/config/default.jsonc b/config/default.jsonc index 124a926..8cdef78 100644 --- a/config/default.jsonc +++ b/config/default.jsonc @@ -28,7 +28,9 @@ // Limits each request from the API process to OpenSearch. "openSearchRequestTimeoutMs": "20000", // Limits total processing time for a /search request. - "requestDeadlineMs": "30000" + "requestDeadlineMs": "30000", + // Limits the minimal OpenSearch request used by /api/ready-check. + "readinessTimeoutMs": "5000" }, // Access logging settings. Error responses are always logged. "requestLogging": { diff --git a/config/runtime-config.test.js b/config/runtime-config.test.js index 67946b3..350a6b1 100644 --- a/config/runtime-config.test.js +++ b/config/runtime-config.test.js @@ -102,6 +102,7 @@ describe('runtime container and compose configuration', () => { '"openSearchRequestTimeoutMs": "SearchOpenSearchRequestTimeoutMs"' ); expect(customEnvironmentVariables).toContain('"requestDeadlineMs": "SearchRequestDeadlineMs"'); + expect(customEnvironmentVariables).toContain('"readinessTimeoutMs": "SearchReadinessTimeoutMs"'); expect(customEnvironmentVariables).toContain('"successSampleRate": "RequestLogSuccessSampleRate"'); expect(customEnvironmentVariables).toContain('"healthChecks": "RequestLogHealthChecks"'); expect(customEnvironmentVariables).toContain('"webSearchEnabled": "WebSearchEnabled"'); diff --git a/controllers/middleware.js b/controllers/middleware.js index 446634f..351e7af 100644 --- a/controllers/middleware.js +++ b/controllers/middleware.js @@ -90,6 +90,10 @@ function hasInvalidCvssMetricsShape(value) { ); } +function countWhitespaceDelimitedTokens(value) { + return value.trim().split(/\s+/).length; +} + function parseCveIds(value) { if (typeof value !== 'string') return null; @@ -142,11 +146,29 @@ function validateParams(req, res, next) { } } + for (const [param, maxLength] of Object.entries(constants.SEARCH_STRING_LENGTH_LIMITS)) { + if (typeof params[param] === 'string' && params[param].length > maxLength) { + return res.status(400).json(error.searchFieldLengthError(param, maxLength)); + } + } + + if ( + typeof params.keywordSearch === 'string' && + countWhitespaceDelimitedTokens(params.keywordSearch) > constants.MAX_KEYWORD_SEARCH_TOKENS + ) { + return res + .status(400) + .json(error.searchFieldTokenLimitError('keywordSearch', constants.MAX_KEYWORD_SEARCH_TOKENS)); + } + if (hasParam(params, 'cveIds')) { const cveIds = parseCveIds(params.cveIds); if (!cveIds) { return res.status(400).json(error.cveIdsFormatError()); } + if (cveIds.some((cveId) => cveId.length > constants.MAX_CVE_ID_LENGTH)) { + return res.status(400).json(error.cveIdLengthError()); + } if (cveIds.length > constants.MAX_CVE_IDS) { return res.status(400).json(error.cveIdsLimitError()); } @@ -193,6 +215,18 @@ function validateParams(req, res, next) { if (virtualMatchString.hasVersionComponent) { return res.status(400).json(error.virtualMatchStringVersionComponentError()); } + + if (hasParam(params, 'versionStart') && hasParam(params, 'versionEnd')) { + const comparison = cpe.compareCpeVersions(params.versionStart, params.versionEnd); + const hasEmptyEqualRange = + comparison === 0 && + (params.versionStartType.toLowerCase() !== 'including' || + params.versionEndType.toLowerCase() !== 'including'); + + if (comparison > 0 || hasEmptyEqualRange) { + return res.status(400).json(error.versionRangeOrderError()); + } + } } for (const { param, allowed, buildError } of severityValidators) { diff --git a/controllers/middleware.test.js b/controllers/middleware.test.js index c4df38f..b036f2f 100644 --- a/controllers/middleware.test.js +++ b/controllers/middleware.test.js @@ -1,4 +1,5 @@ const mw = require('./middleware.js'); +const constants = require('../utils/constants'); describe('middleware validation function', () => { let req, res, next, status, json; @@ -30,6 +31,10 @@ describe('middleware validation function', () => { expect(next.mock.calls[0]).toHaveLength(0); } + function padBetween(prefix, suffix, length, character = 'a') { + return `${prefix}${character.repeat(length - prefix.length - suffix.length)}${suffix}`; + } + test('ValidateParams returns an error when keywordExactMatch is true without keywordSearch', () => { validate({ keywordExactMatch: true @@ -172,6 +177,55 @@ describe('middleware validation function', () => { expectValidationSuccess(); }); + test('ValidateParams returns an error when versionStart exceeds versionEnd', () => { + validate({ + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + versionStart: '3.0', + versionStartType: 'including', + versionEnd: '2.0', + versionEndType: 'including' + }); + + expectValidationError( + 'VERSION_RANGE_ORDER', + 'The versionStart parameter must be less than versionEnd. Equal values require both bounds to be including.' + ); + }); + + test.each([ + ['excluding', 'including'], + ['including', 'excluding'], + ['excluding', 'excluding'] + ])( + 'ValidateParams returns an error for an empty equal version range with %s/%s bounds', + (versionStartType, versionEndType) => { + validate({ + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + versionStart: '2.0', + versionStartType, + versionEnd: '2.0', + versionEndType + }); + + expectValidationError( + 'VERSION_RANGE_ORDER', + 'The versionStart parameter must be less than versionEnd. Equal values require both bounds to be including.' + ); + } + ); + + test('ValidateParams accepts an equal inclusive version range', () => { + validate({ + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + versionStart: '2.0', + versionStartType: 'including', + versionEnd: '2.0', + versionEndType: 'including' + }); + + expectValidationSuccess(); + }); + test('ValidateParams returns an error when version range params are used without virtualMatchString', () => { validate({ versionStart: '2.2', @@ -364,6 +418,108 @@ describe('middleware validation function', () => { expectValidationError('CVE_IDS_LIMIT', 'The cveIds parameter cannot contain more than 100 CVE IDs.'); }); + test.each(Object.entries(constants.SEARCH_STRING_LENGTH_LIMITS))( + 'ValidateParams returns an error when %s exceeds its maximum length.', + (param, maxLength) => { + validate({ + [param]: 'a'.repeat(maxLength + 1) + }); + + expectValidationError( + 'SEARCH_FIELD_LENGTH', + `The ${param} field must be ${maxLength} characters or fewer.` + ); + } + ); + + test.each([ + ['keywordSearch', 'a'.repeat(constants.SEARCH_STRING_LENGTH_LIMITS.keywordSearch)], + ['cweId', 'a'.repeat(constants.SEARCH_STRING_LENGTH_LIMITS.cweId)], + ['sourceIdentifier', 'a'.repeat(constants.SEARCH_STRING_LENGTH_LIMITS.sourceIdentifier)] + ])('ValidateParams accepts %s at its maximum length.', (param, value) => { + validate({ + [param]: value + }); + + expectValidationSuccess(); + }); + + test('ValidateParams accepts cveIds and each CVE ID at their maximum lengths.', () => { + const idPrefix = 'CVE-2024-'; + const longId = `${idPrefix}${'1'.repeat(constants.MAX_CVE_ID_LENGTH - idPrefix.length)}`; + const shortId = `${idPrefix}${'2'.repeat(32 - idPrefix.length)}`; + const mediumId = `${idPrefix}${'3'.repeat(33 - idPrefix.length)}`; + const cveIds = [...Array(62).fill(longId), shortId, mediumId].join(','); + + expect(cveIds).toHaveLength(constants.SEARCH_STRING_LENGTH_LIMITS.cveIds); + validate({ cveIds }); + + expectValidationSuccess(); + }); + + test('ValidateParams returns an error when an individual CVE ID exceeds its maximum length.', () => { + const idPrefix = 'CVE-2024-'; + + validate({ + cveIds: `${idPrefix}${'1'.repeat(constants.MAX_CVE_ID_LENGTH - idPrefix.length + 1)}` + }); + + expectValidationError( + 'CVE_ID_LENGTH', + `Each CVE ID in the cveIds field must be ${constants.MAX_CVE_ID_LENGTH} characters or fewer.` + ); + }); + + test.each([ + [ + 'cpeName', + padBetween('cpe:2.3:a:', ':product:1.0:*:*:*:*:*:*:*', constants.SEARCH_STRING_LENGTH_LIMITS.cpeName) + ], + [ + 'virtualMatchString', + padBetween('cpe:2.3:a:', ':product', constants.SEARCH_STRING_LENGTH_LIMITS.virtualMatchString) + ] + ])('ValidateParams accepts %s at its maximum length.', (param, value) => { + validate({ + [param]: value + }); + + expectValidationSuccess(); + }); + + test.each(['versionStart', 'versionEnd'])('ValidateParams accepts %s at its maximum length.', (versionParam) => { + const typeParam = `${versionParam}Type`; + + validate({ + virtualMatchString: 'cpe:2.3:a:vendor:product', + [versionParam]: '1'.repeat(constants.SEARCH_STRING_LENGTH_LIMITS[versionParam]), + [typeParam]: 'including' + }); + + expectValidationSuccess(); + }); + + test('ValidateParams accepts keywordSearch at the maximum term count.', () => { + validate({ + keywordSearch: Array(constants.MAX_KEYWORD_SEARCH_TOKENS).fill('a').join(' ') + }); + + expectValidationSuccess(); + }); + + test('ValidateParams returns an error when keywordSearch exceeds the maximum term count.', () => { + validate({ + keywordSearch: Array(constants.MAX_KEYWORD_SEARCH_TOKENS + 1) + .fill('a') + .join(' ') + }); + + expectValidationError( + 'SEARCH_FIELD_TOKEN_LIMIT', + `The keywordSearch field must contain ${constants.MAX_KEYWORD_SEARCH_TOKENS} or fewer terms.` + ); + }); + test.each([ ['true string', 'true'], ['false string', 'false'], diff --git a/controllers/searchController.js b/controllers/searchController.js index bf2eb1a..5deabdb 100644 --- a/controllers/searchController.js +++ b/controllers/searchController.js @@ -4,10 +4,13 @@ const cpe = require('../utils/cpe'); const searchPerformance = require('../utils/searchPerformance'); const searchDeadline = require('../utils/searchDeadline'); const searchPostFilterCache = require('../utils/searchPostFilterCache'); +const searchInFlightRegistry = require('../utils/searchInFlightRegistry'); const searchPostFilterWorkQueue = require('../utils/searchPostFilterWorkQueue'); const searchProvider = require('../utils/searchProvider'); const searchQueryBuilder = require('../utils/searchQueryBuilder'); const error = new errors.errors(); +const openSearchUnavailableErrorNames = new Set(['ConnectionError', 'NoLivingConnectionsError']); +const openSearchUnavailableStatusCodes = new Set([502, 503, 504]); function hasSearchPostFilters(postFilters) { return Boolean(postFilters?.cpeName || postFilters?.virtualMatchString || postFilters?.virtualMatchStringRange); @@ -33,7 +36,12 @@ function filterSearchHits(resp, postFilters) { hitsContainer, candidateCount: hitsContainer.hits.length, filteredHits: hitsContainer.hits.filter((hit) => { - if (postFilters?.cpeName && !cpe.hitAppliesToCpeName(hit, postFilters.cpeName, cpeEvaluationContext)) { + if ( + postFilters?.cpeName && + !cpe.hitAppliesToCpeName(hit, postFilters.cpeName, cpeEvaluationContext, { + requireVulnerable: postFilters.isVulnerable === true + }) + ) { return false; } @@ -91,6 +99,18 @@ function buildPostFilteredResponse( return buildPagedSearchResponse(resp, filteredHits.length, pagedHits); } +function getOpenSearchErrorStatusCode(err) { + return Number(err?.statusCode ?? err?.meta?.statusCode ?? err?.meta?.body?.status); +} + +function isOpenSearchUnavailableError(err) { + return ( + searchProvider.isSearchProviderContractError?.(err) === true || + openSearchUnavailableErrorNames.has(err?.name) || + openSearchUnavailableStatusCodes.has(getOpenSearchErrorStatusCode(err)) + ); +} + function buildWebSearchSort(sort) { if (!sort) return undefined; @@ -154,20 +174,32 @@ function combineTook(...values) { return numericValues.reduce((sum, value) => sum + value, 0); } -async function executeStandardSearch( - searchManager, - searchOptions, - queryStringObjects, - postFilters, - responseOptions, - metrics -) { +function recordOpenSearchStart(metrics, operation, state) { + metrics.openSearchOperations ??= []; + metrics.openSearchOperations.push(operation); + metrics.maxOpenSearchActiveRequests = Math.max(metrics.maxOpenSearchActiveRequests ?? 0, state.activeCount); +} + +async function executeOpenSearch(body, operation, signal, metrics) { + return searchProvider.searchOpenSearch(body, { + signal, + operation, + onStart: (state) => recordOpenSearchStart(metrics, operation, state) + }); +} + +async function executeStandardSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, signal) { metrics.strategy = 'standard'; const searchStart = searchPerformance.markTime(); - const searchResp = await searchManager.apiSearch(null, searchOptions, queryStringObjects); + const body = searchQueryBuilder.buildOpenSearchBody(searchOptions, queryStringObjects); + const searchData = await executeOpenSearch(body, 'standard', signal, metrics); + const searchResp = { + status: 'ok', + data: searchData + }; metrics.openSearchElapsedMs = searchPerformance.elapsedMs(searchStart); - metrics.openSearchTookMs = searchResp?.data?.took; + metrics.openSearchTookMs = searchData?.took; const postFilterStart = searchPerformance.markTime(); const resp = applySearchPostFilters(searchResp, postFilters, responseOptions, metrics); @@ -176,39 +208,38 @@ async function executeStandardSearch( return resp; } -async function fetchFullHitsByCveIds(searchManager, cveIds, metrics) { +async function fetchFullHitsByCveIds(cveIds, metrics, signal) { if (!cveIds.length) return []; - const cacheKey = searchPostFilterCache.buildInFlightKey('pageFetch', cveIds); - const inFlightResult = searchPostFilterCache.getInFlight(cacheKey); - if (inFlightResult) { - metrics.pageFetchInFlightHit = true; + const inFlightKey = searchInFlightRegistry.buildKey('pageFetch', cveIds); + const waitStart = searchPerformance.markTime(); + const inFlight = searchInFlightRegistry.acquire( + inFlightKey, + async (operationSignal) => { + const { searchOptions, queryStringObjects } = searchQueryBuilder.buildCveIdsSearchRequest(cveIds); + const body = searchQueryBuilder.buildOpenSearchBody(searchOptions, queryStringObjects); + const pageFetchStart = searchPerformance.markTime(); + const pageData = await executeOpenSearch(body, 'page-fetch', operationSignal, metrics); + metrics.pageFetchElapsedMs = searchPerformance.elapsedMs(pageFetchStart); + metrics.pageFetchTookMs = pageData?.took; + + return orderHitsByCveIds(pageData?.hits?.hits ?? [], cveIds); + }, + { signal } + ); - const waitStart = searchPerformance.markTime(); - const result = await inFlightResult; + metrics.pageFetchInFlightHit = inFlight.shared; + const result = await inFlight.promise; + if (inFlight.shared) { metrics.pageFetchWaitElapsedMs = searchPerformance.elapsedMs(waitStart); - - return result; } - metrics.pageFetchInFlightHit = false; - - const { searchOptions, queryStringObjects } = searchQueryBuilder.buildCveIdsSearchRequest(cveIds); - const promise = (async () => { - const pageFetchStart = searchPerformance.markTime(); - const pageResp = await searchManager.apiSearch(null, searchOptions, queryStringObjects); - metrics.pageFetchElapsedMs = searchPerformance.elapsedMs(pageFetchStart); - metrics.pageFetchTookMs = pageResp?.data?.took; - - return orderHitsByCveIds(pageResp?.data?.hits?.hits ?? [], cveIds); - })(); - - return searchPostFilterCache.setInFlight(cacheKey, promise); + return result; } -async function buildPostFilterCandidateResult(candidateBody, postFilters, metrics) { +async function buildPostFilterCandidateResult(candidateBody, postFilters, metrics, signal) { const candidateStart = searchPerformance.markTime(); - const candidateData = await searchProvider.searchRaw(candidateBody); + const candidateData = await executeOpenSearch(candidateBody, 'cpe-candidate', signal, metrics); metrics.openSearchElapsedMs = searchPerformance.elapsedMs(candidateStart); metrics.openSearchTookMs = candidateData?.took; @@ -226,7 +257,7 @@ async function buildPostFilterCandidateResult(candidateBody, postFilters, metric }; } -async function getPostFilterCandidateResult(cacheKey, candidateBody, postFilters, metrics) { +async function getPostFilterCandidateResult(cacheKey, candidateBody, postFilters, metrics, signal) { const cachedResult = searchPostFilterCache.get(cacheKey); if (cachedResult) { metrics.cacheHit = true; @@ -234,55 +265,58 @@ async function getPostFilterCandidateResult(cacheKey, candidateBody, postFilters return cachedResult; } - const inFlightResult = searchPostFilterCache.getInFlight(cacheKey); - if (inFlightResult) { - metrics.cacheHit = false; - metrics.inFlightHit = true; + metrics.cacheHit = false; + const inFlightKey = searchInFlightRegistry.buildKey('candidate', cacheKey); + const queueStart = searchPerformance.markTime(); + const waitStart = searchPerformance.markTime(); + const inFlight = searchInFlightRegistry.acquire( + inFlightKey, + (operationSignal) => + searchPostFilterWorkQueue + .run( + () => { + metrics.coldSearchQueueWaitElapsedMs = searchPerformance.elapsedMs(queueStart); + metrics.coldSearchQueueState = searchPostFilterWorkQueue.getState(); + return buildPostFilterCandidateResult(candidateBody, postFilters, metrics, operationSignal); + }, + { signal: operationSignal } + ) + .then((result) => { + searchPostFilterCache.set(cacheKey, result); + return result; + }), + { signal } + ); - const waitStart = searchPerformance.markTime(); - const result = await inFlightResult; + metrics.inFlightHit = inFlight.shared; + const result = await inFlight.promise; + if (inFlight.shared) { metrics.inFlightWaitElapsedMs = searchPerformance.elapsedMs(waitStart); - - return result; } - metrics.cacheHit = false; - metrics.inFlightHit = false; - - const queueStart = searchPerformance.markTime(); - const promise = searchPostFilterWorkQueue - .run(() => { - metrics.coldSearchQueueWaitElapsedMs = searchPerformance.elapsedMs(queueStart); - metrics.coldSearchQueueState = searchPostFilterWorkQueue.getState(); - return buildPostFilterCandidateResult(candidateBody, postFilters, metrics); - }) - .then((result) => { - searchPostFilterCache.set(cacheKey, result); - return result; - }); - - return searchPostFilterCache.setInFlight(cacheKey, promise); + return result; } async function executeOptimizedPostFilterSearch( - searchManager, searchOptions, queryStringObjects, postFilters, responseOptions, - metrics + metrics, + signal ) { metrics.strategy = 'cpe-candidate'; const candidateBody = searchQueryBuilder.buildCpeCandidateSearchBody(searchOptions, queryStringObjects); const cacheKey = searchPostFilterCache.buildCacheKey(candidateBody, postFilters); - const candidateResult = await getPostFilterCandidateResult(cacheKey, candidateBody, postFilters, metrics); + const candidateResult = await getPostFilterCandidateResult(cacheKey, candidateBody, postFilters, metrics, signal); + searchDeadline.throwIfAborted(signal); const pageCveIds = paginateHits(candidateResult.filteredCveIds, responseOptions); metrics.candidateCount = candidateResult.candidateCount; metrics.filteredCount = candidateResult.filteredCount ?? candidateResult.filteredCveIds.length; metrics.returnedCount = pageCveIds.length; - const fullPageHits = await fetchFullHitsByCveIds(searchManager, pageCveIds, metrics); + const fullPageHits = await fetchFullHitsByCveIds(pageCveIds, metrics, signal); const response = { data: { took: combineTook(metrics.openSearchTookMs, metrics.pageFetchTookMs), @@ -298,25 +332,18 @@ async function executeOptimizedPostFilterSearch( return buildPagedSearchResponse(response, metrics.filteredCount, fullPageHits); } -async function executeSearch(searchManager, searchOptions, queryStringObjects, postFilters, responseOptions, metrics) { +async function executeSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, signal) { if (!hasSearchPostFilters(postFilters)) { - return executeStandardSearch( - searchManager, - searchOptions, - queryStringObjects, - postFilters, - responseOptions, - metrics - ); + return executeStandardSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, signal); } return executeOptimizedPostFilterSearch( - searchManager, searchOptions, queryStringObjects, postFilters, responseOptions, - metrics + metrics, + signal ); } @@ -326,37 +353,40 @@ async function searchHandler(req, res, next) { requestId: req.requestId }; const handlerStart = searchPerformance.markTime(); + const lifecycle = searchDeadline.createSearchRequestLifecycle( + req, + res, + searchProvider.getSearchRequestDeadlineMs() + ); let searchOptions; let postFilters; let responseOptions; try { - const searchManager = searchProvider.getAdvancedSearchManager(); const searchRequest = searchQueryBuilder.buildSearchRequest(params); searchOptions = searchRequest.searchOptions; postFilters = searchRequest.postFilters; responseOptions = searchRequest.responseOptions; - const resp = await searchDeadline.withSearchDeadline( - executeSearch( - searchManager, - searchOptions, - searchRequest.queryStringObjects, - postFilters, - responseOptions, - metrics - ), - searchProvider.getSearchRequestDeadlineMs() + const resp = await executeSearch( + searchOptions, + searchRequest.queryStringObjects, + postFilters, + responseOptions, + metrics, + lifecycle.signal ); + lifecycle.throwIfAborted(); const pagingOptions = responseOptions ?? searchOptions; metrics.outcome = 'success'; metrics.statusCode = 200; //Calculate current page and total pages to append to JSON output const totalValue = resp.data.hits.total.value; - let pageSize = Math.min(totalValue, pagingOptions.resultsPerPage); - const page = pageSize === 0 ? 0 : Math.floor(pagingOptions.from / pageSize) + 1; - const totalPages = pageSize === 0 ? 0 : Math.ceil(totalValue / pageSize); + const requestedPageSize = pagingOptions.resultsPerPage; + const pageSize = Math.min(totalValue, requestedPageSize); + const page = totalValue === 0 ? 0 : Math.floor(pagingOptions.from / requestedPageSize) + 1; + const totalPages = totalValue === 0 ? 0 : Math.ceil(totalValue / requestedPageSize); return res.status(200).json({ page, @@ -365,6 +395,13 @@ async function searchHandler(req, res, next) { ...resp }); } catch (err) { + if (searchDeadline.isSearchClientDisconnectedError(err)) { + metrics.outcome = 'client_disconnected'; + metrics.statusCode = 499; + metrics.abortReason = 'client_disconnect'; + return; + } + if (searchPostFilterWorkQueue.isQueueFullError(err)) { metrics.outcome = 'queue_full'; metrics.statusCode = 429; @@ -388,19 +425,23 @@ async function searchHandler(req, res, next) { metrics.outcome = 'search_timeout'; metrics.statusCode = 504; metrics.timeoutSource = err?.name === 'TimeoutError' ? 'opensearch' : 'request_deadline'; + metrics.abortReason = metrics.timeoutSource; return res.status(504).json(error.searchTimeoutError()); } - if (err?.name === 'ConnectionError') { - metrics.outcome = 'connection_error'; - metrics.statusCode = 400; - return res.status(400).json(error.openSearchError()); + if (isOpenSearchUnavailableError(err)) { + metrics.outcome = 'opensearch_unavailable'; + metrics.statusCode = 503; + if (typeof res.set === 'function') res.set('Retry-After', '5'); + + return res.status(503).json(error.openSearchError()); } metrics.outcome = 'upstream_error'; metrics.statusCode = 502; return res.status(502).json(error.upstreamSearchError()); } finally { + lifecycle.complete(); metrics.totalElapsedMs = searchPerformance.elapsedMs(handlerStart); searchPerformance.logSearchPerformance(metrics, searchOptions, responseOptions, postFilters); } diff --git a/controllers/searchController.test.js b/controllers/searchController.test.js index 64f8648..1e5fe87 100644 --- a/controllers/searchController.test.js +++ b/controllers/searchController.test.js @@ -1,6 +1,20 @@ +const { EventEmitter } = require('events'); + const mockApiSearch = jest.fn(); const mockBasicSearch = jest.fn(); const mockSearchRaw = jest.fn(); +const mockSearchOpenSearch = jest.fn((body, options) => { + const operation = options?.operation; + const search = operation === 'cpe-candidate' ? mockSearchRaw : mockApiSearch; + options?.onStart?.({ + activeCount: 1, + activeByOperation: { + [operation]: 1 + } + }); + + return Promise.resolve(search(body, options)).then((result) => result?.data ?? result); +}); const mockShouldLogSearchPerformance = jest.fn(); const mockGetPostFilterCacheTtlMs = jest.fn(() => 60000); const mockGetPostFilterCacheMaxEntries = jest.fn(() => 100); @@ -8,9 +22,7 @@ const mockGetPostFilterColdSearchConcurrency = jest.fn(() => 10); const mockGetPostFilterColdSearchMaxQueueDepth = jest.fn(() => 500); const mockGetPostFilterColdSearchMaxWaitMs = jest.fn(() => 10000); const mockGetSearchRequestDeadlineMs = jest.fn(() => 30000); -const mockGetAdvancedSearchManager = jest.fn(() => ({ - apiSearch: mockApiSearch -})); +const mockIsSearchProviderContractError = jest.fn((err) => err?.code === 'SEARCH_PROVIDER_CONTRACT'); const mockGetBasicSearchManager = jest.fn(() => ({ search: mockBasicSearch })); @@ -24,7 +36,6 @@ jest.mock('cve-core', () => { }); jest.mock('../utils/searchProvider', () => ({ - getAdvancedSearchManager: mockGetAdvancedSearchManager, getBasicSearchManager: mockGetBasicSearchManager, getPostFilterCacheMaxEntries: mockGetPostFilterCacheMaxEntries, getPostFilterCacheTtlMs: mockGetPostFilterCacheTtlMs, @@ -32,11 +43,13 @@ jest.mock('../utils/searchProvider', () => ({ getPostFilterColdSearchMaxQueueDepth: mockGetPostFilterColdSearchMaxQueueDepth, getPostFilterColdSearchMaxWaitMs: mockGetPostFilterColdSearchMaxWaitMs, getSearchRequestDeadlineMs: mockGetSearchRequestDeadlineMs, - searchRaw: mockSearchRaw, + isSearchProviderContractError: mockIsSearchProviderContractError, + searchOpenSearch: mockSearchOpenSearch, shouldLogSearchPerformance: mockShouldLogSearchPerformance })); const controller = require('./searchController.js'); +const searchInFlightRegistry = require('../utils/searchInFlightRegistry'); const searchPostFilterCache = require('../utils/searchPostFilterCache'); function createResponse() { @@ -49,6 +62,7 @@ function createResponse() { function advancedResponse(totalValue, hits = []) { return { + status: 'ok', data: { hits: { total: { @@ -134,9 +148,9 @@ describe('Search controller SEARCH handler', () => { mockApiSearch.mockReset(); mockBasicSearch.mockReset(); mockSearchRaw.mockReset(); + mockSearchOpenSearch.mockClear(); mockShouldLogSearchPerformance.mockReset(); mockShouldLogSearchPerformance.mockReturnValue(false); - mockGetAdvancedSearchManager.mockClear(); mockGetBasicSearchManager.mockClear(); mockGetPostFilterCacheTtlMs.mockClear(); mockGetPostFilterCacheMaxEntries.mockClear(); @@ -144,6 +158,7 @@ describe('Search controller SEARCH handler', () => { mockGetPostFilterColdSearchMaxQueueDepth.mockClear(); mockGetPostFilterColdSearchMaxWaitMs.mockClear(); mockGetSearchRequestDeadlineMs.mockClear(); + mockIsSearchProviderContractError.mockClear(); mockGetPostFilterCacheTtlMs.mockReturnValue(60000); mockGetPostFilterCacheMaxEntries.mockReturnValue(100); mockGetPostFilterColdSearchConcurrency.mockReturnValue(10); @@ -151,6 +166,7 @@ describe('Search controller SEARCH handler', () => { mockGetPostFilterColdSearchMaxWaitMs.mockReturnValue(10000); mockGetSearchRequestDeadlineMs.mockReturnValue(30000); searchPostFilterCache.clear(); + searchInFlightRegistry.clear(); jest.spyOn(console, 'log').mockImplementation(() => {}); res = createResponse(); @@ -158,6 +174,7 @@ describe('Search controller SEARCH handler', () => { }); afterEach(() => { + searchInFlightRegistry.clear(); jest.restoreAllMocks(); }); @@ -182,48 +199,53 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockGetAdvancedSearchManager).toHaveBeenCalledTimes(1); - expect(mockApiSearch).toHaveBeenCalledWith( - null, { - filters: [ - { - terms: { - 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] - } - } - ], - rangeObjects: [ - { - range: { - 'cveMetadata.datePublished': { - gte: '2024-01-01T00:00:00.000Z', - lte: '2024-01-31T00:00:00.000Z' + query: { + bool: { + must: [ + { + simple_query_string: { + query: 'remote* code*', + default_operator: 'AND', + fields: ['containers.cna.descriptions.value'] + } } - } - }, - { - range: { - 'cveMetadata.dateUpdated': { - gte: '2024-02-01T00:00:00.000Z', - lte: '2024-02-29T00:00:00.000Z' + ], + filter: [ + { + terms: { + 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] + } + }, + { + range: { + 'cveMetadata.datePublished': { + gte: '2024-01-01T00:00:00.000Z', + lte: '2024-01-31T00:00:00.000Z' + } + } + }, + { + range: { + 'cveMetadata.dateUpdated': { + gte: '2024-02-01T00:00:00.000Z', + lte: '2024-02-29T00:00:00.000Z' + } + } } - } + ] } - ], - resultsPerPage: 10, + }, + track_total_hits: true, + size: 10, from: 20 }, - [ - { - simple_query_string: { - query: 'remote* code*', - default_operator: 'AND', - fields: ['containers.cna.descriptions.value'] - } - } - ] + expect.objectContaining({ + operation: 'standard', + signal: expect.any(AbortSignal), + onStart: expect.any(Function) + }) ); expect(res.status).toHaveBeenCalledWith(200); @@ -249,19 +271,16 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][1]).toStrictEqual({ - filters: [ + expect(mockApiSearch.mock.calls[0][0].query.bool).toStrictEqual({ + must: [], + filter: [ { match_phrase: { 'containers.cna.descriptions.value': 'remote code' } } - ], - rangeObjects: [], - resultsPerPage: 20, - from: 0 + ] }); - expect(mockApiSearch.mock.calls[0][2]).toStrictEqual([]); }); test('SEARCH builds an exact terms filter for cveIds', async () => { @@ -277,14 +296,14 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][1].filters).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ { terms: { 'cveMetadata.cveId.keyword': ['CVE-2024-0001', 'CVE-2024-0002'] } } ]); - expect(mockApiSearch.mock.calls[0][2]).toStrictEqual([]); + expect(mockApiSearch.mock.calls[0][0].query.bool.must).toStrictEqual([]); }); test('SEARCH builds rejected-state exclusions for noRejected', async () => { @@ -300,7 +319,7 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][1].filters).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ { bool: { must_not: [ @@ -322,7 +341,7 @@ describe('Search controller SEARCH handler', () => { } } ]); - expect(mockApiSearch.mock.calls[0][2]).toStrictEqual([]); + expect(mockApiSearch.mock.calls[0][0].query.bool.must).toStrictEqual([]); }); test('SEARCH processes virtualMatchString into a structured CPE criteria filter', async () => { @@ -341,6 +360,7 @@ describe('Search controller SEARCH handler', () => { expect(mockSearchRaw.mock.calls[0][0]._source).toStrictEqual([ 'cveMetadata.cveId', 'containers.cna.cpeApplicability.nodes.cpeMatch.criteria', + 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionStartIncluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionStartExcluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionEndIncluding', @@ -419,8 +439,8 @@ describe('Search controller SEARCH handler', () => { } }); expect(filters[2]).toStrictEqual({ - match_phrase: { - 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable': 'true' + term: { + 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable': true } }); expect(body.size).toBe(10000); @@ -428,6 +448,34 @@ describe('Search controller SEARCH handler', () => { expect(mockApiSearch).not.toHaveBeenCalled(); }); + test('SEARCH requires the matching CPE entry itself to be vulnerable', async () => { + const matchingNotVulnerable = cpeHit('CVE-2024-0001', { + criteria: 'cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*', + vulnerable: false + }); + matchingNotVulnerable._source.containers.cna.cpeApplicability.nodes[0].cpeMatch.push({ + criteria: 'cpe:2.3:a:other:product:1.0:*:*:*:*:*:*:*', + vulnerable: true + }); + mockSearchRaw.mockResolvedValue(rawResponse(1, [matchingNotVulnerable])); + + await controller.SEARCH( + { + query: { + cpeName: 'cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*', + isVulnerable: true + } + }, + res, + next + ); + + expect(res.status).toHaveBeenCalledWith(200); + expect(res.json.mock.calls[0][0].data.hits.total.value).toBe(0); + expect(res.json.mock.calls[0][0].data.hits.hits).toStrictEqual([]); + expect(mockApiSearch).not.toHaveBeenCalled(); + }); + test('SEARCH applies semantic CPE version filtering before building response metadata', async () => { const applicable = cpeHit('CVE-2024-0001', { criteria: 'cpe:2.3:a:vendor:product:*:*:*:*:*:*:*:*', @@ -468,20 +516,27 @@ describe('Search controller SEARCH handler', () => { ); expect(mockApiSearch).toHaveBeenCalledWith( - null, { - filters: [ - { - terms: { - 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] - } + query: { + bool: { + must: [], + filter: [ + { + terms: { + 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] + } + } + ] } - ], - rangeObjects: [], - resultsPerPage: 1, + }, + track_total_hits: true, + size: 1, from: 0 }, - [] + expect.objectContaining({ + operation: 'page-fetch', + signal: expect.any(AbortSignal) + }) ); expect(res.status).toHaveBeenCalledWith(200); expect(res.json).toHaveBeenCalledWith({ @@ -541,18 +596,24 @@ describe('Search controller SEARCH handler', () => { expect(mockSearchRaw.mock.calls[0][0].size).toBe(10000); expect(mockSearchRaw.mock.calls[0][0].from).toBe(0); - expect(mockApiSearch.mock.calls[0][1]).toStrictEqual({ - filters: [ - { - terms: { - 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] - } + expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ + { + terms: { + 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] } - ], - rangeObjects: [], - resultsPerPage: 1, + } + ]); + expect(mockApiSearch.mock.calls[0][0]).toMatchObject({ + track_total_hits: true, + size: 1, from: 0 }); + expect(mockApiSearch.mock.calls[0][1]).toEqual( + expect.objectContaining({ + operation: 'page-fetch', + signal: expect.any(AbortSignal) + }) + ); expect(res.status).toHaveBeenCalledWith(200); expect(res.json).toHaveBeenCalledWith({ page: 1, @@ -606,16 +667,16 @@ describe('Search controller SEARCH handler', () => { expect(mockSearchRaw).toHaveBeenCalledTimes(1); expect(mockApiSearch).toHaveBeenCalledTimes(2); - expect(mockApiSearch.mock.calls[1][1]).toStrictEqual({ - filters: [ - { - terms: { - 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] - } + expect(mockApiSearch.mock.calls[1][0].query.bool.filter).toStrictEqual([ + { + terms: { + 'cveMetadata.cveId.keyword': ['CVE-2024-0001'] } - ], - rangeObjects: [], - resultsPerPage: 1, + } + ]); + expect(mockApiSearch.mock.calls[1][0]).toMatchObject({ + track_total_hits: true, + size: 1, from: 0 }); expect(secondRes.status).toHaveBeenCalledWith(200); @@ -695,6 +756,56 @@ describe('Search controller SEARCH handler', () => { }); }); + test('SEARCH keeps shared CPE work active when one client disconnects', async () => { + const candidateSearch = deferred(); + const applicable = cpeHit('CVE-2024-0001', { + criteria: 'cpe:2.3:o:linux:linux_kernel:2.5:*:*:*:*:*:*:*' + }); + const fullApplicable = { + _id: 'CVE-2024-0001', + _source: { + cveMetadata: { + cveId: 'CVE-2024-0001' + } + } + }; + const firstReq = new EventEmitter(); + const secondReq = new EventEmitter(); + const query = { + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + resultsPerPage: 1, + pageNumber: 1 + }; + firstReq.query = query; + secondReq.query = query; + const firstRes = createResponse(); + const secondRes = createResponse(); + let operationSignal; + + mockSearchRaw.mockImplementation((_body, { signal }) => { + operationSignal = signal; + return candidateSearch.promise; + }); + mockApiSearch.mockResolvedValue(advancedResponse(1, [fullApplicable])); + + const firstSearch = controller.SEARCH(firstReq, firstRes, next); + await Promise.resolve(); + const secondSearch = controller.SEARCH(secondReq, secondRes, next); + await Promise.resolve(); + + firstReq.emit('aborted'); + await firstSearch; + + expect(operationSignal.aborted).toBe(false); + candidateSearch.resolve(rawResponse(1, [applicable])); + await secondSearch; + + expect(mockSearchRaw).toHaveBeenCalledTimes(1); + expect(mockApiSearch).toHaveBeenCalledTimes(1); + expect(firstRes.status).not.toHaveBeenCalled(); + expect(secondRes.status).toHaveBeenCalledWith(200); + }); + test('SEARCH applies semantic virtualMatchString filtering before building response metadata', async () => { const applicable = cpeHit('CVE-2024-0001', { criteria: 'cpe:2.3:o:microsoft:windows_10_1809:-:*:*:*:*:*:x86:*' @@ -923,6 +1034,8 @@ describe('Search controller SEARCH handler', () => { optimizedFallback: false, cacheHit: false, inFlightHit: false, + openSearchOperations: ['standard'], + maxOpenSearchActiveRequests: 1, postFilters: [], requestedFrom: 0, requestedSize: 1, @@ -945,7 +1058,7 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][1].filters).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ { term: { 'containers.cna.metrics.cvssV3_0.baseSeverity.keyword': 'HIGH' @@ -972,7 +1085,7 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][1].filters).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ { bool: { filter: [ @@ -1017,7 +1130,7 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][1].filters).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ { bool: { filter: [ @@ -1074,7 +1187,7 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][2]).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.must).toStrictEqual([ { simple_query_string: { query: 'c\\+\\+* path\\:\\/tmp* \\(test\\)*', @@ -1098,7 +1211,7 @@ describe('Search controller SEARCH handler', () => { next ); - expect(mockApiSearch.mock.calls[0][2]).toStrictEqual([ + expect(mockApiSearch.mock.calls[0][0].query.bool.must).toStrictEqual([ { simple_query_string: { query: '', @@ -1194,10 +1307,37 @@ describe('Search controller SEARCH handler', () => { }); }); - test('SEARCH returns openSearchError when cve-core throws a ConnectionError', async () => { - mockApiSearch.mockRejectedValue({ - name: 'ConnectionError' + test('SEARCH calculates an out-of-range page from the requested page size', async () => { + const resp = advancedResponse(5); + mockApiSearch.mockResolvedValue(resp); + + await controller.SEARCH( + { + query: { + resultsPerPage: 10, + pageNumber: 2 + } + }, + res, + next + ); + + expect(res.status).toHaveBeenCalledWith(200); + expect(res.json).toHaveBeenCalledWith({ + page: 2, + pageSize: 5, + totalPages: 1, + ...resp }); + }); + + test.each([ + ['ConnectionError', { name: 'ConnectionError' }], + ['NoLivingConnectionsError', { name: 'NoLivingConnectionsError' }], + ['cve-core provider contract error', { code: 'SEARCH_PROVIDER_CONTRACT' }], + ['OpenSearch 503 response', { name: 'ResponseError', meta: { statusCode: 503 } }] + ])('SEARCH returns a retryable 503 when OpenSearch is unavailable through %s', async (_caseName, thrownError) => { + mockApiSearch.mockRejectedValue(thrownError); await controller.SEARCH( { @@ -1208,7 +1348,8 @@ describe('Search controller SEARCH handler', () => { next ); - expect(res.status).toHaveBeenCalledWith(400); + expect(res.status).toHaveBeenCalledWith(503); + expect(res.set).toHaveBeenCalledWith('Retry-After', '5'); expect(res.json).toHaveBeenCalledWith({ error: 'OPEN_SEARCH_ERROR', message: @@ -1217,8 +1358,8 @@ describe('Search controller SEARCH handler', () => { expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({ event: 'search.performance', requestId: 'connection-error-request', - outcome: 'connection_error', - statusCode: 400 + outcome: 'opensearch_unavailable', + statusCode: 503 }); }); @@ -1253,7 +1394,12 @@ describe('Search controller SEARCH handler', () => { test('SEARCH returns a sanitized 504 when the overall request deadline is exceeded', async () => { const timers = captureTimers(); - mockApiSearch.mockReturnValue(new Promise(() => {})); + mockApiSearch.mockImplementation( + (_body, { signal }) => + new Promise((resolve, reject) => { + signal.addEventListener('abort', () => reject(signal.reason), { once: true }); + }) + ); const search = controller.SEARCH( { @@ -1279,10 +1425,63 @@ describe('Search controller SEARCH handler', () => { requestId: 'request-deadline-request', outcome: 'search_timeout', statusCode: 504, - timeoutSource: 'request_deadline' + timeoutSource: 'request_deadline', + abortReason: 'request_deadline' }); }); + test('SEARCH stops without writing a response when the client disconnects', async () => { + const req = new EventEmitter(); + req.requestId = 'client-disconnect-request'; + req.query = {}; + mockApiSearch.mockImplementation( + (_body, { signal }) => + new Promise((resolve, reject) => { + signal.addEventListener('abort', () => reject(signal.reason), { once: true }); + }) + ); + + const search = controller.SEARCH(req, res, next); + await Promise.resolve(); + req.emit('aborted'); + await search; + + expect(res.status).not.toHaveBeenCalled(); + expect(res.json).not.toHaveBeenCalled(); + expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({ + event: 'search.performance', + requestId: 'client-disconnect-request', + outcome: 'client_disconnected', + statusCode: 499, + abortReason: 'client_disconnect', + strategy: 'standard' + }); + }); + + test('SEARCH does not start a page fetch after a client disconnects from a CPE search', async () => { + const req = new EventEmitter(); + req.requestId = 'cpe-client-disconnect-request'; + req.query = { + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + resultsPerPage: 1, + pageNumber: 1 + }; + mockSearchRaw.mockImplementation(() => { + req.emit('aborted'); + return rawResponse(1, [ + cpeHit('CVE-2024-0001', { + criteria: 'cpe:2.3:o:linux:linux_kernel:2.5:*:*:*:*:*:*:*' + }) + ]); + }); + + await controller.SEARCH(req, res, next); + + expect(mockApiSearch).not.toHaveBeenCalled(); + expect(res.status).not.toHaveBeenCalled(); + expect(res.json).not.toHaveBeenCalled(); + }); + test('SEARCH returns sanitized errors from cve-core', async () => { const err = { name: 'SearchError', @@ -1324,11 +1523,12 @@ describe('Search controller WEB_SEARCH handler', () => { mockApiSearch.mockReset(); mockBasicSearch.mockReset(); mockSearchRaw.mockReset(); + mockSearchOpenSearch.mockClear(); mockShouldLogSearchPerformance.mockReset(); mockShouldLogSearchPerformance.mockReturnValue(false); - mockGetAdvancedSearchManager.mockClear(); mockGetBasicSearchManager.mockClear(); searchPostFilterCache.clear(); + searchInFlightRegistry.clear(); jest.spyOn(console, 'log').mockImplementation(() => {}); res = createResponse(); @@ -1336,6 +1536,7 @@ describe('Search controller WEB_SEARCH handler', () => { }); afterEach(() => { + searchInFlightRegistry.clear(); jest.restoreAllMocks(); }); diff --git a/routes/index.js b/routes/index.js index 7604253..7cc0568 100644 --- a/routes/index.js +++ b/routes/index.js @@ -5,6 +5,8 @@ const mw = require('../controllers/middleware'); const controller = require('../controllers/searchController'); const constants = require('../utils/constants'); const featureFlags = require('../utils/featureFlags'); +const searchDeadline = require('../utils/searchDeadline'); +const searchProvider = require('../utils/searchProvider'); const searchRateLimiter = require('../utils/searchRateLimiter'); const error = new errors.errors(); @@ -21,6 +23,30 @@ function healthCheck(req, res) { res.status(200).send(); } +async function readinessCheck(req, res) { + const lifecycle = searchDeadline.createSearchRequestLifecycle( + req, + res, + searchProvider.getOpenSearchReadinessTimeoutMs() + ); + + try { + await searchProvider.checkOpenSearchReadiness({ + signal: lifecycle.signal + }); + lifecycle.throwIfAborted(); + + return res.status(200).send(); + } catch (err) { + if (searchDeadline.isSearchClientDisconnectedError(err)) return; + + if (typeof res.set === 'function') res.set('Retry-After', '5'); + return res.status(503).json(error.searchNotReadyError()); + } finally { + lifecycle.complete(); + } +} + function parseError(req, res, next) { const err = validationResult(req).formatWith(({ location, msg, param, value, nestedErrors }) => { return msg; @@ -95,6 +121,20 @@ function rejectSearchQueryParams(req, res, next) { next(); } +function hasRequestBody(req) { + const contentLength = Number(req.headers['content-length']); + + return req.headers['transfer-encoding'] !== undefined || (Number.isFinite(contentLength) && contentLength > 0); +} + +function requireSearchJsonContentType(req, res, next) { + if (hasRequestBody(req) && !req.is('application/json')) { + return res.status(415).json(error.searchContentTypeError()); + } + + next(); +} + function requireWebSearchEnabled(req, res, next) { if (!featureFlags.isWebSearchEnabled()) { return res.status(410).json(error.webSearchDisabledError()); @@ -206,9 +246,15 @@ headers: { } } } +#swagger.responses[400] = { +description: 'Malformed JSON, invalid request body, parameter combination, or version range', +} #swagger.responses[404] = { description: 'Invalid cpeName value', } +#swagger.responses[415] = { +description: 'A non-empty request body did not use the application/json media type with UTF-8 encoding', +} #swagger.responses[429] = { description: 'Application rate limit exceeded, or the cold CPE search queue is full or exceeded its maximum wait', headers: { @@ -232,10 +278,22 @@ headers: { } } } +#swagger.responses[503] = { +description: 'OpenSearch is unavailable', +headers: { + "Retry-After": { + description: "Seconds before retrying an unavailable OpenSearch dependency", + schema: { + type: "integer" + } + } +} +} #swagger.responses[504] = { description: 'Search exceeded its OpenSearch or overall processing deadline', } */ + requireSearchJsonContentType, limitSearchRequests, rejectSearchQueryParams, body().custom(isSearchBodyObject).withMessage(error.searchBodyError()), @@ -268,13 +326,13 @@ description: 'Search exceeded its OpenSearch or overall processing deadline', controller.SEARCH ); -// Endpoint used for AWS health checks +// Process liveness endpoint router.get( '/api/health-check', /* #swagger.operationId = 'healthCheck' -#swagger.summary = "Health check for AWS monitoring" -#swagger.description = "Health check for AWS monitoring" +#swagger.summary = "Process liveness check" +#swagger.description = "Returns 200 when the API process is running. This endpoint does not check OpenSearch." #swagger.responses[200] = { description: 'OK', headers: { @@ -290,6 +348,45 @@ headers: { healthCheck ); +// OpenSearch dependency readiness endpoint +router.get( + '/api/ready-check', + /* +#swagger.operationId = 'readinessCheck' +#swagger.summary = "OpenSearch readiness check" +#swagger.description = "Sends a minimal zero-result search to the configured CVE index. Returns 200 when the index is available with search permission, or a sanitized 503 response when configuration, connectivity, timeout, permission, or index availability checks fail." +#swagger.responses[200] = { +description: 'API is ready to serve search requests', +headers: { + "X-Request-ID": { + description: "Request correlation ID", + schema: { + type: "string" + } + } +} +} +#swagger.responses[503] = { +description: 'Search service is not ready', +headers: { + "Retry-After": { + description: "Seconds before retrying the readiness check", + schema: { + type: "integer" + } + }, + "X-Request-ID": { + description: "Request correlation ID", + schema: { + type: "string" + } + } +} +} +*/ + readinessCheck +); + router.post( '/webSearch', /* diff --git a/routes/index.test.js b/routes/index.test.js index 482f09e..c1cd095 100644 --- a/routes/index.test.js +++ b/routes/index.test.js @@ -24,11 +24,17 @@ jest.mock('../utils/featureFlags', () => ({ })); const mockLimitSearchRequests = jest.fn((req, res, next) => next()); +const mockCheckOpenSearchReadiness = jest.fn(); jest.mock('../utils/searchRateLimiter', () => ({ createSearchRateLimiter: jest.fn(() => mockLimitSearchRequests) })); +jest.mock('../utils/searchProvider', () => ({ + checkOpenSearchReadiness: mockCheckOpenSearchReadiness, + getOpenSearchReadinessTimeoutMs: jest.fn(() => 5000) +})); + const mw = require('../controllers/middleware'); const controller = require('../controllers/searchController'); const featureFlags = require('../utils/featureFlags'); @@ -42,6 +48,7 @@ describe('routes/index router', () => { jest.clearAllMocks(); featureFlags.isWebSearchEnabled.mockReturnValue(true); mockLimitSearchRequests.mockImplementation((req, res, next) => next()); + mockCheckOpenSearchReadiness.mockResolvedValue(true); }); test('GET /api/health-check returns 200', async () => { @@ -51,6 +58,35 @@ describe('routes/index router', () => { expect(response.status).toBe(200); expect(response.text).toBe(''); + expect(mockCheckOpenSearchReadiness).not.toHaveBeenCalled(); + }); + + test('GET /api/ready-check returns 200 when the configured OpenSearch index exists', async () => { + const response = await request(app, { + path: '/api/ready-check' + }); + + expect(response.status).toBe(200); + expect(response.text).toBe(''); + expect(mockCheckOpenSearchReadiness).toHaveBeenCalledWith({ + signal: expect.any(AbortSignal) + }); + }); + + test('GET /api/ready-check returns a sanitized 503 when OpenSearch is unavailable', async () => { + mockCheckOpenSearchReadiness.mockRejectedValue(new Error('sensitive internal OpenSearch details')); + + const response = await request(app, { + path: '/api/ready-check' + }); + + expect(response.status).toBe(503); + expect(response.headers['retry-after']).toBe('5'); + expect(response.body).toStrictEqual({ + error: 'SEARCH_NOT_READY', + message: 'Search service is not ready. Please try again later.' + }); + expect(response.text).not.toContain('sensitive internal OpenSearch details'); }); test('POST /search reaches middleware and controller for valid JSON body fields', async () => { @@ -76,6 +112,42 @@ describe('routes/index router', () => { }); }); + test('POST /search permits an omitted request body', async () => { + const response = await request(app, { + method: 'POST', + path: '/search' + }); + + expect(response.status).toBe(200); + expect(mockLimitSearchRequests).toHaveBeenCalledTimes(1); + expect(mw.validateParams).toHaveBeenCalledTimes(1); + expect(controller.SEARCH).toHaveBeenCalledTimes(1); + expect(response.body).toStrictEqual({ + route: 'search', + params: {} + }); + }); + + test('POST /search rejects a non-empty body with an unsupported media type before rate limiting', async () => { + const response = await request(app, { + method: 'POST', + path: '/search', + rawBody: '{"keywordSearch":"test"}', + headers: { + 'Content-Type': 'text/plain' + } + }); + + expect(response.status).toBe(415); + expect(response.body).toStrictEqual({ + error: 'SEARCH_CONTENT_TYPE', + message: 'The /search request body must use the application/json media type with UTF-8 encoding.' + }); + expect(mockLimitSearchRequests).not.toHaveBeenCalled(); + expect(mw.validateParams).not.toHaveBeenCalled(); + expect(controller.SEARCH).not.toHaveBeenCalled(); + }); + test('POST /search returns a rate-limit response before validation and controller work', async () => { mockLimitSearchRequests.mockImplementation((req, res) => res.status(429).json({ diff --git a/routes/search.integration.test.js b/routes/search.integration.test.js index 740f9ad..5319044 100644 --- a/routes/search.integration.test.js +++ b/routes/search.integration.test.js @@ -1,16 +1,15 @@ const { request } = require('../test/http'); +const constants = require('../utils/constants'); -const mockApiSearch = jest.fn(); +const mockSearchOpenSearch = jest.fn(); const mockBasicSearch = jest.fn(); jest.mock('../utils/searchProvider', () => ({ - getAdvancedSearchManager: jest.fn(() => ({ - apiSearch: mockApiSearch - })), getBasicSearchManager: jest.fn(() => ({ search: mockBasicSearch })), getSearchRequestDeadlineMs: jest.fn(() => 30000), + searchOpenSearch: mockSearchOpenSearch, shouldLogSearchPerformance: jest.fn(() => false) })); @@ -48,12 +47,12 @@ function basicResponse({ total = 0, hits = [] } = {}) { describe('search route integration', () => { beforeEach(() => { - mockApiSearch.mockReset(); + mockSearchOpenSearch.mockReset(); mockBasicSearch.mockReset(); }); test('POST /search validates, builds the OpenSearch request, and formats pagination metadata', async () => { - mockApiSearch.mockResolvedValue(advancedResponse(1, [{ _id: 'CVE-2024-0001' }])); + mockSearchOpenSearch.mockResolvedValue(advancedResponse(1, [{ _id: 'CVE-2024-0001' }]).data); const response = await request(createApp(), { method: 'POST', @@ -66,23 +65,31 @@ describe('search route integration', () => { }); expect(response.status).toBe(200); - expect(mockApiSearch).toHaveBeenCalledWith( - null, + expect(mockSearchOpenSearch).toHaveBeenCalledWith( { - filters: [], - rangeObjects: [], - resultsPerPage: 1, + query: { + bool: { + must: [ + { + simple_query_string: { + query: 'remote* code*', + default_operator: 'AND', + fields: ['containers.cna.descriptions.value'] + } + } + ], + filter: [] + } + }, + track_total_hits: true, + size: 1, from: 0 }, - [ - { - simple_query_string: { - query: 'remote* code*', - default_operator: 'AND', - fields: ['containers.cna.descriptions.value'] - } - } - ] + expect.objectContaining({ + operation: 'standard', + signal: expect.any(AbortSignal), + onStart: expect.any(Function) + }) ); expect(response.body.page).toBe(1); expect(response.body.pageSize).toBe(1); @@ -103,7 +110,48 @@ describe('search route integration', () => { error: 'DATE_TIME_FORMAT', message: "The pubStartDate parameter must match the following format: yyyy-MM-ddTHH:mm:ss.SSS'Z'" }); - expect(mockApiSearch).not.toHaveBeenCalled(); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + + test('POST /search rejects excessive keyword terms before invoking OpenSearch', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { + keywordSearch: Array(constants.MAX_KEYWORD_SEARCH_TOKENS + 1) + .fill('term') + .join(' ') + } + }); + + expect(response.status).toBe(400); + expect(response.body).toStrictEqual({ + error: 'SEARCH_FIELD_TOKEN_LIMIT', + message: `The keywordSearch field must contain ${constants.MAX_KEYWORD_SEARCH_TOKENS} or fewer terms.` + }); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + + test('POST /search rejects an inverted virtualMatchString version range before invoking OpenSearch', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + versionStart: '3.0', + versionStartType: 'including', + versionEnd: '2.0', + versionEndType: 'including' + } + }); + + expect(response.status).toBe(400); + expect(response.body).toStrictEqual({ + error: 'VERSION_RANGE_ORDER', + message: + 'The versionStart parameter must be less than versionEnd. Equal values require both bounds to be including.' + }); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); }); test('POST /webSearch normalizes body fields and formats BasicSearchManager responses', async () => { diff --git a/routes/swagger.js b/routes/swagger.js index ea6f64f..5a78b14 100644 --- a/routes/swagger.js +++ b/routes/swagger.js @@ -15,9 +15,10 @@ function searchRequestProperty(description, schema) { const searchRequestProperties = { keywordSearch: searchRequestProperty( - "Returns CVE records where a word or phrase is included in the record's description. Multiple keywords operate like an AND statement, returning records whose descriptions include every keyword.", + `Returns CVE records where a word or phrase is included in the record's description. Multiple keywords operate like an AND statement, returning records whose descriptions include every keyword. Limited to ${constants.SEARCH_STRING_LENGTH_LIMITS.keywordSearch} characters and ${constants.MAX_KEYWORD_SEARCH_TOKENS} whitespace-delimited terms.`, { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.keywordSearch, example: 'remote code execution' } ), @@ -29,9 +30,10 @@ const searchRequestProperties = { } ), cveIds: searchRequestProperty( - `Returns one or more CVE records identified by a comma-separated list of CVE IDs. Limited to ${constants.MAX_CVE_IDS} CVE IDs.`, + `Returns one or more CVE records identified by a comma-separated list of CVE IDs. Limited to ${constants.MAX_CVE_IDS} CVE IDs, ${constants.MAX_CVE_ID_LENGTH} characters per ID, and ${constants.SEARCH_STRING_LENGTH_LIMITS.cveIds} characters overall.`, { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.cveIds, example: 'CVE-2024-1234,CVE-2024-5678' } ), @@ -41,6 +43,7 @@ const searchRequestProperties = { }), cweId: searchRequestProperty('Returns CVE records that include the provided Common Weakness Enumeration ID.', { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.cweId, example: 'CWE-79' }), cvssV2Severity: searchRequestProperty( @@ -120,17 +123,19 @@ const searchRequestProperties = { ), sourceIdentifier: searchRequestProperty('Returns CVE records that contain the provided source identifier.', { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.sourceIdentifier, example: 'security@example.org' }), cpeName: searchRequestProperty( 'Returns CVE records whose CPE Match Criteria apply to the provided CPE 2.3 asset name. The part, vendor, product, and version components must contain values other than *. The concrete version can match stored criteria with the same version or criteria with a wildcard version and version range fields. Invalid cpeName values return HTTP 404. Use virtualMatchString for partial CPE Match String searches.', { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.cpeName, example: 'cpe:2.3:a:freetype:freetype:2.8.1:*:*:*:*:*:*:*' } ), isVulnerable: searchRequestProperty( - 'When true, returns CVE records whose CPE Match Criteria match cpeName and are marked vulnerable. cpeName is required when isVulnerable is true. virtualMatchString cannot be provided with isVulnerable.', + 'When true, returns CVE records where the same CPE Match Criteria entry both matches cpeName and is marked vulnerable. False or omitted does not apply a vulnerability restriction. cpeName is required when isVulnerable is true. virtualMatchString cannot be provided with isVulnerable.', { type: 'boolean', default: false @@ -188,9 +193,10 @@ const searchRequestProperties = { } ), versionStart: searchRequestProperty( - 'Sets the starting version bound for a CPE Match String range search. versionStartType and virtualMatchString are required when versionStart is provided. virtualMatchString must not include a version component.', + 'Sets the starting version bound for a CPE Match String range search. versionStartType and virtualMatchString are required when versionStart is provided. When both bounds are provided, versionStart must be less than versionEnd; equal values require both bounds to be including. virtualMatchString must not include a version component.', { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.versionStart, example: '2.2' } ), @@ -202,9 +208,10 @@ const searchRequestProperties = { } ), versionEnd: searchRequestProperty( - 'Sets the ending version bound for a CPE Match String range search. versionEndType and virtualMatchString are required when versionEnd is provided. virtualMatchString must not include a version component.', + 'Sets the ending version bound for a CPE Match String range search. versionEndType and virtualMatchString are required when versionEnd is provided. When both bounds are provided, versionStart must be less than versionEnd; equal values require both bounds to be including. virtualMatchString must not include a version component.', { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.versionEnd, example: '2.6' } ), @@ -219,6 +226,7 @@ const searchRequestProperties = { 'Returns CVE records whose CPE Match Criteria are compatible with the provided CPE 2.3 match string. Omitted trailing components and * components act as broad matches. When used with versionStart or versionEnd, provide a partial CPE 2.3 match string without a version component.', { type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.virtualMatchString, example: 'cpe:2.3:o:microsoft:windows_10_1809' } ) diff --git a/server.js b/server.js index d76f476..4c94a56 100644 --- a/server.js +++ b/server.js @@ -7,10 +7,13 @@ const http = require('http'); const logger = require('morgan'); const path = require('path'); const router = require('./routes/index'); +const errors = require('./utils/errors'); const requestLogging = require('./utils/requestLogging'); dotenv.load(); +const error = new errors.errors(); + const swaggerSetupOptions = { swaggerOptions: { supportedSubmitMethods: [] @@ -35,19 +38,32 @@ function createRequestLogger() { }); } +function handleRequestError(err, req, res, next) { + if (err?.type === 'entity.parse.failed') { + return res.status(400).json(error.malformedJsonError()); + } + + if (err?.status === 415 && req.path === '/search') { + return res.status(415).json(error.searchContentTypeError()); + } + + next(err); +} + function createApp() { const app = express(); app.use(requestLogging.assignRequestId); app.use(createRequestLogger()); + app.use(cors()); app.use(express.static(path.join(__dirname, 'public'))); app.use(express.json()); - app.use(cors()); app.get('/api-docs/openapi.json', (req, res) => res.json(openApiSpecification)); app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(openApiSpecification, swaggerSetupOptions)); app.use('/', router); + app.use(handleRequestError); return app; } diff --git a/server.test.js b/server.test.js index 377fb70..51ce3c8 100644 --- a/server.test.js +++ b/server.test.js @@ -142,6 +142,44 @@ describe('server application bootstrap', () => { }); }); + test('createApp returns a sanitized error for malformed JSON', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + rawBody: '{"keywordSearch":', + headers: { + 'Content-Type': 'application/json' + } + }); + + expect(response.status).toBe(400); + expect(response.body).toStrictEqual({ + error: 'MALFORMED_JSON', + message: 'The request body contains malformed JSON.' + }); + expect(response.headers['access-control-allow-origin']).toBe('*'); + expect(mockSearch).not.toHaveBeenCalled(); + }); + + test('createApp returns a sanitized error for unsupported JSON encoding', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + rawBody: '{"keywordSearch":"apache"}', + headers: { + 'Content-Type': 'application/json; charset=iso-8859-1' + } + }); + + expect(response.status).toBe(415); + expect(response.body).toStrictEqual({ + error: 'SEARCH_CONTENT_TYPE', + message: 'The /search request body must use the application/json media type with UTF-8 encoding.' + }); + expect(response.headers['access-control-allow-origin']).toBe('*'); + expect(mockSearch).not.toHaveBeenCalled(); + }); + test('getRuntimeConfig preserves baseURL behavior when PORT is set without BASE_URL', () => { delete process.env.BASE_URL; process.env.PORT = '4567'; diff --git a/test/http.js b/test/http.js index f437097..716afe9 100644 --- a/test/http.js +++ b/test/http.js @@ -8,10 +8,24 @@ function createRouterApp(router) { return app; } -function request(app, { method = 'GET', path, body, headers = {} }) { +function hasHeader(headers, name) { + return Object.keys(headers).some((header) => header.toLowerCase() === name.toLowerCase()); +} + +function request(app, { method = 'GET', path, body, rawBody, headers = {} }) { return new Promise((resolve, reject) => { const server = app.listen(0, () => { - const payload = body ? JSON.stringify(body) : undefined; + const payload = + rawBody !== undefined ? String(rawBody) : body !== undefined ? JSON.stringify(body) : undefined; + const requestHeaders = { ...headers }; + if (payload !== undefined) { + if (rawBody === undefined && !hasHeader(requestHeaders, 'content-type')) { + requestHeaders['Content-Type'] = 'application/json'; + } + if (!hasHeader(requestHeaders, 'content-length')) { + requestHeaders['Content-Length'] = Buffer.byteLength(payload); + } + } const { port } = server.address(); const req = http.request( { @@ -19,15 +33,7 @@ function request(app, { method = 'GET', path, body, headers = {} }) { port, path, method, - headers: { - ...headers, - ...(payload - ? { - 'Content-Type': 'application/json', - 'Content-Length': Buffer.byteLength(payload) - } - : {}) - } + headers: requestHeaders }, (res) => { let text = ''; @@ -61,7 +67,7 @@ function request(app, { method = 'GET', path, body, headers = {} }) { server.close(() => reject(err)); }); - if (payload) { + if (payload !== undefined) { req.write(payload); } diff --git a/utils/constants.js b/utils/constants.js index 6a98784..e3f824e 100644 --- a/utils/constants.js +++ b/utils/constants.js @@ -5,6 +5,18 @@ const MAX_CVSS_METRICS_LENGTH = 256; const MAX_CVSS_METRIC_TOKENS = 32; const MAX_CVSS_METRIC_TOKEN_LENGTH = 32; const MAX_CVE_IDS = 100; +const MAX_CVE_ID_LENGTH = 64; +const MAX_KEYWORD_SEARCH_TOKENS = 64; +const SEARCH_STRING_LENGTH_LIMITS = Object.freeze({ + keywordSearch: 512, + cveIds: 4096, + cweId: 32, + sourceIdentifier: 512, + cpeName: 2048, + versionStart: 256, + versionEnd: 256, + virtualMatchString: 2048 +}); const SEARCH_POST_FILTER_CACHE_TTL_MS = 60 * 1000; const SEARCH_POST_FILTER_CACHE_MAX_ENTRIES = 100; const SEARCH_POST_FILTER_COLD_SEARCH_CONCURRENCY = 10; @@ -12,6 +24,7 @@ const SEARCH_POST_FILTER_COLD_SEARCH_MAX_QUEUE_DEPTH = 500; const SEARCH_POST_FILTER_COLD_SEARCH_MAX_WAIT_MS = 10 * 1000; const SEARCH_OPEN_SEARCH_REQUEST_TIMEOUT_MS = 20 * 1000; const SEARCH_REQUEST_DEADLINE_MS = 30 * 1000; +const SEARCH_READINESS_TIMEOUT_MS = 5 * 1000; const rejectedCveStates = ['REJECT', 'REJECTED', 'Rejected']; const rejectedCveStateFields = ['cveMetadata.state', 'containers.cna.x_legacyV4Record.CVE_data_meta.STATE']; @@ -125,6 +138,9 @@ module.exports = { MAX_CVSS_METRIC_TOKENS, MAX_CVSS_METRIC_TOKEN_LENGTH, MAX_CVE_IDS, + MAX_CVE_ID_LENGTH, + MAX_KEYWORD_SEARCH_TOKENS, + SEARCH_STRING_LENGTH_LIMITS, SEARCH_POST_FILTER_CACHE_TTL_MS, SEARCH_POST_FILTER_CACHE_MAX_ENTRIES, SEARCH_POST_FILTER_COLD_SEARCH_CONCURRENCY, @@ -132,6 +148,7 @@ module.exports = { SEARCH_POST_FILTER_COLD_SEARCH_MAX_WAIT_MS, SEARCH_OPEN_SEARCH_REQUEST_TIMEOUT_MS, SEARCH_REQUEST_DEADLINE_MS, + SEARCH_READINESS_TIMEOUT_MS, RESULTS_PER_PAGE, REJECTED_CVE_STATES: rejectedCveStates, CPE_VERSION_RANGE_TYPES: cpeVersionRangeTypes, diff --git a/utils/cpe.js b/utils/cpe.js index c4f3bc5..293daf8 100644 --- a/utils/cpe.js +++ b/utils/cpe.js @@ -245,10 +245,12 @@ function compareCpeVersions(a, b) { const tokenB = tokensB[i]; if (tokenA === undefined) { - return /^0+$/.test(tokenB) ? 0 : -1; + if (/^0+$/.test(tokenB)) continue; + return -1; } if (tokenB === undefined) { - return /^0+$/.test(tokenA) ? 0 : 1; + if (/^0+$/.test(tokenA)) continue; + return 1; } if (tokenA === tokenB) continue; @@ -365,11 +367,12 @@ function rangeBoundaryMatches(assetVersion, cpeMatch) { return true; } -function cpeMatchAppliesToCpeName(cpeMatch, cpeName, context) { +function cpeMatchAppliesToCpeName(cpeMatch, cpeName, context, { requireVulnerable = false } = {}) { const asset = cpeName?.valid ? cpeName : parseCpeName(cpeName); const criteria = getParsedCpeMatchCriteria(cpeMatch?.criteria, context); if (!asset.valid || !criteria.valid) return false; + if (requireVulnerable && cpeMatch?.vulnerable !== true) return false; for (const index of [0, 1, 2]) { if (!componentMatches(asset.components[index], criteria.components[index])) { @@ -453,8 +456,8 @@ function getCpeMatchesFromHit(hit) { return matches; } -function hitAppliesToCpeName(hit, cpeName, context) { - return getCpeMatchesFromHit(hit).some((match) => cpeMatchAppliesToCpeName(match, cpeName, context)); +function hitAppliesToCpeName(hit, cpeName, context, options) { + return getCpeMatchesFromHit(hit).some((match) => cpeMatchAppliesToCpeName(match, cpeName, context, options)); } function hitOverlapsVirtualMatchStringRange(hit, virtualMatchStringRange, context) { diff --git a/utils/cpe.test.js b/utils/cpe.test.js index 5344929..041728e 100644 --- a/utils/cpe.test.js +++ b/utils/cpe.test.js @@ -159,6 +159,10 @@ describe('CPE helpers', () => { test('compareCpeVersions compares numeric version segments semantically', () => { expect(cpe.compareCpeVersions('2.10', '2.9')).toBeGreaterThan(0); expect(cpe.compareCpeVersions('10.05.1', '10.5.1')).toBe(0); + expect(cpe.compareCpeVersions('1', '1.0')).toBe(0); + expect(cpe.compareCpeVersions('1.0.0', '1')).toBe(0); + expect(cpe.compareCpeVersions('1', '1.0.1')).toBeLessThan(0); + expect(cpe.compareCpeVersions('1.0.1', '1')).toBeGreaterThan(0); expect(cpe.compareCpeVersions('1.0', '1.0.1')).toBeLessThan(0); }); @@ -187,6 +191,28 @@ describe('CPE helpers', () => { ).toBe(false); }); + test('cpeMatchAppliesToCpeName requires vulnerability on the matching entry when requested', () => { + const cpeName = 'cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*'; + const matchingCriteria = 'cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*'; + + expect( + cpe.cpeMatchAppliesToCpeName({ criteria: matchingCriteria, vulnerable: true }, cpeName, undefined, { + requireVulnerable: true + }) + ).toBe(true); + expect( + cpe.cpeMatchAppliesToCpeName({ criteria: matchingCriteria, vulnerable: false }, cpeName, undefined, { + requireVulnerable: true + }) + ).toBe(false); + expect( + cpe.cpeMatchAppliesToCpeName({ criteria: matchingCriteria }, cpeName, undefined, { + requireVulnerable: true + }) + ).toBe(false); + expect(cpe.cpeMatchAppliesToCpeName({ criteria: matchingCriteria, vulnerable: false }, cpeName)).toBe(true); + }); + test('cpeMatchOverlapsVirtualMatchStringRange matches exact criteria inside the requested range', () => { const virtualMatchStringRange = { matchString: cpe.parseVirtualMatchString('cpe:2.3:o:linux:linux_kernel'), diff --git a/utils/errors.js b/utils/errors.js index 6c4654d..4c126e0 100644 --- a/utils/errors.js +++ b/utils/errors.js @@ -70,6 +70,13 @@ class errors { return makeError('VERSION_RANGE_TYPE', `The ${param} parameter must be including or excluding.`); } + versionRangeOrderError() { + return makeError( + 'VERSION_RANGE_ORDER', + 'The versionStart parameter must be less than versionEnd. Equal values require both bounds to be including.' + ); + } + virtualMatchStringVersionRangeFormatError() { return makeError( 'VIRTUAL_MATCH_STRING_RANGE_FORMAT', @@ -121,6 +128,17 @@ class errors { return makeError('SEARCH_BODY_OBJECT', 'The /search request body must be a JSON object.'); } + searchContentTypeError() { + return makeError( + 'SEARCH_CONTENT_TYPE', + 'The /search request body must use the application/json media type with UTF-8 encoding.' + ); + } + + malformedJsonError() { + return makeError('MALFORMED_JSON', 'The request body contains malformed JSON.'); + } + searchQueryUnsupportedError() { return makeError( 'SEARCH_QUERY_UNSUPPORTED', @@ -132,6 +150,21 @@ class errors { return makeError('SEARCH_FIELD_STRING', `The ${field} field must be a non-empty string.`); } + searchFieldLengthError(field, maxLength) { + return makeError('SEARCH_FIELD_LENGTH', `The ${field} field must be ${maxLength} characters or fewer.`); + } + + searchFieldTokenLimitError(field, maxTokens) { + return makeError('SEARCH_FIELD_TOKEN_LIMIT', `The ${field} field must contain ${maxTokens} or fewer terms.`); + } + + cveIdLengthError() { + return makeError( + 'CVE_ID_LENGTH', + `Each CVE ID in the cveIds field must be ${constants.MAX_CVE_ID_LENGTH} characters or fewer.` + ); + } + cvssV2SeverityError(severity) { return makeError( 'INVALID_CVSSV2_SEVERITY', @@ -240,6 +273,10 @@ class errors { ); } + searchNotReadyError() { + return makeError('SEARCH_NOT_READY', 'Search service is not ready. Please try again later.'); + } + upstreamSearchError() { return makeError('UPSTREAM_SEARCH_ERROR', 'Search backend returned an error. Please try again later.'); } diff --git a/utils/errors.test.js b/utils/errors.test.js index 494b1b2..682af30 100644 --- a/utils/errors.test.js +++ b/utils/errors.test.js @@ -99,6 +99,15 @@ describe('error response helpers', () => { message: 'The versionEndType parameter must be including or excluding.' } ], + [ + 'versionRangeOrderError', + [], + { + error: 'VERSION_RANGE_ORDER', + message: + 'The versionStart parameter must be less than versionEnd. Equal values require both bounds to be including.' + } + ], [ 'virtualMatchStringVersionRangeFormatError', [], @@ -198,6 +207,14 @@ describe('error response helpers', () => { message: 'Too many search requests. Please retry later.' } ], + [ + 'searchNotReadyError', + [], + { + error: 'SEARCH_NOT_READY', + message: 'Search service is not ready. Please try again later.' + } + ], [ 'searchBodyError', [], @@ -206,6 +223,22 @@ describe('error response helpers', () => { message: 'The /search request body must be a JSON object.' } ], + [ + 'searchContentTypeError', + [], + { + error: 'SEARCH_CONTENT_TYPE', + message: 'The /search request body must use the application/json media type with UTF-8 encoding.' + } + ], + [ + 'malformedJsonError', + [], + { + error: 'MALFORMED_JSON', + message: 'The request body contains malformed JSON.' + } + ], [ 'searchQueryUnsupportedError', [], diff --git a/utils/requestLogging.js b/utils/requestLogging.js index a363e83..57e7248 100644 --- a/utils/requestLogging.js +++ b/utils/requestLogging.js @@ -4,6 +4,7 @@ const { AppConfig } = require('cve-core'); const DEFAULT_SUCCESS_SAMPLE_RATE = 0.1; const DEFAULT_LOG_HEALTH_CHECKS = false; const REQUEST_ID_PATTERN = /^[A-Za-z0-9._:-]{1,128}$/; +const HEALTH_CHECK_PATHS = new Set(['/api/health-check', '/api/ready-check']); let requestLoggingConfig; @@ -74,7 +75,7 @@ function assignRequestId(req, res, next) { function shouldSkipAccessLog(req, res, random = Math.random) { if (res.statusCode >= 400) return false; - if (getRedactedRequestPath(req) === '/api/health-check' && !shouldLogHealthChecks()) { + if (HEALTH_CHECK_PATHS.has(getRedactedRequestPath(req)) && !shouldLogHealthChecks()) { return true; } diff --git a/utils/requestLogging.test.js b/utils/requestLogging.test.js index 9d90656..1cae11e 100644 --- a/utils/requestLogging.test.js +++ b/utils/requestLogging.test.js @@ -63,33 +63,39 @@ describe('requestLogging', () => { expect(next).toHaveBeenCalledTimes(1); }); - test('shouldSkipAccessLog suppresses successful health checks by default', () => { - expect( - requestLogging.shouldSkipAccessLog( - { - path: '/api/health-check' - }, - { - statusCode: 200 - }, - () => 0 - ) - ).toBe(true); - }); + test.each(['/api/health-check', '/api/ready-check'])( + 'shouldSkipAccessLog suppresses successful health check %s by default', + (path) => { + expect( + requestLogging.shouldSkipAccessLog( + { + path + }, + { + statusCode: 200 + }, + () => 0 + ) + ).toBe(true); + } + ); - test('shouldSkipAccessLog retains failed health checks', () => { - expect( - requestLogging.shouldSkipAccessLog( - { - path: '/api/health-check' - }, - { - statusCode: 500 - }, - () => 1 - ) - ).toBe(false); - }); + test.each(['/api/health-check', '/api/ready-check'])( + 'shouldSkipAccessLog retains failed health check %s', + (path) => { + expect( + requestLogging.shouldSkipAccessLog( + { + path + }, + { + statusCode: 500 + }, + () => 1 + ) + ).toBe(false); + } + ); test('shouldSkipAccessLog samples successful requests at the configured rate', () => { mockAppConfigHas.mockImplementation((path) => path === 'requestLogging.successSampleRate'); diff --git a/utils/searchDeadline.js b/utils/searchDeadline.js index e0fafc0..ecb33c4 100644 --- a/utils/searchDeadline.js +++ b/utils/searchDeadline.js @@ -6,30 +6,95 @@ class SearchDeadlineExceededError extends Error { } } +class SearchClientDisconnectedError extends Error { + constructor() { + super('Search client disconnected.'); + this.name = 'SearchClientDisconnectedError'; + this.code = 'SEARCH_CLIENT_DISCONNECTED'; + } +} + function isSearchTimeoutError(err) { return err instanceof SearchDeadlineExceededError || err?.code === 'SEARCH_TIMEOUT' || err?.name === 'TimeoutError'; } -async function withSearchDeadline(promise, timeoutMs) { - const startedAt = Date.now(); - let timeoutId; - const timeoutPromise = new Promise((_, reject) => { - timeoutId = setTimeout(() => reject(new SearchDeadlineExceededError()), timeoutMs); - timeoutId.unref?.(); +function isSearchClientDisconnectedError(err) { + return err instanceof SearchClientDisconnectedError || err?.code === 'SEARCH_CLIENT_DISCONNECTED'; +} + +function getAbortReason(signal) { + if (signal?.reason instanceof Error) return signal.reason; + + const err = new Error('Search request aborted.'); + err.name = 'AbortError'; + err.code = 'SEARCH_ABORTED'; + return err; +} + +function throwIfAborted(signal) { + if (signal?.aborted) throw getAbortReason(signal); +} + +function addOnceListener(emitter, eventName, listener) { + if (typeof emitter?.once !== 'function') return () => {}; + + emitter.once(eventName, listener); + return () => emitter.removeListener?.(eventName, listener); +} + +function createSearchRequestLifecycle(req, res, timeoutMs) { + const controller = new AbortController(); + let completed = false; + + const abort = (reason) => { + if (!completed && !controller.signal.aborted) { + controller.abort(reason); + } + }; + const removeRequestAbortedListener = addOnceListener(req, 'aborted', () => + abort(new SearchClientDisconnectedError()) + ); + const removeResponseCloseListener = addOnceListener(res, 'close', () => { + if (res?.writableEnded !== true) { + abort(new SearchClientDisconnectedError()); + } }); + const timeoutId = setTimeout(() => abort(new SearchDeadlineExceededError()), timeoutMs); + timeoutId.unref?.(); - try { - const result = await Promise.race([promise, timeoutPromise]); - if (Date.now() - startedAt >= timeoutMs) throw new SearchDeadlineExceededError(); + function complete() { + if (completed) return; - return result; - } finally { + completed = true; clearTimeout(timeoutId); + removeRequestAbortedListener(); + removeResponseCloseListener(); } + + function throwIfRequestAborted() { + if ( + !controller.signal.aborted && + (req?.aborted === true || (res?.destroyed === true && res?.writableEnded !== true)) + ) { + abort(new SearchClientDisconnectedError()); + } + + throwIfAborted(controller.signal); + } + + return { + complete, + signal: controller.signal, + throwIfAborted: throwIfRequestAborted + }; } module.exports = { + createSearchRequestLifecycle, + getAbortReason, + isSearchClientDisconnectedError, isSearchTimeoutError, + SearchClientDisconnectedError, SearchDeadlineExceededError, - withSearchDeadline + throwIfAborted }; diff --git a/utils/searchDeadline.test.js b/utils/searchDeadline.test.js index 944cb80..cafbf3c 100644 --- a/utils/searchDeadline.test.js +++ b/utils/searchDeadline.test.js @@ -1,3 +1,4 @@ +const { EventEmitter } = require('events'); const searchDeadline = require('./searchDeadline'); function captureTimers() { @@ -18,37 +19,90 @@ function captureTimers() { return timers; } +function createLifecycle(timeoutMs = 30000) { + const req = new EventEmitter(); + const res = new EventEmitter(); + res.writableEnded = false; + + return { + lifecycle: searchDeadline.createSearchRequestLifecycle(req, res, timeoutMs), + req, + res + }; +} + describe('searchDeadline', () => { afterEach(() => { jest.restoreAllMocks(); }); - test('returns a search result completed before the deadline', async () => { - await expect(searchDeadline.withSearchDeadline(Promise.resolve('result'), 30000)).resolves.toBe('result'); - }); - - test('rejects a search that exceeds the deadline', async () => { + test('aborts the request lifecycle when the deadline expires', () => { const timers = captureTimers(); - const pendingSearch = new Promise(() => {}); - const result = searchDeadline.withSearchDeadline(pendingSearch, 30000); - const timeoutExpectation = expect(result).rejects.toMatchObject({ + const { lifecycle } = createLifecycle(); + + timers.find(({ delay }) => delay === 30000).callback(); + + expect(lifecycle.signal.aborted).toBe(true); + expect(lifecycle.signal.reason).toMatchObject({ name: 'SearchDeadlineExceededError', code: 'SEARCH_TIMEOUT' }); + }); + + test('complete clears lifecycle listeners and prevents a late deadline abort', () => { + const timers = captureTimers(); + const { lifecycle, req, res } = createLifecycle(); + lifecycle.complete(); timers.find(({ delay }) => delay === 30000).callback(); - await timeoutExpectation; + expect(lifecycle.signal.aborted).toBe(false); + expect(req.listenerCount('aborted')).toBe(0); + expect(res.listenerCount('close')).toBe(0); + expect(clearTimeout).toHaveBeenCalledTimes(1); }); - test('rejects work that completes after the deadline before the timer can run', async () => { - jest.spyOn(Date, 'now').mockReturnValueOnce(0).mockReturnValueOnce(30001); + test('aborts when the request emits aborted', () => { + captureTimers(); + const { lifecycle, req } = createLifecycle(); - await expect(searchDeadline.withSearchDeadline(Promise.resolve('late result'), 30000)).rejects.toMatchObject({ - code: 'SEARCH_TIMEOUT' + req.emit('aborted'); + + expect(lifecycle.signal.reason).toMatchObject({ + name: 'SearchClientDisconnectedError', + code: 'SEARCH_CLIENT_DISCONNECTED' }); }); + test('aborts when the response closes before it is written', () => { + captureTimers(); + const { lifecycle, res } = createLifecycle(); + + res.emit('close'); + + expect(lifecycle.signal.reason).toMatchObject({ + code: 'SEARCH_CLIENT_DISCONNECTED' + }); + }); + + test('does not abort when a completed response closes', () => { + captureTimers(); + const { lifecycle, res } = createLifecycle(); + res.writableEnded = true; + + res.emit('close'); + + expect(lifecycle.signal.aborted).toBe(false); + }); + + test('detects an already-destroyed response before writing', () => { + captureTimers(); + const { lifecycle, res } = createLifecycle(); + res.destroyed = true; + + expect(() => lifecycle.throwIfAborted()).toThrow(searchDeadline.SearchClientDisconnectedError); + }); + test.each([ [new searchDeadline.SearchDeadlineExceededError()], [{ code: 'SEARCH_TIMEOUT' }], @@ -56,4 +110,19 @@ describe('searchDeadline', () => { ])('recognizes search timeout error %#', (err) => { expect(searchDeadline.isSearchTimeoutError(err)).toBe(true); }); + + test.each([[new searchDeadline.SearchClientDisconnectedError()], [{ code: 'SEARCH_CLIENT_DISCONNECTED' }]])( + 'recognizes client disconnect error %#', + (err) => { + expect(searchDeadline.isSearchClientDisconnectedError(err)).toBe(true); + } + ); + + test('throwIfAborted throws the lifecycle abort reason', () => { + const controller = new AbortController(); + const reason = new searchDeadline.SearchDeadlineExceededError(); + controller.abort(reason); + + expect(() => searchDeadline.throwIfAborted(controller.signal)).toThrow(reason); + }); }); diff --git a/utils/searchInFlightRegistry.js b/utils/searchInFlightRegistry.js new file mode 100644 index 0000000..07f4e85 --- /dev/null +++ b/utils/searchInFlightRegistry.js @@ -0,0 +1,118 @@ +const searchDeadline = require('./searchDeadline'); + +const entries = new Map(); + +class SearchOperationAbandonedError extends Error { + constructor() { + super('Shared search operation no longer has active waiters.'); + this.name = 'SearchOperationAbandonedError'; + this.code = 'SEARCH_OPERATION_ABANDONED'; + } +} + +function buildKey(type, value) { + return `${type}:${JSON.stringify(value)}`; +} + +function settleEntry(key, entry) { + entry.settled = true; + if (entries.get(key) === entry) { + entries.delete(key); + } +} + +function createEntry(key, createOperation) { + const controller = new AbortController(); + const entry = { + controller, + key, + promise: undefined, + settled: false, + waiterCount: 0 + }; + + entries.set(key, entry); + entry.promise = Promise.resolve().then(() => createOperation(controller.signal)); + entry.promise.then( + () => settleEntry(key, entry), + () => settleEntry(key, entry) + ); + + return entry; +} + +function releaseWaiter(entry) { + entry.waiterCount = Math.max(0, entry.waiterCount - 1); + if (!entry.settled && entry.waiterCount === 0 && !entry.controller.signal.aborted) { + settleEntry(entry.key, entry); + entry.controller.abort(new SearchOperationAbandonedError()); + } +} + +function waitForEntry(entry, signal) { + entry.waiterCount += 1; + + return new Promise((resolve, reject) => { + let settled = false; + + const finish = (callback, value) => { + if (settled) return; + + settled = true; + signal?.removeEventListener('abort', onAbort); + releaseWaiter(entry); + callback(value); + }; + const onAbort = () => finish(reject, searchDeadline.getAbortReason(signal)); + + if (signal?.aborted) { + onAbort(); + return; + } + + signal?.addEventListener('abort', onAbort, { once: true }); + entry.promise.then( + (value) => finish(resolve, value), + (err) => finish(reject, err) + ); + }); +} + +function acquire(key, createOperation, { signal } = {}) { + searchDeadline.throwIfAborted(signal); + + let entry = entries.get(key); + const shared = Boolean(entry); + if (!entry) { + entry = createEntry(key, createOperation); + } + + return { + promise: waitForEntry(entry, signal), + shared + }; +} + +function clear() { + for (const entry of entries.values()) { + if (!entry.controller.signal.aborted) { + entry.controller.abort(new SearchOperationAbandonedError()); + } + } + entries.clear(); +} + +function getState() { + return { + operationCount: entries.size, + waiterCount: [...entries.values()].reduce((total, entry) => total + entry.waiterCount, 0) + }; +} + +module.exports = { + acquire, + buildKey, + clear, + getState, + SearchOperationAbandonedError +}; diff --git a/utils/searchInFlightRegistry.test.js b/utils/searchInFlightRegistry.test.js new file mode 100644 index 0000000..aed7ad9 --- /dev/null +++ b/utils/searchInFlightRegistry.test.js @@ -0,0 +1,133 @@ +const searchDeadline = require('./searchDeadline'); +const searchInFlightRegistry = require('./searchInFlightRegistry'); + +function deferred() { + const state = {}; + state.promise = new Promise((resolve, reject) => { + state.resolve = resolve; + state.reject = reject; + }); + return state; +} + +describe('searchInFlightRegistry', () => { + beforeEach(() => { + searchInFlightRegistry.clear(); + }); + + afterEach(() => { + searchInFlightRegistry.clear(); + }); + + test('coalesces waiters for the same operation key', async () => { + const operation = deferred(); + const createOperation = jest.fn(() => operation.promise); + const first = searchInFlightRegistry.acquire('same-key', createOperation); + const second = searchInFlightRegistry.acquire('same-key', createOperation); + await Promise.resolve(); + + expect(first.shared).toBe(false); + expect(second.shared).toBe(true); + expect(createOperation).toHaveBeenCalledTimes(1); + expect(searchInFlightRegistry.getState()).toStrictEqual({ + operationCount: 1, + waiterCount: 2 + }); + + operation.resolve('result'); + + await expect(first.promise).resolves.toBe('result'); + await expect(second.promise).resolves.toBe('result'); + expect(searchInFlightRegistry.getState()).toStrictEqual({ + operationCount: 0, + waiterCount: 0 + }); + }); + + test('keeps shared work active while another waiter remains', async () => { + const operation = deferred(); + const firstController = new AbortController(); + const secondController = new AbortController(); + let operationSignal; + const createOperation = jest.fn((signal) => { + operationSignal = signal; + return operation.promise; + }); + const first = searchInFlightRegistry.acquire('same-key', createOperation, { + signal: firstController.signal + }); + const second = searchInFlightRegistry.acquire('same-key', createOperation, { + signal: secondController.signal + }); + await Promise.resolve(); + + firstController.abort(new searchDeadline.SearchClientDisconnectedError()); + + await expect(first.promise).rejects.toMatchObject({ + code: 'SEARCH_CLIENT_DISCONNECTED' + }); + expect(operationSignal.aborted).toBe(false); + expect(searchInFlightRegistry.getState().waiterCount).toBe(1); + + operation.resolve('result'); + await expect(second.promise).resolves.toBe('result'); + }); + + test('aborts shared work after its final waiter cancels', async () => { + const operation = deferred(); + const controller = new AbortController(); + let operationSignal; + const acquired = searchInFlightRegistry.acquire( + 'same-key', + (signal) => { + operationSignal = signal; + return operation.promise; + }, + { + signal: controller.signal + } + ); + await Promise.resolve(); + + controller.abort(new searchDeadline.SearchDeadlineExceededError()); + + await expect(acquired.promise).rejects.toMatchObject({ + code: 'SEARCH_TIMEOUT' + }); + expect(operationSignal.reason).toMatchObject({ + code: 'SEARCH_OPERATION_ABANDONED' + }); + expect(searchInFlightRegistry.getState().operationCount).toBe(0); + + const replacementOperation = jest.fn(() => Promise.resolve('replacement')); + const replacement = searchInFlightRegistry.acquire('same-key', replacementOperation); + + expect(replacement.shared).toBe(false); + await expect(replacement.promise).resolves.toBe('replacement'); + expect(replacementOperation).toHaveBeenCalledTimes(1); + + operation.reject(operationSignal.reason); + await expect(operation.promise).rejects.toMatchObject({ + code: 'SEARCH_OPERATION_ABANDONED' + }); + }); + + test('does not start work for an already-aborted waiter', () => { + const controller = new AbortController(); + const createOperation = jest.fn(); + controller.abort(new searchDeadline.SearchDeadlineExceededError()); + + expect(() => + searchInFlightRegistry.acquire('same-key', createOperation, { + signal: controller.signal + }) + ).toThrow(searchDeadline.SearchDeadlineExceededError); + expect(createOperation).not.toHaveBeenCalled(); + }); + + test('buildKey preserves operation type boundaries', () => { + expect(searchInFlightRegistry.buildKey('candidate', ['CVE-2024-0001'])).not.toBe( + searchInFlightRegistry.buildKey('pageFetch', ['CVE-2024-0001']) + ); + }); +}); diff --git a/utils/searchPerformance.js b/utils/searchPerformance.js index 26ebd22..1240cfc 100644 --- a/utils/searchPerformance.js +++ b/utils/searchPerformance.js @@ -32,6 +32,7 @@ function logSearchPerformance(metrics, searchOptions, responseOptions, postFilte outcome, statusCode: metrics.statusCode, timeoutSource: metrics.timeoutSource, + abortReason: metrics.abortReason, strategy: metrics.strategy, cacheStatus: getCacheStatus(metrics, postFilters), coldSearchQueueWaitElapsedMs: roundMs(metrics.coldSearchQueueWaitElapsedMs), @@ -45,6 +46,8 @@ function logSearchPerformance(metrics, searchOptions, responseOptions, postFilte cacheHit: metrics.cacheHit === true, inFlightHit: metrics.inFlightHit === true, pageFetchInFlightHit: metrics.pageFetchInFlightHit === true, + openSearchOperations: metrics.openSearchOperations ?? [], + maxOpenSearchActiveRequests: metrics.maxOpenSearchActiveRequests, postFilters: Object.keys(postFilters ?? {}), requestedFrom: responseOptions?.from ?? searchOptions?.from, requestedSize: responseOptions?.resultsPerPage ?? searchOptions?.resultsPerPage, diff --git a/utils/searchPerformance.test.js b/utils/searchPerformance.test.js index b4f1cfa..5ec42f1 100644 --- a/utils/searchPerformance.test.js +++ b/utils/searchPerformance.test.js @@ -28,6 +28,8 @@ describe('searchPerformance', () => { cacheHit: true, inFlightHit: true, pageFetchInFlightHit: true, + openSearchOperations: ['cpe-candidate', 'page-fetch'], + maxOpenSearchActiveRequests: 3, candidateCount: 10000, filteredCount: 4, returnedCount: 1, @@ -72,6 +74,8 @@ describe('searchPerformance', () => { cacheHit: true, inFlightHit: true, pageFetchInFlightHit: true, + openSearchOperations: ['cpe-candidate', 'page-fetch'], + maxOpenSearchActiveRequests: 3, postFilters: ['virtualMatchStringRange'], requestedFrom: 0, requestedSize: 1, diff --git a/utils/searchPostFilterCache.js b/utils/searchPostFilterCache.js index a573859..f7ed9d8 100644 --- a/utils/searchPostFilterCache.js +++ b/utils/searchPostFilterCache.js @@ -1,7 +1,6 @@ const searchProvider = require('./searchProvider'); const entries = new Map(); -const inFlightEntries = new Map(); function normalizeForKey(value) { if (Array.isArray(value)) return value.map(normalizeForKey); @@ -29,15 +28,6 @@ function buildCacheKey(candidateBody, postFilters) { ); } -function buildInFlightKey(type, value) { - return JSON.stringify( - normalizeForKey({ - type, - value - }) - ); -} - function get(cacheKey, now = Date.now()) { const entry = entries.get(cacheKey); if (!entry) return null; @@ -70,36 +60,14 @@ function set(cacheKey, value, now = Date.now()) { } } -function getInFlight(cacheKey) { - return inFlightEntries.get(cacheKey); -} - -function setInFlight(cacheKey, promise) { - inFlightEntries.set(cacheKey, promise); - - const clear = () => { - if (inFlightEntries.get(cacheKey) === promise) { - inFlightEntries.delete(cacheKey); - } - }; - - promise.then(clear, clear); - - return promise; -} - function clear() { entries.clear(); - inFlightEntries.clear(); } module.exports = { buildCacheKey, - buildInFlightKey, clear, get, - getInFlight, normalizeForKey, - set, - setInFlight + set }; diff --git a/utils/searchPostFilterCache.test.js b/utils/searchPostFilterCache.test.js index 50a4737..77aff79 100644 --- a/utils/searchPostFilterCache.test.js +++ b/utils/searchPostFilterCache.test.js @@ -1,14 +1,5 @@ const searchPostFilterCache = require('./searchPostFilterCache'); -function deferred() { - const state = {}; - state.promise = new Promise((resolve, reject) => { - state.resolve = resolve; - state.reject = reject; - }); - return state; -} - describe('searchPostFilterCache', () => { beforeEach(() => { searchPostFilterCache.clear(); @@ -55,12 +46,6 @@ describe('searchPostFilterCache', () => { expect(left).toBe(right); }); - test('buildInFlightKey is stable for equivalent structured values', () => { - expect(searchPostFilterCache.buildInFlightKey('pageFetch', ['CVE-2024-0001'])).toBe( - searchPostFilterCache.buildInFlightKey('pageFetch', ['CVE-2024-0001']) - ); - }); - test('get returns cached values until the entry expires', () => { const key = 'cache-key'; const value = { @@ -73,33 +58,4 @@ describe('searchPostFilterCache', () => { expect(searchPostFilterCache.get(key, 1001)).toBe(value); expect(searchPostFilterCache.get(key, 61001)).toBeNull(); }); - - test('setInFlight returns an existing in-flight promise until it resolves', async () => { - const pending = deferred(); - - searchPostFilterCache.setInFlight('cache-key', pending.promise); - - expect(searchPostFilterCache.getInFlight('cache-key')).toBe(pending.promise); - - pending.resolve({ - candidateCount: 1, - filteredCveIds: ['CVE-2024-0001'] - }); - await pending.promise; - await Promise.resolve(); - - expect(searchPostFilterCache.getInFlight('cache-key')).toBeUndefined(); - }); - - test('setInFlight clears rejected in-flight promises', async () => { - const pending = deferred(); - - searchPostFilterCache.setInFlight('cache-key', pending.promise); - - pending.reject(new Error('candidate search failed')); - await expect(pending.promise).rejects.toThrow('candidate search failed'); - await Promise.resolve(); - - expect(searchPostFilterCache.getInFlight('cache-key')).toBeUndefined(); - }); }); diff --git a/utils/searchPostFilterWorkQueue.js b/utils/searchPostFilterWorkQueue.js index b89ae86..8180f97 100644 --- a/utils/searchPostFilterWorkQueue.js +++ b/utils/searchPostFilterWorkQueue.js @@ -1,4 +1,5 @@ const searchProvider = require('./searchProvider'); +const searchDeadline = require('./searchDeadline'); let activeCount = 0; const queue = []; @@ -58,11 +59,13 @@ function drain() { while (activeCount < maxConcurrent && queue.length) { const item = queue.shift(); clearTimeout(item.timeoutId); + item.signal?.removeEventListener('abort', item.onAbort); activeCount += 1; void (async () => { try { - item.resolve(await item.task()); + searchDeadline.throwIfAborted(item.signal); + item.resolve(await item.task(item.signal)); } catch (err) { item.reject(err); } finally { @@ -73,8 +76,15 @@ function drain() { } } -function run(task) { +function run(task, { signal } = {}) { return new Promise((resolve, reject) => { + try { + searchDeadline.throwIfAborted(signal); + } catch (err) { + reject(err); + return; + } + if (activeCount >= getMaxConcurrent() && queue.length >= getMaxQueueDepth()) { reject(new SearchPostFilterQueueFullError(getState())); return; @@ -85,15 +95,28 @@ function run(task) { resolve, reject, queuedAt: process.hrtime.bigint(), - timeoutId: undefined + timeoutId: undefined, + signal, + onAbort: undefined }; if (activeCount >= getMaxConcurrent()) { + item.onAbort = () => { + const queueIndex = queue.indexOf(item); + if (queueIndex === -1) return; + + queue.splice(queueIndex, 1); + clearTimeout(item.timeoutId); + reject(searchDeadline.getAbortReason(signal)); + drain(); + }; + signal?.addEventListener('abort', item.onAbort, { once: true }); item.timeoutId = setTimeout(() => { const queueIndex = queue.indexOf(item); if (queueIndex === -1) return; queue.splice(queueIndex, 1); + signal?.removeEventListener('abort', item.onAbort); const waitElapsedMs = Number(process.hrtime.bigint() - item.queuedAt) / 1e6; reject(new SearchPostFilterQueueTimeoutError(getState(), waitElapsedMs)); drain(); @@ -108,7 +131,10 @@ function run(task) { function resetForTests() { activeCount = 0; - queue.forEach((item) => clearTimeout(item.timeoutId)); + queue.forEach((item) => { + clearTimeout(item.timeoutId); + item.signal?.removeEventListener('abort', item.onAbort); + }); queue.length = 0; } diff --git a/utils/searchPostFilterWorkQueue.test.js b/utils/searchPostFilterWorkQueue.test.js index b8406f3..02ef0b1 100644 --- a/utils/searchPostFilterWorkQueue.test.js +++ b/utils/searchPostFilterWorkQueue.test.js @@ -9,6 +9,7 @@ jest.mock('./searchProvider', () => ({ })); const searchPostFilterWorkQueue = require('./searchPostFilterWorkQueue'); +const searchDeadline = require('./searchDeadline'); function deferred() { const state = {}; @@ -152,4 +153,48 @@ describe('searchPostFilterWorkQueue', () => { first.resolve('first'); await expect(firstRun).resolves.toBe('first'); }); + + test('removes queued work when its operation is aborted', async () => { + const first = deferred(); + const controller = new AbortController(); + const secondTask = jest.fn(); + + const firstRun = searchPostFilterWorkQueue.run(() => first.promise); + const secondRun = searchPostFilterWorkQueue.run(secondTask, { + signal: controller.signal + }); + await Promise.resolve(); + + controller.abort(new searchDeadline.SearchDeadlineExceededError()); + + await expect(secondRun).rejects.toMatchObject({ + code: 'SEARCH_TIMEOUT' + }); + expect(secondTask).not.toHaveBeenCalled(); + expect(searchPostFilterWorkQueue.getState().queuedCount).toBe(0); + + first.resolve('first'); + await expect(firstRun).resolves.toBe('first'); + }); + + test('passes the operation signal to active work', async () => { + const controller = new AbortController(); + const task = jest.fn( + (signal) => + new Promise((resolve, reject) => { + signal.addEventListener('abort', () => reject(signal.reason), { once: true }); + }) + ); + const run = searchPostFilterWorkQueue.run(task, { + signal: controller.signal + }); + await Promise.resolve(); + + controller.abort(new searchDeadline.SearchDeadlineExceededError()); + + await expect(run).rejects.toMatchObject({ + code: 'SEARCH_TIMEOUT' + }); + expect(task).toHaveBeenCalledWith(controller.signal); + }); }); diff --git a/utils/searchProvider.js b/utils/searchProvider.js index e056387..929ec6b 100644 --- a/utils/searchProvider.js +++ b/utils/searchProvider.js @@ -1,10 +1,20 @@ -const { AdvancedSearchManager, AppConfig, BasicSearchManager, SearchProviderSpec, SearchReader } = require('cve-core'); +const { AppConfig, BasicSearchManager, SearchProviderSpec, SearchReader } = require('cve-core'); const constants = require('./constants'); +const searchDeadline = require('./searchDeadline'); + +class SearchProviderContractError extends Error { + constructor() { + super('Installed cve-core SearchReader does not expose the required OpenSearch client contract.'); + this.name = 'SearchProviderContractError'; + this.code = 'SEARCH_PROVIDER_CONTRACT'; + } +} let searchProviderConfig; -let advancedSearchManager; let basicSearchManager; let searchReader; +let activeOpenSearchRequestCount = 0; +const activeOpenSearchRequestsByOperation = new Map(); function configureClientRequestTimeout(client) { if (client?.transport) { @@ -75,14 +85,6 @@ function getSearchProviderConfig() { return searchProviderConfig; } -function getAdvancedSearchManager() { - if (!advancedSearchManager) { - advancedSearchManager = configureManagerRequestTimeout(new AdvancedSearchManager(getSearchProviderConfig())); - } - - return advancedSearchManager; -} - function getBasicSearchManager() { if (!basicSearchManager) { basicSearchManager = configureManagerRequestTimeout(new BasicSearchManager(getSearchProviderConfig())); @@ -91,34 +93,169 @@ function getBasicSearchManager() { return basicSearchManager; } +function validateSearchReader(reader) { + const hasIndex = typeof reader?._cveIndex === 'string' && reader._cveIndex.trim() !== ''; + const hasSearchClient = typeof reader?._client?.search === 'function'; + + if (!hasIndex || !hasSearchClient) { + throw new SearchProviderContractError(); + } + + return reader; +} + function getSearchReader() { if (!searchReader) { const config = getSearchProviderConfig(); - searchReader = new SearchReader(config.providerEndpoint, config.index, { - ignore_ssl_cert_check_errors: config.allowUnknownSslCerts - }); - configureClientRequestTimeout(searchReader._client); + const reader = validateSearchReader( + new SearchReader(config.providerEndpoint, config.index, { + ignore_ssl_cert_check_errors: config.allowUnknownSslCerts + }) + ); + configureClientRequestTimeout(reader._client); + searchReader = reader; } return searchReader; } -async function searchRaw(body) { +function getOpenSearchRequestState() { + return { + activeCount: activeOpenSearchRequestCount, + activeByOperation: Object.fromEntries(activeOpenSearchRequestsByOperation) + }; +} + +function beginOpenSearchRequest(operation) { + activeOpenSearchRequestCount += 1; + activeOpenSearchRequestsByOperation.set(operation, (activeOpenSearchRequestsByOperation.get(operation) ?? 0) + 1); + + return getOpenSearchRequestState(); +} + +function endOpenSearchRequest(operation) { + activeOpenSearchRequestCount = Math.max(0, activeOpenSearchRequestCount - 1); + + const operationCount = activeOpenSearchRequestsByOperation.get(operation) ?? 0; + if (operationCount <= 1) { + activeOpenSearchRequestsByOperation.delete(operation); + } else { + activeOpenSearchRequestsByOperation.set(operation, operationCount - 1); + } +} + +function awaitAbortableRequest(request, signal) { + if (!signal) return Promise.resolve(request); + + searchDeadline.throwIfAborted(signal); + + return new Promise((resolve, reject) => { + let settled = false; + + const finish = (callback, value) => { + if (settled) return; + + settled = true; + signal.removeEventListener('abort', onAbort); + callback(value); + }; + const onAbort = () => { + try { + request?.abort?.(); + } catch (_err) { + // The request still rejects with the lifecycle's sanitized abort reason. + } + finish(reject, searchDeadline.getAbortReason(signal)); + }; + + signal.addEventListener('abort', onAbort, { once: true }); + Promise.resolve(request).then( + (response) => finish(resolve, response), + (err) => finish(reject, signal.aborted ? searchDeadline.getAbortReason(signal) : err) + ); + }); +} + +async function executeOpenSearchRequest(createRequest, { signal, operation, onStart } = {}) { + searchDeadline.throwIfAborted(signal); + const state = beginOpenSearchRequest(operation); + + try { + onStart?.(state); + const response = await awaitAbortableRequest(createRequest(), signal); + + return response; + } finally { + endOpenSearchRequest(operation); + } +} + +async function searchOpenSearch(body, { signal, operation = 'search', onStart } = {}) { + searchDeadline.throwIfAborted(signal); const reader = getSearchReader(); - const response = await reader._client.search( - { - index: reader._cveIndex, - body - }, + const response = await executeOpenSearchRequest( + () => + reader._client.search( + { + index: reader._cveIndex, + body + }, + { + maxRetries: 0, + requestTimeout: getOpenSearchRequestTimeoutMs() + } + ), { - maxRetries: 0, - requestTimeout: getOpenSearchRequestTimeoutMs() + signal, + operation, + onStart } ); return response.body; } +async function checkOpenSearchReadiness({ signal } = {}) { + searchDeadline.throwIfAborted(signal); + const reader = getSearchReader(); + await executeOpenSearchRequest( + () => + reader._client.search( + { + index: reader._cveIndex, + body: { + query: { + match_all: {} + }, + size: 0, + terminate_after: 1, + track_total_hits: false + } + }, + { + maxRetries: 0, + requestTimeout: getOpenSearchReadinessTimeoutMs() + } + ), + { + signal, + operation: 'readiness' + } + ); + + return true; +} + +function isSearchProviderContractError(err) { + return err instanceof SearchProviderContractError || err?.code === 'SEARCH_PROVIDER_CONTRACT'; +} + +function getOpenSearchReadinessTimeoutMs() { + return getIntegerConfig('search.readinessTimeoutMs', constants.SEARCH_READINESS_TIMEOUT_MS, { + min: 1 + }); +} + function shouldLogSearchPerformance() { if (!AppConfig.has('search.logPerformanceTimings')) return false; @@ -169,16 +306,19 @@ function getSearchRequestDeadlineMs() { function resetSearchProviderForTests() { searchProviderConfig = undefined; - advancedSearchManager = undefined; basicSearchManager = undefined; searchReader = undefined; + activeOpenSearchRequestCount = 0; + activeOpenSearchRequestsByOperation.clear(); } module.exports = { - getAdvancedSearchManager, + checkOpenSearchReadiness, getBasicSearchManager, getAllowUnknownSslCerts, getIntegerConfig, + getOpenSearchRequestState, + getOpenSearchReadinessTimeoutMs, getPostFilterCacheMaxEntries, getPostFilterCacheTtlMs, getPostFilterColdSearchConcurrency, @@ -188,8 +328,11 @@ module.exports = { getSearchRequestDeadlineMs, getSearchProviderConfig, getSearchReader, + isSearchProviderContractError, normalizeBoolean, resetSearchProviderForTests, - searchRaw, - shouldLogSearchPerformance + searchOpenSearch, + shouldLogSearchPerformance, + validateSearchReader, + SearchProviderContractError }; diff --git a/utils/searchProvider.test.js b/utils/searchProvider.test.js index 56899d7..c817ad9 100644 --- a/utils/searchProvider.test.js +++ b/utils/searchProvider.test.js @@ -1,14 +1,3 @@ -const mockAdvancedSearchManager = jest.fn(function AdvancedSearchManager(spec) { - this.spec = spec; - this._searchReader = { - _client: { - transport: { - maxRetries: 3, - requestTimeout: 30000 - } - } - }; -}); const mockBasicSearchManager = jest.fn(function BasicSearchManager(spec) { this.spec = spec; this._searchReader = { @@ -41,7 +30,6 @@ const mockSearchProviderSpec = { }; jest.mock('cve-core', () => ({ - AdvancedSearchManager: mockAdvancedSearchManager, AppConfig: { get: mockAppConfigGet, has: mockAppConfigHas @@ -52,11 +40,20 @@ jest.mock('cve-core', () => ({ })); const searchProvider = require('./searchProvider'); +const searchDeadline = require('./searchDeadline'); + +function deferred() { + const state = {}; + state.promise = new Promise((resolve, reject) => { + state.resolve = resolve; + state.reject = reject; + }); + return state; +} describe('searchProvider', () => { beforeEach(() => { searchProvider.resetSearchProviderForTests(); - mockAdvancedSearchManager.mockClear(); mockBasicSearchManager.mockClear(); mockSearchReader.mockClear(); mockClientSearch.mockReset(); @@ -91,22 +88,6 @@ describe('searchProvider', () => { expect(mockGetDefaultSearchProviderSpec).toHaveBeenCalledTimes(1); }); - test('reuses one advanced search manager per process', () => { - const first = searchProvider.getAdvancedSearchManager(); - const second = searchProvider.getAdvancedSearchManager(); - - expect(first).toBe(second); - expect(mockAdvancedSearchManager).toHaveBeenCalledTimes(1); - expect(mockAdvancedSearchManager).toHaveBeenCalledWith({ - providerEndpoint: 'https://search.example', - index: 'test-index', - allowUnknownSslCerts: false - }); - expect(mockGetDefaultSearchProviderSpec).toHaveBeenCalledTimes(1); - expect(first._searchReader._client.transport.maxRetries).toBe(0); - expect(first._searchReader._client.transport.requestTimeout).toBe(20000); - }); - test('reuses one basic search manager per process', () => { const first = searchProvider.getBasicSearchManager(); const second = searchProvider.getBasicSearchManager(); @@ -123,7 +104,7 @@ describe('searchProvider', () => { expect(first._searchReader._client.transport.requestTimeout).toBe(20000); }); - test('reuses one search reader per process for raw OpenSearch searches', async () => { + test('reuses one search reader per process for OpenSearch searches', async () => { mockClientSearch.mockResolvedValue({ body: { hits: { @@ -142,7 +123,11 @@ describe('searchProvider', () => { match_all: {} } }; - const result = await searchProvider.searchRaw(body); + const onStart = jest.fn(); + const result = await searchProvider.searchOpenSearch(body, { + operation: 'standard', + onStart + }); expect(first).toBe(second); expect(mockSearchReader).toHaveBeenCalledTimes(1); @@ -169,6 +154,207 @@ describe('searchProvider', () => { hits: [] } }); + expect(onStart).toHaveBeenCalledWith({ + activeCount: 1, + activeByOperation: { + standard: 1 + } + }); + expect(searchProvider.getOpenSearchRequestState()).toStrictEqual({ + activeCount: 0, + activeByOperation: {} + }); + }); + + test('checks OpenSearch readiness through the configured CVE index', async () => { + mockClientSearch.mockResolvedValue({ + body: { + hits: { + hits: [] + } + } + }); + + await expect(searchProvider.checkOpenSearchReadiness()).resolves.toBe(true); + + expect(mockClientSearch).toHaveBeenCalledWith( + { + index: 'test-index', + body: { + query: { + match_all: {} + }, + size: 0, + terminate_after: 1, + track_total_hits: false + } + }, + { + maxRetries: 0, + requestTimeout: 5000 + } + ); + expect(searchProvider.getOpenSearchRequestState()).toStrictEqual({ + activeCount: 0, + activeByOperation: {} + }); + }); + + test('fails readiness when the configured CVE index is unavailable', async () => { + const unavailable = new Error('index unavailable'); + mockClientSearch.mockRejectedValue(unavailable); + + await expect(searchProvider.checkOpenSearchReadiness()).rejects.toBe(unavailable); + }); + + test('aborts the readiness transport request when its lifecycle is canceled', async () => { + const pending = deferred(); + const request = pending.promise; + request.abort = jest.fn(); + mockClientSearch.mockReturnValue(request); + const controller = new AbortController(); + const readiness = searchProvider.checkOpenSearchReadiness({ + signal: controller.signal + }); + await Promise.resolve(); + + controller.abort(new searchDeadline.SearchDeadlineExceededError()); + + await expect(readiness).rejects.toMatchObject({ + code: 'SEARCH_TIMEOUT' + }); + expect(request.abort).toHaveBeenCalledTimes(1); + expect(searchProvider.getOpenSearchRequestState()).toStrictEqual({ + activeCount: 0, + activeByOperation: {} + }); + }); + + test('does not construct a SearchReader for an already-canceled readiness check', async () => { + const controller = new AbortController(); + controller.abort(new searchDeadline.SearchClientDisconnectedError()); + + await expect( + searchProvider.checkOpenSearchReadiness({ + signal: controller.signal + }) + ).rejects.toMatchObject({ + code: 'SEARCH_CLIENT_DISCONNECTED' + }); + expect(mockSearchReader).not.toHaveBeenCalled(); + expect(mockClientSearch).not.toHaveBeenCalled(); + }); + + test('rejects and does not cache an incompatible cve-core SearchReader', () => { + mockSearchReader.mockImplementationOnce(() => ({ + _cveIndex: 'test-index', + _client: { + transport: {} + } + })); + + expect(() => searchProvider.getSearchReader()).toThrow(searchProvider.SearchProviderContractError); + expect(searchProvider.getSearchReader()).toBeDefined(); + expect(mockSearchReader).toHaveBeenCalledTimes(2); + }); + + test('aborts an active OpenSearch request when the lifecycle signal aborts', async () => { + const pending = deferred(); + const request = pending.promise; + request.abort = jest.fn(); + mockClientSearch.mockReturnValue(request); + const controller = new AbortController(); + const search = searchProvider.searchOpenSearch( + { + query: { + match_all: {} + } + }, + { + signal: controller.signal, + operation: 'standard' + } + ); + await Promise.resolve(); + + controller.abort(new searchDeadline.SearchDeadlineExceededError()); + + await expect(search).rejects.toMatchObject({ + code: 'SEARCH_TIMEOUT' + }); + expect(request.abort).toHaveBeenCalledTimes(1); + expect(searchProvider.getOpenSearchRequestState()).toStrictEqual({ + activeCount: 0, + activeByOperation: {} + }); + }); + + test('tracks concurrent OpenSearch requests by operation type', async () => { + const candidate = deferred(); + const pageFetch = deferred(); + const candidateStart = jest.fn(); + const pageFetchStart = jest.fn(); + mockClientSearch.mockReturnValueOnce(candidate.promise).mockReturnValueOnce(pageFetch.promise); + + const candidateSearch = searchProvider.searchOpenSearch( + {}, + { + operation: 'cpe-candidate', + onStart: candidateStart + } + ); + const pageSearch = searchProvider.searchOpenSearch( + {}, + { + operation: 'page-fetch', + onStart: pageFetchStart + } + ); + + expect(candidateStart).toHaveBeenCalledWith({ + activeCount: 1, + activeByOperation: { + 'cpe-candidate': 1 + } + }); + expect(pageFetchStart).toHaveBeenCalledWith({ + activeCount: 2, + activeByOperation: { + 'cpe-candidate': 1, + 'page-fetch': 1 + } + }); + + candidate.resolve({ body: { hits: { hits: [] } } }); + pageFetch.resolve({ body: { hits: { hits: [] } } }); + await Promise.all([candidateSearch, pageSearch]); + + expect(searchProvider.getOpenSearchRequestState()).toStrictEqual({ + activeCount: 0, + activeByOperation: {} + }); + }); + + test('does not start an OpenSearch request for an already-aborted signal', async () => { + const controller = new AbortController(); + controller.abort(new searchDeadline.SearchClientDisconnectedError()); + + await expect( + searchProvider.searchOpenSearch( + { + query: { + match_all: {} + } + }, + { + signal: controller.signal + } + ) + ).rejects.toMatchObject({ + code: 'SEARCH_CLIENT_DISCONNECTED' + }); + expect(mockSearchReader).not.toHaveBeenCalled(); + expect(mockClientSearch).not.toHaveBeenCalled(); }); test('throws when OpenSearch endpoint or index are missing', () => { @@ -222,7 +408,8 @@ describe('searchProvider', () => { 'search.postFilterColdSearchMaxQueueDepth', 'search.postFilterColdSearchMaxWaitMs', 'search.openSearchRequestTimeoutMs', - 'search.requestDeadlineMs' + 'search.requestDeadlineMs', + 'search.readinessTimeoutMs' ].includes(key) ); mockAppConfigGet.mockImplementation((key) => { @@ -233,7 +420,8 @@ describe('searchProvider', () => { 'search.postFilterColdSearchMaxQueueDepth': '50', 'search.postFilterColdSearchMaxWaitMs': '15000', 'search.openSearchRequestTimeoutMs': '25000', - 'search.requestDeadlineMs': '45000' + 'search.requestDeadlineMs': '45000', + 'search.readinessTimeoutMs': '7000' }; return values[key]; }); @@ -245,6 +433,7 @@ describe('searchProvider', () => { expect(searchProvider.getPostFilterColdSearchMaxWaitMs()).toBe(15000); expect(searchProvider.getOpenSearchRequestTimeoutMs()).toBe(25000); expect(searchProvider.getSearchRequestDeadlineMs()).toBe(45000); + expect(searchProvider.getOpenSearchReadinessTimeoutMs()).toBe(7000); }); test('falls back to CPE post-filter runtime defaults when config values are invalid', () => { @@ -258,5 +447,6 @@ describe('searchProvider', () => { expect(searchProvider.getPostFilterColdSearchMaxWaitMs()).toBe(10000); expect(searchProvider.getOpenSearchRequestTimeoutMs()).toBe(20000); expect(searchProvider.getSearchRequestDeadlineMs()).toBe(30000); + expect(searchProvider.getOpenSearchReadinessTimeoutMs()).toBe(5000); }); }); diff --git a/utils/searchQueryBuilder.js b/utils/searchQueryBuilder.js index b2e1534..83104ce 100644 --- a/utils/searchQueryBuilder.js +++ b/utils/searchQueryBuilder.js @@ -10,6 +10,7 @@ const modDatePath = 'cveMetadata.dateUpdated'; const cpeCandidateSourceFields = [ 'cveMetadata.cveId', 'containers.cna.cpeApplicability.nodes.cpeMatch.criteria', + 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionStartIncluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionStartExcluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionEndIncluding', @@ -97,6 +98,7 @@ function buildPostFilters(params) { const virtualMatchStringRangePostFilter = getVirtualMatchStringRangePostFilter(params); if (cpeNamePostFilter) postFilters.cpeName = cpeNamePostFilter; + if (cpeNamePostFilter && isTrueValue(params?.isVulnerable)) postFilters.isVulnerable = true; if (virtualMatchStringRangePostFilter) postFilters.virtualMatchStringRange = virtualMatchStringRangePostFilter; else if (virtualMatchStringPostFilter) postFilters.virtualMatchString = virtualMatchStringPostFilter; @@ -204,6 +206,10 @@ function processParams(params) { return allParams; } +function isTrueValue(value) { + return value === true || String(value).toLowerCase() === 'true'; +} + function addWildCard(searchString) { const terms = String(searchString ?? '') .trim() @@ -293,6 +299,17 @@ function getFilterFields(params) { return; } + if (param === 'isVulnerable') { + if (isTrueValue(params[param])) { + fieldsToReturn.push({ + term: { + [fieldSpec]: true + } + }); + } + return; + } + if (param === 'virtualMatchString') { const filter = buildVirtualMatchStringFilter(params[param]); if (filter) fieldsToReturn.push(filter); diff --git a/utils/searchQueryBuilder.test.js b/utils/searchQueryBuilder.test.js index 4f09cfd..d8f9e31 100644 --- a/utils/searchQueryBuilder.test.js +++ b/utils/searchQueryBuilder.test.js @@ -90,6 +90,36 @@ describe('searchQueryBuilder buildSearchRequest()', () => { expect(output.postFilters.cpeName.version).toBe('2.10'); }); + test('buildSearchRequest preserves true isVulnerable for same-entry semantic filtering', () => { + const output = queryBuilder.buildSearchRequest({ + cpeName: 'cpe:2.3:a:vendor:product:2.10:*:*:*:*:*:*:*', + isVulnerable: true + }); + + expect(output.postFilters.isVulnerable).toBe(true); + expect(output.searchOptions.filters).toContainEqual({ + term: { + 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable': true + } + }); + }); + + test('buildSearchRequest treats false isVulnerable as no restriction', () => { + const output = queryBuilder.buildSearchRequest({ + cpeName: 'cpe:2.3:a:vendor:product:2.10:*:*:*:*:*:*:*', + isVulnerable: false + }); + + expect(output.postFilters.isVulnerable).toBeUndefined(); + expect(output.searchOptions.filters).not.toContainEqual( + expect.objectContaining({ + term: expect.objectContaining({ + 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable': expect.anything() + }) + }) + ); + }); + test('buildSearchRequest over-fetches virtualMatchString candidates for semantic post-filtering', () => { const output = queryBuilder.buildSearchRequest({ virtualMatchString: 'cpe:2.3:o:microsoft:windows_10_1809:*:*:*:*:*:*:x86:*', @@ -189,6 +219,7 @@ describe('searchQueryBuilder buildSearchRequest()', () => { _source: [ 'cveMetadata.cveId', 'containers.cna.cpeApplicability.nodes.cpeMatch.criteria', + 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionStartIncluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionStartExcluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionEndIncluding', @@ -198,6 +229,41 @@ describe('searchQueryBuilder buildSearchRequest()', () => { ); }); + test('buildOpenSearchBody builds the complete standard search request', () => { + const output = queryBuilder.buildSearchRequest({ + keywordSearch: 'remote code', + cweId: 'CWE-79', + resultsPerPage: 10, + pageNumber: 3 + }); + + expect(queryBuilder.buildOpenSearchBody(output.searchOptions, output.queryStringObjects)).toStrictEqual({ + query: { + bool: { + must: [ + { + simple_query_string: { + query: 'remote* code*', + default_operator: 'AND', + fields: ['containers.cna.descriptions.value'] + } + } + ], + filter: [ + { + match_phrase: { + 'containers.cna.problemTypes.descriptions.cweId': 'CWE-79' + } + } + ] + } + }, + track_total_hits: true, + size: 10, + from: 20 + }); + }); + test('buildCveIdsSearchRequest builds exact page fetch request for selected CVE IDs', () => { expect(queryBuilder.buildCveIdsSearchRequest(['CVE-2024-0002', 'CVE-2024-0001'])).toStrictEqual({ searchOptions: {