diff --git a/.env.example b/.env.example index a90f4b1..823f158 100644 --- a/.env.example +++ b/.env.example @@ -8,6 +8,8 @@ SearchPostFilterCacheMaxEntries=100 SearchPostFilterColdSearchConcurrency=10 SearchPostFilterColdSearchMaxQueueDepth=500 SearchPostFilterColdSearchMaxWaitMs=10000 +SearchInFlightMaxWaitersPerOperation=500 +SearchPostFilterEvaluationChunkSize=250 SearchOpenSearchRequestTimeoutMs=20000 SearchRequestDeadlineMs=30000 SearchReadinessTimeoutMs=5000 diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..a6021ba --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +package-lock.json linguist-generated=true +api-docs/openapi.json linguist-generated=true diff --git a/README.md b/README.md index 6201913..1155e76 100644 --- a/README.md +++ b/README.md @@ -111,6 +111,18 @@ String inputs are bounded before query construction: Requests that exceed a limit return HTTP 400 before contacting OpenSearch. These bounds limit input size and query complexity without changing the documented CPE matching semantics. +CVSS vector filters accept full or partial vectors containing at least one unique `KEY:VALUE` metric token. An optional vector header must identify a version supported by the selected parameter. Requested metric tokens can appear in any order, but all tokens must occur in the same stored vector string. CVSS and CWE filters search CNA-provided data only; ADP-provided enrichment is not currently included. + +Publication and last-modified date filters support one-sided ranges. When both values are supplied, `pubStartDate` must be less than or equal to `pubEndDate`, and `lastModStartDate` must be less than or equal to `lastModEndDate`. + +### Search Ordering and Pagination + +`POST /search` uses deterministic ordering so records do not move arbitrarily between offset-based pages. Searches with scored query terms are ordered by relevance descending and then CVE ID ascending. Filter-only searches are ordered by CVE ID ascending. + +`pageSize` reports the number of CVE records actually returned in the current response. `page` is the requested one-based page, except a search with no matches reports page `0`. An accepted page beyond the available matches returns an empty `hits` array and page size `0`. + +Pagination is limited to complete pages inside the first 10,000 results: `from + resultsPerPage` must not exceed 10,000. `totalPages` reports the smaller of the pages implied by the match count and the complete pages accessible inside that window. For example, a page size of 30 allows at most 333 pages. The match total itself is not reduced by this pagination calculation. Point-in-time search and deep pagination remain deferred. + Use `cveIds` to request up to 100 CVE records by comma-separated CVE ID. ```bash @@ -127,11 +139,11 @@ curl -X POST "http://localhost:3000/search" \ -d '{"noRejected":true}' ``` -When filtering with `cpeName`, provide a CPE 2.3 name with non-wildcard `part`, `vendor`, `product`, and `version` components. Invalid `cpeName` values return HTTP 404. The API compares the provided CPE Name against CPE Match Criteria, including exact match strings and version range match strings when the indexed CVE data includes range fields. Use `virtualMatchString` when you intentionally need to search for partial CPE Match String values without those required components. +When filtering with `cpeName`, provide a CPE 2.3 name with part `a`, `h`, or `o` and non-wildcard `vendor`, `product`, and `version` components. Invalid `cpeName` values return HTTP 404. The API compares the provided CPE Name against CPE Match Criteria, including exact match strings and version range match strings when the indexed CVE data includes range fields. Use `virtualMatchString` when you intentionally need to search for partial CPE Match String values without those required components. ### CPE Name Search -Use `cpeName` when a user has a concrete asset CPE Name and needs CVE records whose CPE Match Criteria apply to that asset version. The `part`, `vendor`, `product`, and `version` components must all be concrete values, not `*`. +Use `cpeName` when a user has a concrete asset CPE Name and needs CVE records whose CPE Match Criteria apply to that asset version. The part must be `a`, `h`, or `o`; the `vendor`, `product`, and `version` components must all be concrete values, not `*`. Valid `cpeName` example: @@ -157,7 +169,7 @@ Set `isVulnerable` to `true` with `cpeName` to require the matching CPE Match Cr ### Virtual Match String Search -Use `virtualMatchString` when a user needs CPE Match Criteria compatibility matching instead of concrete asset matching. String literal components are compared without regard to lexical case. Omitted trailing components and `*` components act as broad matches. +Use `virtualMatchString` when a user needs CPE Match Criteria compatibility matching instead of concrete asset matching. String literal components are compared without regard to lexical case. When a part is supplied, it must be `a`, `h`, `o`, `*`, or `-`. Omitted trailing components and `*` components act as broad matches. Partial product example: @@ -247,9 +259,9 @@ Every request receives an `X-Request-ID` response header. A caller-provided `X-R Access logs omit query strings and request bodies. By default, the API retains 10% of successful non-health-check access logs, suppresses successful `/api/health-check` and `/api/ready-check` access logs, and retains every HTTP 4xx or 5xx access log. Set `RequestLogSuccessSampleRate` from `0` through `1` to change successful-request sampling. Set `RequestLogHealthChecks=true` to retain successful health-check logs. -The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, queue wait timeout, search timeout, client disconnect, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, cancellation reason, and queue state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included. +The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, shared-operation waiter saturation, queue wait timeout, search timeout, client disconnect, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, cancellation reason, and bounded queue or waiter state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included. -Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable. +Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch operation types, maximum observed active OpenSearch requests in the process, shared waiter counts, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable. Each `/search` request owns a lifecycle cancellation signal. When its overall deadline expires, the API aborts the active OpenSearch transport request before returning HTTP 504. When the caller disconnects, the API aborts work that is needed only by that caller and does not attempt to write a response. This prevents timed-out or abandoned standard searches from continuing until the transport timeout. @@ -277,13 +289,17 @@ The cache is intentionally small and short-lived: - Cold-search concurrency: cold CPE candidate queries are limited to 10 per process before reaching OpenSearch - Cold-search queue depth: up to 500 unique cold searches can wait per process behind the concurrency limit - Cold-search queue wait: up to 10 seconds before a waiting search returns HTTP 429 +- Shared operation waiters: up to 500 callers can wait on one candidate or page-fetch operation per process +- Post-filter evaluation chunk: 250 candidates between cancellation checks and event-loop yields - OpenSearch request timeout: 20 seconds for each operation - Overall `/search` deadline: 30 seconds - Eviction: oldest entries are removed when the cache exceeds its max size -When the cold-search queue is full, new unique CPE post-filter searches fail closed with HTTP 429 and a `SEARCH_QUEUE_FULL` response. Identical requests for a search that is already in flight still wait on the shared in-flight result and do not consume additional queue slots. +The admission queue applies only to cold CPE candidate searches. Standard `/search` requests and page fetches are not placed in this queue. When the cold-search queue is full, new unique CPE post-filter searches fail closed with HTTP 429 and a `SEARCH_QUEUE_FULL` response. Identical requests for a search that is already in flight still wait on the shared in-flight result and do not consume additional queue slots, up to the per-operation waiter limit. Additional callers fail closed with HTTP 429 and `SEARCH_IN_FLIGHT_WAITER_LIMIT`. -When a cold search waits in the queue for more than 10 seconds, it fails closed with HTTP 429 and a `SEARCH_QUEUE_TIMEOUT` response. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed 20 seconds. The same response is returned when a `/search` request exceeds the 30-second overall processing deadline, and the active OpenSearch request is aborted. +When a cold search waits in the queue for more than 10 seconds, it fails closed with HTTP 429 and a `SEARCH_QUEUE_TIMEOUT` response. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed 20 seconds. Standard searches use the shorter of that timeout and the request's remaining overall deadline. Shared candidate and page-fetch operations retain the independent OpenSearch timeout so one caller's remaining time does not shorten work still needed by other callers. Every caller retains its own 30-second overall deadline. + +Semantic filtering runs in bounded chunks and yields to the Node.js event loop between chunks. This lets disconnect and deadline cancellation signals interrupt a large candidate evaluation instead of waiting for the full set to finish synchronously. Healthy dev timing logs showed the impact for repeated identical `virtualMatchString` range searches: @@ -298,9 +314,11 @@ Healthy dev timing logs showed the impact for repeated identical `virtualMatchSt | Avg candidates scanned | 10000 | cached from prior result | | Avg filtered CVEs | 4 | 4 | -This helps most when multiple users submit the same expensive CPE search within the TTL. The first request is a cold cache miss; later identical requests on the same running task are warm cache hits. If identical cold requests arrive concurrently before the first one finishes, the later requests wait on the same candidate query instead of starting duplicate 10,000-candidate OpenSearch requests. Disconnecting one waiter does not cancel work still needed by another waiter; the shared operation is aborted when its final waiter disconnects or times out. Each active request still fetches its requested page of full CVE records after the shared candidate result is available. +This helps most when multiple users submit the same expensive CPE search within the TTL. The first request is a cold cache miss; later equivalent requests on the same running task are warm cache hits. Equivalent requests share keys across JSON property order, CVE ID order and duplicates, CVSS metric token order, and case differences in case-insensitive CPE values. If identical cold requests arrive concurrently before the first one finishes, the later requests wait on the same candidate query instead of starting duplicate 10,000-candidate OpenSearch requests. Disconnecting one waiter does not cancel work still needed by another waiter; the shared operation is aborted when its final waiter disconnects or times out. Each active request still fetches its requested page of full CVE records after the shared candidate result is available. + +The cache and cold-search queue are a temporary per-process proof of concept. They do not share state across ECS tasks, so each task warms and admits work independently depending on load balancer routing. Track cache-hit and in-flight-share rates, cold-search queue waits and rejections, cold-versus-warm latency, and OpenSearch time to decide their long-term value. If the benefit is low, remove this layer; if it is material, replace it with an AWS-managed shared cache and admission-control design. -The cache does not share entries across ECS tasks. With multiple tasks, each task warms independently depending on load balancer routing. Steady repeated traffic can warm every active task, but unique one-off searches still pay the cold-cache cost. The request lifecycle and in-flight waiter registry are separate from this result cache, so the cache can later be replaced with Redis or ElastiCache without changing cancellation behavior. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, OpenSearch timeout, and overall request deadline can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchOpenSearchRequestTimeoutMs`, and `SearchRequestDeadlineMs`. +The request lifecycle and in-flight waiter registry are separate from the result cache, so a later shared-cache replacement does not need to change caller cancellation behavior. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, shared waiter limit, evaluation chunk size, OpenSearch timeout, and overall request deadline can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchInFlightMaxWaitersPerOperation`, `SearchPostFilterEvaluationChunkSize`, `SearchOpenSearchRequestTimeoutMs`, and `SearchRequestDeadlineMs`. Docker runtime smoke checks can be run after building the image. The default `Dockerfile` avoids BuildKit `RUN --mount` syntax for AWS builders that do not support it: diff --git a/api-docs/openapi.json b/api-docs/openapi.json index 61279e4..22f7452 100644 --- a/api-docs/openapi.json +++ b/api-docs/openapi.json @@ -33,22 +33,57 @@ "type": "string" } } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/SearchResponse" + } + } } }, "400": { - "description": "Malformed JSON, invalid request body, parameter combination, or version range" + "description": "Malformed JSON or invalid request body, parameter, CVSS vector, date range, or version range", + "content": { + "application/json": { + "schema": { + "oneOf": [ + { + "$ref": "#/components/schemas/ErrorResponse" + }, + { + "$ref": "#/components/schemas/ValidationErrorList" + } + ] + } + } + } }, "404": { - "description": "Invalid cpeName value" + "description": "Invalid cpeName value", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorResponse" + } + } + } }, "415": { - "description": "A non-empty request body did not use the application/json media type with UTF-8 encoding" + "description": "A non-empty request body did not use the application/json media type with UTF-8 encoding", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorResponse" + } + } + } }, "429": { - "description": "Application rate limit exceeded, or the cold CPE search queue is full or exceeded its maximum wait", + "description": "Application rate limit exceeded, the cold CPE search queue is full or exceeded its maximum wait, or a shared semantic search operation reached its waiter limit", "headers": { "Retry-After": { - "description": "Seconds until a rate-limited request can be retried", + "description": "Seconds until a busy or rate-limited request can be retried", "schema": { "type": "integer" } @@ -65,6 +100,23 @@ "type": "string" } } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorResponse" + } + } + } + }, + "502": { + "description": "OpenSearch returned an invalid response or an unexpected upstream error occurred", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorResponse" + } + } } }, "503": { @@ -76,10 +128,24 @@ "type": "integer" } } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorResponse" + } + } } }, "504": { - "description": "Search exceeded its OpenSearch or overall processing deadline" + "description": "Search exceeded its OpenSearch or overall processing deadline", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorResponse" + } + } + } } }, "requestBody": { @@ -327,7 +393,7 @@ "type": "string", "maxLength": 32, "example": "CWE-79", - "description": "Returns CVE records that include the provided Common Weakness Enumeration ID." + "description": "Returns CVE records whose CNA data includes the provided Common Weakness Enumeration ID. ADP-provided CWE data is not searched." }, "cvssV2Severity": { "type": "string", @@ -336,7 +402,7 @@ "MEDIUM", "HIGH" ], - "description": "Returns CVE records that contain the provided CVSS v2 qualitative severity rating." + "description": "Returns CVE records whose CNA data contains the provided CVSS v2 qualitative severity rating. ADP-provided CVSS data is not searched." }, "cvssV3Severity": { "type": "string", @@ -346,7 +412,7 @@ "HIGH", "CRITICAL" ], - "description": "Returns CVE records that contain the provided CVSS v3.0 or v3.1 qualitative severity rating." + "description": "Returns CVE records whose CNA data contains the provided CVSS v3.0 or v3.1 qualitative severity rating. ADP-provided CVSS data is not searched." }, "cvssV3_0Severity": { "type": "string", @@ -356,7 +422,7 @@ "HIGH", "CRITICAL" ], - "description": "Returns CVE records that contain the provided CVSS v3.0 qualitative severity rating." + "description": "Returns CVE records whose CNA data contains the provided CVSS v3.0 qualitative severity rating. ADP-provided CVSS data is not searched." }, "cvssV3_1Severity": { "type": "string", @@ -366,7 +432,7 @@ "HIGH", "CRITICAL" ], - "description": "Returns CVE records that contain the provided CVSS v3.1 qualitative severity rating." + "description": "Returns CVE records whose CNA data contains the provided CVSS v3.1 qualitative severity rating. ADP-provided CVSS data is not searched." }, "cvssV4Severity": { "type": "string", @@ -376,43 +442,43 @@ "HIGH", "CRITICAL" ], - "description": "Returns CVE records that contain the provided CVSS v4.0 qualitative severity rating." + "description": "Returns CVE records whose CNA data contains the provided CVSS v4.0 qualitative severity rating. ADP-provided CVSS data is not searched." }, "cvssV2Metrics": { "type": "string", "maxLength": 256, "example": "AV:N/AC:L/Au:N/C:P/I:P/A:P", - "description": "Returns CVE records that contain the provided CVSS v2 vector string. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." + "description": "Returns CVE records whose CNA data contains a CVSS v2 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." }, "cvssV3Metrics": { "type": "string", "maxLength": 256, "example": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", - "description": "Returns CVE records that contain the provided CVSS v3.0 or v3.1 vector string. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." + "description": "Returns CVE records whose CNA data contains a CVSS v3.0 or v3.1 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 3.0 or 3.1 and restricts the searched version. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." }, "cvssV3_0Metrics": { "type": "string", "maxLength": 256, "example": "CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", - "description": "Returns CVE records that contain the provided CVSS v3.0 vector string. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." + "description": "Returns CVE records whose CNA data contains a CVSS v3.0 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 3.0. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." }, "cvssV3_1Metrics": { "type": "string", "maxLength": 256, "example": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H", - "description": "Returns CVE records that contain the provided CVSS v3.1 vector string. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." + "description": "Returns CVE records whose CNA data contains a CVSS v3.1 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 3.1. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." }, "cvssV4Metrics": { "type": "string", "maxLength": 256, "example": "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:H/VA:H/SC:N/SI:N/SA:N", - "description": "Returns CVE records that contain the provided CVSS v4.0 vector string. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." + "description": "Returns CVE records whose CNA data contains a CVSS v4.0 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 4.0. Values are limited to 256 characters, 32 slash-delimited metric tokens, and 32 characters per metric token." }, "cpeName": { "type": "string", "maxLength": 2048, "example": "cpe:2.3:a:freetype:freetype:2.8.1:*:*:*:*:*:*:*", - "description": "Returns CVE records whose CPE Match Criteria apply to the provided CPE 2.3 asset name. CPE string literal comparisons are insensitive to lexical case. The part, vendor, product, and version components must contain values other than *. Every concrete non-version component must be compatible with the same CPE Match Criteria entry that satisfies the version and vulnerability checks; optional * components remain broad matches. The concrete version can match stored criteria with the same version or criteria with a wildcard version and version range fields. Numeric version segments in ranges ignore leading zeros; exact CPE Match Criteria preserve leading zeros. Version qualifiers use Maven-style ordering, with prerelease qualifiers before an unqualified release and sp or unknown qualifiers after it. The version - represents the CPE logical value NA; numeric range boundaries on wildcard-version criteria do not exclude it. Invalid cpeName values return HTTP 404. Use virtualMatchString for partial CPE Match String searches." + "description": "Returns CVE records whose CPE Match Criteria apply to the provided CPE 2.3 asset name. CPE string literal comparisons are insensitive to lexical case. The part must be a, h, or o; vendor, product, and version must contain values other than *. Every concrete non-version component must be compatible with the same CPE Match Criteria entry that satisfies the version and vulnerability checks; optional * components remain broad matches. The concrete version can match stored criteria with the same version or criteria with a wildcard version and version range fields. Numeric version segments in ranges ignore leading zeros; exact CPE Match Criteria preserve leading zeros. Version qualifiers use Maven-style ordering, with prerelease qualifiers before an unqualified release and sp or unknown qualifiers after it. The version - represents the CPE logical value NA; numeric range boundaries on wildcard-version criteria do not exclude it. Invalid cpeName values return HTTP 404. Use virtualMatchString for partial CPE Match String searches." }, "isVulnerable": { "type": "boolean", @@ -423,25 +489,25 @@ "type": "string", "format": "date-time", "example": "2024-01-01T00:00:00.000Z", - "description": "Returns CVE records with dateUpdated timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ." + "description": "Returns CVE records with dateUpdated timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When lastModEndDate is also provided, this value must be less than or equal to it." }, "lastModEndDate": { "type": "string", "format": "date-time", "example": "2024-12-31T23:59:59.999Z", - "description": "Returns CVE records with dateUpdated timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ." + "description": "Returns CVE records with dateUpdated timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When lastModStartDate is also provided, that value must be less than or equal to this one." }, "pubStartDate": { "type": "string", "format": "date-time", "example": "2024-01-01T00:00:00.000Z", - "description": "Returns CVE records with datePublished timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ." + "description": "Returns CVE records with datePublished timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When pubEndDate is also provided, this value must be less than or equal to it." }, "pubEndDate": { "type": "string", "format": "date-time", "example": "2024-12-31T23:59:59.999Z", - "description": "Returns CVE records with datePublished timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ." + "description": "Returns CVE records with datePublished timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When pubStartDate is also provided, that value must be less than or equal to this one." }, "resultsPerPage": { "type": "integer", @@ -490,9 +556,164 @@ "type": "string", "maxLength": 2048, "example": "cpe:2.3:o:microsoft:windows_10_1809", - "description": "Returns CVE records whose CPE Match Criteria are compatible with the provided CPE 2.3 match string. CPE string literal comparisons are insensitive to lexical case. Omitted trailing components and * components act as broad matches. When used with versionStart or versionEnd, provide a partial CPE 2.3 match string without a version component." + "description": "Returns CVE records whose CPE Match Criteria are compatible with the provided CPE 2.3 match string. CPE string literal comparisons are insensitive to lexical case. When supplied, the part must be a, h, o, *, or -. Omitted trailing components and * components act as broad matches. When used with versionStart or versionEnd, provide a partial CPE 2.3 match string without a version component." + } + } + }, + "SearchHitTotal": { + "type": "object", + "required": [ + "value" + ], + "properties": { + "value": { + "type": "integer", + "minimum": 0, + "description": "Number of records matched by the active search strategy." + }, + "relation": { + "type": "string", + "enum": [ + "eq", + "gte" + ], + "description": "Whether value is exact or a lower bound when supplied by OpenSearch." + } + } + }, + "SearchHit": { + "type": "object", + "additionalProperties": true, + "properties": { + "_index": { + "type": "string" + }, + "_id": { + "type": "string" + }, + "_score": { + "type": "number", + "nullable": true + }, + "_source": { + "type": "object", + "additionalProperties": true, + "description": "The matching CVE record." + }, + "sort": { + "type": "array", + "items": {} + } + } + }, + "SearchHits": { + "type": "object", + "required": [ + "total", + "hits" + ], + "properties": { + "total": { + "$ref": "#/components/schemas/SearchHitTotal" + }, + "max_score": { + "type": "number", + "nullable": true + }, + "hits": { + "type": "array", + "items": { + "$ref": "#/components/schemas/SearchHit" + } + } + } + }, + "SearchData": { + "type": "object", + "required": [ + "hits" + ], + "additionalProperties": true, + "properties": { + "took": { + "type": "integer", + "minimum": 0, + "description": "Combined OpenSearch processing time in milliseconds when available." + }, + "timed_out": { + "type": "boolean" + }, + "_shards": { + "type": "object", + "additionalProperties": true + }, + "hits": { + "$ref": "#/components/schemas/SearchHits" + } + } + }, + "SearchResponse": { + "type": "object", + "required": [ + "page", + "pageSize", + "totalPages", + "status", + "data" + ], + "additionalProperties": false, + "properties": { + "page": { + "type": "integer", + "minimum": 0, + "description": "Requested one-based page number, or 0 when the search has no matches." + }, + "pageSize": { + "type": "integer", + "minimum": 0, + "maximum": 500, + "description": "Number of CVE records returned in this response." + }, + "totalPages": { + "type": "integer", + "minimum": 0, + "description": "Number of pages accessible within the 10000-result search window." + }, + "status": { + "type": "string", + "enum": [ + "ok" + ] + }, + "data": { + "$ref": "#/components/schemas/SearchData" } } + }, + "ErrorResponse": { + "type": "object", + "required": [ + "error", + "message" + ], + "additionalProperties": false, + "properties": { + "error": { + "type": "string", + "description": "Stable machine-readable error code." + }, + "message": { + "type": "string", + "description": "Sanitized explanation of the error." + } + } + }, + "ValidationErrorList": { + "type": "array", + "description": "One or more request validation errors.", + "items": { + "$ref": "#/components/schemas/ErrorResponse" + } } } } diff --git a/api-docs/openapi.test.js b/api-docs/openapi.test.js index 0b49ee4..35da775 100644 --- a/api-docs/openapi.test.js +++ b/api-docs/openapi.test.js @@ -17,6 +17,13 @@ function jsonRequestBodySchema(path, method) { .reduce((value, key) => value[key], openapi); } +function resolveSchemaRef(ref) { + return ref + .replace('#/', '') + .split('/') + .reduce((value, key) => value[key], openapi); +} + describe('OpenAPI contract', () => { test('documents the supported route surface', () => { expect(openapi.paths['/search'].post.operationId).toBe('search'); @@ -78,6 +85,7 @@ describe('OpenAPI contract', () => { expect(schema.properties.isVulnerable.description).toContain('same CPE Match Criteria entry'); expect(schema.properties.isVulnerable.description).toContain('False or omitted'); expect(schema.properties.cpeName.description).toContain('Maven-style ordering'); + expect(schema.properties.cpeName.description).toContain('part must be a, h, or o'); expect(schema.properties.cpeName.description).toContain( 'Every concrete non-version component must be compatible with the same CPE Match Criteria entry' ); @@ -102,7 +110,16 @@ describe('OpenAPI contract', () => { expect(schema.properties.virtualMatchString.description).toContain('compatible'); expect(schema.properties.virtualMatchString.description).toContain('insensitive to lexical case'); expect(schema.properties.virtualMatchString.description).toContain('without a version component'); + expect(schema.properties.virtualMatchString.description).toContain('part must be a, h, o, *, or -'); expect(schema.properties.virtualMatchString.example).toBe('cpe:2.3:o:microsoft:windows_10_1809'); + expect(schema.properties.cweId.description).toContain('CNA data'); + expect(schema.properties.cweId.description).toContain('ADP-provided CWE data is not searched'); + expect(schema.properties.cvssV3Metrics.description).toContain('same stored vector string'); + expect(schema.properties.cvssV3Metrics.description).toContain('CNA data'); + expect(schema.properties.cvssV3Metrics.description).toContain('ADP-provided CVSS data is not searched'); + expect(schema.properties.cvssV3Metrics.description).toContain('at least one unique KEY:VALUE token'); + expect(schema.properties.pubStartDate.description).toContain('less than or equal'); + expect(schema.properties.lastModEndDate.description).toContain('less than or equal'); expect(Object.keys(mediaType.examples)).toStrictEqual([ 'allParameters', 'keywordSearch', @@ -133,7 +150,7 @@ describe('OpenAPI contract', () => { const cpeName = openapi.components.schemas.SearchRequest.properties.cpeName; expect(cpeName.description).toContain('CPE Match Criteria'); - expect(cpeName.description).toContain('part, vendor, product, and version'); + expect(cpeName.description).toContain('part must be a, h, or o'); expect(cpeName.description).toContain('version range fields'); expect(cpeName.description).toContain('insensitive to lexical case'); expect(cpeName.description).toContain('exact CPE Match Criteria preserve leading zeros'); @@ -144,13 +161,17 @@ describe('OpenAPI contract', () => { expect(cpeName.example).toBe('cpe:2.3:a:freetype:freetype:2.8.1:*:*:*:*:*:*:*'); expect(openapi.paths['/search'].post.responses['404'].description).toBe('Invalid cpeName value'); expect(openapi.paths['/search'].post.responses['429'].description).toContain('rate limit'); + expect(openapi.paths['/search'].post.responses['429'].description).toContain('waiter limit'); }); test('documents fail-closed request and dependency responses for POST /search', () => { const responses = openapi.paths['/search'].post.responses; expect(responses['400'].description).toContain('Malformed JSON'); + expect(responses['400'].description).toContain('CVSS vector'); + expect(responses['400'].description).toContain('date range'); expect(responses['415'].description).toContain('application/json'); + expect(responses['502'].description).toContain('invalid response'); expect(responses['503'].description).toContain('OpenSearch'); expect(responses['503'].headers['Retry-After']).toStrictEqual({ description: 'Seconds before retrying an unavailable OpenSearch dependency', @@ -160,6 +181,34 @@ describe('OpenAPI contract', () => { }); }); + test('documents the complete POST /search success and error response shapes', () => { + const responses = openapi.paths['/search'].post.responses; + const successRef = responses['200'].content['application/json'].schema.$ref; + const successSchema = resolveSchemaRef(successRef); + + expect(successRef).toBe('#/components/schemas/SearchResponse'); + expect(successSchema.required).toStrictEqual(['page', 'pageSize', 'totalPages', 'status', 'data']); + expect(successSchema.properties.pageSize.description).toContain('returned'); + expect(successSchema.properties.totalPages.description).toContain(`${constants.MAX_SEARCH_WINDOW}`); + expect(successSchema.properties.status.enum).toStrictEqual(['ok']); + expect(openapi.components.schemas.SearchHits.required).toStrictEqual(['total', 'hits']); + + expect(responses['400'].content['application/json'].schema.oneOf).toStrictEqual([ + { + $ref: '#/components/schemas/ErrorResponse' + }, + { + $ref: '#/components/schemas/ValidationErrorList' + } + ]); + + ['404', '415', '429', '502', '503', '504'].forEach((status) => { + expect(responses[status].content['application/json'].schema).toStrictEqual({ + $ref: '#/components/schemas/ErrorResponse' + }); + }); + }); + test('documents request correlation headers', () => { expect(openapi.paths['/search'].post.responses['200'].headers['X-Request-ID']).toStrictEqual({ description: 'Request correlation ID', diff --git a/config/custom-environment-variables.jsonc b/config/custom-environment-variables.jsonc index 3470a97..8898e50 100644 --- a/config/custom-environment-variables.jsonc +++ b/config/custom-environment-variables.jsonc @@ -17,6 +17,8 @@ "postFilterColdSearchConcurrency": "SearchPostFilterColdSearchConcurrency", "postFilterColdSearchMaxQueueDepth": "SearchPostFilterColdSearchMaxQueueDepth", "postFilterColdSearchMaxWaitMs": "SearchPostFilterColdSearchMaxWaitMs", + "inFlightMaxWaitersPerOperation": "SearchInFlightMaxWaitersPerOperation", + "postFilterEvaluationChunkSize": "SearchPostFilterEvaluationChunkSize", "openSearchRequestTimeoutMs": "SearchOpenSearchRequestTimeoutMs", "requestDeadlineMs": "SearchRequestDeadlineMs", "readinessTimeoutMs": "SearchReadinessTimeoutMs" diff --git a/config/default.jsonc b/config/default.jsonc index 8cdef78..58208d9 100644 --- a/config/default.jsonc +++ b/config/default.jsonc @@ -25,6 +25,10 @@ "postFilterColdSearchMaxQueueDepth": "500", // Limits how long a cold CPE candidate search can wait in the per-process queue. "postFilterColdSearchMaxWaitMs": "10000", + // Limits callers sharing one candidate or page-fetch operation per Node.js process. + "inFlightMaxWaitersPerOperation": "500", + // Allows cancellation checks between batches of semantic CPE candidate evaluations. + "postFilterEvaluationChunkSize": "250", // Limits each request from the API process to OpenSearch. "openSearchRequestTimeoutMs": "20000", // Limits total processing time for a /search request. diff --git a/config/runtime-config.test.js b/config/runtime-config.test.js index 350a6b1..d07d258 100644 --- a/config/runtime-config.test.js +++ b/config/runtime-config.test.js @@ -98,6 +98,12 @@ describe('runtime container and compose configuration', () => { expect(customEnvironmentVariables).toContain( '"postFilterColdSearchMaxWaitMs": "SearchPostFilterColdSearchMaxWaitMs"' ); + expect(customEnvironmentVariables).toContain( + '"inFlightMaxWaitersPerOperation": "SearchInFlightMaxWaitersPerOperation"' + ); + expect(customEnvironmentVariables).toContain( + '"postFilterEvaluationChunkSize": "SearchPostFilterEvaluationChunkSize"' + ); expect(customEnvironmentVariables).toContain( '"openSearchRequestTimeoutMs": "SearchOpenSearchRequestTimeoutMs"' ); diff --git a/controllers/middleware.js b/controllers/middleware.js index 8eedf02..524d052 100644 --- a/controllers/middleware.js +++ b/controllers/middleware.js @@ -1,6 +1,7 @@ const errors = require('../utils/errors'); const constants = require('../utils/constants'); const cpe = require('../utils/cpe'); +const cvss = require('../utils/cvss'); const error = new errors.errors(); const cvssV2SeverityEnum = ['LOW', 'MEDIUM', 'HIGH']; @@ -38,6 +39,10 @@ const severityValidators = [ const cvssMetricsParams = ['cvssV2Metrics', 'cvssV3Metrics', 'cvssV3_0Metrics', 'cvssV3_1Metrics', 'cvssV4Metrics']; const dateParams = ['lastModStartDate', 'lastModEndDate', 'pubStartDate', 'pubEndDate']; +const dateRangePairs = [ + ['lastModStartDate', 'lastModEndDate'], + ['pubStartDate', 'pubEndDate'] +]; const versionRangeParams = ['versionStart', 'versionStartType', 'versionEnd', 'versionEndType']; const versionRangeValueParams = ['versionStart', 'versionEnd']; const versionRangeTypeParams = ['versionStartType', 'versionEndType']; @@ -71,25 +76,6 @@ function hasAnyParam(params, paramList) { return paramList.some((param) => hasParam(params, param)); } -function hasInvalidCvssMetricsShape(value) { - const input = String(value ?? ''); - if (input.length > constants.MAX_CVSS_METRICS_LENGTH) return true; - - const tokens = input - .split('/') - .map((t) => t.trim()) - .filter(Boolean) - .filter((t) => { - const up = t.toUpperCase(); - return !up.startsWith('CVSS:') && !up.startsWith('CVSS#'); - }); - - return ( - tokens.length > constants.MAX_CVSS_METRIC_TOKENS || - tokens.some((t) => t.length > constants.MAX_CVSS_METRIC_TOKEN_LENGTH) - ); -} - function countWhitespaceDelimitedTokens(value) { return value.trim().split(/\s+/).length; } @@ -290,8 +276,15 @@ function validateParams(req, res, next) { } for (const param of cvssMetricsParams) { - if (params?.[param] && hasInvalidCvssMetricsShape(params[param])) { - return res.status(400).json(error.cvssMetricsLimitError(param)); + if (!params?.[param]) continue; + + const parsed = cvss.parseCvssMetrics(params[param], param); + if (!parsed.valid) { + const responseBody = + parsed.reason === 'limit' + ? error.cvssMetricsLimitError(param) + : error.cvssMetricsFormatError(param); + return res.status(400).json(responseBody); } } @@ -302,6 +295,16 @@ function validateParams(req, res, next) { } } + for (const [startParam, endParam] of dateRangePairs) { + if ( + params?.[startParam] && + params?.[endParam] && + new Date(params[startParam]).getTime() > new Date(params[endParam]).getTime() + ) { + return res.status(400).json(error.dateRangeOrderError(startParam, endParam)); + } + } + next(); } catch (err) { next(err); diff --git a/controllers/middleware.test.js b/controllers/middleware.test.js index 5079be4..0261e9b 100644 --- a/controllers/middleware.test.js +++ b/controllers/middleware.test.js @@ -93,6 +93,7 @@ describe('middleware validation function', () => { test.each([ ['malformed CPE name', 'not-a-cpe'], + ['unsupported part', 'cpe:2.3:x:vendor:product:1.0:*:*:*:*:*:*:*'], ['wildcard part', 'cpe:2.3:*:vendor:product:1.0:*:*:*:*:*:*:*'], ['wildcard vendor', 'cpe:2.3:a:*:product:1.0:*:*:*:*:*:*:*'], ['wildcard product', 'cpe:2.3:a:vendor:*:1.0:*:*:*:*:*:*:*'], @@ -104,11 +105,19 @@ describe('middleware validation function', () => { expectValidationError( 'INVALID_CPE_NAME', - 'The cpeName parameter must be a CPE 2.3 name with non-wildcard part, vendor, product, and version components.', + 'The cpeName parameter must be a CPE 2.3 name with part a, h, or o and non-wildcard vendor, product, and version components.', 404 ); }); + test('ValidateParams accepts a cpeName with an uppercase valid part', () => { + validate({ + cpeName: 'cpe:2.3:A:vendor:product:1.0:*:*:*:*:*:*:*' + }); + + expectValidationSuccess(); + }); + test('ValidateParams accepts an incomplete cpeName when required components are present', () => { validate({ cpeName: 'cpe:2.3:a:vendor:product:1.0' @@ -802,6 +811,23 @@ describe('middleware validation function', () => { ); }); + test.each([ + ['header-only value', 'cvssV3Metrics', 'CVSS:3.1'], + ['slash-only value', 'cvssV3Metrics', '////'], + ['malformed token', 'cvssV4Metrics', 'AV'], + ['duplicate metric', 'cvssV2Metrics', 'AV:N/av:L'], + ['incompatible version header', 'cvssV3_0Metrics', 'CVSS:3.1/AV:N'] + ])('ValidateParams rejects a CVSS metrics %s.', (_caseName, param, value) => { + validate({ + [param]: value + }); + + expectValidationError( + 'CVSS_METRICS_FORMAT', + `The ${param} parameter must contain at least one unique KEY:VALUE metric token and use a compatible CVSS version header when a header is provided.` + ); + }); + test.each([['lastModStartDate'], ['lastModEndDate'], ['pubStartDate'], ['pubEndDate']])( 'ValidateParams returns an error when an invalid dateTime value is passed for %s.', (dateParam) => { @@ -816,6 +842,33 @@ describe('middleware validation function', () => { } ); + test.each([ + ['lastModStartDate', 'lastModEndDate'], + ['pubStartDate', 'pubEndDate'] + ])('ValidateParams rejects a reversed %s and %s range.', (startParam, endParam) => { + validate({ + [startParam]: '2024-01-02T00:00:00.000Z', + [endParam]: '2024-01-01T00:00:00.000Z' + }); + + expectValidationError( + 'DATE_RANGE_ORDER', + `The ${startParam} parameter must be less than or equal to ${endParam}.` + ); + }); + + test.each([ + ['lastModStartDate', 'lastModEndDate'], + ['pubStartDate', 'pubEndDate'] + ])('ValidateParams accepts equal %s and %s values.', (startParam, endParam) => { + validate({ + [startParam]: '2024-01-01T00:00:00.000Z', + [endParam]: '2024-01-01T00:00:00.000Z' + }); + + expectValidationSuccess(); + }); + test.each([ ['2024'], ['2024-01'], diff --git a/controllers/searchController.js b/controllers/searchController.js index 5deabdb..f7e00e0 100644 --- a/controllers/searchController.js +++ b/controllers/searchController.js @@ -8,6 +8,7 @@ const searchInFlightRegistry = require('../utils/searchInFlightRegistry'); const searchPostFilterWorkQueue = require('../utils/searchPostFilterWorkQueue'); const searchProvider = require('../utils/searchProvider'); const searchQueryBuilder = require('../utils/searchQueryBuilder'); +const constants = require('../utils/constants'); const error = new errors.errors(); const openSearchUnavailableErrorNames = new Set(['ConnectionError', 'NoLivingConnectionsError']); const openSearchUnavailableStatusCodes = new Set([502, 503, 504]); @@ -20,47 +21,86 @@ function getHitsContainer(resp) { return resp?.data?.hits; } -function filterSearchHits(resp, postFilters) { - const hitsContainer = getHitsContainer(resp); - const cpeEvaluationContext = cpe.createEvaluationContext(); +function hitMatchesPostFilters(hit, postFilters, cpeEvaluationContext) { + if ( + postFilters?.cpeName && + !cpe.hitAppliesToCpeName(hit, postFilters.cpeName, cpeEvaluationContext, { + requireVulnerable: postFilters.isVulnerable === true + }) + ) { + return false; + } - if (!Array.isArray(hitsContainer?.hits)) { - return { - hitsContainer, - candidateCount: 0, - filteredHits: [] - }; + if ( + postFilters?.virtualMatchString && + !cpe.hitMatchesVirtualMatchString(hit, postFilters.virtualMatchString, cpeEvaluationContext) + ) { + return false; + } + + if ( + postFilters?.virtualMatchStringRange && + !cpe.hitOverlapsVirtualMatchStringRange(hit, postFilters.virtualMatchStringRange, cpeEvaluationContext) + ) { + return false; } + return true; +} + +function buildEmptyFilteredResult(hitsContainer) { + return { + hitsContainer, + candidateCount: 0, + filteredHits: [] + }; +} + +function filterSearchHits(resp, postFilters) { + const hitsContainer = getHitsContainer(resp); + if (!Array.isArray(hitsContainer?.hits)) return buildEmptyFilteredResult(hitsContainer); + + const cpeEvaluationContext = cpe.createEvaluationContext(); return { hitsContainer, candidateCount: hitsContainer.hits.length, - filteredHits: hitsContainer.hits.filter((hit) => { - if ( - postFilters?.cpeName && - !cpe.hitAppliesToCpeName(hit, postFilters.cpeName, cpeEvaluationContext, { - requireVulnerable: postFilters.isVulnerable === true - }) - ) { - return false; - } + filteredHits: hitsContainer.hits.filter((hit) => hitMatchesPostFilters(hit, postFilters, cpeEvaluationContext)) + }; +} - if ( - postFilters?.virtualMatchString && - !cpe.hitMatchesVirtualMatchString(hit, postFilters.virtualMatchString, cpeEvaluationContext) - ) { - return false; - } +function yieldToEventLoop() { + return new Promise((resolve) => setImmediate(resolve)); +} + +async function filterSearchHitsInChunks(resp, postFilters, signal, chunkSize) { + const hitsContainer = getHitsContainer(resp); + if (!Array.isArray(hitsContainer?.hits)) return buildEmptyFilteredResult(hitsContainer); + + const effectiveChunkSize = Math.max(1, chunkSize ?? searchProvider.getPostFilterEvaluationChunkSize()); + const cpeEvaluationContext = cpe.createEvaluationContext(); + const filteredHits = []; - if ( - postFilters?.virtualMatchStringRange && - !cpe.hitOverlapsVirtualMatchStringRange(hit, postFilters.virtualMatchStringRange, cpeEvaluationContext) - ) { - return false; + for (let start = 0; start < hitsContainer.hits.length; start += effectiveChunkSize) { + searchDeadline.throwIfAborted(signal); + const end = Math.min(start + effectiveChunkSize, hitsContainer.hits.length); + + for (let index = start; index < end; index += 1) { + const hit = hitsContainer.hits[index]; + if (hitMatchesPostFilters(hit, postFilters, cpeEvaluationContext)) { + filteredHits.push(hit); } + } - return true; - }) + if (end < hitsContainer.hits.length) { + await yieldToEventLoop(); + } + } + + searchDeadline.throwIfAborted(signal); + return { + hitsContainer, + candidateCount: hitsContainer.hits.length, + filteredHits }; } @@ -90,6 +130,29 @@ function buildPagedSearchResponse(resp, totalValue, pagedHits) { }; } +function buildPaginationMetadata(resp, pagingOptions) { + const totalValue = resp.data.hits.total.value; + const requestedPageSize = pagingOptions.resultsPerPage; + const returnedHits = resp.data.hits.hits; + + if (totalValue === 0) { + return { + page: 0, + pageSize: 0, + totalPages: 0 + }; + } + + return { + page: Math.floor(pagingOptions.from / requestedPageSize) + 1, + pageSize: returnedHits.length, + totalPages: Math.min( + Math.ceil(totalValue / requestedPageSize), + Math.floor(constants.MAX_SEARCH_WINDOW / requestedPageSize) + ) + }; +} + function buildPostFilteredResponse( resp, filteredHits, @@ -103,6 +166,14 @@ function getOpenSearchErrorStatusCode(err) { return Number(err?.statusCode ?? err?.meta?.statusCode ?? err?.meta?.body?.status); } +function recordUpstreamFailure(metrics, err, failureType) { + const upstreamStatusCode = getOpenSearchErrorStatusCode(err); + if (Number.isInteger(upstreamStatusCode) && upstreamStatusCode > 0) { + metrics.upstreamStatusCode = upstreamStatusCode; + } + metrics.upstreamFailureType = failureType; +} + function isOpenSearchUnavailableError(err) { return ( searchProvider.isSearchProviderContractError?.(err) === true || @@ -180,20 +251,28 @@ function recordOpenSearchStart(metrics, operation, state) { metrics.maxOpenSearchActiveRequests = Math.max(metrics.maxOpenSearchActiveRequests ?? 0, state.activeCount); } -async function executeOpenSearch(body, operation, signal, metrics) { +async function executeOpenSearch(body, operation, signal, metrics, requestTimeoutMs) { return searchProvider.searchOpenSearch(body, { signal, operation, + requestTimeoutMs, onStart: (state) => recordOpenSearchStart(metrics, operation, state) }); } -async function executeStandardSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, signal) { +async function executeStandardSearch( + searchOptions, + queryStringObjects, + postFilters, + responseOptions, + metrics, + lifecycle +) { metrics.strategy = 'standard'; const searchStart = searchPerformance.markTime(); const body = searchQueryBuilder.buildOpenSearchBody(searchOptions, queryStringObjects); - const searchData = await executeOpenSearch(body, 'standard', signal, metrics); + const searchData = await executeOpenSearch(body, 'standard', lifecycle.signal, metrics, lifecycle.getRemainingMs()); const searchResp = { status: 'ok', data: searchData @@ -225,10 +304,14 @@ async function fetchFullHitsByCveIds(cveIds, metrics, signal) { return orderHitsByCveIds(pageData?.hits?.hits ?? [], cveIds); }, - { signal } + { + signal, + maxWaiters: searchProvider.getInFlightMaxWaitersPerOperation() + } ); metrics.pageFetchInFlightHit = inFlight.shared; + metrics.pageFetchInFlightWaiterCount = inFlight.waiterCount; const result = await inFlight.promise; if (inFlight.shared) { metrics.pageFetchWaitElapsedMs = searchPerformance.elapsedMs(waitStart); @@ -247,7 +330,7 @@ async function buildPostFilterCandidateResult(candidateBody, postFilters, metric data: candidateData }; const postFilterStart = searchPerformance.markTime(); - const { candidateCount, filteredHits } = filterSearchHits(candidateResp, postFilters); + const { candidateCount, filteredHits } = await filterSearchHitsInChunks(candidateResp, postFilters, signal); metrics.postFilterElapsedMs = searchPerformance.elapsedMs(postFilterStart); return { @@ -285,10 +368,14 @@ async function getPostFilterCandidateResult(cacheKey, candidateBody, postFilters searchPostFilterCache.set(cacheKey, result); return result; }), - { signal } + { + signal, + maxWaiters: searchProvider.getInFlightMaxWaitersPerOperation() + } ); metrics.inFlightHit = inFlight.shared; + metrics.inFlightWaiterCount = inFlight.waiterCount; const result = await inFlight.promise; if (inFlight.shared) { metrics.inFlightWaitElapsedMs = searchPerformance.elapsedMs(waitStart); @@ -318,6 +405,7 @@ async function executeOptimizedPostFilterSearch( const fullPageHits = await fetchFullHitsByCveIds(pageCveIds, metrics, signal); const response = { + status: 'ok', data: { took: combineTook(metrics.openSearchTookMs, metrics.pageFetchTookMs), hits: { @@ -332,9 +420,16 @@ async function executeOptimizedPostFilterSearch( return buildPagedSearchResponse(response, metrics.filteredCount, fullPageHits); } -async function executeSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, signal) { +async function executeSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, lifecycle) { if (!hasSearchPostFilters(postFilters)) { - return executeStandardSearch(searchOptions, queryStringObjects, postFilters, responseOptions, metrics, signal); + return executeStandardSearch( + searchOptions, + queryStringObjects, + postFilters, + responseOptions, + metrics, + lifecycle + ); } return executeOptimizedPostFilterSearch( @@ -343,7 +438,7 @@ async function executeSearch(searchOptions, queryStringObjects, postFilters, res postFilters, responseOptions, metrics, - signal + lifecycle.signal ); } @@ -374,24 +469,17 @@ async function searchHandler(req, res, next) { postFilters, responseOptions, metrics, - lifecycle.signal + lifecycle ); lifecycle.throwIfAborted(); const pagingOptions = responseOptions ?? searchOptions; metrics.outcome = 'success'; metrics.statusCode = 200; - //Calculate current page and total pages to append to JSON output - const totalValue = resp.data.hits.total.value; - const requestedPageSize = pagingOptions.resultsPerPage; - const pageSize = Math.min(totalValue, requestedPageSize); - const page = totalValue === 0 ? 0 : Math.floor(pagingOptions.from / requestedPageSize) + 1; - const totalPages = totalValue === 0 ? 0 : Math.ceil(totalValue / requestedPageSize); + const pagination = buildPaginationMetadata(resp, pagingOptions); return res.status(200).json({ - page, - pageSize, - totalPages, + ...pagination, ...resp }); } catch (err) { @@ -421,7 +509,44 @@ async function searchHandler(req, res, next) { return res.status(429).json(error.searchQueueTimeoutError()); } + if (searchInFlightRegistry.isWaiterLimitError(err)) { + metrics.outcome = 'in_flight_waiter_limit'; + metrics.statusCode = 429; + metrics.inFlightState = err.state; + if (typeof res.set === 'function') res.set('Retry-After', '5'); + + return res.status(429).json(error.searchInFlightWaiterLimitError()); + } + + if (searchProvider.isOpenSearchResponseError?.(err)) { + const failureType = err?.failureType ?? 'malformed_response'; + recordUpstreamFailure(metrics, err, failureType); + + if (failureType === 'timeout') { + metrics.outcome = 'search_timeout'; + metrics.statusCode = 504; + metrics.timeoutSource = 'opensearch'; + return res.status(504).json(error.searchTimeoutError()); + } + + metrics.outcome = 'upstream_invalid_response'; + metrics.statusCode = 502; + return res.status(502).json(error.upstreamSearchError()); + } + + if (getOpenSearchErrorStatusCode(err) === 429) { + recordUpstreamFailure(metrics, err, 'throttled'); + metrics.outcome = 'opensearch_throttled'; + metrics.statusCode = 503; + if (typeof res.set === 'function') res.set('Retry-After', '5'); + + return res.status(503).json(error.openSearchError()); + } + if (searchDeadline.isSearchTimeoutError(err)) { + if (err?.name === 'TimeoutError') { + recordUpstreamFailure(metrics, err, 'timeout'); + } metrics.outcome = 'search_timeout'; metrics.statusCode = 504; metrics.timeoutSource = err?.name === 'TimeoutError' ? 'opensearch' : 'request_deadline'; @@ -430,6 +555,11 @@ async function searchHandler(req, res, next) { } if (isOpenSearchUnavailableError(err)) { + recordUpstreamFailure( + metrics, + err, + searchProvider.isSearchProviderContractError?.(err) ? 'provider_contract' : 'unavailable' + ); metrics.outcome = 'opensearch_unavailable'; metrics.statusCode = 503; if (typeof res.set === 'function') res.set('Retry-After', '5'); @@ -437,6 +567,7 @@ async function searchHandler(req, res, next) { return res.status(503).json(error.openSearchError()); } + recordUpstreamFailure(metrics, err, 'upstream_error'); metrics.outcome = 'upstream_error'; metrics.statusCode = 502; return res.status(502).json(error.upstreamSearchError()); @@ -512,6 +643,7 @@ async function webSearchHandler(req, res, next) { module.exports = { applySearchPostFilters, buildWebSearchSort, + filterSearchHitsInChunks, SEARCH: searchHandler, WEB_SEARCH: webSearchHandler }; diff --git a/controllers/searchController.test.js b/controllers/searchController.test.js index 2a116f5..69fb4f2 100644 --- a/controllers/searchController.test.js +++ b/controllers/searchController.test.js @@ -21,7 +21,10 @@ const mockGetPostFilterCacheMaxEntries = jest.fn(() => 100); const mockGetPostFilterColdSearchConcurrency = jest.fn(() => 10); const mockGetPostFilterColdSearchMaxQueueDepth = jest.fn(() => 500); const mockGetPostFilterColdSearchMaxWaitMs = jest.fn(() => 10000); +const mockGetInFlightMaxWaitersPerOperation = jest.fn(() => 500); +const mockGetPostFilterEvaluationChunkSize = jest.fn(() => 250); const mockGetSearchRequestDeadlineMs = jest.fn(() => 30000); +const mockIsOpenSearchResponseError = jest.fn((err) => err?.code === 'OPEN_SEARCH_RESPONSE_INVALID'); const mockIsSearchProviderContractError = jest.fn((err) => err?.code === 'SEARCH_PROVIDER_CONTRACT'); const mockGetBasicSearchManager = jest.fn(() => ({ search: mockBasicSearch @@ -42,7 +45,10 @@ jest.mock('../utils/searchProvider', () => ({ getPostFilterColdSearchConcurrency: mockGetPostFilterColdSearchConcurrency, getPostFilterColdSearchMaxQueueDepth: mockGetPostFilterColdSearchMaxQueueDepth, getPostFilterColdSearchMaxWaitMs: mockGetPostFilterColdSearchMaxWaitMs, + getInFlightMaxWaitersPerOperation: mockGetInFlightMaxWaitersPerOperation, + getPostFilterEvaluationChunkSize: mockGetPostFilterEvaluationChunkSize, getSearchRequestDeadlineMs: mockGetSearchRequestDeadlineMs, + isOpenSearchResponseError: mockIsOpenSearchResponseError, isSearchProviderContractError: mockIsSearchProviderContractError, searchOpenSearch: mockSearchOpenSearch, shouldLogSearchPerformance: mockShouldLogSearchPerformance @@ -78,6 +84,23 @@ function rawResponse(totalValue, hits = []) { return advancedResponse(totalValue, hits).data; } +function cvssRegexp(field, value) { + return { + regexp: { + [`${field}.keyword`]: { + value, + case_insensitive: true, + flags: 'INTERSECTION|ANYSTRING', + rewrite: 'constant_score' + } + } + }; +} + +function cvssValue(...tokens) { + return tokens.map((token) => `(@/)?${token}(/@)?`).join('&'); +} + function basicResponse({ ok = true, status = 'ok', total = 0, hits = [], notes = [], errors = [] } = {}) { return { status, @@ -157,13 +180,18 @@ describe('Search controller SEARCH handler', () => { mockGetPostFilterColdSearchConcurrency.mockClear(); mockGetPostFilterColdSearchMaxQueueDepth.mockClear(); mockGetPostFilterColdSearchMaxWaitMs.mockClear(); + mockGetInFlightMaxWaitersPerOperation.mockClear(); + mockGetPostFilterEvaluationChunkSize.mockClear(); mockGetSearchRequestDeadlineMs.mockClear(); + mockIsOpenSearchResponseError.mockClear(); mockIsSearchProviderContractError.mockClear(); mockGetPostFilterCacheTtlMs.mockReturnValue(60000); mockGetPostFilterCacheMaxEntries.mockReturnValue(100); mockGetPostFilterColdSearchConcurrency.mockReturnValue(10); mockGetPostFilterColdSearchMaxQueueDepth.mockReturnValue(500); mockGetPostFilterColdSearchMaxWaitMs.mockReturnValue(10000); + mockGetInFlightMaxWaitersPerOperation.mockReturnValue(500); + mockGetPostFilterEvaluationChunkSize.mockReturnValue(250); mockGetSearchRequestDeadlineMs.mockReturnValue(30000); searchPostFilterCache.clear(); searchInFlightRegistry.clear(); @@ -237,6 +265,18 @@ describe('Search controller SEARCH handler', () => { ] } }, + sort: [ + { + _score: { + order: 'desc' + } + }, + { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } + } + ], track_total_hits: true, size: 10, from: 20 @@ -251,7 +291,7 @@ describe('Search controller SEARCH handler', () => { expect(res.status).toHaveBeenCalledWith(200); expect(res.json).toHaveBeenCalledWith({ page: 3, - pageSize: 10, + pageSize: 1, totalPages: 3, ...resp }); @@ -531,6 +571,13 @@ describe('Search controller SEARCH handler', () => { ] } }, + sort: [ + { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } + } + ], track_total_hits: true, size: 1, from: 0 @@ -545,6 +592,7 @@ describe('Search controller SEARCH handler', () => { page: 1, pageSize: 1, totalPages: 1, + status: 'ok', data: { hits: { total: { @@ -781,6 +829,7 @@ describe('Search controller SEARCH handler', () => { page: 1, pageSize: 1, totalPages: 1, + status: 'ok', data: { hits: { total: { @@ -846,6 +895,7 @@ describe('Search controller SEARCH handler', () => { page: 1, pageSize: 1, totalPages: 1, + status: 'ok', data: { took: undefined, hits: { @@ -906,6 +956,7 @@ describe('Search controller SEARCH handler', () => { page: 1, pageSize: 1, totalPages: 1, + status: 'ok', data: { took: undefined, hits: { @@ -1011,6 +1062,7 @@ describe('Search controller SEARCH handler', () => { page: 1, pageSize: 1, totalPages: 1, + status: 'ok', data: { hits: { total: { @@ -1022,6 +1074,30 @@ describe('Search controller SEARCH handler', () => { }); }); + test('chunked semantic filtering stops after cancellation between batches', async () => { + const hits = [ + cpeHit('CVE-2024-0001', { + criteria: 'cpe:2.3:o:microsoft:windows_10_1809:-:*:*:*:*:*:x86:*' + }), + cpeHit('CVE-2024-0002', { + criteria: 'cpe:2.3:o:microsoft:windows_10_1809:-:*:*:*:*:*:x86:*' + }) + ]; + const abortController = new AbortController(); + const filtering = controller.filterSearchHitsInChunks( + advancedResponse(2, hits), + { + virtualMatchString: 'cpe:2.3:o:microsoft:windows_10_1809' + }, + abortController.signal, + 1 + ); + + abortController.abort(new Error('cancelled')); + + await expect(filtering).rejects.toThrow('cancelled'); + }); + test('SEARCH returns 429 when the cold semantic post-filter queue is full', async () => { const candidateSearch = deferred(); const firstRes = createResponse(); @@ -1144,6 +1220,53 @@ describe('Search controller SEARCH handler', () => { await firstSearch; }); + test('SEARCH bounds callers sharing one semantic candidate operation', async () => { + const candidateSearch = deferred(); + const firstRes = createResponse(); + const secondRes = createResponse(); + const req = { + query: { + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + resultsPerPage: '1', + pageNumber: '1' + } + }; + + mockGetInFlightMaxWaitersPerOperation.mockReturnValue(1); + mockSearchRaw.mockReturnValue(candidateSearch.promise); + + const firstSearch = controller.SEARCH(req, firstRes, next); + await Promise.resolve(); + await controller.SEARCH( + { + requestId: 'waiter-limit-request', + ...req + }, + secondRes, + next + ); + + expect(secondRes.set).toHaveBeenCalledWith('Retry-After', '5'); + expect(secondRes.status).toHaveBeenCalledWith(429); + expect(secondRes.json).toHaveBeenCalledWith({ + error: 'SEARCH_IN_FLIGHT_WAITER_LIMIT', + message: 'Search service is busy. Please retry later.' + }); + expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({ + event: 'search.performance', + requestId: 'waiter-limit-request', + outcome: 'in_flight_waiter_limit', + statusCode: 429, + inFlightState: { + waiterCount: 1, + maxWaiters: 1 + } + }); + + candidateSearch.resolve(rawResponse(0, [])); + await firstSearch; + }); + test('SEARCH returns sanitized upstream errors when optimized post-filter search fails', async () => { mockSearchRaw.mockRejectedValue(new Error('candidate search failed')); @@ -1167,6 +1290,33 @@ describe('Search controller SEARCH handler', () => { }); }); + test('SEARCH does not cache a partial OpenSearch candidate response', async () => { + mockSearchRaw.mockRejectedValue({ + code: 'OPEN_SEARCH_RESPONSE_INVALID', + failureType: 'partial_shards' + }); + const req = { + query: { + virtualMatchString: 'cpe:2.3:o:microsoft:windows_10_1809', + resultsPerPage: '1', + pageNumber: '1' + } + }; + + await controller.SEARCH(req, res, next); + const secondRes = createResponse(); + await controller.SEARCH(req, secondRes, next); + + expect(mockSearchRaw).toHaveBeenCalledTimes(2); + for (const response of [res, secondRes]) { + expect(response.status).toHaveBeenCalledWith(502); + expect(response.json).toHaveBeenCalledWith({ + error: 'UPSTREAM_SEARCH_ERROR', + message: 'Search backend returned an error. Please try again later.' + }); + } + }); + test('SEARCH logs redacted performance timing when enabled', async () => { mockShouldLogSearchPerformance.mockReturnValue(true); mockApiSearch.mockResolvedValue(advancedResponse(1, [{ _id: 'CVE-2024-0001' }])); @@ -1248,33 +1398,7 @@ describe('Search controller SEARCH handler', () => { ); expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV2_0.vectorString.keyword': { - value: '*AV:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV2_0.vectorString.keyword': { - value: '*AC:L*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV2_0.vectorString.keyword': { - value: '*Au:N*' - } - } - } - ] - } - } + cvssRegexp('containers.cna.metrics.cvssV2_0.vectorString', cvssValue('AC:L', 'Au:N', 'AV:N')) ]); }); @@ -1293,46 +1417,8 @@ describe('Search controller SEARCH handler', () => { ); expect(mockApiSearch.mock.calls[0][0].query.bool.filter).toStrictEqual([ - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV3_0.vectorString.keyword': { - value: '*AV:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV3_0.vectorString.keyword': { - value: '*AC:L*' - } - } - } - ] - } - }, - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV3_1.vectorString.keyword': { - value: '*PR:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV3_1.vectorString.keyword': { - value: '*UI:R*' - } - } - } - ] - } - } + cvssRegexp('containers.cna.metrics.cvssV3_0.vectorString', cvssValue('AC:L', 'AV:N')), + cvssRegexp('containers.cna.metrics.cvssV3_1.vectorString', cvssValue('PR:N', 'UI:R')) ]); }); @@ -1471,7 +1557,7 @@ describe('Search controller SEARCH handler', () => { }); }); - test('SEARCH calculates an out-of-range page from the requested page size', async () => { + test('SEARCH reports zero returned records for an out-of-range page', async () => { const resp = advancedResponse(5); mockApiSearch.mockResolvedValue(resp); @@ -1489,12 +1575,36 @@ describe('Search controller SEARCH handler', () => { expect(res.status).toHaveBeenCalledWith(200); expect(res.json).toHaveBeenCalledWith({ page: 2, - pageSize: 5, + pageSize: 0, totalPages: 1, ...resp }); }); + test('SEARCH limits advertised pages to complete pages inside the search window', async () => { + const resp = advancedResponse(25000, [{ _id: 'CVE-2024-0001' }]); + mockApiSearch.mockResolvedValue(resp); + + await controller.SEARCH( + { + query: { + resultsPerPage: 30, + pageNumber: 333 + } + }, + res, + next + ); + + expect(res.status).toHaveBeenCalledWith(200); + expect(res.json).toHaveBeenCalledWith({ + page: 333, + pageSize: 1, + totalPages: 333, + ...resp + }); + }); + test.each([ ['ConnectionError', { name: 'ConnectionError' }], ['NoLivingConnectionsError', { name: 'NoLivingConnectionsError' }], @@ -1516,8 +1626,7 @@ describe('Search controller SEARCH handler', () => { expect(res.set).toHaveBeenCalledWith('Retry-After', '5'); expect(res.json).toHaveBeenCalledWith({ error: 'OPEN_SEARCH_ERROR', - message: - 'OpenSearch refused connection, please check that OpenSearch is running and accessible then try again.' + message: 'Search backend is temporarily unavailable. Please try again later.' }); expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({ event: 'search.performance', @@ -1527,6 +1636,108 @@ describe('Search controller SEARCH handler', () => { }); }); + test('SEARCH returns a retryable sanitized 503 when OpenSearch throttles the request', async () => { + mockApiSearch.mockRejectedValue({ + name: 'ResponseError', + message: 'sensitive throttling details', + meta: { + statusCode: 429, + body: { + reason: 'sensitive shard details' + } + } + }); + + await controller.SEARCH( + { + requestId: 'throttled-request', + query: {} + }, + res, + next + ); + + expect(res.status).toHaveBeenCalledWith(503); + expect(res.set).toHaveBeenCalledWith('Retry-After', '5'); + expect(res.json).toHaveBeenCalledWith({ + error: 'OPEN_SEARCH_ERROR', + message: 'Search backend is temporarily unavailable. Please try again later.' + }); + const logEntry = console.log.mock.calls[0][0]; + expect(JSON.parse(logEntry)).toMatchObject({ + event: 'search.performance', + requestId: 'throttled-request', + outcome: 'opensearch_throttled', + statusCode: 503, + upstreamStatusCode: 429, + upstreamFailureType: 'throttled' + }); + expect(logEntry).not.toContain('sensitive throttling details'); + expect(logEntry).not.toContain('sensitive shard details'); + }); + + test('SEARCH returns a sanitized 504 for a timed-out OpenSearch response', async () => { + mockApiSearch.mockRejectedValue({ + code: 'OPEN_SEARCH_RESPONSE_INVALID', + failureType: 'timeout' + }); + + await controller.SEARCH( + { + requestId: 'response-timeout-request', + query: {} + }, + res, + next + ); + + expect(res.status).toHaveBeenCalledWith(504); + expect(res.json).toHaveBeenCalledWith({ + error: 'SEARCH_TIMEOUT', + message: 'Search request timed out. Please try again later.' + }); + expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({ + event: 'search.performance', + requestId: 'response-timeout-request', + outcome: 'search_timeout', + statusCode: 504, + timeoutSource: 'opensearch', + upstreamFailureType: 'timeout' + }); + }); + + test.each(['partial_shards', 'malformed_response'])( + 'SEARCH returns a sanitized 502 for an OpenSearch %s response', + async (failureType) => { + mockApiSearch.mockRejectedValue({ + code: 'OPEN_SEARCH_RESPONSE_INVALID', + failureType + }); + + await controller.SEARCH( + { + requestId: `${failureType}-request`, + query: {} + }, + res, + next + ); + + expect(res.status).toHaveBeenCalledWith(502); + expect(res.json).toHaveBeenCalledWith({ + error: 'UPSTREAM_SEARCH_ERROR', + message: 'Search backend returned an error. Please try again later.' + }); + expect(JSON.parse(console.log.mock.calls[0][0])).toMatchObject({ + event: 'search.performance', + requestId: `${failureType}-request`, + outcome: 'upstream_invalid_response', + statusCode: 502, + upstreamFailureType: failureType + }); + } + ); + test('SEARCH returns a sanitized 504 when OpenSearch times out', async () => { mockApiSearch.mockRejectedValue({ name: 'TimeoutError', diff --git a/routes/cpeName.test.js b/routes/cpeName.test.js index 89cc369..4e0d82e 100644 --- a/routes/cpeName.test.js +++ b/routes/cpeName.test.js @@ -35,7 +35,7 @@ describe('POST /search cpeName route behavior', () => { expect(response.body).toStrictEqual({ error: 'INVALID_CPE_NAME', message: - 'The cpeName parameter must be a CPE 2.3 name with non-wildcard part, vendor, product, and version components.' + 'The cpeName parameter must be a CPE 2.3 name with part a, h, or o and non-wildcard vendor, product, and version components.' }); expect(controller.SEARCH).not.toHaveBeenCalled(); }); diff --git a/routes/index.js b/routes/index.js index adca945..658d590 100644 --- a/routes/index.js +++ b/routes/index.js @@ -243,22 +243,57 @@ headers: { type: "string" } } +}, +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/SearchResponse" + } + } } } #swagger.responses[400] = { -description: 'Malformed JSON, invalid request body, parameter combination, or version range', +description: 'Malformed JSON or invalid request body, parameter, CVSS vector, date range, or version range', +content: { + "application/json": { + schema: { + oneOf: [ + { + $ref: "#/components/schemas/ErrorResponse" + }, + { + $ref: "#/components/schemas/ValidationErrorList" + } + ] + } + } +} } #swagger.responses[404] = { description: 'Invalid cpeName value', +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/ErrorResponse" + } + } +} } #swagger.responses[415] = { description: 'A non-empty request body did not use the application/json media type with UTF-8 encoding', +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/ErrorResponse" + } + } +} } #swagger.responses[429] = { -description: 'Application rate limit exceeded, or the cold CPE search queue is full or exceeded its maximum wait', +description: 'Application rate limit exceeded, the cold CPE search queue is full or exceeded its maximum wait, or a shared semantic search operation reached its waiter limit', headers: { "Retry-After": { - description: "Seconds until a rate-limited request can be retried", + description: "Seconds until a busy or rate-limited request can be retried", schema: { type: "integer" } @@ -275,6 +310,23 @@ headers: { type: "string" } } +}, +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/ErrorResponse" + } + } +} +} +#swagger.responses[502] = { +description: 'OpenSearch returned an invalid response or an unexpected upstream error occurred', +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/ErrorResponse" + } + } } } #swagger.responses[503] = { @@ -286,10 +338,24 @@ headers: { type: "integer" } } +}, +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/ErrorResponse" + } + } } } #swagger.responses[504] = { description: 'Search exceeded its OpenSearch or overall processing deadline', +content: { + "application/json": { + schema: { + $ref: "#/components/schemas/ErrorResponse" + } + } +} } */ requireSearchJsonContentType, diff --git a/routes/search.integration.test.js b/routes/search.integration.test.js index 5319044..d4ff51a 100644 --- a/routes/search.integration.test.js +++ b/routes/search.integration.test.js @@ -8,6 +8,8 @@ jest.mock('../utils/searchProvider', () => ({ getBasicSearchManager: jest.fn(() => ({ search: mockBasicSearch })), + getInFlightMaxWaitersPerOperation: jest.fn(() => 500), + getPostFilterEvaluationChunkSize: jest.fn(() => 250), getSearchRequestDeadlineMs: jest.fn(() => 30000), searchOpenSearch: mockSearchOpenSearch, shouldLogSearchPerformance: jest.fn(() => false) @@ -81,6 +83,18 @@ describe('search route integration', () => { filter: [] } }, + sort: [ + { + _score: { + order: 'desc' + } + }, + { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } + } + ], track_total_hits: true, size: 1, from: 0 @@ -94,6 +108,7 @@ describe('search route integration', () => { expect(response.body.page).toBe(1); expect(response.body.pageSize).toBe(1); expect(response.body.totalPages).toBe(1); + expect(response.body.status).toBe('ok'); }); test('POST /search rejects invalid dates before invoking OpenSearch', async () => { @@ -113,6 +128,60 @@ describe('search route integration', () => { expect(mockSearchOpenSearch).not.toHaveBeenCalled(); }); + test('POST /search rejects reversed date ranges before invoking OpenSearch', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { + pubStartDate: '2024-02-01T00:00:00.000Z', + pubEndDate: '2024-01-01T00:00:00.000Z' + } + }); + + expect(response.status).toBe(400); + expect(response.body).toStrictEqual({ + error: 'DATE_RANGE_ORDER', + message: 'The pubStartDate parameter must be less than or equal to pubEndDate.' + }); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + + test('POST /search rejects CVSS headers without metric tokens before invoking OpenSearch', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { + cvssV3Metrics: 'CVSS:3.1' + } + }); + + expect(response.status).toBe(400); + expect(response.body).toStrictEqual({ + error: 'CVSS_METRICS_FORMAT', + message: + 'The cvssV3Metrics parameter must contain at least one unique KEY:VALUE metric token and use a compatible CVSS version header when a header is provided.' + }); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + + test('POST /search rejects unsupported CPE parts before invoking OpenSearch', async () => { + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { + cpeName: 'cpe:2.3:x:vendor:product:1.0:*:*:*:*:*:*:*' + } + }); + + expect(response.status).toBe(404); + expect(response.body).toStrictEqual({ + error: 'INVALID_CPE_NAME', + message: + 'The cpeName parameter must be a CPE 2.3 name with part a, h, or o and non-wildcard vendor, product, and version components.' + }); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + test('POST /search rejects excessive keyword terms before invoking OpenSearch', async () => { const response = await request(createApp(), { method: 'POST', diff --git a/routes/swagger.js b/routes/swagger.js index 1cf65db..ba704cc 100644 --- a/routes/swagger.js +++ b/routes/swagger.js @@ -42,48 +42,51 @@ const searchRequestProperties = { type: 'boolean', default: false }), - cweId: searchRequestProperty('Returns CVE records that include the provided Common Weakness Enumeration ID.', { - type: 'string', - maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.cweId, - example: 'CWE-79' - }), + cweId: searchRequestProperty( + 'Returns CVE records whose CNA data includes the provided Common Weakness Enumeration ID. ADP-provided CWE data is not searched.', + { + type: 'string', + maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.cweId, + example: 'CWE-79' + } + ), cvssV2Severity: searchRequestProperty( - 'Returns CVE records that contain the provided CVSS v2 qualitative severity rating.', + 'Returns CVE records whose CNA data contains the provided CVSS v2 qualitative severity rating. ADP-provided CVSS data is not searched.', { type: 'string', enum: ['LOW', 'MEDIUM', 'HIGH'] } ), cvssV3Severity: searchRequestProperty( - 'Returns CVE records that contain the provided CVSS v3.0 or v3.1 qualitative severity rating.', + 'Returns CVE records whose CNA data contains the provided CVSS v3.0 or v3.1 qualitative severity rating. ADP-provided CVSS data is not searched.', { type: 'string', enum: ['LOW', 'MEDIUM', 'HIGH', 'CRITICAL'] } ), cvssV3_0Severity: searchRequestProperty( - 'Returns CVE records that contain the provided CVSS v3.0 qualitative severity rating.', + 'Returns CVE records whose CNA data contains the provided CVSS v3.0 qualitative severity rating. ADP-provided CVSS data is not searched.', { type: 'string', enum: ['LOW', 'MEDIUM', 'HIGH', 'CRITICAL'] } ), cvssV3_1Severity: searchRequestProperty( - 'Returns CVE records that contain the provided CVSS v3.1 qualitative severity rating.', + 'Returns CVE records whose CNA data contains the provided CVSS v3.1 qualitative severity rating. ADP-provided CVSS data is not searched.', { type: 'string', enum: ['LOW', 'MEDIUM', 'HIGH', 'CRITICAL'] } ), cvssV4Severity: searchRequestProperty( - 'Returns CVE records that contain the provided CVSS v4.0 qualitative severity rating.', + 'Returns CVE records whose CNA data contains the provided CVSS v4.0 qualitative severity rating. ADP-provided CVSS data is not searched.', { type: 'string', enum: ['LOW', 'MEDIUM', 'HIGH', 'CRITICAL'] } ), cvssV2Metrics: searchRequestProperty( - `Returns CVE records that contain the provided CVSS v2 vector string. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, + `Returns CVE records whose CNA data contains a CVSS v2 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, { type: 'string', maxLength: constants.MAX_CVSS_METRICS_LENGTH, @@ -91,7 +94,7 @@ const searchRequestProperties = { } ), cvssV3Metrics: searchRequestProperty( - `Returns CVE records that contain the provided CVSS v3.0 or v3.1 vector string. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, + `Returns CVE records whose CNA data contains a CVSS v3.0 or v3.1 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 3.0 or 3.1 and restricts the searched version. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, { type: 'string', maxLength: constants.MAX_CVSS_METRICS_LENGTH, @@ -99,7 +102,7 @@ const searchRequestProperties = { } ), cvssV3_0Metrics: searchRequestProperty( - `Returns CVE records that contain the provided CVSS v3.0 vector string. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, + `Returns CVE records whose CNA data contains a CVSS v3.0 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 3.0. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, { type: 'string', maxLength: constants.MAX_CVSS_METRICS_LENGTH, @@ -107,7 +110,7 @@ const searchRequestProperties = { } ), cvssV3_1Metrics: searchRequestProperty( - `Returns CVE records that contain the provided CVSS v3.1 vector string. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, + `Returns CVE records whose CNA data contains a CVSS v3.1 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 3.1. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, { type: 'string', maxLength: constants.MAX_CVSS_METRICS_LENGTH, @@ -115,7 +118,7 @@ const searchRequestProperties = { } ), cvssV4Metrics: searchRequestProperty( - `Returns CVE records that contain the provided CVSS v4.0 vector string. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, + `Returns CVE records whose CNA data contains a CVSS v4.0 vector with every provided metric token in the same stored vector string. ADP-provided CVSS data is not searched. Full and partial vectors are accepted, tokens are order independent, and at least one unique KEY:VALUE token is required. When present, the header must specify CVSS 4.0. Values are limited to ${constants.MAX_CVSS_METRICS_LENGTH} characters, ${constants.MAX_CVSS_METRIC_TOKENS} slash-delimited metric tokens, and ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters per metric token.`, { type: 'string', maxLength: constants.MAX_CVSS_METRICS_LENGTH, @@ -123,7 +126,7 @@ const searchRequestProperties = { } ), cpeName: searchRequestProperty( - 'Returns CVE records whose CPE Match Criteria apply to the provided CPE 2.3 asset name. CPE string literal comparisons are insensitive to lexical case. The part, vendor, product, and version components must contain values other than *. Every concrete non-version component must be compatible with the same CPE Match Criteria entry that satisfies the version and vulnerability checks; optional * components remain broad matches. The concrete version can match stored criteria with the same version or criteria with a wildcard version and version range fields. Numeric version segments in ranges ignore leading zeros; exact CPE Match Criteria preserve leading zeros. Version qualifiers use Maven-style ordering, with prerelease qualifiers before an unqualified release and sp or unknown qualifiers after it. The version - represents the CPE logical value NA; numeric range boundaries on wildcard-version criteria do not exclude it. Invalid cpeName values return HTTP 404. Use virtualMatchString for partial CPE Match String searches.', + 'Returns CVE records whose CPE Match Criteria apply to the provided CPE 2.3 asset name. CPE string literal comparisons are insensitive to lexical case. The part must be a, h, or o; vendor, product, and version must contain values other than *. Every concrete non-version component must be compatible with the same CPE Match Criteria entry that satisfies the version and vulnerability checks; optional * components remain broad matches. The concrete version can match stored criteria with the same version or criteria with a wildcard version and version range fields. Numeric version segments in ranges ignore leading zeros; exact CPE Match Criteria preserve leading zeros. Version qualifiers use Maven-style ordering, with prerelease qualifiers before an unqualified release and sp or unknown qualifiers after it. The version - represents the CPE logical value NA; numeric range boundaries on wildcard-version criteria do not exclude it. Invalid cpeName values return HTTP 404. Use virtualMatchString for partial CPE Match String searches.', { type: 'string', maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.cpeName, @@ -138,7 +141,7 @@ const searchRequestProperties = { } ), lastModStartDate: searchRequestProperty( - 'Returns CVE records with dateUpdated timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ.', + 'Returns CVE records with dateUpdated timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When lastModEndDate is also provided, this value must be less than or equal to it.', { type: 'string', format: 'date-time', @@ -146,7 +149,7 @@ const searchRequestProperties = { } ), lastModEndDate: searchRequestProperty( - 'Returns CVE records with dateUpdated timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ.', + 'Returns CVE records with dateUpdated timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When lastModStartDate is also provided, that value must be less than or equal to this one.', { type: 'string', format: 'date-time', @@ -154,7 +157,7 @@ const searchRequestProperties = { } ), pubStartDate: searchRequestProperty( - 'Returns CVE records with datePublished timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ.', + 'Returns CVE records with datePublished timestamps on or after this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When pubEndDate is also provided, this value must be less than or equal to it.', { type: 'string', format: 'date-time', @@ -162,7 +165,7 @@ const searchRequestProperties = { } ), pubEndDate: searchRequestProperty( - 'Returns CVE records with datePublished timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ.', + 'Returns CVE records with datePublished timestamps on or before this value. Use the format yyyy-MM-ddTHH:mm:ss.SSSZ. When pubStartDate is also provided, that value must be less than or equal to this one.', { type: 'string', format: 'date-time', @@ -219,7 +222,7 @@ const searchRequestProperties = { } ), virtualMatchString: searchRequestProperty( - 'Returns CVE records whose CPE Match Criteria are compatible with the provided CPE 2.3 match string. CPE string literal comparisons are insensitive to lexical case. Omitted trailing components and * components act as broad matches. When used with versionStart or versionEnd, provide a partial CPE 2.3 match string without a version component.', + 'Returns CVE records whose CPE Match Criteria are compatible with the provided CPE 2.3 match string. CPE string literal comparisons are insensitive to lexical case. When supplied, the part must be a, h, o, *, or -. Omitted trailing components and * components act as broad matches. When used with versionStart or versionEnd, provide a partial CPE 2.3 match string without a version component.', { type: 'string', maxLength: constants.SEARCH_STRING_LENGTH_LIMITS.virtualMatchString, @@ -228,6 +231,143 @@ const searchRequestProperties = { ) }; +const searchResponseSchemas = { + SearchHitTotal: { + type: 'object', + required: ['value'], + properties: { + value: { + type: 'integer', + minimum: 0, + description: 'Number of records matched by the active search strategy.' + }, + relation: { + type: 'string', + enum: ['eq', 'gte'], + description: 'Whether value is exact or a lower bound when supplied by OpenSearch.' + } + } + }, + SearchHit: { + type: 'object', + additionalProperties: true, + properties: { + _index: { + type: 'string' + }, + _id: { + type: 'string' + }, + _score: { + type: 'number', + nullable: true + }, + _source: { + type: 'object', + additionalProperties: true, + description: 'The matching CVE record.' + }, + sort: { + type: 'array', + items: {} + } + } + }, + SearchHits: { + type: 'object', + required: ['total', 'hits'], + properties: { + total: { + $ref: '#/components/schemas/SearchHitTotal' + }, + max_score: { + type: 'number', + nullable: true + }, + hits: { + type: 'array', + items: { + $ref: '#/components/schemas/SearchHit' + } + } + } + }, + SearchData: { + type: 'object', + required: ['hits'], + additionalProperties: true, + properties: { + took: { + type: 'integer', + minimum: 0, + description: 'Combined OpenSearch processing time in milliseconds when available.' + }, + timed_out: { + type: 'boolean' + }, + _shards: { + type: 'object', + additionalProperties: true + }, + hits: { + $ref: '#/components/schemas/SearchHits' + } + } + }, + SearchResponse: { + type: 'object', + required: ['page', 'pageSize', 'totalPages', 'status', 'data'], + additionalProperties: false, + properties: { + page: { + type: 'integer', + minimum: 0, + description: 'Requested one-based page number, or 0 when the search has no matches.' + }, + pageSize: { + type: 'integer', + minimum: 0, + maximum: constants.MAX_RESULTS_LIMIT, + description: 'Number of CVE records returned in this response.' + }, + totalPages: { + type: 'integer', + minimum: 0, + description: `Number of pages accessible within the ${constants.MAX_SEARCH_WINDOW}-result search window.` + }, + status: { + type: 'string', + enum: ['ok'] + }, + data: { + $ref: '#/components/schemas/SearchData' + } + } + }, + ErrorResponse: { + type: 'object', + required: ['error', 'message'], + additionalProperties: false, + properties: { + error: { + type: 'string', + description: 'Stable machine-readable error code.' + }, + message: { + type: 'string', + description: 'Sanitized explanation of the error.' + } + } + }, + ValidationErrorList: { + type: 'array', + description: 'One or more request validation errors.', + items: { + $ref: '#/components/schemas/ErrorResponse' + } + } +}; + const doc = { openapi: '3.0.2', info: { @@ -251,7 +391,8 @@ const doc = { description: 'Optional filters and pagination controls for a CVE record search.', additionalProperties: false, properties: searchRequestProperties - } + }, + ...searchResponseSchemas } } }; diff --git a/utils/constants.js b/utils/constants.js index 85ae9f3..696bebd 100644 --- a/utils/constants.js +++ b/utils/constants.js @@ -21,6 +21,8 @@ const SEARCH_POST_FILTER_CACHE_MAX_ENTRIES = 100; const SEARCH_POST_FILTER_COLD_SEARCH_CONCURRENCY = 10; const SEARCH_POST_FILTER_COLD_SEARCH_MAX_QUEUE_DEPTH = 500; const SEARCH_POST_FILTER_COLD_SEARCH_MAX_WAIT_MS = 10 * 1000; +const SEARCH_IN_FLIGHT_MAX_WAITERS_PER_OPERATION = 500; +const SEARCH_POST_FILTER_EVALUATION_CHUNK_SIZE = 250; const SEARCH_OPEN_SEARCH_REQUEST_TIMEOUT_MS = 20 * 1000; const SEARCH_REQUEST_DEADLINE_MS = 30 * 1000; const SEARCH_READINESS_TIMEOUT_MS = 5 * 1000; @@ -29,6 +31,13 @@ const rejectedCveStates = ['REJECT', 'REJECTED', 'Rejected']; const rejectedCveStateFields = ['cveMetadata.state', 'containers.cna.x_legacyV4Record.CVE_data_meta.STATE']; const cpeVersionRangeTypes = ['including', 'excluding']; +const cvssVectorVersionFields = Object.freeze({ + '2.0': ['containers.cna.metrics.cvssV2_0.vectorString'], + '3.0': ['containers.cna.metrics.cvssV3_0.vectorString'], + 3.1: ['containers.cna.metrics.cvssV3_1.vectorString'], + '4.0': ['containers.cna.metrics.cvssV4_0.vectorString'] +}); + const cpeMatchFields = { criteria: 'containers.cna.cpeApplicability.nodes.cpeMatch.criteria', vulnerable: 'containers.cna.cpeApplicability.nodes.cpeMatch.vulnerable', @@ -68,15 +77,15 @@ const fields = { noRejected: rejectedCveStateFields, cweId: 'containers.cna.problemTypes.descriptions.cweId', cvssV2Severity: 'containers.cna.metrics.cvssV2_0.baseScore', - cvssV2Metrics: 'containers.cna.metrics.cvssV2_0.vectorString', + cvssV2Metrics: cvssVectorVersionFields['2.0'][0], cvssV3_0Severity: 'containers.cna.metrics.cvssV3_0.baseSeverity', - cvssV3_0Metrics: 'containers.cna.metrics.cvssV3_0.vectorString', + cvssV3_0Metrics: cvssVectorVersionFields['3.0'][0], cvssV3_1Severity: 'containers.cna.metrics.cvssV3_1.baseSeverity', - cvssV3_1Metrics: 'containers.cna.metrics.cvssV3_1.vectorString', + cvssV3_1Metrics: cvssVectorVersionFields['3.1'][0], cvssV3Severity: ['containers.cna.metrics.cvssV3_0.baseSeverity', 'containers.cna.metrics.cvssV3_1.baseSeverity'], - cvssV3Metrics: ['containers.cna.metrics.cvssV3_0.vectorString', 'containers.cna.metrics.cvssV3_1.vectorString'], + cvssV3Metrics: [...cvssVectorVersionFields['3.0'], ...cvssVectorVersionFields['3.1']], cvssV4Severity: 'containers.cna.metrics.cvssV4_0.baseSeverity', - cvssV4Metrics: 'containers.cna.metrics.cvssV4_0.vectorString', + cvssV4Metrics: cvssVectorVersionFields['4.0'][0], cpeName: cpeMatchFields.criteria, isVulnerable: cpeMatchFields.vulnerable, virtualMatchString: cpeMatchFields.criteria @@ -142,6 +151,8 @@ module.exports = { SEARCH_POST_FILTER_COLD_SEARCH_CONCURRENCY, SEARCH_POST_FILTER_COLD_SEARCH_MAX_QUEUE_DEPTH, SEARCH_POST_FILTER_COLD_SEARCH_MAX_WAIT_MS, + SEARCH_IN_FLIGHT_MAX_WAITERS_PER_OPERATION, + SEARCH_POST_FILTER_EVALUATION_CHUNK_SIZE, SEARCH_OPEN_SEARCH_REQUEST_TIMEOUT_MS, SEARCH_REQUEST_DEADLINE_MS, SEARCH_READINESS_TIMEOUT_MS, @@ -149,6 +160,7 @@ module.exports = { REJECTED_CVE_STATES: rejectedCveStates, CPE_VERSION_RANGE_TYPES: cpeVersionRangeTypes, CPE_MATCH_FIELDS: cpeMatchFields, + CVSS_VECTOR_VERSION_FIELDS: cvssVectorVersionFields, FIELDS: fields, SEARCH_CONTROL_PARAMS: searchControlParams, SEARCH_QUERY_PARAMS: searchQueryParams diff --git a/utils/cpe.js b/utils/cpe.js index 7301566..97f0620 100644 --- a/utils/cpe.js +++ b/utils/cpe.js @@ -2,6 +2,7 @@ const CPE_23_PREFIX = ['cpe', '2.3']; const CPE_COMPONENT_COUNT = 11; const CPE_NOT_APPLICABLE = '-'; const REQUIRED_CPE_COMPONENT_INDEXES = [0, 1, 2, 3]; +const VALID_CPE_PARTS = new Set(['a', 'h', 'o']); const VERSION_QUALIFIERS = ['alpha', 'beta', 'milestone', 'rc', 'snapshot', '', 'sp']; const VERSION_QUALIFIER_ALIASES = { a: 'alpha', @@ -27,6 +28,9 @@ function parseCpe23(input) { if (components.length < 4 || components.length > CPE_COMPONENT_COUNT) { return { valid: false, reason: 'component_count' }; } + if (!isValidCpePart(components[0], { allowLogicalValues: true })) { + return { valid: false, reason: 'part' }; + } while (components.length < CPE_COMPONENT_COUNT) { components.push('*'); @@ -72,6 +76,7 @@ function splitOnUnescapedColon(s) { function parseCpeName(input) { const parsed = parseCpe23(input); if (!parsed.valid) return parsed; + if (!isValidCpePart(parsed.part)) return { valid: false, reason: 'part' }; const hasRequiredComponents = REQUIRED_CPE_COMPONENT_INDEXES.every((index) => { const component = parsed.components[index]; @@ -120,6 +125,9 @@ function parseVirtualMatchString(input) { if (rawComponents.length > CPE_COMPONENT_COUNT) { return { valid: false, reason: 'component_count' }; } + if (rawComponents.length && !isValidCpePart(rawComponents[0], { allowLogicalValues: true })) { + return { valid: false, reason: 'part' }; + } const hasVersionComponent = rawComponents.length >= 4; const components = [...rawComponents]; @@ -143,6 +151,13 @@ function isValidCpeName(input) { return parseCpeName(input).valid; } +function isValidCpePart(part, { allowLogicalValues = false } = {}) { + const normalized = String(part ?? '').toLowerCase(); + if (VALID_CPE_PARTS.has(normalized)) return true; + + return allowLogicalValues && ['*', CPE_NOT_APPLICABLE].includes(normalized); +} + function escapeLuceneRegexpLiteral(input) { return String(input ?? '').replace(/([.?+*|{}[\]()"\\#@&<>~])/g, '\\$1'); } diff --git a/utils/cpe.test.js b/utils/cpe.test.js index 916a0d9..caa429e 100644 --- a/utils/cpe.test.js +++ b/utils/cpe.test.js @@ -34,6 +34,7 @@ describe('CPE helpers', () => { test.each([ 'not-a-cpe', 'cpe:2.3:a:vendor:product', + 'cpe:2.3:x:vendor:product:1.0:*:*:*:*:*:*:*', 'cpe:2.3:*:vendor:product:1.0:*:*:*:*:*:*:*', 'cpe:2.3:a:*:product:1.0:*:*:*:*:*:*:*', 'cpe:2.3:a:vendor:*:1.0:*:*:*:*:*:*:*', @@ -44,6 +45,10 @@ describe('CPE helpers', () => { expect(cpe.isValidCpeName(value)).toBe(false); }); + test('parseCpeName accepts a valid uppercase part value', () => { + expect(cpe.parseCpeName('cpe:2.3:A:vendor:product:1.0:*:*:*:*:*:*:*').valid).toBe(true); + }); + test('parseCpeName preserves escaped colons inside required components', () => { const parsed = cpe.parseCpeName('cpe:2.3:a:ven\\:dor:product\\:edition:1.0:*:*:*:*:*:*:*'); @@ -79,6 +84,13 @@ describe('CPE helpers', () => { expect(parsed.components).toStrictEqual(['*', '*', '*', '*', '*', '*', '*', '*', '*', '*', '*']); }); + test('parseVirtualMatchString rejects an unsupported concrete part value', () => { + expect(cpe.parseVirtualMatchString('cpe:2.3:x:vendor:product')).toStrictEqual({ + valid: false, + reason: 'part' + }); + }); + test('splitOnUnescapedColon preserves escaped colons inside CPE components', () => { expect(cpe.splitOnUnescapedColon('a:vendor:product\\:edition:1.0')).toStrictEqual([ 'a', diff --git a/utils/cvss.js b/utils/cvss.js new file mode 100644 index 0000000..e56a291 --- /dev/null +++ b/utils/cvss.js @@ -0,0 +1,77 @@ +const constants = require('./constants'); + +const CVSS_HEADER_PATTERN = /^CVSS[:#](2\.0|3\.0|3\.1|4\.0)$/i; +const CVSS_METRIC_TOKEN_PATTERN = /^([A-Za-z][A-Za-z0-9_]*):([^/:\s]+)$/; +const CVSS_PARAM_VERSIONS = Object.freeze({ + cvssV2Metrics: ['2.0'], + cvssV3Metrics: ['3.0', '3.1'], + cvssV3_0Metrics: ['3.0'], + cvssV3_1Metrics: ['3.1'], + cvssV4Metrics: ['4.0'] +}); + +function invalid(reason) { + return { + valid: false, + reason + }; +} + +function parseCvssMetrics(value, param) { + const input = String(value ?? '').trim(); + if (input.length > constants.MAX_CVSS_METRICS_LENGTH) return invalid('limit'); + + const parts = input.split('/').map((part) => part.trim()); + if (!parts.length || parts.some((part) => !part)) return invalid('format'); + + let version; + const headerMatch = parts[0].match(CVSS_HEADER_PATTERN); + if (headerMatch) { + version = headerMatch[1]; + parts.shift(); + } else if (/^CVSS[:#]/i.test(parts[0])) { + return invalid('format'); + } + + const allowedVersions = CVSS_PARAM_VERSIONS[param]; + if (!allowedVersions || (version && !allowedVersions.includes(version))) return invalid('format'); + if (!parts.length) return invalid('format'); + if (parts.length > constants.MAX_CVSS_METRIC_TOKENS) return invalid('limit'); + if (parts.some((part) => part.length > constants.MAX_CVSS_METRIC_TOKEN_LENGTH)) return invalid('limit'); + + const metricKeys = new Set(); + const tokens = []; + + for (const part of parts) { + if (/^CVSS[:#]/i.test(part)) return invalid('format'); + + const tokenMatch = part.match(CVSS_METRIC_TOKEN_PATTERN); + if (!tokenMatch) return invalid('format'); + + const key = tokenMatch[1].toUpperCase(); + if (metricKeys.has(key)) return invalid('format'); + + metricKeys.add(key); + tokens.push({ + key, + value: tokenMatch[2], + raw: part + }); + } + + tokens.sort((left, right) => { + if (left.key < right.key) return -1; + if (left.key > right.key) return 1; + return 0; + }); + + return { + valid: true, + version, + tokens + }; +} + +module.exports = { + parseCvssMetrics +}; diff --git a/utils/cvss.test.js b/utils/cvss.test.js new file mode 100644 index 0000000..17bdbbc --- /dev/null +++ b/utils/cvss.test.js @@ -0,0 +1,70 @@ +const cvss = require('./cvss'); + +describe('cvss', () => { + test.each([ + ['cvssV2Metrics', 'AV:N/AC:L/Au:N'], + ['cvssV2Metrics', 'CVSS:2.0/AV:N/AC:L'], + ['cvssV3Metrics', 'CVSS:3.0/AV:N/AC:L'], + ['cvssV3Metrics', 'CVSS:3.1/AC:L/AV:N'], + ['cvssV4Metrics', 'CVSS#4.0/AV:N/AC:L'] + ])('parseCvssMetrics accepts %s value %s', (param, value) => { + expect(cvss.parseCvssMetrics(value, param)).toMatchObject({ + valid: true, + tokens: expect.any(Array) + }); + }); + + test.each([ + ['header only', 'cvssV3Metrics', 'CVSS:3.1'], + ['empty slash tokens', 'cvssV3Metrics', '////'], + ['missing metric key', 'cvssV3Metrics', ':N'], + ['missing metric value', 'cvssV3Metrics', 'AV:'], + ['missing metric separator', 'cvssV3Metrics', 'AV'], + ['duplicate metric key', 'cvssV3Metrics', 'AV:N/av:L'], + ['header after a metric', 'cvssV3Metrics', 'AV:N/CVSS:3.1'], + ['unsupported header version', 'cvssV3Metrics', 'CVSS:4.0/AV:N'], + ['wrong parameter version', 'cvssV3_0Metrics', 'CVSS:3.1/AV:N'] + ])('parseCvssMetrics rejects %s', (_caseName, param, value) => { + expect(cvss.parseCvssMetrics(value, param)).toStrictEqual({ + valid: false, + reason: 'format' + }); + }); + + test.each([ + ['overall length', 'A'.repeat(257)], + ['token count', Array.from({ length: 33 }, (_value, index) => `T${index}:A`).join('/')], + ['token length', `AV:${'A'.repeat(33)}`] + ])('parseCvssMetrics enforces the %s limit', (_caseName, value) => { + expect(cvss.parseCvssMetrics(value, 'cvssV3Metrics')).toStrictEqual({ + valid: false, + reason: 'limit' + }); + }); + + test('parseCvssMetrics returns the version and structured tokens', () => { + expect(cvss.parseCvssMetrics('CVSS:3.1/AC:L/AV:N', 'cvssV3Metrics')).toStrictEqual({ + valid: true, + version: '3.1', + tokens: [ + { + key: 'AC', + value: 'L', + raw: 'AC:L' + }, + { + key: 'AV', + value: 'N', + raw: 'AV:N' + } + ] + }); + }); + + test('parseCvssMetrics canonicalizes metric token order', () => { + const first = cvss.parseCvssMetrics('AV:N/AC:L', 'cvssV3Metrics'); + const second = cvss.parseCvssMetrics('AC:L/AV:N', 'cvssV3Metrics'); + + expect(first).toStrictEqual(second); + }); +}); diff --git a/utils/errors.js b/utils/errors.js index 4c126e0..824b804 100644 --- a/utils/errors.js +++ b/utils/errors.js @@ -105,7 +105,7 @@ class errors { invalidCpeNameError() { return makeError( 'INVALID_CPE_NAME', - 'The cpeName parameter must be a CPE 2.3 name with non-wildcard part, vendor, product, and version components.' + 'The cpeName parameter must be a CPE 2.3 name with part a, h, or o and non-wildcard vendor, product, and version components.' ); } @@ -225,6 +225,13 @@ class errors { ); } + cvssMetricsFormatError(param) { + return makeError( + 'CVSS_METRICS_FORMAT', + `The ${param} parameter must contain at least one unique KEY:VALUE metric token and use a compatible CVSS version header when a header is provided.` + ); + } + dateTimeError(dateTime) { return makeError( 'DATE_TIME_FORMAT', @@ -232,6 +239,10 @@ class errors { ); } + dateRangeOrderError(startParam, endParam) { + return makeError('DATE_RANGE_ORDER', `The ${startParam} parameter must be less than or equal to ${endParam}.`); + } + webSearchQueryError() { return makeError('WEB_SEARCH_QUERY_REQUIRED', 'The query field is required and must be a non-empty string.'); } @@ -267,10 +278,7 @@ class errors { } openSearchError() { - return makeError( - 'OPEN_SEARCH_ERROR', - 'OpenSearch refused connection, please check that OpenSearch is running and accessible then try again.' - ); + return makeError('OPEN_SEARCH_ERROR', 'Search backend is temporarily unavailable. Please try again later.'); } searchNotReadyError() { @@ -289,6 +297,10 @@ class errors { return makeError('SEARCH_QUEUE_TIMEOUT', 'Search service is busy. Please retry later.'); } + searchInFlightWaiterLimitError() { + return makeError('SEARCH_IN_FLIGHT_WAITER_LIMIT', 'Search service is busy. Please retry later.'); + } + searchTimeoutError() { return makeError('SEARCH_TIMEOUT', 'Search request timed out. Please try again later.'); } diff --git a/utils/errors.test.js b/utils/errors.test.js index 682af30..3258bf4 100644 --- a/utils/errors.test.js +++ b/utils/errors.test.js @@ -148,7 +148,7 @@ describe('error response helpers', () => { { error: 'INVALID_CPE_NAME', message: - 'The cpeName parameter must be a CPE 2.3 name with non-wildcard part, vendor, product, and version components.' + 'The cpeName parameter must be a CPE 2.3 name with part a, h, or o and non-wildcard vendor, product, and version components.' } ], [ @@ -191,6 +191,14 @@ describe('error response helpers', () => { message: 'Search service is busy. Please retry later.' } ], + [ + 'searchInFlightWaiterLimitError', + [], + { + error: 'SEARCH_IN_FLIGHT_WAITER_LIMIT', + message: 'Search service is busy. Please retry later.' + } + ], [ 'searchTimeoutError', [], @@ -331,6 +339,15 @@ describe('error response helpers', () => { message: `The cvssV3Metrics parameter must be ${constants.MAX_CVSS_METRICS_LENGTH} characters or fewer, contain ${constants.MAX_CVSS_METRIC_TOKENS} or fewer slash-delimited metric tokens, and each metric token must be ${constants.MAX_CVSS_METRIC_TOKEN_LENGTH} characters or fewer.` } ], + [ + 'cvssMetricsFormatError', + ['cvssV3Metrics'], + { + error: 'CVSS_METRICS_FORMAT', + message: + 'The cvssV3Metrics parameter must contain at least one unique KEY:VALUE metric token and use a compatible CVSS version header when a header is provided.' + } + ], [ 'dateTimeError', ['pubStartDate'], @@ -339,6 +356,14 @@ describe('error response helpers', () => { message: "The pubStartDate parameter must match the following format: yyyy-MM-ddTHH:mm:ss.SSS'Z'" } ], + [ + 'dateRangeOrderError', + ['pubStartDate', 'pubEndDate'], + { + error: 'DATE_RANGE_ORDER', + message: 'The pubStartDate parameter must be less than or equal to pubEndDate.' + } + ], [ 'webSearchQueryError', [], @@ -400,8 +425,7 @@ describe('error response helpers', () => { [], { error: 'OPEN_SEARCH_ERROR', - message: - 'OpenSearch refused connection, please check that OpenSearch is running and accessible then try again.' + message: 'Search backend is temporarily unavailable. Please try again later.' } ], [ diff --git a/utils/searchDeadline.js b/utils/searchDeadline.js index ecb33c4..d30efec 100644 --- a/utils/searchDeadline.js +++ b/utils/searchDeadline.js @@ -44,6 +44,7 @@ function addOnceListener(emitter, eventName, listener) { function createSearchRequestLifecycle(req, res, timeoutMs) { const controller = new AbortController(); + const deadlineAt = Date.now() + timeoutMs; let completed = false; const abort = (reason) => { @@ -82,8 +83,22 @@ function createSearchRequestLifecycle(req, res, timeoutMs) { throwIfAborted(controller.signal); } + function getRemainingMs() { + throwIfRequestAborted(); + + const remainingMs = deadlineAt - Date.now(); + if (remainingMs <= 0) { + abort(new SearchDeadlineExceededError()); + throwIfAborted(controller.signal); + } + + return Math.max(1, Math.floor(remainingMs)); + } + return { complete, + deadlineAt, + getRemainingMs, signal: controller.signal, throwIfAborted: throwIfRequestAborted }; diff --git a/utils/searchDeadline.test.js b/utils/searchDeadline.test.js index cafbf3c..53b962c 100644 --- a/utils/searchDeadline.test.js +++ b/utils/searchDeadline.test.js @@ -103,6 +103,24 @@ describe('searchDeadline', () => { expect(() => lifecycle.throwIfAborted()).toThrow(searchDeadline.SearchClientDisconnectedError); }); + test('reports the absolute deadline and remaining request budget', () => { + captureTimers(); + jest.spyOn(Date, 'now').mockReturnValueOnce(1000).mockReturnValueOnce(11000); + const { lifecycle } = createLifecycle(); + + expect(lifecycle.deadlineAt).toBe(31000); + expect(lifecycle.getRemainingMs()).toBe(20000); + }); + + test('fails closed when the remaining request budget has elapsed', () => { + captureTimers(); + jest.spyOn(Date, 'now').mockReturnValueOnce(1000).mockReturnValueOnce(31000); + const { lifecycle } = createLifecycle(); + + expect(() => lifecycle.getRemainingMs()).toThrow(searchDeadline.SearchDeadlineExceededError); + expect(lifecycle.signal.aborted).toBe(true); + }); + test.each([ [new searchDeadline.SearchDeadlineExceededError()], [{ code: 'SEARCH_TIMEOUT' }], diff --git a/utils/searchInFlightRegistry.js b/utils/searchInFlightRegistry.js index 07f4e85..4a20c59 100644 --- a/utils/searchInFlightRegistry.js +++ b/utils/searchInFlightRegistry.js @@ -10,6 +10,15 @@ class SearchOperationAbandonedError extends Error { } } +class SearchInFlightWaiterLimitError extends Error { + constructor(state) { + super('Shared search operation reached its waiter limit.'); + this.name = 'SearchInFlightWaiterLimitError'; + this.code = 'SEARCH_IN_FLIGHT_WAITER_LIMIT'; + this.state = state; + } +} + function buildKey(type, value) { return `${type}:${JSON.stringify(value)}`; } @@ -78,7 +87,7 @@ function waitForEntry(entry, signal) { }); } -function acquire(key, createOperation, { signal } = {}) { +function acquire(key, createOperation, { signal, maxWaiters } = {}) { searchDeadline.throwIfAborted(signal); let entry = entries.get(key); @@ -87,12 +96,24 @@ function acquire(key, createOperation, { signal } = {}) { entry = createEntry(key, createOperation); } + if (Number.isInteger(maxWaiters) && maxWaiters > 0 && entry.waiterCount >= maxWaiters) { + throw new SearchInFlightWaiterLimitError({ + waiterCount: entry.waiterCount, + maxWaiters + }); + } + return { promise: waitForEntry(entry, signal), - shared + shared, + waiterCount: entry.waiterCount }; } +function isWaiterLimitError(err) { + return err instanceof SearchInFlightWaiterLimitError || err?.code === 'SEARCH_IN_FLIGHT_WAITER_LIMIT'; +} + function clear() { for (const entry of entries.values()) { if (!entry.controller.signal.aborted) { @@ -114,5 +135,7 @@ module.exports = { buildKey, clear, getState, + isWaiterLimitError, + SearchInFlightWaiterLimitError, SearchOperationAbandonedError }; diff --git a/utils/searchInFlightRegistry.test.js b/utils/searchInFlightRegistry.test.js index aed7ad9..a5c256d 100644 --- a/utils/searchInFlightRegistry.test.js +++ b/utils/searchInFlightRegistry.test.js @@ -27,7 +27,9 @@ describe('searchInFlightRegistry', () => { await Promise.resolve(); expect(first.shared).toBe(false); + expect(first.waiterCount).toBe(1); expect(second.shared).toBe(true); + expect(second.waiterCount).toBe(2); expect(createOperation).toHaveBeenCalledTimes(1); expect(searchInFlightRegistry.getState()).toStrictEqual({ operationCount: 1, @@ -125,6 +127,41 @@ describe('searchInFlightRegistry', () => { expect(createOperation).not.toHaveBeenCalled(); }); + test('rejects callers beyond the per-operation waiter limit without cancelling active work', async () => { + const operation = deferred(); + const createOperation = jest.fn(() => operation.promise); + const first = searchInFlightRegistry.acquire('same-key', createOperation, { + maxWaiters: 1 + }); + + let limitError; + try { + searchInFlightRegistry.acquire('same-key', createOperation, { + maxWaiters: 1 + }); + } catch (err) { + limitError = err; + } + + expect(limitError).toMatchObject({ + code: 'SEARCH_IN_FLIGHT_WAITER_LIMIT', + state: { + waiterCount: 1, + maxWaiters: 1 + } + }); + expect( + searchInFlightRegistry.isWaiterLimitError(new searchInFlightRegistry.SearchInFlightWaiterLimitError({})) + ).toBe(true); + expect(searchInFlightRegistry.getState()).toStrictEqual({ + operationCount: 1, + waiterCount: 1 + }); + + operation.resolve('result'); + await expect(first.promise).resolves.toBe('result'); + }); + test('buildKey preserves operation type boundaries', () => { expect(searchInFlightRegistry.buildKey('candidate', ['CVE-2024-0001'])).not.toBe( searchInFlightRegistry.buildKey('pageFetch', ['CVE-2024-0001']) diff --git a/utils/searchPerformance.js b/utils/searchPerformance.js index 1240cfc..19c690f 100644 --- a/utils/searchPerformance.js +++ b/utils/searchPerformance.js @@ -33,8 +33,11 @@ function logSearchPerformance(metrics, searchOptions, responseOptions, postFilte statusCode: metrics.statusCode, timeoutSource: metrics.timeoutSource, abortReason: metrics.abortReason, + upstreamStatusCode: metrics.upstreamStatusCode, + upstreamFailureType: metrics.upstreamFailureType, strategy: metrics.strategy, cacheStatus: getCacheStatus(metrics, postFilters), + inFlightState: metrics.inFlightState, coldSearchQueueWaitElapsedMs: roundMs(metrics.coldSearchQueueWaitElapsedMs), coldSearchQueueState: metrics.coldSearchQueueState, totalElapsedMs: roundMs(metrics.totalElapsedMs) @@ -60,8 +63,10 @@ function logSearchPerformance(metrics, searchOptions, responseOptions, postFilte openSearchElapsedMs: roundMs(metrics.openSearchElapsedMs), postFilterElapsedMs: roundMs(metrics.postFilterElapsedMs), inFlightWaitElapsedMs: roundMs(metrics.inFlightWaitElapsedMs), + inFlightWaiterCount: metrics.inFlightWaiterCount, pageFetchElapsedMs: roundMs(metrics.pageFetchElapsedMs), - pageFetchWaitElapsedMs: roundMs(metrics.pageFetchWaitElapsedMs) + pageFetchWaitElapsedMs: roundMs(metrics.pageFetchWaitElapsedMs), + pageFetchInFlightWaiterCount: metrics.pageFetchInFlightWaiterCount }); } diff --git a/utils/searchPerformance.test.js b/utils/searchPerformance.test.js index 5ec42f1..b4d9a75 100644 --- a/utils/searchPerformance.test.js +++ b/utils/searchPerformance.test.js @@ -44,8 +44,10 @@ describe('searchPerformance', () => { maxConcurrent: 4 }, inFlightWaitElapsedMs: 5.678, + inFlightWaiterCount: 4, pageFetchElapsedMs: 10.234, pageFetchWaitElapsedMs: 3.456, + pageFetchInFlightWaiterCount: 2, totalElapsedMs: 150.567 }, { @@ -94,8 +96,10 @@ describe('searchPerformance', () => { maxConcurrent: 4 }, inFlightWaitElapsedMs: 5.68, + inFlightWaiterCount: 4, pageFetchElapsedMs: 10.23, pageFetchWaitElapsedMs: 3.46, + pageFetchInFlightWaiterCount: 2, totalElapsedMs: 150.57 }); }); @@ -132,6 +136,10 @@ describe('searchPerformance', () => { maxQueueDepth: 500, maxWaitMs: 10000 }, + inFlightState: { + waiterCount: 500, + maxWaiters: 500 + }, totalElapsedMs: 10001.456 }, undefined, @@ -150,6 +158,10 @@ describe('searchPerformance', () => { statusCode: 429, strategy: 'cpe-candidate', cacheStatus: 'miss', + inFlightState: { + waiterCount: 500, + maxWaiters: 500 + }, coldSearchQueueWaitElapsedMs: 10000.12, coldSearchQueueState: { activeCount: 10, diff --git a/utils/searchPostFilterCache.js b/utils/searchPostFilterCache.js index f7ed9d8..adfd906 100644 --- a/utils/searchPostFilterCache.js +++ b/utils/searchPostFilterCache.js @@ -1,31 +1,48 @@ const searchProvider = require('./searchProvider'); const entries = new Map(); +const UNORDERED_QUERY_ARRAY_KEYS = new Set(['filter', 'must', 'must_not', 'should']); -function normalizeForKey(value) { - if (Array.isArray(value)) return value.map(normalizeForKey); +function compareNormalizedValues(left, right) { + const leftValue = JSON.stringify(left); + const rightValue = JSON.stringify(right); + + if (leftValue < rightValue) return -1; + if (leftValue > rightValue) return 1; + return 0; +} + +function normalizeForKey(value, { parentKey, lowercaseStrings = false } = {}) { + if (Array.isArray(value)) { + const normalized = value.map((item) => normalizeForKey(item, { lowercaseStrings })); + return UNORDERED_QUERY_ARRAY_KEYS.has(parentKey) ? normalized.sort(compareNormalizedValues) : normalized; + } if (value && typeof value === 'object') { + const caseInsensitiveValue = value.case_insensitive === true; return Object.keys(value) .sort() .reduce((normalized, key) => { if (value[key] !== undefined) { - normalized[key] = normalizeForKey(value[key]); + normalized[key] = normalizeForKey(value[key], { + parentKey: key, + lowercaseStrings: lowercaseStrings || (caseInsensitiveValue && key === 'value') + }); } return normalized; }, {}); } + if (lowercaseStrings && typeof value === 'string') return value.toLowerCase(); + return value; } function buildCacheKey(candidateBody, postFilters) { - return JSON.stringify( - normalizeForKey({ - candidateBody, - postFilters - }) - ); + return JSON.stringify({ + candidateBody: normalizeForKey(candidateBody), + postFilters: normalizeForKey(postFilters, { lowercaseStrings: true }) + }); } function get(cacheKey, now = Date.now()) { diff --git a/utils/searchPostFilterCache.test.js b/utils/searchPostFilterCache.test.js index 77aff79..4fd81c6 100644 --- a/utils/searchPostFilterCache.test.js +++ b/utils/searchPostFilterCache.test.js @@ -46,6 +46,91 @@ describe('searchPostFilterCache', () => { expect(left).toBe(right); }); + test('buildCacheKey is stable for equivalent boolean clause ordering', () => { + const left = searchPostFilterCache.buildCacheKey( + { + query: { + bool: { + filter: [{ term: { first: true } }, { term: { second: true } }] + } + } + }, + {} + ); + const right = searchPostFilterCache.buildCacheKey( + { + query: { + bool: { + filter: [{ term: { second: true } }, { term: { first: true } }] + } + } + }, + {} + ); + + expect(left).toBe(right); + }); + + test('buildCacheKey normalizes case-insensitive CPE values', () => { + const left = searchPostFilterCache.buildCacheKey( + { + query: { + prefix: { + criteria: { + value: 'cpe:2.3:A:Vendor:Product:', + case_insensitive: true + } + } + } + }, + { + virtualMatchString: { + raw: 'cpe:2.3:A:Vendor:Product' + } + } + ); + const right = searchPostFilterCache.buildCacheKey( + { + query: { + prefix: { + criteria: { + value: 'cpe:2.3:a:vendor:product:', + case_insensitive: true + } + } + } + }, + { + virtualMatchString: { + raw: 'cpe:2.3:a:vendor:product' + } + } + ); + + expect(left).toBe(right); + }); + + test('buildCacheKey preserves positional array ordering', () => { + const left = searchPostFilterCache.buildCacheKey( + {}, + { + cpeName: { + components: ['a', 'vendor', 'product'] + } + } + ); + const right = searchPostFilterCache.buildCacheKey( + {}, + { + cpeName: { + components: ['vendor', 'a', 'product'] + } + } + ); + + expect(left).not.toBe(right); + }); + test('get returns cached values until the entry expires', () => { const key = 'cache-key'; const value = { diff --git a/utils/searchProvider.js b/utils/searchProvider.js index 929ec6b..18c0ea2 100644 --- a/utils/searchProvider.js +++ b/utils/searchProvider.js @@ -10,6 +10,21 @@ class SearchProviderContractError extends Error { } } +const OPEN_SEARCH_RESPONSE_FAILURE_TYPES = Object.freeze({ + MALFORMED_RESPONSE: 'malformed_response', + PARTIAL_SHARDS: 'partial_shards', + TIMEOUT: 'timeout' +}); + +class OpenSearchResponseError extends Error { + constructor(failureType) { + super(`OpenSearch returned an invalid ${failureType} response.`); + this.name = 'OpenSearchResponseError'; + this.code = 'OPEN_SEARCH_RESPONSE_INVALID'; + this.failureType = failureType; + } +} + let searchProviderConfig; let basicSearchManager; let searchReader; @@ -104,6 +119,42 @@ function validateSearchReader(reader) { return reader; } +function throwOpenSearchResponseError(failureType) { + throw new OpenSearchResponseError(failureType); +} + +function validateOpenSearchResponse(body) { + if (!body || typeof body !== 'object' || Array.isArray(body)) { + throwOpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.MALFORMED_RESPONSE); + } + + if (body.timed_out === true) { + throwOpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.TIMEOUT); + } + + if (body._shards !== undefined) { + if (!body._shards || typeof body._shards !== 'object' || Array.isArray(body._shards)) { + throwOpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.MALFORMED_RESPONSE); + } + + if (body._shards.failed !== undefined) { + const failedShards = Number(body._shards.failed); + if (!Number.isInteger(failedShards) || failedShards < 0) { + throwOpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.MALFORMED_RESPONSE); + } + if (failedShards > 0) { + throwOpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.PARTIAL_SHARDS); + } + } + } + + if (!Array.isArray(body.hits?.hits)) { + throwOpenSearchResponseError(OPEN_SEARCH_RESPONSE_FAILURE_TYPES.MALFORMED_RESPONSE); + } + + return body; +} + function getSearchReader() { if (!searchReader) { const config = getSearchProviderConfig(); @@ -190,7 +241,18 @@ async function executeOpenSearchRequest(createRequest, { signal, operation, onSt } } -async function searchOpenSearch(body, { signal, operation = 'search', onStart } = {}) { +function getEffectiveOpenSearchRequestTimeoutMs(requestTimeoutMs) { + const configuredTimeoutMs = getOpenSearchRequestTimeoutMs(); + const requestedTimeoutMs = Number(requestTimeoutMs); + + if (!Number.isFinite(requestedTimeoutMs) || requestedTimeoutMs < 1) { + return configuredTimeoutMs; + } + + return Math.min(configuredTimeoutMs, Math.max(1, Math.floor(requestedTimeoutMs))); +} + +async function searchOpenSearch(body, { signal, operation = 'search', onStart, requestTimeoutMs } = {}) { searchDeadline.throwIfAborted(signal); const reader = getSearchReader(); const response = await executeOpenSearchRequest( @@ -198,11 +260,12 @@ async function searchOpenSearch(body, { signal, operation = 'search', onStart } reader._client.search( { index: reader._cveIndex, + allow_partial_search_results: false, body }, { maxRetries: 0, - requestTimeout: getOpenSearchRequestTimeoutMs() + requestTimeout: getEffectiveOpenSearchRequestTimeoutMs(requestTimeoutMs) } ), { @@ -212,17 +275,18 @@ async function searchOpenSearch(body, { signal, operation = 'search', onStart } } ); - return response.body; + return validateOpenSearchResponse(response.body); } async function checkOpenSearchReadiness({ signal } = {}) { searchDeadline.throwIfAborted(signal); const reader = getSearchReader(); - await executeOpenSearchRequest( + const response = await executeOpenSearchRequest( () => reader._client.search( { index: reader._cveIndex, + allow_partial_search_results: false, body: { query: { match_all: {} @@ -242,6 +306,7 @@ async function checkOpenSearchReadiness({ signal } = {}) { operation: 'readiness' } ); + validateOpenSearchResponse(response.body); return true; } @@ -250,6 +315,10 @@ function isSearchProviderContractError(err) { return err instanceof SearchProviderContractError || err?.code === 'SEARCH_PROVIDER_CONTRACT'; } +function isOpenSearchResponseError(err) { + return err instanceof OpenSearchResponseError || err?.code === 'OPEN_SEARCH_RESPONSE_INVALID'; +} + function getOpenSearchReadinessTimeoutMs() { return getIntegerConfig('search.readinessTimeoutMs', constants.SEARCH_READINESS_TIMEOUT_MS, { min: 1 @@ -294,6 +363,22 @@ function getPostFilterColdSearchMaxWaitMs() { ); } +function getInFlightMaxWaitersPerOperation() { + return getIntegerConfig( + 'search.inFlightMaxWaitersPerOperation', + constants.SEARCH_IN_FLIGHT_MAX_WAITERS_PER_OPERATION, + { min: 1 } + ); +} + +function getPostFilterEvaluationChunkSize() { + return getIntegerConfig( + 'search.postFilterEvaluationChunkSize', + constants.SEARCH_POST_FILTER_EVALUATION_CHUNK_SIZE, + { min: 1 } + ); +} + function getOpenSearchRequestTimeoutMs() { return getIntegerConfig('search.openSearchRequestTimeoutMs', constants.SEARCH_OPEN_SEARCH_REQUEST_TIMEOUT_MS, { min: 1 @@ -319,20 +404,26 @@ module.exports = { getIntegerConfig, getOpenSearchRequestState, getOpenSearchReadinessTimeoutMs, + getInFlightMaxWaitersPerOperation, getPostFilterCacheMaxEntries, getPostFilterCacheTtlMs, getPostFilterColdSearchConcurrency, getPostFilterColdSearchMaxQueueDepth, getPostFilterColdSearchMaxWaitMs, + getPostFilterEvaluationChunkSize, getOpenSearchRequestTimeoutMs, getSearchRequestDeadlineMs, getSearchProviderConfig, getSearchReader, + isOpenSearchResponseError, isSearchProviderContractError, normalizeBoolean, resetSearchProviderForTests, searchOpenSearch, shouldLogSearchPerformance, + validateOpenSearchResponse, validateSearchReader, + OPEN_SEARCH_RESPONSE_FAILURE_TYPES, + OpenSearchResponseError, SearchProviderContractError }; diff --git a/utils/searchProvider.test.js b/utils/searchProvider.test.js index c817ad9..8d06bed 100644 --- a/utils/searchProvider.test.js +++ b/utils/searchProvider.test.js @@ -137,6 +137,7 @@ describe('searchProvider', () => { expect(mockClientSearch).toHaveBeenCalledWith( { index: 'test-index', + allow_partial_search_results: false, body }, { @@ -166,6 +167,97 @@ describe('searchProvider', () => { }); }); + test('caps an OpenSearch request timeout to the remaining request budget', async () => { + mockClientSearch.mockResolvedValue({ + body: { + hits: { + hits: [] + } + } + }); + + await searchProvider.searchOpenSearch( + { + query: { + match_all: {} + } + }, + { + requestTimeoutMs: 1500 + } + ); + + expect(mockClientSearch.mock.calls[0][1]).toStrictEqual({ + maxRetries: 0, + requestTimeout: 1500 + }); + }); + + test('does not extend an OpenSearch request beyond the configured timeout', async () => { + mockClientSearch.mockResolvedValue({ + body: { + hits: { + hits: [] + } + } + }); + + await searchProvider.searchOpenSearch( + { + query: { + match_all: {} + } + }, + { + requestTimeoutMs: 30000 + } + ); + + expect(mockClientSearch.mock.calls[0][1]).toStrictEqual({ + maxRetries: 0, + requestTimeout: 20000 + }); + }); + + test.each([ + [ + 'timed-out', + { + timed_out: true, + hits: { + hits: [] + } + }, + 'timeout' + ], + [ + 'partial-shard', + { + _shards: { + failed: 1 + }, + hits: { + hits: [] + } + }, + 'partial_shards' + ], + [ + 'malformed', + { + hits: {} + }, + 'malformed_response' + ] + ])('rejects a %s OpenSearch response', async (_caseName, body, failureType) => { + mockClientSearch.mockResolvedValue({ body }); + + await expect(searchProvider.searchOpenSearch({ query: { match_all: {} } })).rejects.toMatchObject({ + code: 'OPEN_SEARCH_RESPONSE_INVALID', + failureType + }); + }); + test('checks OpenSearch readiness through the configured CVE index', async () => { mockClientSearch.mockResolvedValue({ body: { @@ -180,6 +272,7 @@ describe('searchProvider', () => { expect(mockClientSearch).toHaveBeenCalledWith( { index: 'test-index', + allow_partial_search_results: false, body: { query: { match_all: {} @@ -200,6 +293,24 @@ describe('searchProvider', () => { }); }); + test('fails readiness when OpenSearch reports a partial response', async () => { + mockClientSearch.mockResolvedValue({ + body: { + _shards: { + failed: 1 + }, + hits: { + hits: [] + } + } + }); + + await expect(searchProvider.checkOpenSearchReadiness()).rejects.toMatchObject({ + code: 'OPEN_SEARCH_RESPONSE_INVALID', + failureType: 'partial_shards' + }); + }); + test('fails readiness when the configured CVE index is unavailable', async () => { const unavailable = new Error('index unavailable'); mockClientSearch.mockRejectedValue(unavailable); @@ -407,6 +518,8 @@ describe('searchProvider', () => { 'search.postFilterColdSearchConcurrency', 'search.postFilterColdSearchMaxQueueDepth', 'search.postFilterColdSearchMaxWaitMs', + 'search.inFlightMaxWaitersPerOperation', + 'search.postFilterEvaluationChunkSize', 'search.openSearchRequestTimeoutMs', 'search.requestDeadlineMs', 'search.readinessTimeoutMs' @@ -419,6 +532,8 @@ describe('searchProvider', () => { 'search.postFilterColdSearchConcurrency': '6', 'search.postFilterColdSearchMaxQueueDepth': '50', 'search.postFilterColdSearchMaxWaitMs': '15000', + 'search.inFlightMaxWaitersPerOperation': '125', + 'search.postFilterEvaluationChunkSize': '100', 'search.openSearchRequestTimeoutMs': '25000', 'search.requestDeadlineMs': '45000', 'search.readinessTimeoutMs': '7000' @@ -431,6 +546,8 @@ describe('searchProvider', () => { expect(searchProvider.getPostFilterColdSearchConcurrency()).toBe(6); expect(searchProvider.getPostFilterColdSearchMaxQueueDepth()).toBe(50); expect(searchProvider.getPostFilterColdSearchMaxWaitMs()).toBe(15000); + expect(searchProvider.getInFlightMaxWaitersPerOperation()).toBe(125); + expect(searchProvider.getPostFilterEvaluationChunkSize()).toBe(100); expect(searchProvider.getOpenSearchRequestTimeoutMs()).toBe(25000); expect(searchProvider.getSearchRequestDeadlineMs()).toBe(45000); expect(searchProvider.getOpenSearchReadinessTimeoutMs()).toBe(7000); @@ -445,6 +562,8 @@ describe('searchProvider', () => { expect(searchProvider.getPostFilterColdSearchConcurrency()).toBe(10); expect(searchProvider.getPostFilterColdSearchMaxQueueDepth()).toBe(500); expect(searchProvider.getPostFilterColdSearchMaxWaitMs()).toBe(10000); + expect(searchProvider.getInFlightMaxWaitersPerOperation()).toBe(500); + expect(searchProvider.getPostFilterEvaluationChunkSize()).toBe(250); expect(searchProvider.getOpenSearchRequestTimeoutMs()).toBe(20000); expect(searchProvider.getSearchRequestDeadlineMs()).toBe(30000); expect(searchProvider.getOpenSearchReadinessTimeoutMs()).toBe(5000); diff --git a/utils/searchQueryBuilder.js b/utils/searchQueryBuilder.js index 07894bb..f5c59ec 100644 --- a/utils/searchQueryBuilder.js +++ b/utils/searchQueryBuilder.js @@ -1,5 +1,6 @@ const constants = require('./constants'); const cpe = require('./cpe'); +const cvss = require('./cvss'); const { MAX_RESULTS_LIMIT, RESULTS_PER_PAGE } = constants; @@ -16,6 +17,26 @@ const cpeCandidateSourceFields = [ 'containers.cna.cpeApplicability.nodes.cpeMatch.versionEndIncluding', 'containers.cna.cpeApplicability.nodes.cpeMatch.versionEndExcluding' ]; +const cveIdAscendingSort = { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } +}; + +function buildDeterministicSort(queryStringObjects) { + const sort = []; + + if (queryStringObjects.length) { + sort.push({ + _score: { + order: 'desc' + } + }); + } + + sort.push(cveIdAscendingSort); + return sort; +} function buildSearchRequest(params) { const processedParams = processParams(params); @@ -50,14 +71,16 @@ function buildSearchRequest(params) { } function buildOpenSearchBody(searchOptions, queryStringObjects = [], { sourceIncludes, trackTotalHits = true } = {}) { + const normalizedQueryStringObjects = Array.isArray(queryStringObjects) ? queryStringObjects : []; const queryObj = { - must: Array.isArray(queryStringObjects) ? [...queryStringObjects] : [], + must: [...normalizedQueryStringObjects], filter: [...(searchOptions?.filters ?? []), ...(searchOptions?.rangeObjects ?? [])] }; const body = { query: { bool: queryObj }, + sort: buildDeterministicSort(normalizedQueryStringObjects), track_total_hits: trackTotalHits, size: searchOptions.resultsPerPage, from: searchOptions.from @@ -182,13 +205,17 @@ function buildDateRangeObjects(params) { return rangeObjects; } +function getOrderedSearchParamNames(params) { + return constants.SEARCH_QUERY_PARAMS.filter((param) => Object.prototype.hasOwnProperty.call(params, param)); +} + function processParams(params) { let allParams = { queryStrings: {}, matchPhrases: {} }; - Object.keys(params).forEach((param) => { + getOrderedSearchParamNames(params).forEach((param) => { if ( param === 'keywordSearch' && (!params['keywordExactMatch'] || String(params['keywordExactMatch']).toLowerCase() === 'false') @@ -231,7 +258,7 @@ function escapeQueryStringTerm(term, { escapeWildcards = false } = {}) { function buildQueryStrings(params) { const queryStringObjects = []; - Object.keys(params).forEach((param) => { + getOrderedSearchParamNames(params).forEach((param) => { if (param === 'virtualMatchString') return; const fieldSpec = constants.FIELDS[param]; @@ -274,7 +301,7 @@ function getFilterFields(params) { const cvssSeverityParams = new Set(['cvssV3Severity', 'cvssV3_0Severity', 'cvssV3_1Severity', 'cvssV4Severity']); - Object.keys(params).forEach((param) => { + getOrderedSearchParamNames(params).forEach((param) => { if (!validFields.includes(param)) return; const fieldSpec = constants.FIELDS[param]; @@ -320,14 +347,12 @@ function getFilterFields(params) { const r = cvssV2SeverityToRange(params[param]); if (!r) return; - fieldsToReturn.push({ - range: { [fieldSpec]: r } - }); + fieldsToReturn.push(buildRangeAcrossFields(r, fieldSpec)); return; } if (cvssVectorParams.has(param)) { - const filter = buildCvssWildcardFilter(params[param], fieldSpec); + const filter = buildCvssRegexpFilter(params[param], param, fieldSpec); if (filter) fieldsToReturn.push(filter); return; } @@ -343,8 +368,12 @@ function getFilterFields(params) { return fieldsToReturn; } -function escapeForWildcard(s) { - return String(s).replace(/([\\*?])/g, '\\$1'); +function escapeForRegexp(s) { + return String(s).replace(/([.?+*|{}[\]()"\\#@&<>~])/g, '\\$1'); +} + +function buildCvssTokenRegexp(token) { + return `(@/)?${escapeForRegexp(token)}(/@)?`; } function parseCveIds(value) { @@ -355,7 +384,7 @@ function parseCveIds(value) { } function buildCveIdsFilter(value, field) { - const cveIds = [...new Set(parseCveIds(value))]; + const cveIds = [...new Set(parseCveIds(value))].sort(); if (!cveIds.length) return null; return { @@ -494,29 +523,30 @@ function buildHasAnyRangeFieldFilter(rangeFields) { }; } -function buildCvssWildcardFilter(inputVectorString, fieldSpec) { - const tokens = String(inputVectorString) - .split('/') - .map((t) => t.trim()) - .filter(Boolean) - .filter((t) => { - const up = t.toUpperCase(); - return !up.startsWith('CVSS:') && !up.startsWith('CVSS#'); - }); +function buildCvssRegexpFilter(inputVectorString, param, fieldSpec) { + const parsed = cvss.parseCvssMetrics(inputVectorString, param); + if (!parsed.valid) { + throw new TypeError(`Invalid ${param} value reached query construction.`); + } - if (!tokens.length) return null; + let fields = Array.isArray(fieldSpec) ? fieldSpec : [fieldSpec]; + if (parsed.version) { + const versionFields = constants.CVSS_VECTOR_VERSION_FIELDS[parsed.version] ?? []; + fields = fields.filter((field) => versionFields.includes(field)); + } - const fields = Array.isArray(fieldSpec) ? fieldSpec : [fieldSpec]; + const value = parsed.tokens.map((token) => buildCvssTokenRegexp(token.raw)).join('&'); const perFieldFilter = (fieldPath) => { const keywordField = `${fieldPath}.keyword`; return { - bool: { - filter: tokens.map((t) => ({ - wildcard: { - [keywordField]: { value: `*${escapeForWildcard(t)}*` } - } - })) + regexp: { + [keywordField]: { + value, + case_insensitive: true, + flags: 'INTERSECTION|ANYSTRING', + rewrite: 'constant_score' + } } }; }; @@ -525,12 +555,30 @@ function buildCvssWildcardFilter(inputVectorString, fieldSpec) { return { bool: { - should: fields.map((f) => perFieldFilter(f)), + should: fields.map((field) => perFieldFilter(field)), minimum_should_match: 1 } }; } +function buildRangeAcrossFields(range, fieldSpec) { + const fields = Array.isArray(fieldSpec) ? fieldSpec : [fieldSpec]; + const clauses = fields.map((field) => ({ + range: { + [field]: range + } + })); + + return clauses.length === 1 + ? clauses[0] + : { + bool: { + should: clauses, + minimum_should_match: 1 + } + }; +} + function buildSeverityAcrossFields(value, fieldSpec) { const fields = Array.isArray(fieldSpec) ? fieldSpec : [fieldSpec]; const v = String(value ?? '').toUpperCase(); diff --git a/utils/searchQueryBuilder.test.js b/utils/searchQueryBuilder.test.js index b26f879..e8f4b88 100644 --- a/utils/searchQueryBuilder.test.js +++ b/utils/searchQueryBuilder.test.js @@ -20,6 +20,32 @@ function prefixShould(prefixes) { }; } +function cvssRegexp(field, value) { + return { + regexp: { + [`${field}.keyword`]: { + value, + case_insensitive: true, + flags: 'INTERSECTION|ANYSTRING', + rewrite: 'constant_score' + } + } + }; +} + +function cvssValue(...tokens) { + return tokens.map((token) => `(@/)?${token}(/@)?`).join('&'); +} + +function anyOf(clauses) { + return { + bool: { + should: clauses, + minimum_should_match: 1 + } + }; +} + describe('searchQueryBuilder buildSearchRequest()', () => { test('buildSearchRequest builds filters, query strings, date ranges, and pagination options', () => { const output = queryBuilder.buildSearchRequest({ @@ -75,6 +101,21 @@ describe('searchQueryBuilder buildSearchRequest()', () => { }); }); + test('buildSearchRequest is stable across request property ordering', () => { + const first = queryBuilder.buildSearchRequest({ + keywordSearch: 'remote code', + cveIds: 'CVE-2024-0002,CVE-2024-0001', + cvssV3Metrics: 'AV:N/AC:L' + }); + const second = queryBuilder.buildSearchRequest({ + cvssV3Metrics: 'AC:L/AV:N', + cveIds: 'CVE-2024-0001,CVE-2024-0002', + keywordSearch: 'remote code' + }); + + expect(first).toStrictEqual(second); + }); + test('buildSearchRequest over-fetches bounded CPE candidates and keeps caller pagination for post-filtering', () => { const output = queryBuilder.buildSearchRequest({ cpeName: 'cpe:2.3:a:vendor:product:2.10:*:*:*:*:*:*:*', @@ -215,6 +256,18 @@ describe('searchQueryBuilder buildSearchRequest()', () => { filter: output.searchOptions.filters } }, + sort: [ + { + _score: { + order: 'desc' + } + }, + { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } + } + ], track_total_hits: false, size: 10000, from: 0, @@ -260,19 +313,53 @@ describe('searchQueryBuilder buildSearchRequest()', () => { ] } }, + sort: [ + { + _score: { + order: 'desc' + } + }, + { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } + } + ], track_total_hits: true, size: 10, from: 20 }); }); + test('buildOpenSearchBody orders filter-only results by CVE ID', () => { + expect( + queryBuilder.buildOpenSearchBody( + { + filters: [], + rangeObjects: [], + resultsPerPage: 20, + from: 0 + }, + [] + ) + ).toMatchObject({ + sort: [ + { + 'cveMetadata.cveId.keyword': { + order: 'asc' + } + } + ] + }); + }); + test('buildCveIdsSearchRequest builds exact page fetch request for selected CVE IDs', () => { expect(queryBuilder.buildCveIdsSearchRequest(['CVE-2024-0002', 'CVE-2024-0001'])).toStrictEqual({ searchOptions: { filters: [ { terms: { - 'cveMetadata.cveId.keyword': ['CVE-2024-0002', 'CVE-2024-0001'] + 'cveMetadata.cveId.keyword': ['CVE-2024-0001', 'CVE-2024-0002'] } } ], @@ -360,7 +447,7 @@ describe('searchQueryBuilder getFilterFields() function', () => { test('getFilterFields() builds exact terms filter for comma-separated cveIds.', () => { const output = queryBuilder.getFilterFields({ - cveIds: 'CVE-2024-0001, cve-2024-0002,CVE-2024-0001' + cveIds: 'cve-2024-0002, CVE-2024-0001,CVE-2024-0002' }); expect(output).toStrictEqual([ @@ -426,39 +513,13 @@ describe('searchQueryBuilder getFilterFields() function', () => { ]); }); - test('getFilterFields() builds order-independent wildcard filters for cvssV2 metrics.', () => { + test('getFilterFields() builds a same-vector regexp filter for CNA cvssV2 metrics.', () => { const output = queryBuilder.getFilterFields({ cvssV2Metrics: 'AV:N/AC:L/Au:N' }); expect(output).toStrictEqual([ - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV2_0.vectorString.keyword': { - value: '*AV:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV2_0.vectorString.keyword': { - value: '*AC:L*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV2_0.vectorString.keyword': { - value: '*Au:N*' - } - } - } - ] - } - } + cvssRegexp('containers.cna.metrics.cvssV2_0.vectorString', cvssValue('AC:L', 'Au:N', 'AV:N')) ]); }); @@ -631,96 +692,47 @@ describe('searchQueryBuilder getFilterFields() function', () => { ]); }); - test('getFilterFields() builds order-independent wildcard filters for combined cvssV3 metrics.', () => { + test('getFilterFields() constrains versioned cvssV3 metrics to the matching CNA field.', () => { const output = queryBuilder.getFilterFields({ - cvssV3Metrics: 'CVSS:3.1/AV:N/AC:L' + cvssV3Metrics: 'CVSS:3.1/AC:L/AV:N' }); expect(output).toStrictEqual([ - { - bool: { - should: [ - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV3_0.vectorString.keyword': { - value: '*AV:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV3_0.vectorString.keyword': { - value: '*AC:L*' - } - } - } - ] - } - }, - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV3_1.vectorString.keyword': { - value: '*AV:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV3_1.vectorString.keyword': { - value: '*AC:L*' - } - } - } - ] - } - } - ], - minimum_should_match: 1 - } - } + cvssRegexp('containers.cna.metrics.cvssV3_1.vectorString', cvssValue('AC:L', 'AV:N')) + ]); + }); + + test('getFilterFields() searches both v3 versions when cvssV3Metrics omits a header.', () => { + const output = queryBuilder.getFilterFields({ + cvssV3Metrics: 'AV:N' + }); + + expect(output).toStrictEqual([ + anyOf([ + cvssRegexp('containers.cna.metrics.cvssV3_0.vectorString', cvssValue('AV:N')), + cvssRegexp('containers.cna.metrics.cvssV3_1.vectorString', cvssValue('AV:N')) + ]) ]); }); - test('getFilterFields() escapes wildcard metacharacters in CVSS metric filters.', () => { + test('getFilterFields() escapes regexp metacharacters in CVSS metric filters.', () => { const output = queryBuilder.getFilterFields({ cvssV4Metrics: 'CVSS#4.0/AV:N/AC:*?/UI:P' }); expect(output).toStrictEqual([ - { - bool: { - filter: [ - { - wildcard: { - 'containers.cna.metrics.cvssV4_0.vectorString.keyword': { - value: '*AV:N*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV4_0.vectorString.keyword': { - value: '*AC:\\*\\?*' - } - } - }, - { - wildcard: { - 'containers.cna.metrics.cvssV4_0.vectorString.keyword': { - value: '*UI:P*' - } - } - } - ] - } - } + cvssRegexp('containers.cna.metrics.cvssV4_0.vectorString', cvssValue('AC:\\*\\?', 'AV:N', 'UI:P')) ]); + + expect(JSON.stringify(output)).not.toContain('containers.adp'); + }); + + test('getFilterFields() fails closed when invalid CVSS metrics reach query construction.', () => { + expect(() => + queryBuilder.getFilterFields({ + cvssV3Metrics: 'CVSS:3.1' + }) + ).toThrow('Invalid cvssV3Metrics value reached query construction.'); }); test('getFilterFields() does not build filter objects for incorrect parameters.', () => {