diff --git a/.env.example b/.env.example index a6ac790..cdece5b 100644 --- a/.env.example +++ b/.env.example @@ -5,6 +5,10 @@ OpenSearchDomainEndpoint= SearchOpenSearchRequestTimeoutMs=20000 SearchRequestDeadlineMs=30000 SearchReadinessTimeoutMs=5000 +# PIT traversal is enabled by default; set false to restore bounded semantic searches. +SearchPitEnabled=true +SearchPitBatchSize=1000 +SearchPitKeepAliveMs=60000 RequestLogSuccessSampleRate=0.1 RequestLogHealthChecks=false WebSearchEnabled=true diff --git a/README.md b/README.md index fd53f3c..ca4d56c 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,9 @@ Copy `.env.example` to an untracked `.env` and replace or check the following va - `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000` - `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000` - `SearchReadinessTimeoutMs` - optional timeout for the OpenSearch index readiness check, default `5000` +- `SearchPitEnabled` - semantic PIT traversal, default `true`; set `false` to restore bounded semantic searches +- `SearchPitBatchSize` - PIT candidate batch size, default `1000`; see [PIT Implementation Status](#pit-implementation-status) +- `SearchPitKeepAliveMs` - PIT lifetime in milliseconds, default `60000` - `RequestLogSuccessSampleRate` - optional fraction of successful access logs to retain, default `0.1`; includes health checks when they are enabled for logging - `RequestLogHealthChecks` - optional flag to include successful health checks in access-log sampling, default `false` - `WebSearchEnabled` - optional `/webSearch` availability flag, default `true` @@ -79,7 +82,9 @@ The compose file runs only the Search API with `NODE_ENV=production` and `NODE_C Use Node.js 24 for the repository's current runtime and development tooling. CI and the Docker dependency-build stage use Node.js 24.14.1; the final runtime image uses Distroless Node.js 24. -**Upstream compatibility caveat:** The locked `cve-core` package still declares Node.js `>= 16.20.2 < 22`. That metadata conflicts with this repository's Node.js 24 baseline and can cause installation warnings or failures when strict engine checks are enabled. Resolving the declaration requires coordination with the `cve-core` maintainers; it is not a recommendation to use Node.js 16 or 18 with this repository's current development tooling. +The API depends directly on the official OpenSearch JavaScript client. `utils/searchProvider.js` owns its configuration and reuses one client per process for both endpoints and readiness checks. Query construction remains separate: `utils/searchQueryBuilder.js` implements `/search`, and `utils/webSearchQueryBuilder.js` implements `/webSearch`. + +`utils/appConfig.js` loads `.env` before reading `config/` through node-config. Existing process environment variables take precedence over `.env`; the mappings in `config/custom-environment-variables.jsonc` override deployment-file values. `NODE_CONFIG_ENV` selects a deployment configuration when set, otherwise node-config uses `NODE_ENV`. No credentials or index configuration need to change for this dependency migration. 1. Set `OpenSearchDomainEndpoint` in `.env` to an existing OpenSearch deployment: @@ -111,7 +116,7 @@ The Search API uses `PORT`, defaulting to `http://localhost:3000`. The tracked P `GET /api/health-check` is a process liveness check. It returns HTTP 200 when the Node.js process is running and deliberately does not contact OpenSearch. Use it where a dependency outage should not restart otherwise healthy API tasks. -`GET /api/ready-check` first verifies that the configured CVE index exposes compatible field capabilities for each logical `/search` feature, then sends a minimal zero-result search. CNA, ADP, legacy, and CVSS-version fields that implement the same feature are treated as alternatives, while mapped fields must still use compatible searchable types. This checks mappings, connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 when these checks pass; it does not prove data completeness or capacity for every search. Configuration errors, connection failures, timeouts, incompatible `cve-core` SearchReader contracts, missing logical capabilities, incompatible mapped fields, and a missing index return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. `SearchReadinessTimeoutMs` defaults to five seconds for the overall readiness check, covering both field-capability and search operations. Each individual operation also uses that value as its transport timeout, and the overall deadline can abort it earlier. +`GET /api/ready-check` first verifies that the configured CVE index exposes compatible field capabilities for each logical `/search` feature, then sends a minimal zero-result search. CNA, ADP, legacy, and CVSS-version fields that implement the same feature are treated as alternatives, while mapped fields must still use compatible searchable types. This checks mappings, connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 when these checks pass; it does not prove data completeness or capacity for every search. Configuration errors, connection failures, timeouts, missing logical capabilities, incompatible mapped fields, and a missing index return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. `SearchReadinessTimeoutMs` defaults to five seconds for the overall readiness check, covering both field-capability and search operations. Each individual operation also uses that value as its transport timeout, and the overall deadline can abort it earlier. An AWS load balancer can use `/api/ready-check` to stop routing search traffic to an unready task while the container orchestrator continues using `/api/health-check` for process liveness. Coordinate that probe selection with the infrastructure team before changing deployed health-check behavior. @@ -294,7 +299,11 @@ While enabled, `POST /webSearch` requires a non-empty JSON `query` string. It al - `from` must be an integer greater than or equal to `0` - `size` must be an integer from `1` to `500` -The controller validates successful `BasicSearchManager` responses before returning HTTP 200. A response without the expected status metadata, total-hit count, or hit array is treated as an invalid upstream response and returns a sanitized HTTP 502. +The raw-query builder preserves the original expression and uses `query_string` with an `AND` default operator. Standalone calendar dates such as `2024-01-01` and ranges such as `2024-01-01..2024-01-31` search the supported record/container date fields using inclusive boundaries. Range endpoints can also use year or month precision (`2023..2024`, `2024-01..2024-03`); a standalone year or month remains a text query. Timestamps accept UTC or explicit offsets, and timestamps without a zone are interpreted as UTC. Date-query years are limited to 1900 through 2100. Quoted dates remain literal query expressions. This date syntax is specific to `/webSearch`; `/search` retains its explicit timestamp filters. + +Both endpoints use the shared OpenSearch client and reject incomplete upstream responses. A response with failed shards, a timeout, missing hit data, or an invalid total-hit count cannot produce a successful `/webSearch` response. Backend failures return sanitized HTTP 502 responses; transport timeouts return HTTP 504. + +The `/webSearch` response retains `searchMetadata`, `resultsTotal`, and `data`. Metadata includes `searchStatus`, the response-format `version`, `notes`, `errors`, and `queryTime_ms` when available. The obsolete `cveCore` and `cveMitreCore` package-version fields have been removed. `/search` keeps its existing API-owned response format, warnings, and filter behavior. ## Swagger API docs @@ -325,7 +334,7 @@ npm run coverage npm run quality ``` -An importable Postman collection for manual `/search` contract and filter testing is documented in [test/postman/README.md](test/postman/README.md). The complete collection targets a fully populated index like those used by shared managed environments; data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly. +An importable Postman collection for manual `/search` contract and filter testing is documented in [test/postman/README.md](test/postman/README.md). Set its `expectedSearchPitEnabled` assertion variable to match the server's PIT setting; it defaults to `true` and does not enable PIT. The complete collection targets a fully populated index like those used by shared managed environments; data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly. ## Operational Logging @@ -339,21 +348,59 @@ Deadline checks also compare elapsed monotonic time, so they do not depend on th ## Semantic Post-Filtering -Two separate 10,000-record controls apply to `/search`: +Two separate 10,000-record controls apply to `/search`, depending on the server's PIT setting: - Every requested page must fall within the first 10,000 results. Direct OpenSearch filters still report an exact total that can exceed 10,000, but pages beyond that window are rejected. -- Filters that require semantic post-filtering inspect at most 10,000 search candidates. Their reported total and pagination describe only matches found within that candidate set. +- Only when PIT is disabled, filters that require semantic post-filtering inspect at most 10,000 search candidates. Their reported total and pagination describe only matches found within that candidate set. PIT is enabled by default and removes this candidate cap, not the public pagination limit. -CVSS vector filters, SSVC filters, date-bounded positive `knownExploited` searches, `cpeName`, and `virtualMatchString` searches use semantic post-filtering where OpenSearch cannot guarantee the required same-entry or version-comparison behavior. These searches retrieve up to 10,000 lightweight candidates, apply semantic checks in Node.js, paginate the matching CVE IDs, and fetch the requested page as complete CVE records. Every slash-delimited token supplied to one CVSS vector filter must match a complete token in the same eligible vector; a base token such as `AV:N` does not match a modified token such as `MAV:N`. +CVSS vector filters, SSVC filters, date-bounded positive `knownExploited` searches, `cpeName`, and `virtualMatchString` searches use semantic post-filtering where OpenSearch cannot guarantee the required same-entry or version-comparison behavior. These searches retrieve lightweight candidates, apply semantic checks in Node.js, paginate the matching CVE IDs, and fetch the requested page as complete CVE records. Every slash-delimited token supplied to one CVSS vector filter must match a complete token in the same eligible vector; a base token such as `AV:N` does not match a modified token such as `MAV:N`. -The reported total and pagination are complete only within that bounded candidate set; qualifying records beyond the first 10,000 search candidates may not be represented. When a semantic search reaches that limit, the response sets `pagination.totalIsExact` to `false` and includes a `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Semantic searches that remain below the limit and direct searches set `totalIsExact` to `true`. This preserves the existing bounded post-filter behavior until PIT-based pagination is implemented. A `knownExploited` filter without date boundaries uses a direct query and retains ordinary exact-total behavior. +By default, PIT traversal evaluates successive candidate batches to completion rather than stopping at 10,000 candidates. Successful responses have exact totals and no `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Public pages must still fit entirely within the first 10,000 matching results. `totalPages` is calculated from the full total and may exceed the number of accessible pages; it does not override the pagination window. PIT is server configuration, not a request parameter. No public cursor is returned. See the deployment prerequisites below. -`/search` orders search results by CVE ID so repeated shallow-page requests use a deterministic sort. The response `pagination.pageSize` preserves the requested `resultsPerPage` value, while `pagination.resultsReturned` reports the number of records actually returned and may be smaller for a partial or out-of-range page. Current searches do not use a point-in-time snapshot, so index updates between separate page requests can shift results. Request-scoped PIT traversal would improve consistency within one request, but consistency across separate page requests would additionally require reusing the same snapshot. +With PIT disabled using `SearchPitEnabled=false`, the reported total and pagination are complete only within the bounded candidate set; qualifying records beyond the first 10,000 search candidates may not be represented. When a semantic search reaches that limit, the response sets `pagination.totalIsExact` to `false` and includes a `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Semantic searches that remain below the limit and direct searches set `totalIsExact` to `true`. A `knownExploited` filter without date boundaries uses a direct query and retains ordinary exact-total behavior. + +`/search` orders search results by CVE ID so repeated shallow-page requests use a deterministic sort. The response `pagination.pageSize` preserves the requested `resultsPerPage` value, while `pagination.resultsReturned` reports the number of records actually returned and may be smaller for a partial or out-of-range page. PIT provides a consistent snapshot within one semantic request, including its full-record retrieval. Each subsequent HTTP request opens a new PIT, so index updates between separate page requests can still shift results. Direct searches do not use PIT. The API does not cache, coalesce, queue, rate-limit, or aggregate performance telemetry in process. Each request executes its own required OpenSearch operations. Public rate enforcement, admission control, shared caching, and durable metrics collection must be supplied by the deployment infrastructure. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed the 20-second default timeout. The same response is returned when a `/search` request exceeds the 30-second default overall deadline, and the active OpenSearch request is aborted. +### PIT Implementation Status + +Internal point-in-time (PIT) traversal is enabled by default for semantic `/search` requests, including deadline enforcement, completion/error validation, Swagger descriptions, and mode-aware Postman assertions. Explicit `SearchPitEnabled=false` overrides remain honored and retain the existing bounded semantic path. Deployments adopting the new default must support PIT; unsupported or unauthorized operations return errors without automatic fallback. Direct `/search` queries, the public pagination window, readiness behavior, and `/webSearch` are unchanged. + +Local validation against OpenSearch 3.5.0 covered traversal beyond 10,000 candidates, exact totals and page retrieval, snapshot consistency during index changes, and controlled timeout, cancellation, cleanup, and incomplete-response cases. The Postman collection passed with PIT disabled and enabled, including against the built runtime container. Separate minimal create/search/delete probes confirmed PIT access through deployed application clients in the managed environments. These checks do not establish production capacity, complete managed traversal, or behavior during managed-service restarts and deployments. + +The provider can create a PIT on the configured index, search through it without specifying an index again, and delete that specific context. Creation rejects partial shard snapshots. Every PIT search, including an empty final batch, must account for all shards successfully and report no timeout, early termination, or shard failures. Search shards skipped because they cannot match are allowed and are already included in the successful shard count; skipped creation shards are not allowed. Hit-shape, candidate-order, and full-page checks apply as well. The managed lifecycle helper attempts cleanup after success, failure, or cancellation with an independent timeout of at most five seconds. Cleanup failures emit a fixed warning without PIT IDs or upstream error details and do not replace the operation's result or error. If canceled creation later delivers a PIT ID, cleanup is attempted then; contexts whose IDs never reach the application must expire through their keep-alive period. + +The `/search` lifecycle measures elapsed time with a monotonic clock, so explicit deadline checks do not depend on the timer callback having run. Each PIT creation, candidate-batch search, and full-page retrieval clamps its client transport timeout to the smaller of the configured operation timeout and the remaining overall request budget (with a minimum positive transport limit of one millisecond). Deadline checks prevent new work after expiry and reject late results. PIT keep-alive renewal does not reset this budget. This bounds the application's wait and aborts active transport work; it does not guarantee immediate termination of work already running on OpenSearch. + +The PIT candidate builder preserves existing filters, scope, and restricted source fields while replacing offset pagination with a bounded batch size and optional `search_after`. It keeps ascending `cveMetadata.cveId.keyword` sorting; the provider supplies the PIT ID and lifetime separately. Candidate response validation requires each hit's sort value to match its source CVE ID and advance strictly beyond the previous hit or continuation. These checks reject missing, duplicate, or out-of-order values in returned batches, but cannot detect duplicate IDs skipped at a batch boundary. Index-wide CVE ID uniqueness remains a deployment prerequisite. + +`utils/searchPitTraversal.js` scans until an empty candidate batch confirms exhaustion, even if an earlier batch had no semantic matches or the requested page was already filled. It counts every match but retains only the current candidate batch and the requested page's IDs, resetting CPE parsing state between batches. Existing matching rules are unchanged. Evaluation yields to the event loop every 100 candidates and between batches so request deadlines and disconnects can be observed; an individual synchronous record evaluation is not preemptible. + +Complete page records are retrieved through the same PIT, restored to the selected CVE ID order, and validated for missing, duplicate, or unexpected records. Failed or canceled traversal does not return a partial count, restart with a new snapshot, or fall back to a live or bounded search. PIT timeouts return HTTP 504 `SEARCH_TIMEOUT`; a missing or expired search context returns HTTP 503 `SEARCH_UNAVAILABLE` with `Retry-After: 5`; malformed or incomplete responses return HTTP 502 `UPSTREAM_SEARCH_ERROR`. Upstream messages and PIT IDs are not exposed. Cleanup can add up to five seconds after a failed or canceled operation. Request warnings still precede records; the broad virtual-match warning describes increased processing work instead of a candidate cap when PIT is used. + +The configuration follows the existing `AppConfig` environment overrides: + +- `SearchPitEnabled` maps to `features.searchPitEnabled` and defaults to `true`. Explicit `false` overrides restore the bounded path. Unrecognized boolean strings leave PIT disabled. Existing environment overrides take precedence over the new default; configuration changes require an API restart or redeployment. +- `SearchPitBatchSize` maps to `search.pitBatchSize` and accepts integers from 1 through 10,000, defaulting to 1,000. This is a per-batch size, not a total candidate limit. +- `SearchPitKeepAliveMs` maps to `search.pitKeepAliveMs` and accepts positive safe integers, defaulting to 60,000 milliseconds. Each PIT search refreshes its lifetime. This does not extend the overall request deadline and must fit the deployed cluster's PIT lifetime limits. + +Missing or out-of-range numeric settings fall back to their defaults. The initial batch size and lifetime are starting points for validation, not established production capacity settings. + +Before deploying with PIT enabled, verify these deployment prerequisites: + +1. The actual domain version supports PIT. Amazon OpenSearch Service supports PIT on OpenSearch 2.5 and later; the JavaScript client version is not the server version. +2. The API's effective OpenSearch identity has index-scoped PIT create, search, and delete permissions, together with any required domain-policy permissions. An AWS administrator CLI session or a successful `/api/ready-check` does not establish these permissions for the application. +3. The configured index or alias supports a complete, unique sort order. Current queries sort by `cveMetadata.cveId.keyword`, and readiness checks its searchable/aggregatable keyword capability, not uniqueness. Confirm one searchable document per CVE ID across the resolved indices, with keyword sort values matching source CVE IDs. If that is not the case, revise the sort and continuation validation to use an existing supported unique tiebreaker before enabling traversal. Do not assume a tiebreaker supported by another engine or version is available. +4. Each index supports the chosen per-batch size, and the deployment has headroom for retained PIT segments and open contexts. Verify effective context and lifetime limits with the infrastructure owner; do not assume managed-service settings can be changed. +5. PIT lifecycle, traversal, cancellation, cleanup, and failure handling have been validated against the deployed version and application identity. The helpers use the installed client's `createPit()` and `deletePit()` methods. Local runtime validation and minimal managed PIT access probes have passed; complete managed traversal, failure handling, and capacity validation remain pending. The [Postman PIT-mode guidance](test/postman/README.md#pit-modes) distinguishes response checks from lifecycle and capacity validation. No cache, queue, or telemetry subsystem is being reintroduced. + +Rollback is a server configuration change: set `SearchPitEnabled=false` and restart or redeploy the API so new requests use the bounded path again. This restores the 10,000-candidate limitation and its warnings; it does not preserve PIT totals or snapshots. Existing requests must be allowed to finish cleanup or be canceled, and contexts whose cleanup could not complete expire according to their keep-alive. No index or mapping migration is involved. + +The initial PIT scope is internal semantic candidate traversal only. Public cursors and pagination beyond the existing 10,000-result window are separate work. See the [OpenSearch PIT documentation](https://docs.opensearch.org/latest/search-plugins/searching-data/point-in-time/) and [Amazon OpenSearch Service PIT documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/pit.html) for lifecycle, permission, and resource details. + ## Docker Runtime Checks Docker runtime smoke checks can be run after building the image. The default `Dockerfile` avoids BuildKit `RUN --mount` syntax for AWS builders that do not support it. The example below checks packaging and process liveness only; replace the placeholder endpoint/index and check `/api/ready-check` separately to verify an actual OpenSearch dependency: diff --git a/api-docs/openapi.json b/api-docs/openapi.json index 43f0bec..e293a9f 100644 --- a/api-docs/openapi.json +++ b/api-docs/openapi.json @@ -21,7 +21,7 @@ "CVE Record" ], "summary": "Retrieves all matching CVE records after applying the JSON request body as filters", - "description": "

Expected Behavior

Retrieves all matching CVE records after applying the JSON request body as filters

Result Counts and Pagination

Direct searches report the exact number of matching records, including totals greater than 10,000, but requested pages must remain within the first 10,000 results.

CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. These searches evaluate at most 10,000 search candidates, so their reported totals and pages include only matches found within that candidate set. A warning is returned when that limit is reached.

", + "description": "

Expected Behavior

Retrieves all matching CVE records after applying the JSON request body as filters

Result Counts and Pagination

Direct searches report the exact number of matching records, including totals greater than 10,000, but requested pages must remain within the first 10,000 results.

CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. PIT is enabled by default on the server. With PIT explicitly disabled using SearchPitEnabled=false, these searches evaluate at most 10,000 search candidates, so their reported totals and pages include only matches found within that candidate set. Reaching that limit returns SEMANTIC_CANDIDATE_LIMIT_REACHED and pagination.totalIsExact=false.

With point-in-time (PIT) traversal enabled, semantic searches evaluate all candidate batches before returning exact totals and pagination.totalIsExact=true. Requested pages must still fit entirely within the first 10,000 matching results. Filling the requested page does not stop the scan because the total must also be computed. PIT is a server-side setting, not a request field; clients continue to use pageNumber and resultsPerPage, with no public cursor.

Each PIT semantic request uses one snapshot for its candidates and returned records. Separate page requests use separate snapshots, so changes between requests can still shift results. Incomplete traversal returns a sanitized error, not partial results, and does not restart or fall back to a bounded search. Warnings remain before records in successful responses. Direct searches and /webSearch do not use PIT.

", "operationId": "search", "responses": { "200": { @@ -83,7 +83,7 @@ } }, "502": { - "description": "The search backend returned an invalid or unexpected response", + "description": "UPSTREAM_SEARCH_ERROR: The search backend returned an invalid or unexpected response. This includes malformed or incomplete PIT batches or full-page responses; no partial PIT result is returned.", "content": { "application/json": { "schema": { @@ -93,7 +93,7 @@ } }, "503": { - "description": "The search service is temporarily unavailable", + "description": "SEARCH_UNAVAILABLE: The search service is temporarily unavailable, including a missing or expired PIT search context. PIT traversal is not silently restarted. Retry-After specifies the suggested delay.", "content": { "application/json": { "schema": { @@ -111,7 +111,7 @@ } }, "504": { - "description": "Search exceeded its processing deadline", + "description": "SEARCH_TIMEOUT: Search exceeded its operation timeout or overall processing deadline, or a PIT search reported a timeout. No partial PIT result is returned. PIT cleanup has a separate bounded timeout and can add up to five seconds before the error response.", "content": { "application/json": { "schema": { @@ -275,7 +275,7 @@ "CVE Record" ], "summary": "Raw query search", - "description": "

Expected Behavior

Retrieves matching CVE records using the raw query expression supplied in the JSON request body. Service operators can disable this endpoint through runtime configuration.

", + "description": "

Expected Behavior

Retrieves matching CVE records using the raw query expression supplied in the JSON request body. Service operators can disable this endpoint through runtime configuration. Standalone dates and date ranges search the supported record/container date fields; other expressions use raw query syntax with an AND default operator.

", "operationId": "webSearch", "responses": { "200": { @@ -380,6 +380,7 @@ "properties": { "query": { "type": "string", + "description": "Raw search expression. Examples: remote code execution, apache*, 2024-01-01, or 2024-01-01..2024-01-31. Date ranges are inclusive and allow year/month endpoints; standalone years/months and quoted dates remain text queries. Date-query years must be between 1900 and 2100.", "minLength": 1 }, "sort": { @@ -455,6 +456,7 @@ }, "SearchWarning": { "type": "object", + "description": "Non-fatal search interpretation information. SEMANTIC_CANDIDATE_LIMIT_REACHED applies only to bounded semantic searches with PIT disabled. BROAD_VIRTUAL_MATCH_STRING can occur in either mode; its message describes the candidate limit when PIT is disabled and increased processing time when PIT is enabled.", "additionalProperties": false, "required": [ "code", @@ -526,16 +528,16 @@ "totalResults": { "type": "integer", "minimum": 0, - "description": "For direct searches, the exact number of matching records, which may be greater than 10,000. For semantic post-filtered searches, the number of matches found within the bounded 10,000-candidate set." + "description": "For direct searches, the exact number of matching records, which may be greater than 10,000. For semantic searches with PIT enabled (the default), the exact number of matches across the completed snapshot traversal. With PIT explicitly disabled, the number of matches found within the bounded 10,000-candidate set. Separate page requests use separate snapshots." }, "totalPages": { "type": "integer", "minimum": 0, - "description": "Pages derived from totalResults and pageSize. Semantic post-filtered searches derive this value from matches found within the bounded 10,000-candidate set." + "description": "Pages derived from totalResults and pageSize, using the same bounded or exact total as totalResults. This value can exceed the number of accessible pages: each requested page must still fit entirely within the first 10,000 matching results in either mode." }, "totalIsExact": { "type": "boolean", - "description": "True when totalResults represents every match. False when a semantic search reaches the 10,000-candidate limit and additional matches may exist." + "description": "True for direct searches and successful, fully completed PIT semantic searches (the default mode). With PIT explicitly disabled, false when a semantic search reaches the 10,000-candidate limit and additional matches may exist; true otherwise. A failed PIT traversal returns an error rather than a success with a partial total." } } }, @@ -570,7 +572,7 @@ }, "warnings": { "type": "array", - "description": "Non-fatal conditions that may affect how the results should be interpreted.", + "description": "Non-fatal conditions that may affect how the results should be interpreted. This field is emitted before records. Successful PIT traversal does not return a semantic candidate-limit warning, but other interpretation warnings still apply.", "items": { "$ref": "#/components/schemas/SearchWarning" } @@ -598,6 +600,7 @@ "searchMetadata": { "type": "object", "additionalProperties": true, + "description": "API-owned raw-query metadata. Includes the response-format version, not dependency package versions.", "required": [ "searchStatus", "version", @@ -608,7 +611,18 @@ "type": "string" }, "version": { - "type": "string" + "type": "string", + "description": "Version of the raw-query response format." + }, + "notes": { + "type": "array", + "items": { + "type": "string" + } + }, + "queryTime_ms": { + "type": "number", + "description": "Query processing duration when available." }, "errors": { "type": "array", @@ -633,7 +647,7 @@ }, "SearchRequest": { "type": "object", - "description": "Optional filters and pagination controls for a CVE record search. Enum-valued fields are accepted case-insensitively and normalized to the documented canonical values. When both boundaries of a date range are supplied, the start must be before or equal to the end. Direct searches report exact totals even when they exceed 10,000, although requested pages must remain inside that window. CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. They evaluate at most 10,000 search candidates, so reported totals and pages include only matches found within that candidate set.", + "description": "Optional filters and pagination controls for a CVE record search. Enum-valued fields are accepted case-insensitively and normalized to the documented canonical values. When both boundaries of a date range are supplied, the start must be before or equal to the end. Direct searches report exact totals even when they exceed 10,000, although requested pages must remain inside that window. CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. PIT is enabled by default on the server. All candidate batches must complete before a successful response reports exact totals; traversal failures return an error instead of partial results. With PIT explicitly disabled using SearchPitEnabled=false, these searches evaluate at most 10,000 search candidates, so reported totals and pages include only matches found within that candidate set. Reaching that limit returns SEMANTIC_CANDIDATE_LIMIT_REACHED and totalIsExact=false. Requested pages must still remain within the first 10,000 matching results. PIT is a server-side setting, not a request field.", "additionalProperties": false, "properties": { "scope": { diff --git a/api-docs/openapi.test.js b/api-docs/openapi.test.js index be8da04..5cf415e 100644 --- a/api-docs/openapi.test.js +++ b/api-docs/openapi.test.js @@ -165,6 +165,23 @@ describe('OpenAPI contract', () => { expect(operation.responses['410'].description).toContain('disabled by service configuration'); }); + test('distinguishes bounded semantic searches from internal PIT traversal without adding client controls', () => { + const operation = openapi.paths['/search'].post; + const request = jsonRequestBodySchema('/search', 'post'); + for (const description of [operation.description, request.description]) { + expect(description).toContain('PIT is enabled by default on the server'); + expect(description).toContain('With PIT explicitly disabled using SearchPitEnabled=false'); + expect(description).toContain('SEMANTIC_CANDIDATE_LIMIT_REACHED'); + expect(description).toContain('first 10,000 matching results'); + expect(description).toContain('PIT is a server-side setting, not a request field'); + } + expect(operation.description).toContain('Separate page requests use separate snapshots'); + expect(operation.description).toContain('does not restart or fall back'); + expect(operation.description).toContain('no public cursor'); + expect(operation.description).toContain('Filling the requested page does not stop the scan'); + expect(openapi.components.schemas.SearchWarning.description).toContain('can occur in either mode'); + }); + test('documents cpeName as a CPE Match Criteria filter with required components', () => { const cpeName = openapi.components.schemas.SearchRequest.properties.cpeName; @@ -193,7 +210,11 @@ describe('OpenAPI contract', () => { expect(responses['400'].description).toContain('Malformed JSON'); expect(responses['415'].description).toContain('application/json'); expect(responses['502'].description).toContain('invalid or unexpected response'); + expect(responses['502'].description).toContain('no partial PIT result'); expect(responses['503'].description).toContain('search service'); + expect(responses['503'].description).toContain('missing or expired PIT search context'); + expect(responses['504'].description).toContain('SEARCH_TIMEOUT'); + expect(responses['504'].description).toContain('separate bounded timeout'); expect(responses['503'].headers['Retry-After']).toStrictEqual({ description: 'Seconds before retrying the unavailable search service', schema: { @@ -244,7 +265,11 @@ describe('OpenAPI contract', () => { 'exact number of matching records' ); expect(schemas.SearchPagination.properties.totalResults.description).toContain('10,000-candidate set'); + expect(schemas.SearchPagination.properties.totalResults.description).toContain('completed snapshot traversal'); + expect(schemas.SearchPagination.properties.totalPages.description).toContain('number of accessible pages'); expect(schemas.SearchPagination.properties.totalIsExact.description).toContain('additional matches may exist'); + expect(schemas.SearchPagination.properties.totalIsExact.description).toContain('error rather than a success'); + expect(searchResponse.properties.warnings.description).toContain('emitted before records'); expect(schemas.CveRecord.description).toContain('without search-backend transport metadata'); expect(schemas.ApiError.required).toStrictEqual(['error', 'message']); expect(schemas.ApiErrorResponse.oneOf).toHaveLength(2); diff --git a/config/custom-environment-variables.jsonc b/config/custom-environment-variables.jsonc index 045c52a..61740ce 100644 --- a/config/custom-environment-variables.jsonc +++ b/config/custom-environment-variables.jsonc @@ -1,9 +1,6 @@ { - // this file is used by node-config to map a node-config (AppConfig) - // hierarchy of constants to an environment variable - // Note that much of the environment variables mapped here existed for some time - // without AppConfig, this file bridges the historical uses of those with the new - // as we transition to AppConfig + // Map appConfig settings to environment variables. utils/appConfig.js loads .env + // before node-config resolves these overrides; existing process variables take precedence. "appConfig": { // constants for search capability "search": { @@ -13,22 +10,17 @@ "allowUnknownSslCerts": "OpenSearchAllowUnknownSslCerts", "openSearchRequestTimeoutMs": "SearchOpenSearchRequestTimeoutMs", "requestDeadlineMs": "SearchRequestDeadlineMs", - "readinessTimeoutMs": "SearchReadinessTimeoutMs" + "readinessTimeoutMs": "SearchReadinessTimeoutMs", + "pitBatchSize": "SearchPitBatchSize", + "pitKeepAliveMs": "SearchPitKeepAliveMs" }, "requestLogging": { "successSampleRate": "RequestLogSuccessSampleRate", "healthChecks": "RequestLogHealthChecks" }, "features": { - "webSearchEnabled": "WebSearchEnabled" - }, - // constants for testing node-config - // these values are only used to test node-config in AppConfig.test.int.ts - // DO NOT USE THIS FOR ANYTHING ELSE - "test": { - "appConfigTest": { - "test": "JEST_env_config_test" - } + "webSearchEnabled": "WebSearchEnabled", + "searchPitEnabled": "SearchPitEnabled" } } } diff --git a/config/default.jsonc b/config/default.jsonc index ac30f34..41c41d8 100644 --- a/config/default.jsonc +++ b/config/default.jsonc @@ -3,8 +3,8 @@ // These values are overridable using other *.jsonc (e.g., prod.jsonc) // as well as using environment variables (e.g., in `.env`) // - Each configuration is mapped using custom-environment-variables.jsonc to enable environment variable overrides. - // - For more information, see cve-core/src/adapters/config/AppConfig.ts - // NOTE for consistency, all values need to be strings for proper type when using .env overrides + // - utils/appConfig.js loads .env before reading these files through node-config. + // Values are normalized by their consumers because environment overrides are strings. "appConfig": { // constants for search capability "search": { @@ -16,7 +16,10 @@ // Limits total processing time for a /search request. "requestDeadlineMs": "30000", // Overall /api/ready-check deadline and per-operation timeout for field capabilities and search. - "readinessTimeoutMs": "5000" + "readinessTimeoutMs": "5000", + // Semantic PIT traversal settings; validate capacity for each deployment. + "pitBatchSize": "1000", + "pitKeepAliveMs": "60000" }, // Access logging settings. Error responses are always logged. "requestLogging": { @@ -25,10 +28,12 @@ // Set to TRUE to include successful health checks in the same successSampleRate sampling. "healthChecks": "FALSE" }, - // Feature flags used for operator-controlled endpoint availability. + // Operator-controlled feature flags. "features": { // Set to FALSE only when an environment must disable POST /webSearch. - "webSearchEnabled": "TRUE" + "webSearchEnabled": "TRUE", + // Set to FALSE to restore bounded semantic searches without PIT. + "searchPitEnabled": "TRUE" } } } diff --git a/config/devel.jsonc b/config/devel.jsonc index e718b92..b8c26dd 100644 --- a/config/devel.jsonc +++ b/config/devel.jsonc @@ -1,8 +1,7 @@ { // development configurations // overrides values specified in default.jsonc, read additional comments there - // and in cve-core/src/adapters/config/AppConfig.ts - // NOTE for consistency, all values need to be strings for proper type when using .env overrides + // and in utils/appConfig.js "appConfig": { // constants for search capability "search": { @@ -14,29 +13,6 @@ // TRUE disables verification for isolated local development with self-signed certificates. // DO NOT USE THIS IN ANY PUBLIC OR PRODUCTION ENVIRONMENTS "allowUnknownSslCerts": "TRUE" - }, - // constants for unit, int, e2e testing - "test": { - // constants for testing search capability - "searchTest": { - // many tests for search uses snapshots, which requires CVEs to remain unchanged - // since the live server is updated all the time, a fixture containing fixed CVEs - // is required to keep the test consistent. "fixtures" provides the link - // to the cve-fixtures repository - "fixtures": { - // @todo these constants needs to be in sync in cve-fixtures - // so that testing snapshots are consistent and valid - "name": "fixtures-search-baseline-1086", // release tag - "numCves": "1086" // possible identifier assuming we always add cves to a new release - } - }, - // constants for testing node-config - "appConfigTest": { - // these values are only used to test node-config in AppConfig.test.int.ts - // DO NOT USE THIS FOR ANYTHING ELSE - "two": "2", - "five": "5" - } } } } diff --git a/config/prod.jsonc b/config/prod.jsonc index eb4d49c..175f757 100644 --- a/config/prod.jsonc +++ b/config/prod.jsonc @@ -1,8 +1,7 @@ { // production (example) constants // overrides values specified in default.jsonc, read additional comments there - // and in cve-core/src/adapters/config/AppConfig.ts - // NOTE for consistency, all values need to be strings for proper type when using .env overrides + // and in utils/appConfig.js "appConfig": { "search": { "providerEndpoint": "", // preference is to specify this in a (secret) environment variable on production platforms diff --git a/config/runtime-config.test.js b/config/runtime-config.test.js index c94173e..959c00f 100644 --- a/config/runtime-config.test.js +++ b/config/runtime-config.test.js @@ -91,9 +91,24 @@ describe('runtime container and compose configuration', () => { ); expect(customEnvironmentVariables).toContain('"requestDeadlineMs": "SearchRequestDeadlineMs"'); expect(customEnvironmentVariables).toContain('"readinessTimeoutMs": "SearchReadinessTimeoutMs"'); + expect(customEnvironmentVariables).toContain('"pitBatchSize": "SearchPitBatchSize"'); + expect(customEnvironmentVariables).toContain('"pitKeepAliveMs": "SearchPitKeepAliveMs"'); expect(customEnvironmentVariables).toContain('"successSampleRate": "RequestLogSuccessSampleRate"'); expect(customEnvironmentVariables).toContain('"healthChecks": "RequestLogHealthChecks"'); expect(customEnvironmentVariables).toContain('"webSearchEnabled": "WebSearchEnabled"'); + expect(customEnvironmentVariables).toContain('"searchPitEnabled": "SearchPitEnabled"'); + }); + + test('enables PIT in defaults and the environment example with bounded batch tuning', () => { + const defaults = readProjectFile('config/default.jsonc'); + const envExample = readProjectFile('.env.example'); + + expect(defaults).toContain('"searchPitEnabled": "TRUE"'); + expect(defaults).toContain('"pitBatchSize": "1000"'); + expect(defaults).toContain('"pitKeepAliveMs": "60000"'); + expect(envExample).toMatch(/^SearchPitEnabled=true$/m); + expect(envExample).toMatch(/^SearchPitBatchSize=1000$/m); + expect(envExample).toMatch(/^SearchPitKeepAliveMs=60000$/m); }); test.each([['config/default.jsonc'], ['config/devel.jsonc'], ['config/prod.jsonc']])( diff --git a/controllers/searchController.js b/controllers/searchController.js index a8eec9d..af23d7f 100644 --- a/controllers/searchController.js +++ b/controllers/searchController.js @@ -1,11 +1,13 @@ -const PackageInfo = require('cve-core').packageInfo; const constants = require('../utils/constants'); const errors = require('../utils/errors'); +const featureFlags = require('../utils/featureFlags'); const searchDeadline = require('../utils/searchDeadline'); +const searchPitTraversal = require('../utils/searchPitTraversal'); const searchProvider = require('../utils/searchProvider'); const searchQueryBuilder = require('../utils/searchQueryBuilder'); const searchPostFilters = require('../utils/searchPostFilters'); const searchResponseBuilder = require('../utils/searchResponseBuilder'); +const webSearchQueryBuilder = require('../utils/webSearchQueryBuilder'); const error = new errors.errors(); const openSearchUnavailableErrorNames = new Set(['ConnectionError', 'NoLivingConnectionsError']); const openSearchUnavailableStatusCodes = new Set([502, 503, 504]); @@ -23,7 +25,6 @@ function getOpenSearchErrorStatusCode(err) { function isOpenSearchUnavailableError(err) { return ( - searchProvider.isSearchProviderContractError?.(err) === true || openSearchUnavailableErrorNames.has(err?.name) || openSearchUnavailableStatusCodes.has(getOpenSearchErrorStatusCode(err)) ); @@ -55,28 +56,6 @@ function buildWebSearchSort(sort) { return undefined; } -function getWebSearchSuccessData(searchResponse) { - if (typeof searchResponse?.isOk !== 'function' || searchResponse.isOk() !== true) return null; - - const hits = searchResponse?.data?.hits; - const totalValue = hits?.total?.value; - const notes = searchResponse.notes === undefined ? [] : searchResponse.notes; - const responseErrors = searchResponse.errors === undefined ? [] : searchResponse.errors; - const hasValidMetadata = - typeof searchResponse.status === 'string' && Array.isArray(notes) && Array.isArray(responseErrors); - - if (!hasValidMetadata || !Array.isArray(hits?.hits) || !Number.isSafeInteger(totalValue) || totalValue < 0) { - return null; - } - - return { - hits: hits.hits, - totalValue, - notes, - errors: responseErrors - }; -} - function getHitCveId(hit) { return hit?._source?.cveMetadata?.cveId ?? hit?._id; } @@ -156,11 +135,19 @@ async function executePostFilterSearch(searchOptions, queryStringObjects, postFi }; } -async function executeSearch(searchOptions, queryStringObjects, postFilters, responseOptions, signal) { +async function executeSearch(searchOptions, queryStringObjects, postFilters, responseOptions, lifecycle) { + const { signal, getRemainingTimeMs } = lifecycle; if (!searchPostFilters.hasSearchPostFilters(postFilters)) { return executeStandardSearch(searchOptions, queryStringObjects, signal); } + if (featureFlags.isSearchPitEnabled()) { + return searchPitTraversal.executePitSearch( + { searchOptions, queryStringObjects, postFilters, responseOptions }, + { signal, getRemainingTimeMs } + ); + } + return executePostFilterSearch(searchOptions, queryStringObjects, postFilters, responseOptions, signal); } @@ -181,11 +168,14 @@ async function searchHandler(req, res, next) { searchRequest.queryStringObjects, postFilters, responseOptions, - lifecycle.signal + lifecycle ); lifecycle.throwIfAborted(); const pagingOptions = responseOptions ?? searchOptions; - const warnings = [...searchResponseBuilder.getRequestWarnings(params), ...searchResult.warnings]; + const warnings = [ + ...searchResponseBuilder.getRequestWarnings(params, { usesPit: searchResult.usesPit }), + ...searchResult.warnings + ]; return res.status(200).json( searchResponseBuilder.buildSearchResponse({ @@ -203,7 +193,7 @@ async function searchHandler(req, res, next) { return res.status(504).json(error.searchTimeoutError()); } - if (isOpenSearchUnavailableError(err)) { + if (err?.code === 'SEARCH_PIT_UNAVAILABLE' || isOpenSearchUnavailableError(err)) { if (typeof res.set === 'function') res.set('Retry-After', '5'); return res.status(503).json(error.searchUnavailableError()); @@ -221,13 +211,10 @@ async function webSearchHandler(req, res, next) { const sort = buildWebSearchSort(req.body.sort); // initialize response body to error condition - let status; - let body = { + const body = { searchMetadata: { searchStatus: 'errors', version: kLambdaTimeVersion, - cveCore: PackageInfo['version'], - cveMitreCore: PackageInfo['version'], notes: [], errors: [] }, @@ -236,29 +223,25 @@ async function webSearchHandler(req, res, next) { }; try { - const searchManager = searchProvider.getBasicSearchManager(); const _startTime = Date.now(); - - const searchResponse = await searchManager.search(`${userProvidedQuery}`, { + const request = webSearchQueryBuilder.buildWebSearchRequest(userProvidedQuery, { sort, from: req.body.from, size: req.body.size }); + const searchData = await searchProvider.searchOpenSearch(request.body); const queryTime = Date.now() - _startTime; body.searchMetadata.queryTime_ms = queryTime; - const successData = getWebSearchSuccessData(searchResponse); - if (successData) { - status = 200; - body.searchMetadata.searchStatus = searchResponse.status; - body.searchMetadata.notes = successData.notes; - body.searchMetadata.errors = successData.errors; - body.resultsTotal = successData.totalValue; - body.data = successData.hits; - } else { - status = 502; - body.searchMetadata.errors = [error.upstreamSearchError()]; + const total = searchData?.hits?.total?.value; + if (!Number.isSafeInteger(total) || total < 0 || !Array.isArray(searchData?.hits?.hits)) { + throw new Error('Invalid web search results.'); } + body.searchMetadata.searchStatus = 'ok'; + body.searchMetadata.notes = request.notes; + body.resultsTotal = total; + body.data = searchData.hits.hits; + return res.status(200).json(body); } catch (ex) { const timedOut = searchDeadline.isSearchTimeoutError(ex); const err = timedOut ? error.searchTimeoutError() : error.upstreamSearchError(); @@ -266,8 +249,6 @@ async function webSearchHandler(req, res, next) { return res.status(timedOut ? 504 : 502).json(body); } - - return res.status(status).json(body); } module.exports = { diff --git a/controllers/searchController.test.js b/controllers/searchController.test.js index 8319ec4..b1e800d 100644 --- a/controllers/searchController.test.js +++ b/controllers/searchController.test.js @@ -1,33 +1,23 @@ const { EventEmitter } = require('events'); const mockApiSearch = jest.fn(); -const mockBasicSearch = jest.fn(); const mockSearchRaw = jest.fn(); +const mockIsSearchPitEnabled = jest.fn(() => false); +const mockExecutePitSearch = jest.fn(); const mockSearchOpenSearch = jest.fn((body, options) => { const search = Array.isArray(body?._source) ? mockSearchRaw : mockApiSearch; return Promise.resolve(search(body, options)).then((result) => result?.data ?? result); }); const mockGetSearchRequestDeadlineMs = jest.fn(() => 30000); -const mockIsSearchProviderContractError = jest.fn((err) => err?.code === 'SEARCH_PROVIDER_CONTRACT'); -const mockGetBasicSearchManager = jest.fn(() => ({ - search: mockBasicSearch -})); -jest.mock('cve-core', () => { - return { - packageInfo: { - version: 'test-core-version' - } - }; -}); - jest.mock('../utils/searchProvider', () => ({ - getBasicSearchManager: mockGetBasicSearchManager, getSearchRequestDeadlineMs: mockGetSearchRequestDeadlineMs, - isSearchProviderContractError: mockIsSearchProviderContractError, searchOpenSearch: mockSearchOpenSearch })); +jest.mock('../utils/featureFlags', () => ({ isSearchPitEnabled: mockIsSearchPitEnabled })); +jest.mock('../utils/searchPitTraversal', () => ({ executePitSearch: mockExecutePitSearch })); + const controller = require('./searchController.js'); const constants = require('../utils/constants'); const searchPostFilters = require('../utils/searchPostFilters'); @@ -81,23 +71,6 @@ function searchSuccessResponse(totalResults, hits = [], options = {}) { }; } -function basicResponse({ ok = true, status = 'ok', total = 0, hits = [], notes, errors } = {}) { - return { - status, - notes, - errors, - data: { - hits: { - total: { - value: total - }, - hits - } - }, - isOk: jest.fn(() => ok) - }; -} - function cvssTokenRegexp(field, token) { return { regexp: { @@ -151,12 +124,11 @@ describe('Search controller SEARCH handler', () => { beforeEach(() => { mockApiSearch.mockReset(); - mockBasicSearch.mockReset(); mockSearchRaw.mockReset(); + mockIsSearchPitEnabled.mockReset().mockReturnValue(false); + mockExecutePitSearch.mockReset(); mockSearchOpenSearch.mockClear(); - mockGetBasicSearchManager.mockClear(); mockGetSearchRequestDeadlineMs.mockClear(); - mockIsSearchProviderContractError.mockClear(); mockGetSearchRequestDeadlineMs.mockReturnValue(30000); res = createResponse(); next = jest.fn(); @@ -164,6 +136,101 @@ describe('Search controller SEARCH handler', () => { afterEach(() => jest.restoreAllMocks()); + test('SEARCH delegates semantic searches to PIT and keeps warnings ahead of records', async () => { + const hit = { _source: { cveMetadata: { cveId: 'CVE-2024-0001' } } }; + mockIsSearchPitEnabled.mockReturnValue(true); + mockExecutePitSearch.mockResolvedValue({ + totalResults: 10001, + hits: [hit], + totalIsExact: true, + warnings: [], + usesPit: true + }); + + await controller.SEARCH({ searchParams: { virtualMatchString: 'cpe:2.3:o', resultsPerPage: 1 } }, res, next); + + expect(mockExecutePitSearch).toHaveBeenCalledWith( + expect.objectContaining({ + postFilters: expect.objectContaining({ virtualMatchString: expect.any(Object) }), + responseOptions: { from: 0, resultsPerPage: 1 } + }), + { signal: expect.any(AbortSignal), getRemainingTimeMs: expect.any(Function) } + ); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + expect(res.status).toHaveBeenCalledWith(200); + expect(res.json).toHaveBeenCalledWith( + searchSuccessResponse(10001, [hit], { + pageSize: 1, + warnings: [ + { + code: 'BROAD_VIRTUAL_MATCH_STRING', + message: + 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.', + components: ['vendor', 'product'] + } + ] + }) + ); + expect(Object.keys(res.json.mock.calls[0][0])).toStrictEqual([ + 'apiVersion', + 'status', + 'warnings', + 'pagination', + 'records' + ]); + }); + + test.each([{}, { knownExploited: true }, { cvssV3Severity: 'HIGH' }])( + 'SEARCH leaves direct filters on the ordinary path when PIT is enabled: %p', + async (searchParams) => { + mockIsSearchPitEnabled.mockReturnValue(true); + mockApiSearch.mockResolvedValue(advancedResponse(0)); + + await controller.SEARCH({ searchParams }, res, next); + + expect(res.status).toHaveBeenCalledWith(200); + expect(mockSearchOpenSearch).toHaveBeenCalledTimes(1); + expect(mockExecutePitSearch).not.toHaveBeenCalled(); + } + ); + + test.each([ + [new Error('private upstream details'), 502, 'UPSTREAM_SEARCH_ERROR'], + [ + Object.assign(new Error('private PIT identifier'), { code: 'SEARCH_PIT_UNAVAILABLE' }), + 503, + 'SEARCH_UNAVAILABLE' + ], + [Object.assign(new Error('private timeout details'), { code: 'SEARCH_TIMEOUT' }), 504, 'SEARCH_TIMEOUT'] + ])('SEARCH sanitizes PIT failures without falling back to a bounded search: %p', async (error, status, code) => { + mockIsSearchPitEnabled.mockReturnValue(true); + mockExecutePitSearch.mockRejectedValue(error); + + await controller.SEARCH({ searchParams: { virtualMatchString: 'cpe:2.3:o:linux:linux_kernel' } }, res, next); + + expect(res.status).toHaveBeenCalledWith(status); + expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ error: code })); + expect(JSON.stringify(res.json.mock.calls[0][0])).not.toContain('private'); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + + test('SEARCH does not send a PIT success response after its deadline with a delayed timer callback', async () => { + captureTimers(); + const clock = jest.spyOn(process.hrtime, 'bigint').mockReturnValue(0n); + mockIsSearchPitEnabled.mockReturnValue(true); + mockExecutePitSearch.mockImplementation(async (_request, { getRemainingTimeMs }) => { + expect(getRemainingTimeMs()).toBe(30000); + clock.mockReturnValue(30000000000n); + return { totalResults: 0, hits: [], totalIsExact: true, warnings: [], usesPit: true }; + }); + + await controller.SEARCH({ searchParams: { virtualMatchString: 'cpe:2.3:o:linux:linux_kernel' } }, res, next); + + expect(res.status).toHaveBeenCalledWith(504); + expect(res.json).toHaveBeenCalledWith(expect.objectContaining({ error: 'SEARCH_TIMEOUT' })); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + }); + test('SEARCH builds filters, query strings, date ranges, and pagination options', async () => { const hit = { _id: 'CVE-2024-0001', @@ -901,6 +968,7 @@ describe('Search controller SEARCH handler', () => { ); expect(res.status).toHaveBeenCalledWith(200); + expect(mockExecutePitSearch).not.toHaveBeenCalled(); expect(res.json).toHaveBeenCalledWith( searchSuccessResponse(0, [], { totalIsExact: false, @@ -1161,7 +1229,6 @@ describe('Search controller SEARCH handler', () => { test.each([ ['ConnectionError', { name: 'ConnectionError' }], ['NoLivingConnectionsError', { name: 'NoLivingConnectionsError' }], - ['cve-core provider contract error', { code: 'SEARCH_PROVIDER_CONTRACT' }], ['OpenSearch 503 response', { name: 'ResponseError', meta: { statusCode: 503 } }] ])('SEARCH returns a retryable 503 when OpenSearch is unavailable through %s', async (_caseName, thrownError) => { mockApiSearch.mockRejectedValue(thrownError); @@ -1305,7 +1372,7 @@ describe('Search controller SEARCH handler', () => { expect(res.json).not.toHaveBeenCalled(); }); - test('SEARCH returns sanitized errors from cve-core', async () => { + test('SEARCH returns sanitized upstream errors', async () => { const err = { name: 'SearchError', message: 'bad query', @@ -1338,53 +1405,32 @@ describe('Search controller WEB_SEARCH handler', () => { beforeEach(() => { mockApiSearch.mockReset(); - mockBasicSearch.mockReset(); mockSearchRaw.mockReset(); mockSearchOpenSearch.mockClear(); - mockGetBasicSearchManager.mockClear(); jest.spyOn(console, 'log').mockImplementation(() => {}); - res = createResponse(); next = jest.fn(); }); afterEach(() => jest.restoreAllMocks()); - test('WEB_SEARCH sends cveId ascending sort, paging, and response metadata', async () => { + test('WEB_SEARCH sends sorting and paging and owns its response metadata', async () => { const hits = [{ _id: 'CVE-2024-0001' }, { _id: 'CVE-2024-0002' }]; - const resp = basicResponse({ - total: 2, - hits, - notes: ['note'] - }); - mockBasicSearch.mockResolvedValue(resp); + mockApiSearch.mockResolvedValue(rawResponse(2, hits)); jest.spyOn(Date, 'now').mockReturnValueOnce(1000).mockReturnValueOnce(1042); await controller.WEB_SEARCH( { - body: { - query: 'apache', - sort: { - property: 'cveId', - order: 'asc' - }, - from: 5, - size: 10 - } + body: { query: 'apache', sort: { property: 'cveId', order: 'asc' }, from: 5, size: 10 } }, res, next ); - expect(mockGetBasicSearchManager).toHaveBeenCalledTimes(1); - expect(mockBasicSearch).toHaveBeenCalledWith('apache', { - sort: [ - { - 'cveMetadata.cveId.keyword': { - order: 'asc' - } - } - ], + expect(mockSearchOpenSearch).toHaveBeenCalledWith({ + query: { query_string: { query: 'apache', default_operator: 'AND' } }, + sort: [{ 'cveMetadata.cveId.keyword': { order: 'asc' } }], + track_total_hits: true, from: 5, size: 10 }); @@ -1393,9 +1439,7 @@ describe('Search controller WEB_SEARCH handler', () => { searchMetadata: { searchStatus: 'ok', version: '2025-08-26', - cveCore: 'test-core-version', - cveMitreCore: 'test-core-version', - notes: ['note'], + notes: ['general search text string'], errors: [], queryTime_ms: 42 }, @@ -1405,187 +1449,49 @@ describe('Search controller WEB_SEARCH handler', () => { expect(console.log).not.toHaveBeenCalled(); }); - test('WEB_SEARCH defaults cveId sort to descending when order is not asc', async () => { - mockBasicSearch.mockResolvedValue(basicResponse()); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache', - sort: { - property: 'cveId', - order: 'desc' - } - } - }, - res, - next - ); - - expect(mockBasicSearch.mock.calls[0][1].sort).toStrictEqual([ - { - 'cveMetadata.cveId.keyword': { - order: 'desc' - } - } - ]); - }); - - test('WEB_SEARCH supports relevancy ascending sort', async () => { - mockBasicSearch.mockResolvedValue(basicResponse()); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache', - sort: { - property: 'relevancy', - order: 'asc' - } - } - }, - res, - next - ); - - expect(mockBasicSearch.mock.calls[0][1].sort).toStrictEqual([ - { - _score: { - order: 'asc' - } - } - ]); - }); - - test('WEB_SEARCH supports relevancy descending sort', async () => { - mockBasicSearch.mockResolvedValue(basicResponse()); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache', - sort: { - property: 'relevancy', - order: 'desc' - } - } - }, - res, - next - ); - - expect(mockBasicSearch.mock.calls[0][1].sort).toStrictEqual([ - { - _score: { - order: 'desc' - } - } - ]); - }); - - test('WEB_SEARCH omits sort, from, and size when they are not provided', async () => { - mockBasicSearch.mockResolvedValue(basicResponse()); - - await controller.WEB_SEARCH( - { - body: { - query: 'nginx' - } - }, - res, - next - ); - - expect(mockBasicSearch).toHaveBeenCalledWith('nginx', { - sort: undefined, - from: undefined, - size: undefined - }); + test.each([ + ['cveId', 'asc', 'cveMetadata.cveId.keyword', 'asc'], + ['cveId', 'desc', 'cveMetadata.cveId.keyword', 'desc'], + ['cveId', undefined, 'cveMetadata.cveId.keyword', 'desc'], + ['relevancy', 'asc', '_score', 'asc'], + ['relevancy', 'desc', '_score', 'desc'] + ])('WEB_SEARCH supports %s sorting in %s order', async (property, order, field, expected) => { + mockApiSearch.mockResolvedValue(rawResponse(0)); + await controller.WEB_SEARCH({ body: { query: 'apache', sort: { property, order } } }, res, next); + expect(mockSearchOpenSearch.mock.calls[0][0].sort).toStrictEqual([{ [field]: { order: expected } }]); + expect(res.status).toHaveBeenCalledWith(200); }); - test('WEB_SEARCH returns sanitized metadata when cve-core response is not ok', async () => { - mockBasicSearch.mockResolvedValue( - basicResponse({ - ok: false, - status: 'errors', - total: 3, - hits: [{ _id: 'CVE-2024-0001' }], - notes: ['partial'], - errors: [ - { - code: 1, - url: 'redacted-sensitive-url', - headers: { - authorization: 'redacted-sensitive-header' - } - } - ] - }) - ); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache' - } - }, - res, - next - ); - - expect(res.status).toHaveBeenCalledWith(502); - expect(res.json.mock.calls[0][0]).toStrictEqual({ - searchMetadata: { - searchStatus: 'errors', - version: '2025-08-26', - cveCore: 'test-core-version', - cveMitreCore: 'test-core-version', - notes: [], - errors: [ - { - error: 'UPSTREAM_SEARCH_ERROR', - message: 'Search backend returned an error. Please try again later.' - } - ], - queryTime_ms: expect.any(Number) - }, - resultsTotal: 0, - data: [] + test('WEB_SEARCH uses the legacy default sort and paging', async () => { + mockApiSearch.mockResolvedValue(rawResponse(0)); + await controller.WEB_SEARCH({ body: { query: 'nginx' } }, res, next); + expect(mockSearchOpenSearch.mock.calls[0][0]).toMatchObject({ + sort: [{ 'cveMetadata.cveId.keyword': { order: 'desc' } }], + from: 0, + size: 25 }); }); - test('WEB_SEARCH fails closed when cve-core reports success without a complete hits response', async () => { - mockBasicSearch.mockResolvedValue({ - status: 'ok', - data: {}, - isOk: jest.fn(() => true) - }); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache' - } - }, - res, - next - ); - + test.each([ + {}, + { hits: {} }, + { hits: { total: { value: -1 }, hits: [] } }, + { hits: { total: { value: '1' }, hits: [] } }, + { hits: { total: { value: 1 } } } + ])('WEB_SEARCH rejects malformed totals or hits: %p', async (response) => { + mockApiSearch.mockResolvedValue(response); + await controller.WEB_SEARCH({ body: { query: 'apache' } }, res, next); expect(res.status).toHaveBeenCalledWith(502); - expect(res.json.mock.calls[0][0]).toStrictEqual({ + expect(res.json.mock.calls[0][0]).toMatchObject({ searchMetadata: { searchStatus: 'errors', - version: '2025-08-26', - cveCore: 'test-core-version', - cveMitreCore: 'test-core-version', notes: [], errors: [ { error: 'UPSTREAM_SEARCH_ERROR', message: 'Search backend returned an error. Please try again later.' } - ], - queryTime_ms: expect.any(Number) + ] }, resultsTotal: 0, data: [] @@ -1593,100 +1499,25 @@ describe('Search controller WEB_SEARCH handler', () => { }); test.each([ - ['notes', 'invalid notes'], - ['errors', { message: 'invalid errors' }] - ])('WEB_SEARCH fails closed when successful cve-core metadata has invalid %s', async (field, value) => { - mockBasicSearch.mockResolvedValue({ - ...basicResponse(), - [field]: value - }); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache' - } - }, - res, - next - ); - - expect(res.status).toHaveBeenCalledWith(502); - expect(res.json.mock.calls[0][0].searchMetadata.errors).toStrictEqual([ - { - error: 'UPSTREAM_SEARCH_ERROR', - message: 'Search backend returned an error. Please try again later.' - } - ]); - }); - - test('WEB_SEARCH returns a sanitized error when BasicSearchManager throws', async () => { - mockBasicSearch.mockRejectedValue(new Error('boom')); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache' - } - }, - res, - next - ); - - expect(res.status).toHaveBeenCalledWith(502); - expect(res.json.mock.calls[0][0]).toStrictEqual({ - searchMetadata: { - searchStatus: 'errors', - version: '2025-08-26', - cveCore: 'test-core-version', - cveMitreCore: 'test-core-version', - notes: [], - errors: [ - { - error: 'UPSTREAM_SEARCH_ERROR', - message: 'Search backend returned an error. Please try again later.' - } - ] - }, - resultsTotal: 0, - data: [] - }); + [new Error('sensitive backend details'), 502, 'UPSTREAM_SEARCH_ERROR'], + [{ name: 'TimeoutError', message: 'sensitive timeout details' }, 504, 'SEARCH_TIMEOUT'] + ])('WEB_SEARCH sanitizes transport errors: %p', async (err, status, code) => { + mockApiSearch.mockRejectedValue(err); + await controller.WEB_SEARCH({ body: { query: 'apache' } }, res, next); + expect(res.status).toHaveBeenCalledWith(status); + const body = res.json.mock.calls[0][0]; + expect(body.searchMetadata.searchStatus).toBe('errors'); + expect(body.searchMetadata.errors[0].error).toBe(code); + expect(body.searchMetadata).not.toHaveProperty('cveCore'); + expect(body.searchMetadata).not.toHaveProperty('cveMitreCore'); + expect(JSON.stringify(body)).not.toContain('sensitive'); expect(console.log).not.toHaveBeenCalled(); }); - test('WEB_SEARCH returns a sanitized 504 when OpenSearch times out', async () => { - mockBasicSearch.mockRejectedValue({ - name: 'TimeoutError', - message: 'sensitive upstream timeout details' - }); - - await controller.WEB_SEARCH( - { - body: { - query: 'apache' - } - }, - res, - next - ); - - expect(res.status).toHaveBeenCalledWith(504); - expect(res.json.mock.calls[0][0]).toStrictEqual({ - searchMetadata: { - searchStatus: 'errors', - version: '2025-08-26', - cveCore: 'test-core-version', - cveMitreCore: 'test-core-version', - notes: [], - errors: [ - { - error: 'SEARCH_TIMEOUT', - message: 'Search request timed out. Please try again later.' - } - ] - }, - resultsTotal: 0, - data: [] - }); + test('WEB_SEARCH keeps unsupported expressions sanitized without contacting OpenSearch', async () => { + await controller.WEB_SEARCH({ body: { query: '{invalid}' } }, res, next); + expect(res.status).toHaveBeenCalledWith(502); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + expect(res.json.mock.calls[0][0].searchMetadata.errors[0].error).toBe('UPSTREAM_SEARCH_ERROR'); }); }); diff --git a/package-lock.json b/package-lock.json index 70c8592..d3f115f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8,13 +8,15 @@ "name": "cve-search-api", "version": "1.0.1-beta", "dependencies": { + "@opensearch-project/opensearch": "2.13.0", + "config": "4.4.1", "cors": "^2.8.5", - "cve-core": "github:CVEProject/cve-core.git#dev-package-test", - "dotenv": "^6.1.0", + "dotenv": "^16.6.1", "express": "^4.22.2", "express-validator": "^7.2.1", - "morgan": "~1.12.0", - "swagger-ui-express": "^5.0.1" + "morgan": "~1.12.1", + "swagger-ui-express": "^5.0.1", + "validator": "^13.15.26" }, "devDependencies": { "@eslint/js": "^10.0.1", @@ -496,15 +498,6 @@ "@babel/core": "^7.0.0-0" } }, - "node_modules/@babel/runtime": { - "version": "7.29.2", - "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz", - "integrity": "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g==", - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, "node_modules/@babel/template": { "version": "7.29.7", "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.29.7.tgz", @@ -578,15 +571,6 @@ "dev": true, "license": "MIT" }, - "node_modules/@commander-js/extra-typings": { - "version": "10.0.3", - "resolved": "https://registry.npmjs.org/@commander-js/extra-typings/-/extra-typings-10.0.3.tgz", - "integrity": "sha512-OIw28QV/GlP8k0B5CJTRsl8IyNvd0R8C8rfo54Yz9P388vCNDgdNrFlKxZTGqps+5j6lSw3Ss9JTQwcur1w1oA==", - "license": "MIT", - "peerDependencies": { - "commander": "10.0.x" - } - }, "node_modules/@eslint-community/eslint-utils": { "version": "4.9.1", "resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.9.1.tgz", @@ -799,15 +783,6 @@ "url": "https://github.com/sponsors/nzakas" } }, - "node_modules/@isaacs/cliui": { - "version": "9.0.0", - "resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-9.0.0.tgz", - "integrity": "sha512-AokJm4tuBHillT+FpMtxQ60n8ObyXBatq7jD2/JA9dxbDDokKQm8KMht5ibGzLVU9IJDIKK4TPKgMHEYMn3lMg==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=18" - } - }, "node_modules/@istanbuljs/load-nyc-config": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/@istanbuljs/load-nyc-config/-/load-nyc-config-1.1.0.tgz", @@ -1213,38 +1188,6 @@ "@jridgewell/sourcemap-codec": "^1.4.14" } }, - "node_modules/@kwsites/file-exists": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/@kwsites/file-exists/-/file-exists-1.1.1.tgz", - "integrity": "sha512-m9/5YGR18lIwxSFDwfE3oA7bWuq9kdau6ugN4H2rJeyhFQZcG9AgSHkQtSD15a8WvTgfz9aikZMrKPHvbpqFiw==", - "license": "MIT", - "dependencies": { - "debug": "^4.1.1" - } - }, - "node_modules/@kwsites/file-exists/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/@kwsites/promise-deferred": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/@kwsites/promise-deferred/-/promise-deferred-1.1.1.tgz", - "integrity": "sha512-GaHYm+c0O9MjZRu0ongGBRbinu8gVAMd2UZjji6jVmqKtZluZnptXGWhz1E8j8D2HJ3f/yMxKAUC0b+57wncIw==", - "license": "MIT" - }, "node_modules/@opensearch-project/opensearch": { "version": "2.13.0", "resolved": "https://registry.npmjs.org/@opensearch-project/opensearch/-/opensearch-2.13.0.tgz", @@ -1287,21 +1230,6 @@ "hasInstallScript": true, "license": "Apache-2.0" }, - "node_modules/@simple-git/args-pathspec": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/@simple-git/args-pathspec/-/args-pathspec-1.0.3.tgz", - "integrity": "sha512-ngJMaHlsWDTfjyq9F3VIQ8b7NXbBLq5j9i5bJ6XLYtD6qlDXT7fdKY2KscWWUF8t18xx052Y/PUO1K1TRc9yKA==", - "license": "MIT" - }, - "node_modules/@simple-git/argv-parser": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/@simple-git/argv-parser/-/argv-parser-1.1.1.tgz", - "integrity": "sha512-Q9lBcfQ+VQCpQqGJFHe5yooOS5hGdLFFbJ5R+R5aDsnkPCahtn1hSkMcORX65J2Z5lxSkD0lQorMsncuBQxYUw==", - "license": "MIT", - "dependencies": { - "@simple-git/args-pathspec": "^1.0.3" - } - }, "node_modules/@sinclair/typebox": { "version": "0.24.51", "resolved": "https://registry.npmjs.org/@sinclair/typebox/-/typebox-0.24.51.tgz", @@ -1509,15 +1437,6 @@ "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" } }, - "node_modules/adm-zip": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.0.tgz", - "integrity": "sha512-XleryMhbuksdKtofnWZ9Sk+4CUTbms4Mb/EU32SZwToAyZ5RgVos/ki8n+yr0LWHOGKuakbXTuuYNHLQjhddgg==", - "license": "MIT", - "engines": { - "node": ">=14.0" - } - }, "node_modules/ajv": { "version": "6.15.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz", @@ -1604,42 +1523,12 @@ "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==", "license": "MIT" }, - "node_modules/asynckit": { - "version": "0.4.0", - "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz", - "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==", - "license": "MIT" - }, "node_modules/aws4": { "version": "1.13.2", "resolved": "https://registry.npmjs.org/aws4/-/aws4-1.13.2.tgz", "integrity": "sha512-lHe62zvbTB5eEABUVi/AwVh0ZKY9rMMDhmm+eeyuuUQbQ3+J+fONVQOZyj+DdrvD4BY33uYniyRJ4UJIaSKAfw==", "license": "MIT" }, - "node_modules/axios": { - "version": "1.18.0", - "resolved": "https://registry.npmjs.org/axios/-/axios-1.18.0.tgz", - "integrity": "sha512-E32NzpYKp++W7XRe52rHiXV2ehxmh3wbdgO7MHeFM+vqxLBYHzt0ElkiImtOBxtOmyp0yoC8C6uESVV84Y2/hw==", - "license": "MIT", - "dependencies": { - "follow-redirects": "^1.16.0", - "form-data": "^4.0.5", - "https-proxy-agent": "^5.0.1", - "proxy-from-env": "^2.1.0" - } - }, - "node_modules/axios-retry": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/axios-retry/-/axios-retry-4.5.0.tgz", - "integrity": "sha512-aR99oXhpEDGo0UuAlYcn2iGRds30k366Zfa05XWScR9QaQD4JYiP3/1Qt1u7YlefUOK+cn0CcwoL1oefavQUlQ==", - "license": "Apache-2.0", - "dependencies": { - "is-retry-allowed": "^2.2.0" - }, - "peerDependencies": { - "axios": "0.x || 1.x" - } - }, "node_modules/babel-jest": { "version": "28.1.3", "resolved": "https://registry.npmjs.org/babel-jest/-/babel-jest-28.1.3.tgz", @@ -1743,6 +1632,7 @@ "version": "4.0.4", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", + "dev": true, "license": "MIT", "engines": { "node": "18 || 20 || >=22" @@ -1852,9 +1742,10 @@ "license": "MIT" }, "node_modules/brace-expansion": { - "version": "5.0.9", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz", - "integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==", + "version": "5.0.12", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.12.tgz", + "integrity": "sha512-YovQ3rzhaLMIrDjNDMkNS01tea93qhEhG5xy8f6+R0l+dw3Ki+5sCoIoI942iuLZTHWogWktgwVDhU09iNEimQ==", + "dev": true, "license": "MIT", "dependencies": { "balanced-match": "^4.0.2" @@ -2134,27 +2025,6 @@ "dev": true, "license": "MIT" }, - "node_modules/combined-stream": { - "version": "1.0.8", - "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz", - "integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==", - "license": "MIT", - "dependencies": { - "delayed-stream": "~1.0.0" - }, - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/commander": { - "version": "10.0.1", - "resolved": "https://registry.npmjs.org/commander/-/commander-10.0.1.tgz", - "integrity": "sha512-y4Mg2tXshplEbSGzx7amzPwKKOCGuoSRP/CjEdwwk0FOGlUbq6lKuoyDZTNZkmxHdJtp54hdfY/JUrdL7Xfdug==", - "license": "MIT", - "engines": { - "node": ">=14" - } - }, "node_modules/config": { "version": "4.4.1", "resolved": "https://registry.npmjs.org/config/-/config-4.4.1.tgz", @@ -2231,6 +2101,7 @@ "version": "7.0.6", "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", "integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==", + "dev": true, "license": "MIT", "dependencies": { "path-key": "^3.1.0", @@ -2241,82 +2112,6 @@ "node": ">= 8" } }, - "node_modules/cve-core": { - "version": "2.2.0-rc4", - "resolved": "git+ssh://git@github.com/CVEProject/cve-core.git#a1640b2979aebb681f151aa20248f87eb4fd8a81", - "license": "(CC0)", - "dependencies": { - "@commander-js/extra-typings": "^10.0.2", - "@opensearch-project/opensearch": "^2.3.1", - "adm-zip": "^0.5.10", - "axios": "^1.6.0", - "axios-retry": "^4.4.1", - "commander": "^10.0.0", - "config": "^4.0.0", - "date-fns": "^2.29.3", - "date-fns-tz": "^2.0.0", - "dotenv": "^16.0.1", - "fetch-retry": "^6.0.0", - "glob": "^11.0.1", - "json-difference": "^1.16.1", - "lodash.clonedeep": "^4.5.0", - "lodash.isequal": "^4.5.0", - "lodash.isinteger": "^4.0.4", - "lodash.map": "^4.6.0", - "lodash.omit": "^4.5.0", - "lodash.reduce": "^4.6.0", - "lodash.truncate": "^4.4.2", - "lodash.unset": "^4.5.2", - "semver": "^7.7.2", - "simple-git": "^3.19.1", - "ts-pattern": "^4.0.5", - "tslib": "~2.4", - "validator": "^13.15.20" - }, - "bin": { - "cves": "dist/index.js" - }, - "engines": { - "node": ">= 16.20.2 < 22" - } - }, - "node_modules/cve-core/node_modules/dotenv": { - "version": "16.6.1", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", - "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://dotenvx.com" - } - }, - "node_modules/date-fns": { - "version": "2.30.0", - "resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz", - "integrity": "sha512-fnULvOpxnC5/Vg3NCiWelDsLiUc9bRwAPs/+LfTLNvetFCtCTN+yQz15C/fs4AwX1R9K5GLtLfn8QW+dWisaAw==", - "license": "MIT", - "dependencies": { - "@babel/runtime": "^7.21.0" - }, - "engines": { - "node": ">=0.11" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/date-fns" - } - }, - "node_modules/date-fns-tz": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/date-fns-tz/-/date-fns-tz-2.0.1.tgz", - "integrity": "sha512-fJCG3Pwx8HUoLhkepdsP7Z5RsucUi+ZBOxyM5d0ZZ6c4SdYustq0VMmOu6Wf7bli+yS/Jwp91TOCqn9jMcVrUA==", - "license": "MIT", - "peerDependencies": { - "date-fns": "2.x" - } - }, "node_modules/dedent": { "version": "0.7.0", "resolved": "https://registry.npmjs.org/dedent/-/dedent-0.7.0.tgz", @@ -2341,15 +2136,6 @@ "node": ">=0.10.0" } }, - "node_modules/delayed-stream": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", - "integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==", - "license": "MIT", - "engines": { - "node": ">=0.4.0" - } - }, "node_modules/depd": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", @@ -2390,12 +2176,15 @@ } }, "node_modules/dotenv": { - "version": "6.2.0", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-6.2.0.tgz", - "integrity": "sha512-HygQCKUBSFl8wKQZBSemMywRWcEDNidvNbjGVyZu3nbZ8qq9ubiPoGLMdRDpfSrpkkm9BXYFkpKxxFX38o/76w==", + "version": "16.6.1", + "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", + "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==", "license": "BSD-2-Clause", "engines": { - "node": ">=6" + "node": ">=12" + }, + "funding": { + "url": "https://dotenvx.com" } }, "node_modules/dunder-proto": { @@ -2494,21 +2283,6 @@ "node": ">= 0.4" } }, - "node_modules/es-set-tostringtag": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz", - "integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "get-intrinsic": "^1.2.6", - "has-tostringtag": "^1.0.2", - "hasown": "^2.0.2" - }, - "engines": { - "node": ">= 0.4" - } - }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -2987,12 +2761,6 @@ "bser": "2.1.1" } }, - "node_modules/fetch-retry": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/fetch-retry/-/fetch-retry-6.0.0.tgz", - "integrity": "sha512-BUFj1aMubgib37I3v4q78fYo63Po7t4HUPTpQ6/QE6yK6cIQrP+W43FYToeTEyg5m2Y7eFUtijUuAv/PDlWuag==", - "license": "MIT" - }, "node_modules/file-entry-cache": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-8.0.0.tgz", @@ -3087,58 +2855,6 @@ "dev": true, "license": "ISC" }, - "node_modules/follow-redirects": { - "version": "1.16.0", - "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.16.0.tgz", - "integrity": "sha512-y5rN/uOsadFT/JfYwhxRS5R7Qce+g3zG97+JrtFZlC9klX/W5hD7iiLzScI4nZqUS7DNUdhPgw4xI8W2LuXlUw==", - "funding": [ - { - "type": "individual", - "url": "https://github.com/sponsors/RubenVerborgh" - } - ], - "license": "MIT", - "engines": { - "node": ">=4.0" - }, - "peerDependenciesMeta": { - "debug": { - "optional": true - } - } - }, - "node_modules/foreground-child": { - "version": "3.3.1", - "resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.3.1.tgz", - "integrity": "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==", - "license": "ISC", - "dependencies": { - "cross-spawn": "^7.0.6", - "signal-exit": "^4.0.1" - }, - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/form-data": { - "version": "4.0.6", - "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz", - "integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==", - "license": "MIT", - "dependencies": { - "asynckit": "^0.4.0", - "combined-stream": "^1.0.8", - "es-set-tostringtag": "^2.1.0", - "hasown": "^2.0.4", - "mime-types": "^2.1.35" - }, - "engines": { - "node": ">= 6" - } - }, "node_modules/forwarded": { "version": "0.2.0", "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", @@ -3268,30 +2984,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/glob": { - "version": "11.1.0", - "resolved": "https://registry.npmjs.org/glob/-/glob-11.1.0.tgz", - "integrity": "sha512-vuNwKSaKiqm7g0THUBu2x7ckSs3XJLXE+2ssL7/MfTGPLLcrJQ/4Uq1CjPTtO5cCIiRxqvN6Twy1qOwhL0Xjcw==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "BlueOak-1.0.0", - "dependencies": { - "foreground-child": "^3.3.1", - "jackspeak": "^4.1.1", - "minimatch": "^10.1.1", - "minipass": "^7.1.2", - "package-json-from-dist": "^1.0.0", - "path-scurry": "^2.0.0" - }, - "bin": { - "glob": "dist/esm/bin.mjs" - }, - "engines": { - "node": "20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/glob-parent": { "version": "5.1.2", "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz", @@ -3359,21 +3051,6 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/has-tostringtag": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz", - "integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==", - "license": "MIT", - "dependencies": { - "has-symbols": "^1.0.3" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/hasown": { "version": "2.0.4", "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", @@ -3402,48 +3079,6 @@ "dev": true, "license": "MIT" }, - "node_modules/https-proxy-agent": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz", - "integrity": "sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==", - "license": "MIT", - "dependencies": { - "agent-base": "6", - "debug": "4" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/https-proxy-agent/node_modules/agent-base": { - "version": "6.0.2", - "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-6.0.2.tgz", - "integrity": "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==", - "license": "MIT", - "dependencies": { - "debug": "4" - }, - "engines": { - "node": ">= 6.0.0" - } - }, - "node_modules/https-proxy-agent/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, "node_modules/human-signals": { "version": "2.1.0", "resolved": "https://registry.npmjs.org/human-signals/-/human-signals-2.1.0.tgz", @@ -3629,18 +3264,6 @@ "node": ">=0.12.0" } }, - "node_modules/is-retry-allowed": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/is-retry-allowed/-/is-retry-allowed-2.2.0.tgz", - "integrity": "sha512-XVm7LOeLpTW4jV19QSH38vkswxoLud8sQ57YwJVTPWdiaI9I8keEhGFpBlslyVsgdQy4Opg8QOLb8YRgsyZiQg==", - "license": "MIT", - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/is-stream": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-2.0.1.tgz", @@ -3658,6 +3281,7 @@ "version": "2.0.0", "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", + "dev": true, "license": "ISC" }, "node_modules/istanbul-lib-coverage": { @@ -3759,21 +3383,6 @@ "node": ">=8" } }, - "node_modules/jackspeak": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-4.2.3.tgz", - "integrity": "sha512-ykkVRwrYvFm1nb2AJfKKYPr0emF6IiXDYUaFx4Zn9ZuIH7MrzEZ3sD5RlqGXNRpHtvUHJyOnCEFxOlNDtGo7wg==", - "license": "BlueOak-1.0.0", - "dependencies": { - "@isaacs/cliui": "^9.0.0" - }, - "engines": { - "node": "20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/jest": { "version": "28.1.3", "resolved": "https://registry.npmjs.org/jest/-/jest-28.1.3.tgz", @@ -4477,15 +4086,6 @@ "dev": true, "license": "MIT" }, - "node_modules/json-difference": { - "version": "1.16.1", - "resolved": "https://registry.npmjs.org/json-difference/-/json-difference-1.16.1.tgz", - "integrity": "sha512-tNVUzBKn2I4BB4Cjn+j+Kq/BfLNwqtEvxhnlyuX5XJmKn9VlfV5MvaTbB/ure/toFgLaD6XglOUt+dd4iMQ3dA==", - "license": "MIT", - "engines": { - "node": ">=18.17.x" - } - }, "node_modules/json-parse-even-better-errors": { "version": "2.3.1", "resolved": "https://registry.npmjs.org/json-parse-even-better-errors/-/json-parse-even-better-errors-2.3.1.tgz", @@ -4598,56 +4198,6 @@ "integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==", "license": "MIT" }, - "node_modules/lodash.clonedeep": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/lodash.clonedeep/-/lodash.clonedeep-4.5.0.tgz", - "integrity": "sha512-H5ZhCF25riFd9uB5UCkVKo61m3S/xZk1x4wA6yp/L3RFP6Z/eHH1ymQcGLo7J3GMPfm0V/7m1tryHuGVxpqEBQ==", - "license": "MIT" - }, - "node_modules/lodash.isequal": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/lodash.isequal/-/lodash.isequal-4.5.0.tgz", - "integrity": "sha512-pDo3lu8Jhfjqls6GkMgpahsF9kCyayhgykjyLMNFTKWrpVdAQtYyB4muAMWozBB4ig/dtWAmsMxLEI8wuz+DYQ==", - "deprecated": "This package is deprecated. Use require('node:util').isDeepStrictEqual instead.", - "license": "MIT" - }, - "node_modules/lodash.isinteger": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/lodash.isinteger/-/lodash.isinteger-4.0.4.tgz", - "integrity": "sha512-DBwtEWN2caHQ9/imiNeEA5ys1JoRtRfY3d7V9wkqtbycnAmTvRRmbHKDV4a0EYc678/dia0jrte4tjYwVBaZUA==", - "license": "MIT" - }, - "node_modules/lodash.map": { - "version": "4.6.0", - "resolved": "https://registry.npmjs.org/lodash.map/-/lodash.map-4.6.0.tgz", - "integrity": "sha512-worNHGKLDetmcEYDvh2stPCrrQRkP20E4l0iIS7F8EvzMqBBi7ltvFN5m1HvTf1P7Jk1txKhvFcmYsCr8O2F1Q==", - "license": "MIT" - }, - "node_modules/lodash.omit": { - "version": "4.5.0", - "resolved": "https://registry.npmjs.org/lodash.omit/-/lodash.omit-4.5.0.tgz", - "integrity": "sha512-XeqSp49hNGmlkj2EJlfrQFIzQ6lXdNro9sddtQzcJY8QaoC2GO0DT7xaIokHeyM+mIT0mPMlPvkYzg2xCuHdZg==", - "deprecated": "This package is deprecated. Use destructuring assignment syntax instead.", - "license": "MIT" - }, - "node_modules/lodash.reduce": { - "version": "4.6.0", - "resolved": "https://registry.npmjs.org/lodash.reduce/-/lodash.reduce-4.6.0.tgz", - "integrity": "sha512-6raRe2vxCYBhpBu+B+TtNGUzah+hQjVdu3E17wfusjyrXBka2nBS8OH/gjVZ5PvHOhWmIZTYri09Z6n/QfnNMw==", - "license": "MIT" - }, - "node_modules/lodash.truncate": { - "version": "4.4.2", - "resolved": "https://registry.npmjs.org/lodash.truncate/-/lodash.truncate-4.4.2.tgz", - "integrity": "sha512-jttmRe7bRse52OsWIMDLaXxWqRAmtIUccAQ3garviCqJjafXOfNMO0yMfNpdD6zbGaTU0P5Nz7e7gAT6cKmJRw==", - "license": "MIT" - }, - "node_modules/lodash.unset": { - "version": "4.18.0", - "resolved": "https://registry.npmjs.org/lodash.unset/-/lodash.unset-4.18.0.tgz", - "integrity": "sha512-7B95k5zjcJtuGuLqxOjT91g0Txwv1PWBlBDGzjW4kM8qgw2qfSgBo1QcIWGUpW9uN/1ZSKX9NHJzVJIgbeR5PQ==", - "license": "MIT" - }, "node_modules/lru-cache": { "version": "5.1.1", "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-5.1.1.tgz", @@ -4788,6 +4338,7 @@ "version": "10.2.5", "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", "integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==", + "dev": true, "license": "BlueOak-1.0.0", "dependencies": { "brace-expansion": "^5.0.5" @@ -4799,19 +4350,10 @@ "url": "https://github.com/sponsors/isaacs" } }, - "node_modules/minipass": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.3.tgz", - "integrity": "sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=16 || 14 >=14.17" - } - }, "node_modules/morgan": { - "version": "1.12.0", - "resolved": "https://registry.npmjs.org/morgan/-/morgan-1.12.0.tgz", - "integrity": "sha512-OHpTRQwn2ezasILW8iKe+Yww1XsfWsZIpUOLF7RDb2g5GwO3trPaRwi7+8BDiJ7HFx2Kg2mfUdCBcVhwYlOz2g==", + "version": "1.12.1", + "resolved": "https://registry.npmjs.org/morgan/-/morgan-1.12.1.tgz", + "integrity": "sha512-tljKC0ex20AjO58Ob/eZ53JloycbVswbVNCHx6V6VLGzqt/w8dIynVGL0G8qVjNKwiA7sYSogCrN6QtJ82IV+g==", "license": "MIT", "dependencies": { "basic-auth": "~2.0.1", @@ -5116,12 +4658,6 @@ "node": ">=6" } }, - "node_modules/package-json-from-dist": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz", - "integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==", - "license": "BlueOak-1.0.0" - }, "node_modules/parse-json": { "version": "5.2.0", "resolved": "https://registry.npmjs.org/parse-json/-/parse-json-5.2.0.tgz", @@ -5174,6 +4710,7 @@ "version": "3.1.1", "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", "integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==", + "dev": true, "license": "MIT", "engines": { "node": ">=8" @@ -5186,31 +4723,6 @@ "dev": true, "license": "MIT" }, - "node_modules/path-scurry": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-2.0.2.tgz", - "integrity": "sha512-3O/iVVsJAPsOnpwWIeD+d6z/7PmqApyQePUtCndjatj/9I5LylHvt5qluFaBT3I5h3r1ejfR056c+FCv+NnNXg==", - "license": "BlueOak-1.0.0", - "dependencies": { - "lru-cache": "^11.0.0", - "minipass": "^7.1.2" - }, - "engines": { - "node": "18 || 20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/path-scurry/node_modules/lru-cache": { - "version": "11.2.7", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.2.7.tgz", - "integrity": "sha512-aY/R+aEsRelme17KGQa/1ZSIpLpNYYrhcrepKTZgE+W3WM16YMCaPwOHLHsmopZHELU0Ojin1lPVxKR0MihncA==", - "license": "BlueOak-1.0.0", - "engines": { - "node": "20 || >=22" - } - }, "node_modules/path-to-regexp": { "version": "0.1.13", "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.13.tgz", @@ -5330,9 +4842,9 @@ } }, "node_modules/proxy-addr": { - "version": "2.0.7", - "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", - "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", + "version": "2.0.8", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz", + "integrity": "sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==", "license": "MIT", "dependencies": { "forwarded": "0.2.0", @@ -5340,15 +4852,10 @@ }, "engines": { "node": ">= 0.10" - } - }, - "node_modules/proxy-from-env": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-2.1.0.tgz", - "integrity": "sha512-cJ+oHTW1VAEa8cJslgmUZrc+sjRKgAKl3Zyse6+PV38hZe/V6Z14TbCuXcan9F9ghlz4QrFr2c92TNF82UkYHA==", - "license": "MIT", - "engines": { - "node": ">=10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" } }, "node_modules/pstree.remy": { @@ -5600,6 +5107,7 @@ "version": "7.8.5", "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", + "dev": true, "license": "ISC", "bin": { "semver": "bin/semver.js" @@ -5692,6 +5200,7 @@ "version": "2.0.0", "resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz", "integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==", + "dev": true, "license": "MIT", "dependencies": { "shebang-regex": "^3.0.0" @@ -5704,6 +5213,7 @@ "version": "3.0.0", "resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz", "integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==", + "dev": true, "license": "MIT", "engines": { "node": ">=8" @@ -5781,52 +5291,6 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/signal-exit": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz", - "integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==", - "license": "ISC", - "engines": { - "node": ">=14" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/simple-git": { - "version": "3.36.0", - "resolved": "https://registry.npmjs.org/simple-git/-/simple-git-3.36.0.tgz", - "integrity": "sha512-cGQjLjK8bxJw4QuYT7gxHw3/IouVESbhahSsHrX97MzCL1gu2u7oy38W6L2ZIGECEfIBG4BabsWDPjBxJENv9Q==", - "license": "MIT", - "dependencies": { - "@kwsites/file-exists": "^1.1.1", - "@kwsites/promise-deferred": "^1.1.1", - "@simple-git/args-pathspec": "^1.0.3", - "@simple-git/argv-parser": "^1.1.0", - "debug": "^4.4.0" - }, - "funding": { - "type": "github", - "url": "https://github.com/steveukx/git-js?sponsor=1" - } - }, - "node_modules/simple-git/node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, "node_modules/simple-update-notifier": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/simple-update-notifier/-/simple-update-notifier-2.0.0.tgz", @@ -6193,18 +5657,6 @@ "nodetouch": "bin/nodetouch.js" } }, - "node_modules/ts-pattern": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/ts-pattern/-/ts-pattern-4.3.0.tgz", - "integrity": "sha512-pefrkcd4lmIVR0LA49Imjf9DYLK8vtWhqBPA3Ya1ir8xCW0O2yjL9dsCVvI7pCodLC5q7smNpEtDR2yVulQxOg==", - "license": "MIT" - }, - "node_modules/tslib": { - "version": "2.4.1", - "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.4.1.tgz", - "integrity": "sha512-tGyy4dAjRIEwI7BzsB0lynWgOpfqjUdq91XXAlIWD2OwKBH7oCl/GZG/HT4BOHrTlPMOASlMQ7veyTqpmRcrNA==", - "license": "0BSD" - }, "node_modules/type-check": { "version": "0.4.0", "resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz", @@ -6381,6 +5833,7 @@ "version": "2.0.2", "resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz", "integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==", + "dev": true, "license": "ISC", "dependencies": { "isexe": "^2.0.0" diff --git a/package.json b/package.json index abbe9f6..b15aafa 100644 --- a/package.json +++ b/package.json @@ -14,13 +14,15 @@ "test": "jest" }, "dependencies": { + "@opensearch-project/opensearch": "2.13.0", + "config": "4.4.1", "cors": "^2.8.5", - "cve-core": "github:CVEProject/cve-core.git#dev-package-test", - "dotenv": "^6.1.0", + "dotenv": "^16.6.1", "express": "^4.22.2", "express-validator": "^7.2.1", - "morgan": "~1.12.0", - "swagger-ui-express": "^5.0.1" + "morgan": "~1.12.1", + "swagger-ui-express": "^5.0.1", + "validator": "^13.15.26" }, "devDependencies": { "@eslint/js": "^10.0.1", @@ -32,15 +34,11 @@ "swagger-autogen": "^2.23.7" }, "overrides": { - "adm-zip": "0.6.0", - "axios": "1.18.0", - "brace-expansion": "5.0.9", + "brace-expansion": "5.0.12", "js-yaml": "4.3.2", "lodash": "4.18.1", - "lodash.unset": "4.18.0", "path-to-regexp": "0.1.13", "picomatch": "2.3.2", - "qs": "6.16.0", - "simple-git": "3.36.0" + "qs": "6.16.0" } } diff --git a/routes/index.js b/routes/index.js index 7e5a5a5..b341283 100644 --- a/routes/index.js +++ b/routes/index.js @@ -71,7 +71,7 @@ function hasBodyField(req, field) { function isWebSearchWindowWithinLimit(_value, { req }) { const from = hasBodyField(req, 'from') ? Number(req.body.from) : 0; - const size = hasBodyField(req, 'size') ? Number(req.body.size) : constants.MAX_RESULTS_LIMIT; + const size = hasBodyField(req, 'size') ? Number(req.body.size) : constants.WEB_SEARCH_RESULTS_PER_PAGE; if (!Number.isFinite(from) || !Number.isFinite(size)) return true; @@ -198,7 +198,9 @@ router.post(

Retrieves all matching CVE records after applying the JSON request body as filters

Result Counts and Pagination

Direct searches report the exact number of matching records, including totals greater than 10,000, but requested pages must remain within the first 10,000 results.

-

CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. These searches evaluate at most 10,000 search candidates, so their reported totals and pages include only matches found within that candidate set. A warning is returned when that limit is reached.

+

CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. PIT is enabled by default on the server. With PIT explicitly disabled using SearchPitEnabled=false, these searches evaluate at most 10,000 search candidates, so their reported totals and pages include only matches found within that candidate set. Reaching that limit returns SEMANTIC_CANDIDATE_LIMIT_REACHED and pagination.totalIsExact=false.

+

With point-in-time (PIT) traversal enabled, semantic searches evaluate all candidate batches before returning exact totals and pagination.totalIsExact=true. Requested pages must still fit entirely within the first 10,000 matching results. Filling the requested page does not stop the scan because the total must also be computed. PIT is a server-side setting, not a request field; clients continue to use pageNumber and resultsPerPage, with no public cursor.

+

Each PIT semantic request uses one snapshot for its candidates and returned records. Separate page requests use separate snapshots, so changes between requests can still shift results. Incomplete traversal returns a sanitized error, not partial results, and does not restart or fall back to a bounded search. Warnings remain before records in successful responses. Direct searches and /webSearch do not use PIT.

#swagger.requestBody = { required: false, content: { @@ -336,7 +338,7 @@ content: { }, } #swagger.responses[502] = { -description: 'The search backend returned an invalid or unexpected response', +description: 'UPSTREAM_SEARCH_ERROR: The search backend returned an invalid or unexpected response. This includes malformed or incomplete PIT batches or full-page responses; no partial PIT result is returned.', content: { "application/json": { schema: { @@ -346,7 +348,7 @@ content: { }, } #swagger.responses[503] = { -description: 'The search service is temporarily unavailable', +description: 'SEARCH_UNAVAILABLE: The search service is temporarily unavailable, including a missing or expired PIT search context. PIT traversal is not silently restarted. Retry-After specifies the suggested delay.', content: { "application/json": { schema: { @@ -364,7 +366,7 @@ headers: { } } #swagger.responses[504] = { -description: 'Search exceeded its processing deadline', +description: 'SEARCH_TIMEOUT: Search exceeded its operation timeout or overall processing deadline, or a PIT search reported a timeout. No partial PIT result is returned. PIT cleanup has a separate bounded timeout and can add up to five seconds before the error response.', content: { "application/json": { schema: { @@ -484,7 +486,7 @@ router.post( #swagger.summary = "Raw query search" #swagger.description = "

Expected Behavior

-

Retrieves matching CVE records using the raw query expression supplied in the JSON request body. Service operators can disable this endpoint through runtime configuration.

+

Retrieves matching CVE records using the raw query expression supplied in the JSON request body. Service operators can disable this endpoint through runtime configuration. Standalone dates and date ranges search the supported record/container date fields; other expressions use raw query syntax with an AND default operator.

#swagger.requestBody = { required: true, content: { @@ -496,6 +498,7 @@ router.post( properties: { query: { type: "string", + description: "Raw search expression. Examples: remote code execution, apache*, 2024-01-01, or 2024-01-01..2024-01-31. Date ranges are inclusive and allow year/month endpoints; standalone years/months and quoted dates remain text queries. Date-query years must be between 1900 and 2100.", minLength: 1 }, sort: { diff --git a/routes/search.integration.test.js b/routes/search.integration.test.js index 63c3958..92908a1 100644 --- a/routes/search.integration.test.js +++ b/routes/search.integration.test.js @@ -2,14 +2,21 @@ const { request } = require('../test/http'); const constants = require('../utils/constants'); const mockSearchOpenSearch = jest.fn(); -const mockBasicSearch = jest.fn(); +const mockSearchOpenSearchPit = jest.fn(); +const mockWithOpenSearchPit = jest.fn(async (operation) => operation('route-test-pit')); +const mockIsSearchPitEnabled = jest.fn(() => false); + +jest.mock('../utils/featureFlags', () => ({ + ...jest.requireActual('../utils/featureFlags'), + isSearchPitEnabled: mockIsSearchPitEnabled +})); jest.mock('../utils/searchProvider', () => ({ - getBasicSearchManager: jest.fn(() => ({ - search: mockBasicSearch - })), getSearchRequestDeadlineMs: jest.fn(() => 30000), - searchOpenSearch: mockSearchOpenSearch + searchOpenSearch: mockSearchOpenSearch, + getSearchPitBatchSize: jest.fn(() => 1), + searchOpenSearchPit: mockSearchOpenSearchPit, + withOpenSearchPit: mockWithOpenSearchPit })); const { createApp } = require('../server'); @@ -27,25 +34,95 @@ function advancedResponse(totalValue, hits = []) { }; } -function basicResponse({ total = 0, hits = [] } = {}) { - return { - status: 'ok', - data: { - hits: { - total: { - value: total - }, - hits - } - }, - isOk: jest.fn(() => true) - }; -} - describe('search route integration', () => { beforeEach(() => { mockSearchOpenSearch.mockReset(); - mockBasicSearch.mockReset(); + mockSearchOpenSearchPit.mockReset(); + mockWithOpenSearchPit.mockClear(); + mockIsSearchPitEnabled.mockReset().mockReturnValue(false); + }); + + test('POST /search traverses semantic PIT batches and returns complete scoped records', async () => { + mockIsSearchPitEnabled.mockReturnValue(true); + const candidate = (id, vector) => ({ + sort: [id], + _source: { + cveMetadata: { cveId: id }, + containers: { adp: [{ metrics: [{ cvssV3_1: { vectorString: vector } }] }] } + } + }); + const fullRecord = { + cveMetadata: { cveId: 'CVE-2024-0002' }, + containers: { cna: { title: 'CNA content' }, adp: [{ title: 'ADP content' }] } + }; + const pitResponse = (hits) => ({ + timed_out: false, + _shards: { total: 1, successful: 1, skipped: 0, failed: 0 }, + hits: { total: { value: hits.length, relation: 'eq' }, hits } + }); + mockSearchOpenSearchPit + .mockResolvedValueOnce(pitResponse([candidate('CVE-2024-0001', 'AV:N/AC:H')])) + .mockResolvedValueOnce(pitResponse([candidate('CVE-2024-0002', 'AV:N/AC:L')])) + .mockResolvedValueOnce(pitResponse([])) + .mockResolvedValueOnce(pitResponse([{ _source: fullRecord }])); + + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { scope: 'adp_only', cvssV3_1Metrics: 'AV:N/AC:L', resultsPerPage: 1 } + }); + + expect(response.status).toBe(200); + expect(response.body.pagination).toStrictEqual({ + page: 1, + pageSize: 1, + resultsReturned: 1, + totalResults: 1, + totalPages: 1, + totalIsExact: true + }); + expect(response.body.records).toStrictEqual([fullRecord]); + expect(Object.keys(response.body)).toStrictEqual(['apiVersion', 'status', 'warnings', 'pagination', 'records']); + expect(response.body.warnings).toStrictEqual([]); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + expect(mockWithOpenSearchPit).toHaveBeenCalledTimes(1); + expect(mockSearchOpenSearchPit).toHaveBeenCalledTimes(4); + expect(mockSearchOpenSearchPit.mock.calls.every(([pitId]) => pitId === 'route-test-pit')).toBe(true); + expect(mockSearchOpenSearchPit.mock.calls[1][1].search_after).toStrictEqual(['CVE-2024-0001']); + }); + + test.each([ + [{ timed_out: true }, 504, 'SEARCH_TIMEOUT'], + [{ _shards: { total: 2, successful: 1, skipped: 0, failed: 0 } }, 502, 'UPSTREAM_SEARCH_ERROR'], + [ + { error: { type: 'search_context_missing_exception', reason: 'private PIT identifier' } }, + 503, + 'SEARCH_UNAVAILABLE' + ] + ])('POST /search fails closed on incomplete PIT responses: %p', async (overrides, status, code) => { + mockIsSearchPitEnabled.mockReturnValue(true); + mockSearchOpenSearchPit.mockResolvedValueOnce({ + timed_out: false, + _shards: { total: 1, successful: 1, skipped: 0, failed: 0 }, + hits: { hits: [] }, + ...overrides + }); + + const response = await request(createApp(), { + method: 'POST', + path: '/search', + body: { cvssV3_1Metrics: 'AV:N/AC:L' } + }); + + expect(response.status).toBe(status); + expect(response.body).toMatchObject({ error: code }); + expect(response.body).not.toHaveProperty('records'); + expect(response.body).not.toHaveProperty('pagination'); + expect(response.text).not.toContain('private'); + if (status === 503) expect(response.headers['retry-after']).toBe('5'); + expect(mockWithOpenSearchPit).toHaveBeenCalledTimes(1); + expect(mockSearchOpenSearchPit).toHaveBeenCalledTimes(1); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); }); test('POST /search validates, builds the OpenSearch request, and formats pagination metadata', async () => { @@ -284,8 +361,49 @@ describe('search route integration', () => { expect(mockSearchOpenSearch).not.toHaveBeenCalled(); }); - test('POST /webSearch normalizes body fields and formats BasicSearchManager responses', async () => { - mockBasicSearch.mockResolvedValue(basicResponse({ total: 1, hits: [{ _id: 'CVE-2024-0002' }] })); + test.each([ + [500, 1, 1], + [4999, 1, 1], + [9500, 500, 500], + [9999, 1, 1], + [9975, undefined, 25] + ])('POST /webSearch preserves page size at offset %s with size %s', async (from, size, expectedSize) => { + mockSearchOpenSearch.mockImplementation( + async (body) => + advancedResponse( + 20000, + Array.from({ length: body.size }, (_, index) => ({ _id: String(index) })) + ).data + ); + + const response = await request(createApp(), { + method: 'POST', + path: '/webSearch', + body: { query: 'apache', from, ...(size === undefined ? {} : { size }) } + }); + + expect(response.status).toBe(200); + expect(mockSearchOpenSearch).toHaveBeenCalledTimes(1); + expect(mockSearchOpenSearch).toHaveBeenCalledWith(expect.objectContaining({ from, size: expectedSize })); + expect(response.body.data).toHaveLength(expectedSize); + }); + + test.each([{ from: 9501, size: 500 }, { from: 9976 }, { from: 10000, size: 1 }, { size: 501 }])( + 'POST /webSearch rejects pagination outside its limits: %j', + async (pagination) => { + const response = await request(createApp(), { + method: 'POST', + path: '/webSearch', + body: { query: 'apache', ...pagination } + }); + + expect(response.status).toBe(400); + expect(mockSearchOpenSearch).not.toHaveBeenCalled(); + } + ); + + test('POST /webSearch normalizes fields, builds DSL, and formats OpenSearch responses', async () => { + mockSearchOpenSearch.mockResolvedValue(advancedResponse(1, [{ _id: 'CVE-2024-0002' }]).data); const response = await request(createApp(), { method: 'POST', @@ -302,7 +420,9 @@ describe('search route integration', () => { }); expect(response.status).toBe(200); - expect(mockBasicSearch).toHaveBeenCalledWith('apache', { + expect(mockSearchOpenSearch).toHaveBeenCalledWith({ + query: { query_string: { query: 'apache', default_operator: 'AND' } }, + track_total_hits: true, sort: [ { _score: { @@ -313,7 +433,9 @@ describe('search route integration', () => { from: 5, size: 10 }); - expect(response.body.searchMetadata.notes).toStrictEqual([]); + expect(response.body.searchMetadata.notes).toStrictEqual(['general search text string']); + expect(response.body.searchMetadata).not.toHaveProperty('cveCore'); + expect(response.body.searchMetadata).not.toHaveProperty('cveMitreCore'); expect(response.body.searchMetadata.errors).toStrictEqual([]); expect(response.body.resultsTotal).toBe(1); expect(response.body.data).toStrictEqual([{ _id: 'CVE-2024-0002' }]); diff --git a/routes/swagger.js b/routes/swagger.js index 2dcc8c8..69ee05c 100644 --- a/routes/swagger.js +++ b/routes/swagger.js @@ -10,7 +10,7 @@ const projectRoot = path.join(__dirname, '..'); const outputFile = path.join(projectRoot, 'api-docs', 'openapi.json'); const routes = [path.join(__dirname, 'index.js')]; const maxSearchWindowLabel = constants.MAX_SEARCH_WINDOW.toLocaleString('en-US'); -const semanticPostFilterDescription = `CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. They evaluate at most ${maxSearchWindowLabel} search candidates, so reported totals and pages include only matches found within that candidate set.`; +const semanticPostFilterDescription = `CVSS vector filters, SSVC filters, date-bounded positive knownExploited searches, cpeName, and virtualMatchString use semantic post-filtering. PIT is enabled by default on the server. All candidate batches must complete before a successful response reports exact totals; traversal failures return an error instead of partial results. With PIT explicitly disabled using SearchPitEnabled=false, these searches evaluate at most ${maxSearchWindowLabel} search candidates, so reported totals and pages include only matches found within that candidate set. Reaching that limit returns SEMANTIC_CANDIDATE_LIMIT_REACHED and totalIsExact=false. Requested pages must still remain within the first ${maxSearchWindowLabel} matching results. PIT is a server-side setting, not a request field.`; function searchRequestProperty(description, schema) { return { @@ -382,6 +382,8 @@ const doc = { }, SearchWarning: { type: 'object', + description: + 'Non-fatal search interpretation information. SEMANTIC_CANDIDATE_LIMIT_REACHED applies only to bounded semantic searches with PIT disabled. BROAD_VIRTUAL_MATCH_STRING can occur in either mode; its message describes the candidate limit when PIT is disabled and increased processing time when PIT is enabled.', additionalProperties: false, required: ['code', 'message'], properties: { @@ -431,16 +433,16 @@ const doc = { totalResults: { type: 'integer', minimum: 0, - description: `For direct searches, the exact number of matching records, which may be greater than ${maxSearchWindowLabel}. For semantic post-filtered searches, the number of matches found within the bounded ${maxSearchWindowLabel}-candidate set.` + description: `For direct searches, the exact number of matching records, which may be greater than ${maxSearchWindowLabel}. For semantic searches with PIT enabled (the default), the exact number of matches across the completed snapshot traversal. With PIT explicitly disabled, the number of matches found within the bounded ${maxSearchWindowLabel}-candidate set. Separate page requests use separate snapshots.` }, totalPages: { type: 'integer', minimum: 0, - description: `Pages derived from totalResults and pageSize. Semantic post-filtered searches derive this value from matches found within the bounded ${maxSearchWindowLabel}-candidate set.` + description: `Pages derived from totalResults and pageSize, using the same bounded or exact total as totalResults. This value can exceed the number of accessible pages: each requested page must still fit entirely within the first ${maxSearchWindowLabel} matching results in either mode.` }, totalIsExact: { type: 'boolean', - description: `True when totalResults represents every match. False when a semantic search reaches the ${maxSearchWindowLabel}-candidate limit and additional matches may exist.` + description: `True for direct searches and successful, fully completed PIT semantic searches (the default mode). With PIT explicitly disabled, false when a semantic search reaches the ${maxSearchWindowLabel}-candidate limit and additional matches may exist; true otherwise. A failed PIT traversal returns an error rather than a success with a partial total.` } } }, @@ -465,7 +467,8 @@ const doc = { }, warnings: { type: 'array', - description: 'Non-fatal conditions that may affect how the results should be interpreted.', + description: + 'Non-fatal conditions that may affect how the results should be interpreted. This field is emitted before records. Successful PIT traversal does not return a semantic candidate-limit warning, but other interpretation warnings still apply.', items: { $ref: '#/components/schemas/SearchWarning' } @@ -489,13 +492,24 @@ const doc = { searchMetadata: { type: 'object', additionalProperties: true, + description: + 'API-owned raw-query metadata. Includes the response-format version, not dependency package versions.', required: ['searchStatus', 'version', 'errors'], properties: { searchStatus: { type: 'string' }, version: { - type: 'string' + type: 'string', + description: 'Version of the raw-query response format.' + }, + notes: { + type: 'array', + items: { type: 'string' } + }, + queryTime_ms: { + type: 'number', + description: 'Query processing duration when available.' }, errors: { type: 'array', diff --git a/server.js b/server.js index 02f4eff..6cc0233 100644 --- a/server.js +++ b/server.js @@ -1,4 +1,4 @@ -const dotenv = require('dotenv'); +require('dotenv/config'); const cors = require('cors'); const express = require('express'); const swaggerUi = require('swagger-ui-express'); @@ -9,8 +9,6 @@ const router = require('./routes/index'); const errors = require('./utils/errors'); const requestLogging = require('./utils/requestLogging'); -dotenv.load(); - const error = new errors.errors(); const swaggerSetupOptions = { diff --git a/test/postman/CVE-Search-API.postman_collection.json b/test/postman/CVE-Search-API.postman_collection.json index c87b1c5..c9ad4c1 100644 --- a/test/postman/CVE-Search-API.postman_collection.json +++ b/test/postman/CVE-Search-API.postman_collection.json @@ -2,11 +2,12 @@ "info": { "_postman_id": "28911974-2b15-4ddf-a88f-3fa6fe001a50", "name": "CVE Search API - Manual Contract Tests", - "description": "Positive filter coverage, intentional combinations, validation failures, and opt-in fault-injection checks for POST /search. The complete collection targets a fully populated index like those used by shared managed environments. Data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly. The separate /webSearch contract is intentionally excluded.", + "description": "Positive filter coverage, intentional combinations, validation failures, and opt-in fault-injection checks for POST /search. Set expectedSearchPitEnabled to match the server configuration (default true); it affects assertions only and does not enable PIT. The complete collection targets a fully populated index like those used by shared managed environments. Data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly. The separate /webSearch contract is intentionally excluded.", "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json" }, "variable": [ { "key": "baseUrl", "value": "http://localhost:4000", "type": "string" }, + { "key": "expectedSearchPitEnabled", "value": "true", "type": "string" }, { "key": "runFaultInjection", "value": "false", "type": "string" } ], "item": [ @@ -122,7 +123,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -137,6 +142,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is at least 1', () => pm.expect(payload.pagination.totalResults).to.be.at.least(1));", "}" ] @@ -166,7 +179,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -181,6 +198,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is at least 1', () => pm.expect(payload.pagination.totalResults).to.be.at.least(1));", "}" ] @@ -216,7 +241,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -231,6 +260,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -272,7 +309,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -287,9 +328,22 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 0', () => pm.expect(payload.pagination.totalResults).to.eql(0));", "pm.test('returned record count is 0', () => pm.expect(payload.records.length).to.eql(0));", - "const expectedWarnings = [{\"code\":\"CVE_IDS_NOT_RETURNED\",\"message\":\"CVE Records do not exist for years before 1999. The listed requested IDs cannot produce results.\",\"cveIds\":[\"CVE-1998-0001\"]}];", + "const expectedWarnings = [{\"code\":\"CVE_IDS_NOT_RETURNED\",\"message\":\"CVE Records do not exist for years before 1999. The listed requested IDs cannot produce results.\",\"cveIds\":[\"CVE-1998-0001\"]}].map((warning) => {", + " if (expectPit && warning.code === 'BROAD_VIRTUAL_MATCH_STRING') {", + " return { ...warning, message: 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.' };", + " }", + " return warning;", + "});", "pm.test('expected warnings are returned', () => pm.expect(payload.warnings).to.deep.include.members(expectedWarnings));", "}" ] @@ -329,7 +383,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -344,10 +402,23 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", - "const expectedWarnings = [{\"code\":\"CVE_IDS_NOT_RETURNED\",\"message\":\"CVE Records do not exist for years before 1999. The listed requested IDs cannot produce results.\",\"cveIds\":[\"CVE-1998-0001\"]}];", + "const expectedWarnings = [{\"code\":\"CVE_IDS_NOT_RETURNED\",\"message\":\"CVE Records do not exist for years before 1999. The listed requested IDs cannot produce results.\",\"cveIds\":[\"CVE-1998-0001\"]}].map((warning) => {", + " if (expectPit && warning.code === 'BROAD_VIRTUAL_MATCH_STRING') {", + " return { ...warning, message: 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.' };", + " }", + " return warning;", + "});", "pm.test('expected warnings are returned', () => pm.expect(payload.warnings).to.deep.include.members(expectedWarnings));", "}" ] @@ -387,7 +458,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -402,6 +477,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 2', () => pm.expect(payload.pagination.totalResults).to.eql(2));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\",\"CVE-2022-31279\"]));", @@ -443,7 +526,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -458,6 +545,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", "pm.test('returned record count is 0', () => pm.expect(payload.records.length).to.eql(0));", "}" @@ -498,7 +593,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -513,6 +612,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", "pm.test('returned record count is 0', () => pm.expect(payload.records.length).to.eql(0));", "}" @@ -559,7 +666,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -574,6 +685,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -615,7 +734,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -630,6 +753,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -671,7 +802,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -686,6 +821,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 0', () => pm.expect(payload.pagination.totalResults).to.eql(0));", "pm.test('returned record count is 0', () => pm.expect(payload.records.length).to.eql(0));", "}" @@ -726,7 +869,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -741,6 +888,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -782,7 +937,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -797,6 +956,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -838,7 +1005,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -853,6 +1024,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2022-31640\"]));", @@ -900,7 +1079,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -915,6 +1098,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -956,7 +1147,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -971,6 +1166,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1012,7 +1215,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1027,6 +1234,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1068,7 +1283,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1083,6 +1302,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1124,7 +1351,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1139,6 +1370,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 2', () => pm.expect(payload.pagination.totalResults).to.eql(2));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\",\"CVE-2022-31279\"]));", @@ -1180,7 +1419,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1195,6 +1438,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1242,7 +1493,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1257,6 +1512,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1298,7 +1561,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1313,6 +1580,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1354,7 +1629,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1369,6 +1648,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1410,7 +1697,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1425,6 +1716,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1466,7 +1765,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1481,6 +1784,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1522,7 +1833,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1537,6 +1852,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1578,7 +1901,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1593,6 +1920,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1634,7 +1969,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1649,6 +1988,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1690,7 +2037,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1705,6 +2056,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1746,7 +2105,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1761,6 +2124,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1802,7 +2173,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1817,6 +2192,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2022-31640\"]));", @@ -1864,7 +2247,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1879,6 +2266,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1920,7 +2315,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1935,6 +2334,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -1976,7 +2383,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -1991,6 +2402,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2032,7 +2451,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2047,6 +2470,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2088,7 +2519,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2103,6 +2538,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2144,7 +2587,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2159,6 +2606,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2206,7 +2661,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2221,6 +2680,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", @@ -2262,7 +2729,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2277,6 +2748,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", @@ -2318,7 +2797,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2333,6 +2816,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2374,7 +2865,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2389,6 +2884,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", @@ -2430,7 +2933,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2445,10 +2952,23 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", - "const expectedWarnings = [{\"code\":\"BROAD_VIRTUAL_MATCH_STRING\",\"message\":\"Wildcard or omitted leading CPE components broaden this search and may cause results to reach the semantic candidate limit.\",\"components\":[\"vendor\",\"product\"]}];", + "const expectedWarnings = [{\"code\":\"BROAD_VIRTUAL_MATCH_STRING\",\"message\":\"Wildcard or omitted leading CPE components broaden this search and may cause results to reach the semantic candidate limit.\",\"components\":[\"vendor\",\"product\"]}].map((warning) => {", + " if (expectPit && warning.code === 'BROAD_VIRTUAL_MATCH_STRING') {", + " return { ...warning, message: 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.' };", + " }", + " return warning;", + "});", "pm.test('expected warnings are returned', () => pm.expect(payload.warnings).to.deep.include.members(expectedWarnings));", "}" ] @@ -2488,7 +3008,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2503,10 +3027,23 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", - "const expectedWarnings = [{\"code\":\"CPE_COMPONENTS_DEFAULTED\",\"message\":\"Omitted trailing cpeName components were treated as wildcard components.\",\"components\":[\"update\",\"edition\",\"language\",\"sw_edition\",\"target_sw\",\"target_hw\",\"other\"]}];", + "const expectedWarnings = [{\"code\":\"CPE_COMPONENTS_DEFAULTED\",\"message\":\"Omitted trailing cpeName components were treated as wildcard components.\",\"components\":[\"update\",\"edition\",\"language\",\"sw_edition\",\"target_sw\",\"target_hw\",\"other\"]}].map((warning) => {", + " if (expectPit && warning.code === 'BROAD_VIRTUAL_MATCH_STRING') {", + " return { ...warning, message: 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.' };", + " }", + " return warning;", + "});", "pm.test('expected warnings are returned', () => pm.expect(payload.warnings).to.deep.include.members(expectedWarnings));", "}" ] @@ -2546,7 +3083,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2561,6 +3102,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", @@ -2602,7 +3151,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2617,6 +3170,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", @@ -2658,7 +3219,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2673,6 +3238,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2025-23022\"]));", @@ -2720,7 +3293,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2735,6 +3312,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2776,7 +3361,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2791,6 +3380,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2832,7 +3429,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2847,6 +3448,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2888,7 +3497,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2903,6 +3516,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -2944,7 +3565,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -2959,6 +3584,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3000,7 +3633,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3015,6 +3652,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3056,7 +3701,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3071,6 +3720,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3118,7 +3775,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3133,6 +3794,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2020-1472\"]));", @@ -3174,7 +3843,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3189,6 +3862,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3230,7 +3911,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3245,6 +3930,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2020-1472\"]));", @@ -3286,7 +3979,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3301,6 +3998,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2020-1472\"]));", @@ -3342,7 +4047,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3357,6 +4066,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2020-1472\"]));", @@ -3404,7 +4121,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3419,6 +4140,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3460,7 +4189,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3475,6 +4208,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3516,7 +4257,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3531,6 +4276,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3572,7 +4325,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3587,6 +4344,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 0', () => pm.expect(payload.pagination.totalResults).to.eql(0));", "pm.test('returned record count is 0', () => pm.expect(payload.records.length).to.eql(0));", "}" @@ -3627,7 +4392,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3642,6 +4411,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2022-38625\"]));", @@ -3683,7 +4460,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3698,6 +4479,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3745,7 +4534,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3760,6 +4553,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2024-5981\"]));", @@ -3801,7 +4602,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3816,6 +4621,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2022-31640\"]));", @@ -3857,7 +4670,11 @@ "if (isExpectedSuccess) {", " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", " const payload = pm.response.json();", + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -3872,6 +4689,14 @@ " pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);", " pm.expect(payload).not.to.have.property('data');", "});", + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + " pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);", + " if (expectPit) {", + " pm.expect(payload.pagination.totalIsExact).to.eql(true);", + " pm.expect(candidateLimitReached).to.eql(false);", + " }", + "});", "pm.test('total is 1', () => pm.expect(payload.pagination.totalResults).to.eql(1));", " const returnedIds = payload.records.map((record) => record?.cveMetadata?.cveId);", "pm.test('expected CVE IDs are returned', () => pm.expect(returnedIds).to.include.members([\"CVE-2020-1472\"]));", @@ -6961,6 +7786,11 @@ "pm.test('expected error message is returned', () => {", " const match = errors.find((entry) => entry.error === expectedError.error);", " pm.expect(match?.message).to.eql(expectedError.message);", + "});", + "pm.test('failed search does not return partial records or totals', () => {", + " const payload = pm.response.json();", + " pm.expect(payload).not.to.have.property('records');", + " pm.expect(payload).not.to.have.property('pagination');", "});" ] } @@ -7020,6 +7850,11 @@ "pm.test('expected error message is returned', () => {", " const match = errors.find((entry) => entry.error === expectedError.error);", " pm.expect(match?.message).to.eql(expectedError.message);", + "});", + "pm.test('failed search does not return partial records or totals', () => {", + " const payload = pm.response.json();", + " pm.expect(payload).not.to.have.property('records');", + " pm.expect(payload).not.to.have.property('pagination');", "});" ] } @@ -7079,6 +7914,11 @@ "pm.test('expected error message is returned', () => {", " const match = errors.find((entry) => entry.error === expectedError.error);", " pm.expect(match?.message).to.eql(expectedError.message);", + "});", + "pm.test('failed search does not return partial records or totals', () => {", + " const payload = pm.response.json();", + " pm.expect(payload).not.to.have.property('records');", + " pm.expect(payload).not.to.have.property('pagination');", "});" ] } @@ -7101,6 +7941,134 @@ } }, "response": [] + }, + { + "name": "PIT snapshot unavailable", + "event": [ + { + "listen": "prerequest", + "script": { + "type": "text/javascript", + "exec": [ + "const environmentSetting = pm.environment.get('runFaultInjection');", + "const setting = environmentSetting === undefined ? pm.collectionVariables.get('runFaultInjection') : environmentSetting;", + "if (setting !== 'true') {", + " console.log('Skipping PIT snapshot unavailable; set runFaultInjection=true to enable it.');", + " pm.execution.skipRequest();", + "}" + ] + } + }, + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "pm.test('status is 503', () => pm.response.to.have.status(503));", + "pm.test('X-Request-ID is present', () => {", + " pm.expect(pm.response.headers.get('X-Request-ID')).to.be.a('string').and.not.empty;", + "});", + "pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", + "const payload = pm.response.json();", + "const errors = Array.isArray(payload) ? payload : [payload];", + "const expectedError = {\"error\":\"SEARCH_UNAVAILABLE\",\"message\":\"Search service is temporarily unavailable. Please try again later.\"};", + "pm.test('expected error code is returned', () => {", + " pm.expect(errors.map((entry) => entry.error)).to.include(expectedError.error);", + "});", + "pm.test('expected error message is returned', () => {", + " const match = errors.find((entry) => entry.error === expectedError.error);", + " pm.expect(match?.message).to.eql(expectedError.message);", + "});", + "pm.test('failed search does not return partial records or totals', () => {", + " const payload = pm.response.json();", + " pm.expect(payload).not.to.have.property('records');", + " pm.expect(payload).not.to.have.property('pagination');", + "});" + ] + } + } + ], + "request": { + "method": "POST", + "header": [{ "key": "Content-Type", "value": "application/json", "type": "text" }], + "url": { + "raw": "{{baseUrl}}/search", + "host": ["{{baseUrl}}"], + "path": ["search"], + "query": [] + }, + "description": "Requires server-side SearchPitEnabled=true. Use an isolated test double or proxy to return a structured search_context_missing_exception during a PIT candidate search. No PIT ID is accepted from this request body. Enable runFaultInjection only for this prepared condition.", + "body": { + "mode": "raw", + "raw": "{\n \"cveIds\": \"CVE-2025-23022\",\n \"virtualMatchString\": \"cpe:2.3:a:freetype:freetype\"\n}", + "options": { "raw": { "language": "json" } } + } + }, + "response": [] + }, + { + "name": "Incomplete PIT candidate batch", + "event": [ + { + "listen": "prerequest", + "script": { + "type": "text/javascript", + "exec": [ + "const environmentSetting = pm.environment.get('runFaultInjection');", + "const setting = environmentSetting === undefined ? pm.collectionVariables.get('runFaultInjection') : environmentSetting;", + "if (setting !== 'true') {", + " console.log('Skipping Incomplete PIT candidate batch; set runFaultInjection=true to enable it.');", + " pm.execution.skipRequest();", + "}" + ] + } + }, + { + "listen": "test", + "script": { + "type": "text/javascript", + "exec": [ + "pm.test('status is 502', () => pm.response.to.have.status(502));", + "pm.test('X-Request-ID is present', () => {", + " pm.expect(pm.response.headers.get('X-Request-ID')).to.be.a('string').and.not.empty;", + "});", + "pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", + "const payload = pm.response.json();", + "const errors = Array.isArray(payload) ? payload : [payload];", + "const expectedError = {\"error\":\"UPSTREAM_SEARCH_ERROR\",\"message\":\"Search backend returned an error. Please try again later.\"};", + "pm.test('expected error code is returned', () => {", + " pm.expect(errors.map((entry) => entry.error)).to.include(expectedError.error);", + "});", + "pm.test('expected error message is returned', () => {", + " const match = errors.find((entry) => entry.error === expectedError.error);", + " pm.expect(match?.message).to.eql(expectedError.message);", + "});", + "pm.test('failed search does not return partial records or totals', () => {", + " const payload = pm.response.json();", + " pm.expect(payload).not.to.have.property('records');", + " pm.expect(payload).not.to.have.property('pagination');", + "});" + ] + } + } + ], + "request": { + "method": "POST", + "header": [{ "key": "Content-Type", "value": "application/json", "type": "text" }], + "url": { + "raw": "{{baseUrl}}/search", + "host": ["{{baseUrl}}"], + "path": ["search"], + "query": [] + }, + "description": "Requires server-side SearchPitEnabled=true. Use an isolated test double or proxy to return an empty PIT batch with timed_out=false but unaccounted shards, such as total=2, successful=1, skipped=0, failed=0. Enable runFaultInjection only for this prepared condition.", + "body": { + "mode": "raw", + "raw": "{\n \"cveIds\": \"CVE-2025-23022\",\n \"virtualMatchString\": \"cpe:2.3:a:freetype:freetype\"\n}", + "options": { "raw": { "language": "json" } } + } + }, + "response": [] } ] } diff --git a/test/postman/README.md b/test/postman/README.md index c4883b6..7807c4d 100644 --- a/test/postman/README.md +++ b/test/postman/README.md @@ -19,11 +19,29 @@ Deployment-specific environment files are intentionally ignored by Git so deploy 2. Import `CVE-Search-API.postman_collection.json` into Postman. 3. Import `local.postman_environment.json` or an ignored deployment-specific environment file. 4. Confirm that `baseUrl` points to the intended API instance. The local environment uses port `4000`, while the API defaults to `3000`; set `PORT=4000` before starting the API or update `baseUrl` accordingly. -5. Open the collection runner and run folders `00` through `16`. +5. Set `expectedSearchPitEnabled` to match the server's PIT configuration (default `true`). Leave `runFaultInjection=false`. +6. Open the collection runner and run folders `00` through `16`. Folders are grouped by service checks, positive filter coverage, intentional combinations, and expected validation failures. Positive search requests assert the response contract and use stable CVE fixtures to check totals or returned IDs. Negative requests assert the exact HTTP status, error code, and error message. -The collection checks that each response includes an `X-Request-ID`. Search responses also must contain the documented API version, status, warnings, pagination, and CVE Record array fields, without search-backend hit wrappers. +The collection checks that each response includes an `X-Request-ID`. Search responses also must contain the documented API version, status, warnings, pagination, and CVE Record array fields in that order, without search-backend hit wrappers. Warnings appear before records. + +## PIT Modes + +`expectedSearchPitEnabled` is a Postman assertion setting, not an API request field or a server feature switch. It defaults to `true` at collection scope. Override it with `true` or `false` in the selected environment, or change the collection value when no environment override exists. Values are case-insensitive; other values fail the assertion. Environment values take precedence over collection values. The collection does not discover the deployed mode automatically. Re-import the regenerated collection to pick up the new default, and check for older environment overrides. + +| Server mode | Postman setting | Successful semantic search expectations | +| ----------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `SearchPitEnabled=true` (default) | `expectedSearchPitEnabled=true` | Completes all candidate batches before returning exact totals. No candidate-limit warning is allowed. The broad virtual-match warning describes increased processing time. | +| `SearchPitEnabled=false` (rollback) | `expectedSearchPitEnabled=false` | Evaluates up to 10,000 candidates. Reaching the limit produces `SEMANTIC_CANDIDATE_LIMIT_REACHED` and `pagination.totalIsExact=false`. Below the limit, the total is exact. The broad virtual-match warning describes the candidate limit. | + +The collection checks that total exactness agrees with candidate-limit warnings in both modes. In PIT mode, every successful search must have an exact total. The fixture-constrained broad virtual-match request checks the mode-specific warning message, helping detect a mismatched Postman expectation without requiring a large scan. + +For an approved two-mode comparison, use an isolated API instance that satisfies the [PIT deployment prerequisites](../../README.md#pit-implementation-status). Explicitly set `SearchPitEnabled=false`, restart the API, set the Postman expectation to `false`, and run folders `00` through `16` for the bounded baseline. Then set `SearchPitEnabled=true` (or remove the override to use the default), restart it, set the Postman expectation to `true`, and repeat the same folders. Only server configuration selects the mode. Restore the server and Postman settings afterwards. This procedure is guidance, not evidence that either run has been performed against your deployment. + +In both modes, requested pages must fit within the first 10,000 matching results, and no public cursor is supported. A total above 10,000 does not make later pages accessible. Each PIT HTTP request opens a new snapshot, so separate page requests can see intervening index updates. + +These small fixture searches verify response behavior and filters, not traversal beyond 10,000 candidates, concurrent index-update consistency, PIT resource cleanup, or production capacity. Those require separately approved controlled fixtures, backend observation, and lifecycle/integration validation. A successful `/api/ready-check` does not verify PIT permissions or index-wide CVE ID uniqueness. Do not run a broad index-wide query solely to prove that PIT is active. ## Environments @@ -32,8 +50,8 @@ The collection uses the `baseUrl` variable and does not embed deployment credent 1. Duplicate `local.postman_environment.json`. 2. Rename the copy for the target deployment using one of the ignored filenames listed above. 3. Give the environment a distinct Postman `id` and `name`. -4. Replace only the `baseUrl` value with the approved deployment URL. -5. Keep `runFaultInjection` set to `false` unless performing an explicitly approved fault test. +4. Replace the `baseUrl` value with the approved deployment URL. Add an `expectedSearchPitEnabled` environment override matching that deployment, or leave the collection default of `true` when the server uses PIT. +5. Keep `runFaultInjection` set to `false` unless performing an explicitly approved fault test. The collection's skip guard honors the selected environment value before the collection value. The deployment file should resemble: @@ -42,6 +60,7 @@ The deployment file should resemble: "name": "CVE Search API - ", "values": [ { "key": "baseUrl", "value": "https://", "type": "default", "enabled": true }, + { "key": "expectedSearchPitEnabled", "value": "true", "type": "default", "enabled": true }, { "key": "runFaultInjection", "value": "false", "type": "default", "enabled": true } ] } @@ -70,25 +89,29 @@ Setting `runFaultInjection` to `true` only enables the requests; it does not cre The selected environment's `runFaultInjection` value takes precedence over the collection variable. When the environment does not define it, the collection default is used. Only the string `true` enables these requests; `false`, an empty value, and other values keep them disabled. Set the environment value back to `false` after each approved fault check. -| Request | Required condition | Expected response | -| ---------------------------- | ------------------------------------------------------------------------------------------- | --------------------------- | -| OpenSearch readiness failure | Make the configured OpenSearch index unavailable to the API. | `503 SEARCH_NOT_READY` | -| Search service unavailable | Make search-backend connectivity unavailable, such as by targeting a closed local port. | `503 SEARCH_UNAVAILABLE` | -| Search timeout | Deliberately lower the request deadline or delay OpenSearch beyond the configured deadline. | `504 SEARCH_TIMEOUT` | -| Unexpected upstream response | Use a test double or proxy that returns an invalid or incomplete OpenSearch response. | `502 UPSTREAM_SEARCH_ERROR` | +| Request | Required condition | Expected response | +| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | +| OpenSearch readiness failure | Make the configured OpenSearch index unavailable to the API. | `503 SEARCH_NOT_READY` | +| Search service unavailable | Make search-backend connectivity unavailable. Permission or index errors can map to different statuses and are not this fixture. | `503 SEARCH_UNAVAILABLE` | +| Search timeout | Deliberately lower the request deadline or delay OpenSearch beyond the configured deadline. | `504 SEARCH_TIMEOUT` | +| Unexpected upstream response | Use a test double or proxy that returns an invalid or incomplete OpenSearch response. | `502 UPSTREAM_SEARCH_ERROR` | +| PIT snapshot unavailable | Enable server-side PIT and inject a structured `search_context_missing_exception` on a PIT candidate request through an isolated test double or proxy. | `503 SEARCH_UNAVAILABLE` | +| Incomplete PIT candidate batch | Enable server-side PIT and inject an empty batch with `timed_out=false` and `_shards: { total: 2, successful: 1, skipped: 0, failed: 0 }`. | `502 UPSTREAM_SEARCH_ERROR` | Permission denial (upstream HTTP 403) and a missing index (upstream HTTP 404) instead map to `502 UPSTREAM_SEARCH_ERROR` on `/search`; they are not valid ways to trigger the collection's `503 SEARCH_UNAVAILABLE` assertion. The readiness endpoint maps either failure to `503 SEARCH_NOT_READY`. These are endpoint-specific error mappings, not contradictory test results. Run a fault check as follows: 1. Target a disposable local environment and prepare exactly one condition from the table. -2. Set `runFaultInjection` to `true` in the selected Postman environment. +2. Set `runFaultInjection` to `true` in the selected Postman environment. For the PIT-specific requests, the server must already have PIT enabled and `expectedSearchPitEnabled` should be `true`. 3. Open folder `17` and send only the request for the prepared condition. 4. Confirm the asserted HTTP status, sanitized error code, message, and `X-Request-ID`. 5. Set `runFaultInjection` back to `false` and restore the normal application and dependency configuration. Do not enable fault injection against a shared or production environment without the target owner's approval. A WAF or load balancer can return a different error contract than the application-level response asserted by this folder. +Postman cannot expire an internal snapshot or manufacture an incomplete shard response by changing ordinary search filters. The PIT requests above verify application behavior only after a controlled backend fault is prepared; they do not create that fault. Do not add PIT IDs to the request body or delete unrelated contexts. PIT failures must not yield partial records or totals, retry with a fresh snapshot, or fall back to the bounded path. Backend lifecycle tests are needed to verify those internal actions. Cleanup is independent of the search deadline and can add up to five seconds before an error is returned. + ## Known Validation Boundary `VERSION_RANGE_VALUE` remains a defined middleware error, but it is not independently reachable through the current HTTP route. Empty or non-string `versionStart` and `versionEnd` values are rejected earlier by the route's shared string validation as `SEARCH_FIELD_STRING`. The collection tests the observable route contract and does not include an unreachable black-box assertion. diff --git a/test/postman/buildCollection.js b/test/postman/buildCollection.js index 60feeb1..6e1d917 100644 --- a/test/postman/buildCollection.js +++ b/test/postman/buildCollection.js @@ -64,7 +64,11 @@ function successTests({ 'if (isExpectedSuccess) {', " pm.test('response is JSON', () => pm.expect(pm.response.headers.get('Content-Type')).to.include('application/json'));", ' const payload = pm.response.json();', + " const pitMode = String(pm.variables.get('expectedSearchPitEnabled') ?? 'true').trim().toLowerCase();", + " const expectPit = pitMode === 'true';", + " pm.test('expectedSearchPitEnabled is true or false', () => pm.expect(['true', 'false']).to.include(pitMode));", "pm.test('search response has the documented shape', () => {", + " pm.expect(Object.keys(payload)).to.eql(['apiVersion', 'status', 'warnings', 'pagination', 'records']);", " pm.expect(payload.apiVersion).to.eql('1.0');", " pm.expect(payload.status).to.eql('ok');", " pm.expect(payload.warnings).to.be.an('array');", @@ -78,6 +82,14 @@ function successTests({ " pm.expect(payload.records).to.be.an('array');", ' pm.expect(payload.pagination.resultsReturned).to.eql(payload.records.length);', " pm.expect(payload).not.to.have.property('data');", + '});', + "pm.test('total exactness agrees with warnings and expected PIT mode', () => {", + " const candidateLimitReached = payload.warnings.some((warning) => warning.code === 'SEMANTIC_CANDIDATE_LIMIT_REACHED');", + ' pm.expect(payload.pagination.totalIsExact).to.eql(!candidateLimitReached);', + ' if (expectPit) {', + ' pm.expect(payload.pagination.totalIsExact).to.eql(true);', + ' pm.expect(candidateLimitReached).to.eql(false);', + ' }', '});' ]; @@ -106,7 +118,12 @@ function successTests({ if (expectedWarnings.length) { tests.push( - `const expectedWarnings = ${JSON.stringify(expectedWarnings)};`, + `const expectedWarnings = ${JSON.stringify(expectedWarnings)}.map((warning) => {`, + " if (expectPit && warning.code === 'BROAD_VIRTUAL_MATCH_STRING') {", + " return { ...warning, message: 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.' };", + ' }', + ' return warning;', + '});', "pm.test('expected warnings are returned', () => pm.expect(payload.warnings).to.deep.include.members(expectedWarnings));" ); } @@ -1166,6 +1183,16 @@ const dateFailures = folder('16 - Expected failures - dates', 'Timestamp format, ]); function faultInjectionItem(name, requestPath, body, status, expectedError, description, method = 'POST') { + const tests = errorTests(status, expectedError); + if (requestPath === '/search') { + tests.push( + "pm.test('failed search does not return partial records or totals', () => {", + ' const payload = pm.response.json();', + " pm.expect(payload).not.to.have.property('records');", + " pm.expect(payload).not.to.have.property('pagination');", + '});' + ); + } return item({ name, description, @@ -1173,7 +1200,7 @@ function faultInjectionItem(name, requestPath, body, status, expectedError, desc requestPath, body, skipUnless: 'runFaultInjection', - tests: errorTests(status, expectedError) + tests }); } @@ -1213,6 +1240,22 @@ const faultInjection = folder( 502, error.upstreamSearchError(), 'Requires an intentionally malformed or unexpected OpenSearch response.' + ), + faultInjectionItem( + 'PIT snapshot unavailable', + '/search', + { cveIds: ids.cpe, virtualMatchString: 'cpe:2.3:a:freetype:freetype' }, + 503, + error.searchUnavailableError(), + 'Requires server-side SearchPitEnabled=true. Use an isolated test double or proxy to return a structured search_context_missing_exception during a PIT candidate search. No PIT ID is accepted from this request body. Enable runFaultInjection only for this prepared condition.' + ), + faultInjectionItem( + 'Incomplete PIT candidate batch', + '/search', + { cveIds: ids.cpe, virtualMatchString: 'cpe:2.3:a:freetype:freetype' }, + 502, + error.upstreamSearchError(), + 'Requires server-side SearchPitEnabled=true. Use an isolated test double or proxy to return an empty PIT batch with timed_out=false but unaccounted shards, such as total=2, successful=1, skipped=0, failed=0. Enable runFaultInjection only for this prepared condition.' ) ] ); @@ -1243,11 +1286,12 @@ const collection = { _postman_id: '28911974-2b15-4ddf-a88f-3fa6fe001a50', name: 'CVE Search API - Manual Contract Tests', description: - 'Positive filter coverage, intentional combinations, validation failures, and opt-in fault-injection checks for POST /search. The complete collection targets a fully populated index like those used by shared managed environments. Data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly. The separate /webSearch contract is intentionally excluded.', + 'Positive filter coverage, intentional combinations, validation failures, and opt-in fault-injection checks for POST /search. Set expectedSearchPitEnabled to match the server configuration (default true); it affects assertions only and does not enable PIT. The complete collection targets a fully populated index like those used by shared managed environments. Data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly. The separate /webSearch contract is intentionally excluded.', schema: 'https://schema.getpostman.com/json/collection/v2.1.0/collection.json' }, variable: [ { key: 'baseUrl', value: 'http://localhost:4000', type: 'string' }, + { key: 'expectedSearchPitEnabled', value: 'true', type: 'string' }, { key: 'runFaultInjection', value: 'false', type: 'string' } ], item: folders diff --git a/utils/appConfig.js b/utils/appConfig.js new file mode 100644 index 0000000..6c72cd6 --- /dev/null +++ b/utils/appConfig.js @@ -0,0 +1,8 @@ +// Load local environment overrides before node-config resolves its deployment files. +require('dotenv/config'); +const config = require('config'); + +module.exports = { + get: (path) => config.get(`appConfig.${path}`), + has: (path) => config.has(`appConfig.${path}`) +}; diff --git a/utils/appConfig.test.js b/utils/appConfig.test.js new file mode 100644 index 0000000..4740419 --- /dev/null +++ b/utils/appConfig.test.js @@ -0,0 +1,111 @@ +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { execFileSync } = require('child_process'); + +const projectRoot = path.resolve(__dirname, '..'); +let directory; + +function readConfig(env = {}) { + const script = ` + const config = require(${JSON.stringify(path.join(__dirname, 'appConfig'))}); + const provider = require(${JSON.stringify(path.join(__dirname, 'searchProvider'))}); + const flags = require(${JSON.stringify(path.join(__dirname, 'featureFlags'))}); + const logging = require(${JSON.stringify(path.join(__dirname, 'requestLogging'))}); + console.log(JSON.stringify({ + endpoint: config.has('search.providerEndpoint') ? config.get('search.providerEndpoint') : null, + index: config.has('search.index') ? config.get('search.index') : null, + insecure: provider.getAllowUnknownSslCerts(), + timeout: provider.getOpenSearchRequestTimeoutMs(), + enabled: flags.isWebSearchEnabled(), + sampleRate: logging.getSuccessSampleRate(), + missing: config.has('missing') + })); + `; + return JSON.parse( + execFileSync(process.execPath, ['-e', script], { + cwd: directory, + env: { + NODE_ENV: 'test', + NODE_CONFIG_DIR: path.join(projectRoot, 'config'), + DOTENV_CONFIG_PATH: path.join(directory, '.env'), + ...env + }, + encoding: 'utf8', + timeout: 10000 + }) + ); +} + +beforeEach(() => { + directory = fs.mkdtempSync(path.join(os.tmpdir(), 'search-config-')); +}); +afterEach(() => fs.rmSync(directory, { recursive: true, force: true })); + +test('default settings load without a dotenv file or endpoint', () => { + expect(readConfig()).toStrictEqual({ + endpoint: null, + index: null, + insecure: false, + timeout: 20000, + enabled: true, + sampleRate: 0.1, + missing: false + }); +}); + +test('dotenv is loaded before deployment selection and custom environment mapping', () => { + fs.writeFileSync( + path.join(directory, '.env'), + [ + 'NODE_CONFIG_ENV=prod', + 'OpenSearchDomainEndpoint=https://dotenv.example', + 'OpenSearchCveIndex=dotenv-index', + 'OpenSearchAllowUnknownSslCerts=FALSE', + 'SearchOpenSearchRequestTimeoutMs=25000', + 'WebSearchEnabled=false', + 'RequestLogSuccessSampleRate=0.25' + ].join('\n') + ); + expect(readConfig()).toStrictEqual({ + endpoint: 'https://dotenv.example', + index: 'dotenv-index', + insecure: false, + timeout: 25000, + enabled: false, + sampleRate: 0.25, + missing: false + }); +}); + +test('process environment overrides dotenv and deployment files', () => { + fs.writeFileSync( + path.join(directory, '.env'), + [ + 'NODE_CONFIG_ENV=devel', + 'OpenSearchDomainEndpoint=https://dotenv.example', + 'OpenSearchCveIndex=dotenv-index', + 'OpenSearchAllowUnknownSslCerts=TRUE' + ].join('\n') + ); + expect( + readConfig({ + NODE_CONFIG_ENV: 'prod', + OpenSearchDomainEndpoint: 'https://process.example', + OpenSearchCveIndex: 'process-index', + OpenSearchAllowUnknownSslCerts: 'FALSE' + }) + ).toMatchObject({ endpoint: 'https://process.example', index: 'process-index', insecure: false }); +}); + +test.each([ + ['devel', 'https://localhost:9200', true], + ['prod', '', false] +])('NODE_CONFIG_ENV selects the %s deployment', (deployment, endpoint, insecure) => { + expect(readConfig({ NODE_CONFIG_ENV: deployment })).toMatchObject({ endpoint, insecure }); +}); + +test('a missing config key retains the node-config error behavior', () => { + const config = require('./appConfig'); + expect(() => config.get('nonexistent')).toThrow('appConfig.nonexistent'); +}); diff --git a/utils/constants.js b/utils/constants.js index 83f7b4d..17e8b6c 100644 --- a/utils/constants.js +++ b/utils/constants.js @@ -1,5 +1,6 @@ const MAX_RESULTS_LIMIT = 500; const RESULTS_PER_PAGE = 20; +const WEB_SEARCH_RESULTS_PER_PAGE = 25; const MAX_SEARCH_WINDOW = 10000; const MAX_CVSS_METRICS_LENGTH = 256; const MAX_CVSS_METRIC_TOKENS = 32; @@ -25,6 +26,10 @@ const SEARCH_OPEN_SEARCH_REQUEST_TIMEOUT_MS = 20 * 1000; const SEARCH_REQUEST_DEADLINE_MS = 30 * 1000; const SEARCH_READINESS_TIMEOUT_MS = 5 * 1000; const MAX_TIMER_TIMEOUT_MS = 2147483647; +const SEARCH_PIT_BATCH_SIZE = 1000; +const SEARCH_PIT_MAX_BATCH_SIZE = 10000; +const SEARCH_PIT_KEEP_ALIVE_MS = 60 * 1000; +const SEARCH_PIT_CLEANUP_TIMEOUT_MS = 5 * 1000; const rejectedCveStates = ['REJECT', 'REJECTED', 'Rejected']; const cpeVersionRangeTypes = ['including', 'excluding']; @@ -138,7 +143,12 @@ module.exports = { SEARCH_REQUEST_DEADLINE_MS, SEARCH_READINESS_TIMEOUT_MS, MAX_TIMER_TIMEOUT_MS, + SEARCH_PIT_BATCH_SIZE, + SEARCH_PIT_MAX_BATCH_SIZE, + SEARCH_PIT_KEEP_ALIVE_MS, + SEARCH_PIT_CLEANUP_TIMEOUT_MS, RESULTS_PER_PAGE, + WEB_SEARCH_RESULTS_PER_PAGE, REJECTED_CVE_STATES: rejectedCveStates, CPE_VERSION_RANGE_TYPES: cpeVersionRangeTypes, CPE_MATCH_FIELDS: cpeMatchFields, diff --git a/utils/featureFlags.js b/utils/featureFlags.js index 61ff67f..0a77a41 100644 --- a/utils/featureFlags.js +++ b/utils/featureFlags.js @@ -1,6 +1,7 @@ -const { AppConfig } = require('cve-core'); +const AppConfig = require('./appConfig'); const DEFAULT_WEB_SEARCH_ENABLED = true; +const DEFAULT_SEARCH_PIT_ENABLED = true; function normalizeBoolean(value, defaultValue) { if (value === undefined || value === null || value === '') return defaultValue; @@ -20,6 +21,13 @@ function isWebSearchEnabled() { return normalizeBoolean(AppConfig.get('features.webSearchEnabled'), false); } +function isSearchPitEnabled() { + if (!AppConfig.has('features.searchPitEnabled')) return DEFAULT_SEARCH_PIT_ENABLED; + + return normalizeBoolean(AppConfig.get('features.searchPitEnabled'), false); +} + module.exports = { + isSearchPitEnabled, isWebSearchEnabled }; diff --git a/utils/featureFlags.test.js b/utils/featureFlags.test.js index 3d47d75..879132f 100644 --- a/utils/featureFlags.test.js +++ b/utils/featureFlags.test.js @@ -1,11 +1,9 @@ const mockAppConfigHas = jest.fn(); const mockAppConfigGet = jest.fn(); -jest.mock('cve-core', () => ({ - AppConfig: { - get: mockAppConfigGet, - has: mockAppConfigHas - } +jest.mock('./appConfig', () => ({ + get: mockAppConfigGet, + has: mockAppConfigHas })); const featureFlags = require('./featureFlags'); @@ -41,4 +39,32 @@ describe('featureFlags', () => { expect(featureFlags.isWebSearchEnabled()).toBe(false); }); + + test('keeps PIT enabled when no override is configured', () => { + expect(featureFlags.isSearchPitEnabled()).toBe(true); + }); + + test.each([ + ['TRUE', true], + [' true ', true], + ['false', false], + [' FALSE ', false], + ['off', false], + ['1', true], + ['0', false], + [true, true], + [false, false], + ['', false], + [undefined, false], + [null, false], + ['invalid', false] + ])('normalizes a configured PIT value of %p', (configuredValue, expected) => { + mockAppConfigHas.mockImplementation((key) => key === 'features.searchPitEnabled'); + mockAppConfigGet.mockImplementation((key) => + key === 'features.searchPitEnabled' ? configuredValue : undefined + ); + + expect(featureFlags.isSearchPitEnabled()).toBe(expected); + expect(featureFlags.isWebSearchEnabled()).toBe(true); + }); }); diff --git a/utils/requestLogging.js b/utils/requestLogging.js index 57e7248..a0fd092 100644 --- a/utils/requestLogging.js +++ b/utils/requestLogging.js @@ -1,5 +1,5 @@ const { randomUUID } = require('crypto'); -const { AppConfig } = require('cve-core'); +const AppConfig = require('./appConfig'); const DEFAULT_SUCCESS_SAMPLE_RATE = 0.1; const DEFAULT_LOG_HEALTH_CHECKS = false; diff --git a/utils/requestLogging.test.js b/utils/requestLogging.test.js index 1cae11e..0821e43 100644 --- a/utils/requestLogging.test.js +++ b/utils/requestLogging.test.js @@ -1,11 +1,9 @@ const mockAppConfigHas = jest.fn(); const mockAppConfigGet = jest.fn(); -jest.mock('cve-core', () => ({ - AppConfig: { - get: mockAppConfigGet, - has: mockAppConfigHas - } +jest.mock('./appConfig', () => ({ + get: mockAppConfigGet, + has: mockAppConfigHas })); const requestLogging = require('./requestLogging'); diff --git a/utils/searchDeadline.js b/utils/searchDeadline.js index df7e4ce..c38ba82 100644 --- a/utils/searchDeadline.js +++ b/utils/searchDeadline.js @@ -33,12 +33,13 @@ function getAbortReason(signal) { return err; } -function throwIfAborted(signal) { +function throwIfAborted(signal, getRemainingTimeMs) { const deadline = requestDeadlines.get(signal); if (deadline && process.hrtime.bigint() >= deadline.expiresAt) { deadline.abort(new SearchDeadlineExceededError()); } if (signal?.aborted) throw getAbortReason(signal); + if (getRemainingTimeMs && getRemainingTimeMs() <= 0) throw new SearchDeadlineExceededError(); } function addOnceListener(emitter, eventName, listener) { @@ -91,8 +92,18 @@ function createSearchRequestLifecycle(req, res, timeoutMs) { throwIfAborted(controller.signal); } + function remainingTimeMs() { + return Math.max(0, Number(expiresAt - process.hrtime.bigint()) / 1e6); + } + + function getRemainingTimeMs() { + throwIfRequestAborted(); + return remainingTimeMs(); + } + return { complete, + getRemainingTimeMs, signal: controller.signal, throwIfAborted: throwIfRequestAborted }; diff --git a/utils/searchDeadline.test.js b/utils/searchDeadline.test.js index 43d3d6e..b2d8a94 100644 --- a/utils/searchDeadline.test.js +++ b/utils/searchDeadline.test.js @@ -86,6 +86,33 @@ describe('searchDeadline', () => { expect(() => lifecycle.throwIfAborted()).not.toThrow(); }); + test('uses monotonic elapsed time and expires even before the timer callback runs', () => { + captureTimers(); + const clock = jest.spyOn(process.hrtime, 'bigint').mockReturnValue(0n); + const { lifecycle } = createLifecycle(); + try { + expect(lifecycle.getRemainingTimeMs()).toBe(30000); + clock.mockReturnValue(12500000000n); + expect(lifecycle.getRemainingTimeMs()).toBe(17500); + clock.mockReturnValue(30000000000n); + + expect(() => lifecycle.getRemainingTimeMs()).toThrow(searchDeadline.SearchDeadlineExceededError); + expect(lifecycle.signal.aborted).toBe(true); + expect(() => lifecycle.throwIfAborted()).toThrow(searchDeadline.SearchDeadlineExceededError); + } finally { + lifecycle.complete(); + } + }); + + test('preserves disconnect cancellation when reading the remaining budget', () => { + captureTimers(); + const { lifecycle, req } = createLifecycle(); + req.aborted = true; + + expect(() => lifecycle.getRemainingTimeMs()).toThrow(searchDeadline.SearchClientDisconnectedError); + lifecycle.complete(); + }); + test('aborts when the request emits aborted', () => { captureTimers(); const { lifecycle, req } = createLifecycle(); diff --git a/utils/searchPitTraversal.js b/utils/searchPitTraversal.js new file mode 100644 index 0000000..adfd0a2 --- /dev/null +++ b/utils/searchPitTraversal.js @@ -0,0 +1,104 @@ +const cpe = require('./cpe'); +const searchDeadline = require('./searchDeadline'); +const searchPostFilters = require('./searchPostFilters'); +const searchProvider = require('./searchProvider'); +const searchQueryBuilder = require('./searchQueryBuilder'); +const { + InvalidOpenSearchResponseError, + validatePitSearchResponse, + validatePitCandidateResponse +} = require('./searchResponseValidator'); + +const EVALUATION_YIELD_INTERVAL = 100; + +async function yieldToEventLoop({ signal, getRemainingTimeMs }) { + await new Promise(setImmediate); + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); +} + +async function fetchPageHits(pitId, cveIds, requestOptions) { + const { signal, getRemainingTimeMs } = requestOptions; + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + if (!cveIds.length) return []; + + const { searchOptions, queryStringObjects } = searchQueryBuilder.buildCveIdsSearchRequest(cveIds); + const body = searchQueryBuilder.buildOpenSearchBody(searchOptions, queryStringObjects); + const response = await searchProvider.searchOpenSearchPit(pitId, body, requestOptions); + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + validatePitSearchResponse(response); + + const hitsById = new Map(response.hits.hits.map((hit) => [hit._source.cveMetadata?.cveId, hit])); + if ( + response.hits.total?.relation !== 'eq' || + response.hits.total?.value !== cveIds.length || + response.hits.hits.length !== cveIds.length || + hitsById.size !== cveIds.length || + cveIds.some((id) => !hitsById.has(id)) + ) { + throw new InvalidOpenSearchResponseError('OpenSearch PIT page did not contain exactly the requested records.'); + } + + return cveIds.map((id) => hitsById.get(id)); +} + +async function executePitSearch( + { searchOptions, queryStringObjects, postFilters, responseOptions }, + requestOptions = {} +) { + const { signal, getRemainingTimeMs } = requestOptions; + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + const batchSize = searchProvider.getSearchPitBatchSize(); + const { from, resultsPerPage } = responseOptions; + + const result = await searchProvider.withOpenSearchPit(async (pitId) => { + let searchAfter; + let totalResults = 0; + const pageCveIds = []; + + while (true) { + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + const body = searchQueryBuilder.buildPitCandidateSearchBody( + searchOptions, + queryStringObjects, + postFilters, + { + batchSize, + searchAfter + } + ); + const response = await searchProvider.searchOpenSearchPit(pitId, body, requestOptions); + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + validatePitCandidateResponse(response, searchAfter); + const hits = response.hits.hits; + if (hits.length > batchSize) { + throw new InvalidOpenSearchResponseError('OpenSearch PIT candidate batch is incomplete or oversized.'); + } + if (!hits.length) break; + + // Keep parsing state batch-local; only page IDs and the count survive each batch. + const context = cpe.createEvaluationContext(); + for (let index = 0; index < hits.length; index += 1) { + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + const hit = hits[index]; + if (searchPostFilters.hitMatches(hit, postFilters, context)) { + if (totalResults >= from && pageCveIds.length < resultsPerPage) { + pageCveIds.push(hit._source.cveMetadata.cveId); + } + totalResults += 1; + } + if ((index + 1) % EVALUATION_YIELD_INTERVAL === 0) await yieldToEventLoop(requestOptions); + } + + searchAfter = [...hits[hits.length - 1].sort]; + await yieldToEventLoop(requestOptions); + } + + const hits = await fetchPageHits(pitId, pageCveIds, requestOptions); + return { totalResults, hits, totalIsExact: true, warnings: [], usesPit: true }; + }, requestOptions); + + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + return result; +} + +module.exports = { executePitSearch }; diff --git a/utils/searchPitTraversal.test.js b/utils/searchPitTraversal.test.js new file mode 100644 index 0000000..679241d --- /dev/null +++ b/utils/searchPitTraversal.test.js @@ -0,0 +1,400 @@ +jest.mock('./searchProvider', () => ({ + getSearchPitBatchSize: jest.fn(), + searchOpenSearchPit: jest.fn(), + searchOpenSearch: jest.fn(), + withOpenSearchPit: jest.fn() +})); + +const cpe = require('./cpe'); +const searchDeadline = require('./searchDeadline'); +const searchPostFilters = require('./searchPostFilters'); +const provider = require('./searchProvider'); +const queryBuilder = require('./searchQueryBuilder'); +const { InvalidOpenSearchResponseError } = require('./searchResponseValidator'); +const { executePitSearch } = require('./searchPitTraversal'); + +const CPE_NAME = 'cpe:2.3:a:vendor:product:2.10:*:*:*:*:*:*:*'; + +function cveId(number) { + return `CVE-2024-${String(number).padStart(5, '0')}`; +} + +function candidate(number, matches = true) { + return { + _source: { + cveMetadata: { cveId: cveId(number) }, + containers: { + cna: { + cpeApplicability: { + nodes: [ + { + cpeMatch: [ + { + criteria: 'cpe:2.3:a:vendor:product:*:*:*:*:*:*:*:*', + versionStartIncluding: matches ? '2.0' : '3.0', + vulnerable: true + } + ] + } + ] + } + } + } + }, + sort: [cveId(number)] + }; +} + +function fullHit(number) { + return { + _source: { + cveMetadata: { cveId: cveId(number) }, + containers: { cna: { title: 'Complete record' }, adp: [{ title: 'Enrichment' }] } + } + }; +} + +function response(hits, overrides = {}) { + return { + timed_out: false, + _shards: { total: 1, successful: 1, skipped: 0, failed: 0 }, + hits: { total: { value: hits.length, relation: 'eq' }, hits }, + ...overrides + }; +} + +function runSearch(params = { cpeName: CPE_NAME }, options) { + return executePitSearch(queryBuilder.buildSearchRequest(params), options); +} + +describe('PIT semantic traversal', () => { + beforeEach(() => { + jest.resetAllMocks(); + provider.getSearchPitBatchSize.mockReturnValue(2); + provider.withOpenSearchPit.mockImplementation(async (operation) => operation('test-pit')); + }); + + afterEach(() => jest.restoreAllMocks()); + + test('counts all matches, retains only the requested page, and retrieves it from the same PIT', async () => { + const contexts = jest.spyOn(cpe, 'createEvaluationContext'); + const signal = new AbortController().signal; + const options = { signal, getRemainingTimeMs: jest.fn(() => 1000) }; + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([candidate(1), candidate(2, false)])) + .mockResolvedValueOnce(response([candidate(3), candidate(4)])) + .mockResolvedValueOnce(response([candidate(5), candidate(6)])) + .mockResolvedValueOnce(response([])) + .mockResolvedValueOnce(response([fullHit(5), fullHit(4)])); + + const result = await runSearch({ cpeName: CPE_NAME, resultsPerPage: 2, pageNumber: 2 }, options); + + expect(result).toStrictEqual({ + totalResults: 5, + hits: [fullHit(4), fullHit(5)], + totalIsExact: true, + warnings: [], + usesPit: true + }); + expect(provider.withOpenSearchPit).toHaveBeenCalledWith(expect.any(Function), options); + expect(provider.searchOpenSearch).not.toHaveBeenCalled(); + const calls = provider.searchOpenSearchPit.mock.calls; + expect(calls).toHaveLength(5); + expect(calls.every(([pitId, _body, passedOptions]) => pitId === 'test-pit' && passedOptions === options)).toBe( + true + ); + expect(Array.from(calls.slice(0, 4), ([, body]) => body.search_after)).toStrictEqual([ + undefined, + [cveId(2)], + [cveId(4)], + [cveId(6)] + ]); + expect(calls[0][1]._source).toContain('containers.cna.cpeApplicability.nodes.cpeMatch.criteria'); + expect(calls[4][1]).not.toHaveProperty('_source'); + expect(calls[4][1].query.bool.filter).toStrictEqual([ + { terms: { 'cveMetadata.cveId.keyword': [cveId(4), cveId(5)] } } + ]); + expect(contexts).toHaveBeenCalledTimes(3); + expect(new Set(contexts.mock.results.map(({ value }) => value)).size).toBe(3); + }); + + test.each([true, false])('traverses beyond 10,000 candidates (all match: %s)', async (allMatch) => { + provider.getSearchPitBatchSize.mockReturnValue(1000); + provider.searchOpenSearchPit.mockImplementation(async (_pitId, body) => { + if (!body._source) return response([fullHit(allMatch ? 1 : 10001)]); + const from = body.search_after ? Number(body.search_after[0].split('-')[2]) : 0; + const length = Math.min(body.size, 10001 - from); + return response( + Array.from({ length }, (_, i) => candidate(from + i + 1, allMatch || from + i + 1 === 10001)) + ); + }); + + const result = await runSearch({ cpeName: CPE_NAME, resultsPerPage: 1 }); + + expect(result.totalResults).toBe(allMatch ? 10001 : 1); + expect(result.hits).toStrictEqual([fullHit(allMatch ? 1 : 10001)]); + expect(result.totalIsExact).toBe(true); + expect(result.warnings).toStrictEqual([]); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(13); + }); + + test.each([ + ['no candidates', [], 1, 0], + ['no matches', [candidate(1, false)], 1, 0], + ['out-of-range page', [candidate(1)], 2, 1] + ])('skips full record retrieval for %s', async (_name, hits, pageNumber, totalResults) => { + provider.searchOpenSearchPit.mockResolvedValueOnce(response(hits)).mockResolvedValueOnce(response([])); + + const result = await runSearch({ cpeName: CPE_NAME, resultsPerPage: 1, pageNumber }); + + expect(result).toMatchObject({ totalResults, hits: [], totalIsExact: true }); + expect(provider.searchOpenSearchPit.mock.calls.every(([, body]) => Array.isArray(body._source))).toBe(true); + }); + + test.each([ + [{ virtualMatchString: 'cpe:2.3:a:vendor:product' }, 2], + [{ virtualMatchString: 'cpe:2.3:a:vendor:product', versionEnd: '2.5', versionEndType: 'excluding' }, 1] + ])('evaluates virtual match candidates with the existing version semantics: %p', async (params, expectedCount) => { + const pageHits = expectedCount === 2 ? [fullHit(1), fullHit(2)] : [fullHit(1)]; + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([candidate(1), candidate(2, false)])) + .mockResolvedValueOnce(response([])) + .mockResolvedValueOnce(response(pageHits)); + + const result = await runSearch(params); + + expect(result.totalResults).toBe(expectedCount); + expect(result.hits).toStrictEqual(pageHits); + }); + + test.each([ + [ + { cvssV3_1Metrics: 'AV:N/AC:L' }, + [{ cvssV3_1: { vectorString: 'AV:N' } }, { cvssV3_1: { vectorString: 'AC:L' } }], + [{ cvssV3_1: { vectorString: 'AV:N/AC:L' } }] + ], + [ + { ssvcExploitation: 'ACTIVE', ssvcAutomatable: 'YES' }, + [ + { other: { type: 'ssvc', content: { options: [{ Exploitation: 'active' }] } } }, + { other: { type: 'ssvc', content: { options: [{ Automatable: 'yes' }] } } } + ], + [{ other: { type: 'ssvc', content: { options: [{ Exploitation: 'active' }, { Automatable: 'yes' }] } } }] + ], + [ + { + knownExploited: true, + kevDateAddedStartDate: '2024-01-01T00:00:00.000Z', + kevDateAddedEndDate: '2024-12-31T00:00:00.000Z' + }, + [ + { other: { type: 'kev', content: { dateAdded: '2023-12-31' } } }, + { other: { type: 'kev', content: { dateAdded: '2025-01-01' } } } + ], + [{ other: { type: 'kev', content: { dateAdded: '2024-06-01' } } }] + ] + ])('retains scope and same-entry semantics across batches for %p', async (params, nonmatching, matching) => { + provider.getSearchPitBatchSize.mockReturnValue(1); + const hits = [nonmatching, matching].map((metrics, index) => { + const hit = candidate(index + 1); + hit._source.containers = { cna: { metrics: matching }, adp: [{ metrics }] }; + return hit; + }); + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([hits[0]])) + .mockResolvedValueOnce(response([hits[1]])) + .mockResolvedValueOnce(response([])) + .mockResolvedValueOnce(response([fullHit(2)])); + + const result = await runSearch({ ...params, scope: 'ADP_ONLY' }); + + expect(result).toMatchObject({ totalResults: 1, hits: [fullHit(2)] }); + expect(provider.searchOpenSearchPit.mock.calls[0][1]._source.join(',')).not.toContain('containers.cna'); + }); + + test.each([ + ['an oversized batch', response([candidate(1), candidate(2), candidate(3)])], + ['early termination', response([], { terminated_early: true })], + [ + 'unaccounted shards in an empty batch', + response([], { _shards: { total: 2, successful: 1, skipped: 0, failed: 0 } }) + ], + ['failed shards', response([], { _shards: { failed: 1 } })], + ['a non-advancing cursor', response([candidate(2), candidate(1)])] + ])('rejects %s instead of returning a partial count', async (_name, body) => { + provider.searchOpenSearchPit.mockResolvedValueOnce(body); + + await expect(runSearch()).rejects.toBeInstanceOf(InvalidOpenSearchResponseError); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(1); + }); + + test('rejects a repeated batch', async () => { + provider.searchOpenSearchPit.mockResolvedValue(response([candidate(1), candidate(2)])); + + await expect(runSearch()).rejects.toBeInstanceOf(InvalidOpenSearchResponseError); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(2); + }); + + test('does not return a partial total when the final empty batch times out', async () => { + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([candidate(1)])) + .mockResolvedValueOnce(response([], { timed_out: true })); + + await expect(runSearch()).rejects.toBeInstanceOf(searchDeadline.SearchDeadlineExceededError); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(2); + expect(provider.searchOpenSearch).not.toHaveBeenCalled(); + }); + + test('checks remaining budget between records even when the abort timer has not run', async () => { + let remaining = 100; + const signal = new AbortController().signal; + provider.searchOpenSearchPit.mockResolvedValueOnce(response([candidate(1), candidate(2)])); + const matches = jest.spyOn(searchPostFilters, 'hitMatches').mockImplementation(() => { + remaining = 0; + return true; + }); + + await expect(runSearch(undefined, { signal, getRemainingTimeMs: () => remaining })).rejects.toBeInstanceOf( + searchDeadline.SearchDeadlineExceededError + ); + expect(signal.aborted).toBe(false); + expect(matches).toHaveBeenCalledTimes(1); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(1); + }); + + test('checks remaining budget after cleanup before returning exact totals', async () => { + let remaining = 100; + provider.searchOpenSearchPit.mockResolvedValueOnce(response([])); + provider.withOpenSearchPit.mockImplementationOnce(async (operation) => { + const result = await operation('test-pit'); + remaining = 0; + return result; + }); + + await expect(runSearch(undefined, { getRemainingTimeMs: () => remaining })).rejects.toBeInstanceOf( + searchDeadline.SearchDeadlineExceededError + ); + }); + + test.each([ + ['missing records', response([fullHit(1)])], + ['truncated hits', response([], { hits: { hits: [fullHit(1)], total: { value: 2, relation: 'eq' } } })], + ['duplicate records', response([fullHit(1), fullHit(1)])], + ['unexpected records', response([fullHit(1), fullHit(3)])], + ['extra records', response([fullHit(1), fullHit(2), fullHit(3)])], + ['missing identity', response([fullHit(1), { _source: {} }])], + ['early termination', response([fullHit(1), fullHit(2)], { terminated_early: true })], + [ + 'inexact total', + response([], { hits: { hits: [fullHit(1), fullHit(2)], total: { value: 2, relation: 'gte' } } }) + ] + ])('rejects a PIT page with %s', async (_name, body) => { + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([candidate(1), candidate(2)])) + .mockResolvedValueOnce(response([])) + .mockResolvedValueOnce(body); + + await expect(runSearch()).rejects.toBeInstanceOf(InvalidOpenSearchResponseError); + expect(provider.searchOpenSearch).not.toHaveBeenCalled(); + }); + + test.each(['create', 'candidate', 'evaluation', 'page'])( + 'propagates %s failures through the managed PIT operation', + async (phase) => { + const error = new Error('upstream failure'); + if (phase === 'create') provider.withOpenSearchPit.mockRejectedValueOnce(error); + if (phase === 'candidate') provider.searchOpenSearchPit.mockRejectedValueOnce(error); + if (phase === 'evaluation') { + provider.searchOpenSearchPit.mockResolvedValueOnce(response([candidate(1)])); + jest.spyOn(searchPostFilters, 'hitMatches').mockImplementation(() => { + throw error; + }); + } + if (phase === 'page') { + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([candidate(1)])) + .mockResolvedValueOnce(response([])) + .mockRejectedValueOnce(error); + } + + await expect(runSearch()).rejects.toBe(error); + expect(provider.withOpenSearchPit).toHaveBeenCalledTimes(1); + expect(provider.searchOpenSearch).not.toHaveBeenCalled(); + } + ); + + test('does not open a PIT for an already canceled request', async () => { + const controller = new AbortController(); + const error = new searchDeadline.SearchClientDisconnectedError(); + controller.abort(error); + + await expect(runSearch(undefined, { signal: controller.signal })).rejects.toBe(error); + expect(provider.withOpenSearchPit).not.toHaveBeenCalled(); + }); + + test.each(['candidate', 'page'])('does not accept a %s response after cancellation', async (phase) => { + const controller = new AbortController(); + const error = new searchDeadline.SearchClientDisconnectedError(); + if (phase === 'page') { + provider.searchOpenSearchPit + .mockResolvedValueOnce(response([candidate(1)])) + .mockResolvedValueOnce(response([])); + } + provider.searchOpenSearchPit.mockImplementationOnce(async () => { + controller.abort(error); + return phase === 'page' ? response([fullHit(1)]) : response([candidate(1)]); + }); + + await expect(runSearch(undefined, { signal: controller.signal })).rejects.toBe(error); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(phase === 'page' ? 3 : 1); + }); + + test('yields between small batches so cancellation can prevent the next search', async () => { + const controller = new AbortController(); + const error = new searchDeadline.SearchDeadlineExceededError(); + provider.searchOpenSearchPit.mockResolvedValueOnce(response([candidate(1)])); + const immediate = setImmediate(() => controller.abort(error)); + + try { + await expect(runSearch(undefined, { signal: controller.signal })).rejects.toBe(error); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(1); + } finally { + clearImmediate(immediate); + } + }); + + test.each([searchDeadline.SearchDeadlineExceededError, searchDeadline.SearchClientDisconnectedError])( + 'yields during evaluation so %p can stop the scan before the next request', + async (ErrorClass) => { + const controller = new AbortController(); + const error = new ErrorClass(); + const matches = jest.spyOn(searchPostFilters, 'hitMatches'); + provider.getSearchPitBatchSize.mockReturnValue(250); + provider.searchOpenSearchPit.mockResolvedValueOnce( + response(Array.from({ length: 250 }, (_, i) => candidate(i + 1))) + ); + const immediate = setImmediate(() => controller.abort(error)); + + try { + await expect(runSearch(undefined, { signal: controller.signal })).rejects.toBe(error); + expect(matches).toHaveBeenCalledTimes(100); + expect(provider.searchOpenSearchPit).toHaveBeenCalledTimes(1); + } finally { + clearImmediate(immediate); + } + } + ); + + test('does not return success if cancellation occurs during lifecycle cleanup', async () => { + const controller = new AbortController(); + const error = new searchDeadline.SearchDeadlineExceededError(); + provider.searchOpenSearchPit.mockResolvedValueOnce(response([])); + provider.withOpenSearchPit.mockImplementationOnce(async (operation) => { + const result = await operation('test-pit'); + controller.abort(error); + return result; + }); + + await expect(runSearch(undefined, { signal: controller.signal })).rejects.toBe(error); + }); +}); diff --git a/utils/searchProvider.js b/utils/searchProvider.js index 03b9855..dd8f5d0 100644 --- a/utils/searchProvider.js +++ b/utils/searchProvider.js @@ -1,34 +1,12 @@ -const { AppConfig, BasicSearchManager, SearchProviderSpec, SearchReader } = require('cve-core'); +const { Client } = require('@opensearch-project/opensearch'); +const AppConfig = require('./appConfig'); const constants = require('./constants'); const searchDeadline = require('./searchDeadline'); const searchReadiness = require('./searchReadiness'); const searchResponseValidator = require('./searchResponseValidator'); -class SearchProviderContractError extends Error { - constructor() { - super('Installed cve-core SearchReader does not expose the required OpenSearch client contract.'); - this.name = 'SearchProviderContractError'; - this.code = 'SEARCH_PROVIDER_CONTRACT'; - } -} - let searchProviderConfig; -let basicSearchManager; -let searchReader; - -function configureClientRequestTimeout(client) { - if (client?.transport) { - client.transport.requestTimeout = getOpenSearchRequestTimeoutMs(); - client.transport.maxRetries = 0; - } - - return client; -} - -function configureManagerRequestTimeout(manager) { - configureClientRequestTimeout(manager?._searchReader?._client); - return manager; -} +let searchClient; function normalizeBoolean(value, defaultValue = false) { if (value === undefined || value === null || value === '') return defaultValue; @@ -45,86 +23,68 @@ function normalizeBoolean(value, defaultValue = false) { function getIntegerConfig(path, defaultValue, { min = 0, max = Number.MAX_SAFE_INTEGER } = {}) { if (!AppConfig.has(path)) return defaultValue; - const value = Number(AppConfig.get(path)); + const configuredValue = AppConfig.get(path); + if (typeof configuredValue !== 'string' && typeof configuredValue !== 'number') return defaultValue; + + const value = Number(configuredValue); if (Number.isInteger(value) && value >= min && value <= max) return value; return defaultValue; } -function getAllowUnknownSslCerts(spec) { +function getAllowUnknownSslCerts() { if (AppConfig.has('search.allowUnknownSslCerts')) { return normalizeBoolean(AppConfig.get('search.allowUnknownSslCerts')); } - return normalizeBoolean(spec?.allowUnknownSslCerts); + return false; } -function normalizeSearchProviderConfig(spec) { - const providerEndpoint = spec?.providerEndpoint; - const index = spec?.index; +function readSearchProviderConfig() { + const providerEndpoint = AppConfig.has('search.providerEndpoint') ? AppConfig.get('search.providerEndpoint') : ''; + const index = AppConfig.has('search.index') ? AppConfig.get('search.index') : ''; - if (!providerEndpoint || !index) { + if ( + typeof providerEndpoint !== 'string' || + !providerEndpoint.trim() || + typeof index !== 'string' || + !index.trim() + ) { throw new Error('OpenSearch providerEndpoint and index must be configured.'); } return Object.freeze({ providerEndpoint, index, - allowUnknownSslCerts: getAllowUnknownSslCerts(spec) + allowUnknownSslCerts: getAllowUnknownSslCerts() }); } function getSearchProviderConfig() { if (!searchProviderConfig) { - const defaultSpec = SearchProviderSpec.getDefaultSearchProviderSpec(); - searchProviderConfig = normalizeSearchProviderConfig(defaultSpec); - defaultSpec.allowUnknownSslCerts = searchProviderConfig.allowUnknownSslCerts; - SearchProviderSpec._sDefaultSearchProviderSpec = defaultSpec; + searchProviderConfig = readSearchProviderConfig(); } return searchProviderConfig; } -function getBasicSearchManager() { - if (!basicSearchManager) { - basicSearchManager = configureManagerRequestTimeout(new BasicSearchManager(getSearchProviderConfig())); - } - - return basicSearchManager; -} - -function validateSearchReader(reader) { - const hasIndex = typeof reader?._cveIndex === 'string' && reader._cveIndex.trim() !== ''; - const hasSearchClient = typeof reader?._client?.search === 'function'; - const hasFieldCapsClient = typeof reader?._client?.fieldCaps === 'function'; - - if (!hasIndex || !hasSearchClient || !hasFieldCapsClient) { - throw new SearchProviderContractError(); - } - - return reader; -} - -function getSearchReader() { - if (!searchReader) { +function getSearchClient() { + if (!searchClient) { const config = getSearchProviderConfig(); - const reader = validateSearchReader( - new SearchReader(config.providerEndpoint, config.index, { - ignore_ssl_cert_check_errors: config.allowUnknownSslCerts - }) - ); - configureClientRequestTimeout(reader._client); - searchReader = reader; + searchClient = new Client({ + node: config.providerEndpoint, + ssl: { rejectUnauthorized: !config.allowUnknownSslCerts }, + requestTimeout: getOpenSearchRequestTimeoutMs(), + maxRetries: 0 + }); } - return searchReader; + return searchClient; } function awaitAbortableRequest(request, signal) { if (!signal) return Promise.resolve(request); - searchDeadline.throwIfAborted(signal); - return new Promise((resolve, reject) => { let settled = false; @@ -149,6 +109,7 @@ function awaitAbortableRequest(request, signal) { (response) => finish(resolve, response), (err) => finish(reject, signal.aborted ? searchDeadline.getAbortReason(signal) : err) ); + if (signal.aborted) onAbort(); }); } @@ -160,12 +121,12 @@ async function executeOpenSearchRequest(createRequest, { signal } = {}) { async function searchOpenSearch(body, { signal } = {}) { searchDeadline.throwIfAborted(signal); - const reader = getSearchReader(); + const client = getSearchClient(); const response = await executeOpenSearchRequest( () => - reader._client.search( + client.search( { - index: reader._cveIndex, + index: getSearchProviderConfig().index, allow_partial_search_results: false, body }, @@ -180,18 +141,152 @@ async function searchOpenSearch(body, { signal } = {}) { return searchResponseValidator.validateOpenSearchResponse(response.body); } +function getPitSearchClient() { + const client = getSearchClient(); + if (typeof client.createPit !== 'function' || typeof client.deletePit !== 'function') { + throw new TypeError('OpenSearch client does not expose the required PIT methods.'); + } + + return client; +} + +function requirePitId(pitId) { + if (!searchResponseValidator.isValidPitId(pitId)) { + throw new TypeError('A single valid PIT identifier is required.'); + } +} + +async function cleanupOpenSearchPit(pitId) { + if (!searchResponseValidator.isValidPitId(pitId)) return; + + try { + await deleteOpenSearchPit(pitId); + } catch (_err) { + console.warn('OpenSearch PIT cleanup failed; the context will expire after its keep-alive period.'); + } +} + +function getPitRequestTimeoutMs(getRemainingTimeMs) { + const configured = getOpenSearchRequestTimeoutMs(); + const remaining = getRemainingTimeMs?.() ?? configured; + if (!Number.isFinite(remaining) || remaining <= 0) throw new searchDeadline.SearchDeadlineExceededError(); + // A zero transport timeout disables the limit; retain a positive millisecond bound. + return Math.max(1, Math.floor(Math.min(configured, remaining))); +} + +async function createOpenSearchPit({ signal, getRemainingTimeMs } = {}) { + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + const client = getPitSearchClient(); + let request; + let response; + + try { + response = await executeOpenSearchRequest( + () => { + request = client.createPit( + { + index: getSearchProviderConfig().index, + keep_alive: `${getSearchPitKeepAliveMs()}ms`, + allow_partial_pit_creation: false + }, + { maxRetries: 0, requestTimeout: getPitRequestTimeoutMs(getRemainingTimeMs) } + ); + return request; + }, + { signal } + ); + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + searchResponseValidator.validatePitCreationResponse(response?.body); + + return response.body.pit_id; + } catch (err) { + if (response) { + await cleanupOpenSearchPit(response.body?.pit_id); + } else if (request) { + // A canceled create may still deliver an ID; release it if it arrives later. + void Promise.resolve(request).then( + (lateResponse) => cleanupOpenSearchPit(lateResponse?.body?.pit_id), + () => {} + ); + } + throw err; + } +} + +async function searchOpenSearchPit(pitId, body, { signal, getRemainingTimeMs } = {}) { + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + requirePitId(pitId); + const client = getPitSearchClient(); + try { + const response = await executeOpenSearchRequest( + () => + client.search( + { + allow_partial_search_results: false, + body: { + ...body, + pit: { id: pitId, keep_alive: `${getSearchPitKeepAliveMs()}ms` } + } + }, + { maxRetries: 0, requestTimeout: getPitRequestTimeoutMs(getRemainingTimeMs) } + ), + { signal } + ); + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + + return searchResponseValidator.validatePitSearchResponse(response?.body); + } catch (err) { + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + searchResponseValidator.throwIfPitContextMissing(err?.meta?.body); + throw err; + } +} + +async function deleteOpenSearchPit(pitId) { + requirePitId(pitId); + const client = getPitSearchClient(); + const controller = new AbortController(); + const timeoutMs = Math.min(getOpenSearchRequestTimeoutMs(), constants.SEARCH_PIT_CLEANUP_TIMEOUT_MS); + const timer = setTimeout(() => controller.abort(new searchDeadline.SearchDeadlineExceededError()), timeoutMs); + timer.unref?.(); + + try { + const response = await executeOpenSearchRequest( + () => client.deletePit({ body: { pit_id: [pitId] } }, { maxRetries: 0, requestTimeout: timeoutMs }), + { signal: controller.signal } + ); + + return searchResponseValidator.validatePitDeletionResponse(response?.body, pitId); + } finally { + clearTimeout(timer); + } +} + +async function withOpenSearchPit(operation, { signal, getRemainingTimeMs } = {}) { + const pitId = await createOpenSearchPit({ signal, getRemainingTimeMs }); + + try { + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + const result = await operation(pitId); + searchDeadline.throwIfAborted(signal, getRemainingTimeMs); + return result; + } finally { + await cleanupOpenSearchPit(pitId); + } +} + async function checkOpenSearchReadiness({ signal } = {}) { searchDeadline.throwIfAborted(signal); - const reader = getSearchReader(); + const client = getSearchClient(); const requestOptions = { maxRetries: 0, requestTimeout: getOpenSearchReadinessTimeoutMs() }; const fieldCapsResponse = await executeOpenSearchRequest( () => - reader._client.fieldCaps( + client.fieldCaps( { - index: reader._cveIndex, + index: getSearchProviderConfig().index, fields: searchReadiness.getRequiredFieldNames().join(','), include_unmapped: true }, @@ -204,9 +299,9 @@ async function checkOpenSearchReadiness({ signal } = {}) { const response = await executeOpenSearchRequest( () => - reader._client.search( + client.search( { - index: reader._cveIndex, + index: getSearchProviderConfig().index, allow_partial_search_results: false, body: { query: { @@ -227,15 +322,6 @@ async function checkOpenSearchReadiness({ signal } = {}) { return true; } -function isSearchProviderContractError(err) { - return ( - err instanceof SearchProviderContractError || - err instanceof searchReadiness.SearchMappingContractError || - err?.code === 'SEARCH_PROVIDER_CONTRACT' || - err?.code === 'SEARCH_MAPPING_CONTRACT' - ); -} - function getOpenSearchReadinessTimeoutMs() { return getIntegerConfig('search.readinessTimeoutMs', constants.SEARCH_READINESS_TIMEOUT_MS, { min: 1, @@ -257,26 +343,41 @@ function getSearchRequestDeadlineMs() { }); } +function getSearchPitBatchSize() { + return getIntegerConfig('search.pitBatchSize', constants.SEARCH_PIT_BATCH_SIZE, { + min: 1, + max: constants.SEARCH_PIT_MAX_BATCH_SIZE + }); +} + +function getSearchPitKeepAliveMs() { + return getIntegerConfig('search.pitKeepAliveMs', constants.SEARCH_PIT_KEEP_ALIVE_MS, { + min: 1, + max: Number.MAX_SAFE_INTEGER + }); +} + function resetSearchProviderForTests() { searchProviderConfig = undefined; - basicSearchManager = undefined; - searchReader = undefined; + searchClient = undefined; } module.exports = { checkOpenSearchReadiness, - getBasicSearchManager, + createOpenSearchPit, + deleteOpenSearchPit, getAllowUnknownSslCerts, getIntegerConfig, getOpenSearchReadinessTimeoutMs, getOpenSearchRequestTimeoutMs, + getSearchPitBatchSize, + getSearchPitKeepAliveMs, getSearchRequestDeadlineMs, getSearchProviderConfig, - getSearchReader, - isSearchProviderContractError, + getSearchClient, normalizeBoolean, resetSearchProviderForTests, searchOpenSearch, - validateSearchReader, - SearchProviderContractError + searchOpenSearchPit, + withOpenSearchPit }; diff --git a/utils/searchProvider.test.js b/utils/searchProvider.test.js index 050644b..8fc837a 100644 --- a/utils/searchProvider.test.js +++ b/utils/searchProvider.test.js @@ -1,44 +1,21 @@ -const mockBasicSearchManager = jest.fn(function BasicSearchManager(spec) { - this.spec = spec; - this._searchReader = { - _client: { - transport: { - maxRetries: 3, - requestTimeout: 30000 - } - } - }; -}); const mockClientSearch = jest.fn(); const mockClientFieldCaps = jest.fn(); -const mockSearchReader = jest.fn(function SearchReader(endpoint, index, options) { - this._openSearchDomainEndpoint = endpoint; - this._cveIndex = index; - this._options = options; - this._client = { - fieldCaps: mockClientFieldCaps, - search: mockClientSearch, - transport: { - maxRetries: 3, - requestTimeout: 30000 - } - }; +const mockClientCreatePit = jest.fn(); +const mockClientDeletePit = jest.fn(); +const mockClient = jest.fn(function Client() { + this.fieldCaps = mockClientFieldCaps; + this.search = mockClientSearch; + this.createPit = mockClientCreatePit; + this.deletePit = mockClientDeletePit; }); -const mockGetDefaultSearchProviderSpec = jest.fn(); const mockAppConfigHas = jest.fn(); const mockAppConfigGet = jest.fn(); -const mockSearchProviderSpec = { - getDefaultSearchProviderSpec: mockGetDefaultSearchProviderSpec -}; - -jest.mock('cve-core', () => ({ - AppConfig: { - get: mockAppConfigGet, - has: mockAppConfigHas - }, - BasicSearchManager: mockBasicSearchManager, - SearchReader: mockSearchReader, - SearchProviderSpec: mockSearchProviderSpec +let mockConfig; + +jest.mock('@opensearch-project/opensearch', () => ({ Client: mockClient })); +jest.mock('./appConfig', () => ({ + get: mockAppConfigGet, + has: mockAppConfigHas })); const searchProvider = require('./searchProvider'); @@ -90,27 +67,38 @@ function fieldCapsResponse() { }; } +function pitCreationResponse(overrides = {}) { + return { body: { pit_id: 'test-pit', _shards: openSearchResponse().body._shards, ...overrides } }; +} + describe('searchProvider', () => { beforeEach(() => { searchProvider.resetSearchProviderForTests(); - mockBasicSearchManager.mockClear(); - mockSearchReader.mockClear(); + mockClient.mockClear(); mockClientSearch.mockReset(); mockClientFieldCaps.mockReset(); mockClientFieldCaps.mockResolvedValue(fieldCapsResponse()); - mockGetDefaultSearchProviderSpec.mockReset(); - mockGetDefaultSearchProviderSpec.mockReturnValue({ - providerEndpoint: 'https://search.example', - index: 'test-index', - allowUnknownSslCerts: false - }); + mockConfig = { + 'search.providerEndpoint': 'https://search.example', + 'search.index': 'test-index', + 'search.allowUnknownSslCerts': false + }; + mockClientCreatePit.mockReset(); + mockClientCreatePit.mockResolvedValue(pitCreationResponse()); + mockClientDeletePit.mockReset(); + mockClientDeletePit.mockResolvedValue({ body: { pits: [{ pit_id: 'test-pit', successful: true }] } }); mockAppConfigHas.mockReset(); mockAppConfigGet.mockReset(); - mockAppConfigHas.mockReturnValue(false); - delete mockSearchProviderSpec._sDefaultSearchProviderSpec; + mockAppConfigHas.mockImplementation((key) => Object.hasOwn(mockConfig, key)); + mockAppConfigGet.mockImplementation((key) => mockConfig[key]); }); - test('resolves search provider config once through cve-core config', () => { + afterEach(() => { + jest.useRealTimers(); + jest.restoreAllMocks(); + }); + + test('resolves immutable search provider config once through application config', () => { const first = searchProvider.getSearchProviderConfig(); const second = searchProvider.getSearchProviderConfig(); @@ -121,35 +109,28 @@ describe('searchProvider', () => { allowUnknownSslCerts: false }); expect(Object.isFrozen(first)).toBe(true); - expect(mockSearchProviderSpec._sDefaultSearchProviderSpec).toStrictEqual({ - providerEndpoint: 'https://search.example', - index: 'test-index', - allowUnknownSslCerts: false - }); - expect(mockGetDefaultSearchProviderSpec).toHaveBeenCalledTimes(1); + expect(mockAppConfigGet).toHaveBeenCalledTimes(3); }); - test('reuses one basic search manager per process', () => { - const first = searchProvider.getBasicSearchManager(); - const second = searchProvider.getBasicSearchManager(); + test('reuses one OpenSearch client per process', () => { + const first = searchProvider.getSearchClient(); + const second = searchProvider.getSearchClient(); expect(first).toBe(second); - expect(mockBasicSearchManager).toHaveBeenCalledTimes(1); - expect(mockBasicSearchManager).toHaveBeenCalledWith({ - providerEndpoint: 'https://search.example', - index: 'test-index', - allowUnknownSslCerts: false + expect(mockClient).toHaveBeenCalledTimes(1); + expect(mockClient).toHaveBeenCalledWith({ + node: 'https://search.example', + ssl: { rejectUnauthorized: true }, + requestTimeout: 20000, + maxRetries: 0 }); - expect(mockGetDefaultSearchProviderSpec).toHaveBeenCalledTimes(1); - expect(first._searchReader._client.transport.maxRetries).toBe(0); - expect(first._searchReader._client.transport.requestTimeout).toBe(20000); }); - test('reuses one search reader per process for OpenSearch searches', async () => { + test('uses the shared client for validated OpenSearch searches', async () => { mockClientSearch.mockResolvedValue(openSearchResponse()); - const first = searchProvider.getSearchReader(); - const second = searchProvider.getSearchReader(); + const first = searchProvider.getSearchClient(); + const second = searchProvider.getSearchClient(); const body = { query: { match_all: {} @@ -158,10 +139,7 @@ describe('searchProvider', () => { const result = await searchProvider.searchOpenSearch(body); expect(first).toBe(second); - expect(mockSearchReader).toHaveBeenCalledTimes(1); - expect(mockSearchReader).toHaveBeenCalledWith('https://search.example', 'test-index', { - ignore_ssl_cert_check_errors: false - }); + expect(mockClient).toHaveBeenCalledTimes(1); expect(mockClientSearch).toHaveBeenCalledWith( { index: 'test-index', @@ -173,8 +151,6 @@ describe('searchProvider', () => { requestTimeout: 20000 } ); - expect(first._client.transport.maxRetries).toBe(0); - expect(first._client.transport.requestTimeout).toBe(20000); expect(result).toStrictEqual(openSearchResponse().body); }); @@ -267,7 +243,7 @@ describe('searchProvider', () => { expect(request.abort).toHaveBeenCalledTimes(1); }); - test('does not construct a SearchReader for an already-canceled readiness check', async () => { + test('does not construct a client for an already-canceled readiness check', async () => { const controller = new AbortController(); controller.abort(new searchDeadline.SearchClientDisconnectedError()); @@ -278,21 +254,57 @@ describe('searchProvider', () => { ).rejects.toMatchObject({ code: 'SEARCH_CLIENT_DISCONNECTED' }); - expect(mockSearchReader).not.toHaveBeenCalled(); + expect(mockClient).not.toHaveBeenCalled(); expect(mockClientSearch).not.toHaveBeenCalled(); }); - test('rejects and does not cache an incompatible cve-core SearchReader', () => { - mockSearchReader.mockImplementationOnce(() => ({ - _cveIndex: 'test-index', - _client: { - transport: {} + test('does not retain a client after failed initialization', () => { + mockClient.mockImplementationOnce(() => { + throw new Error('Invalid endpoint'); + }); + expect(() => searchProvider.getSearchClient()).toThrow('Invalid endpoint'); + expect(searchProvider.getSearchClient()).toBeDefined(); + expect(mockClient).toHaveBeenCalledTimes(2); + }); + + test.each([ + [ + 'timed-out response', + (body) => { + body.timed_out = true; + } + ], + [ + 'partial shards', + (body) => { + body._shards.failed = 1; } - })); + ], + [ + 'missing hit source', + (body) => { + body.hits.hits = [{ _id: 'CVE-2024-0001' }]; + } + ] + ])('rejects a %s for either search endpoint', async (_case, mutate) => { + const response = openSearchResponse(); + mutate(response.body); + mockClientSearch.mockResolvedValue(response); + await expect(searchProvider.searchOpenSearch({ query: { match_all: {} } })).rejects.toMatchObject({ + code: 'INVALID_OPEN_SEARCH_RESPONSE' + }); + }); - expect(() => searchProvider.getSearchReader()).toThrow(searchProvider.SearchProviderContractError); - expect(searchProvider.getSearchReader()).toBeDefined(); - expect(mockSearchReader).toHaveBeenCalledTimes(2); + test.each([undefined, '', 'invalid'])('certificate verification stays enabled for %p', (value) => { + mockConfig['search.allowUnknownSslCerts'] = value; + searchProvider.getSearchClient(); + expect(mockClient.mock.calls[0][0].ssl.rejectUnauthorized).toBe(true); + }); + + test('certificate verification stays enabled when the setting is absent', () => { + delete mockConfig['search.allowUnknownSslCerts']; + searchProvider.getSearchClient(); + expect(mockClient.mock.calls[0][0].ssl.rejectUnauthorized).toBe(true); }); test('aborts an active OpenSearch request when the lifecycle signal aborts', async () => { @@ -339,43 +351,42 @@ describe('searchProvider', () => { ).rejects.toMatchObject({ code: 'SEARCH_CLIENT_DISCONNECTED' }); - expect(mockSearchReader).not.toHaveBeenCalled(); + expect(mockClient).not.toHaveBeenCalled(); expect(mockClientSearch).not.toHaveBeenCalled(); }); - test('throws when OpenSearch endpoint or index are missing', () => { - mockGetDefaultSearchProviderSpec.mockReturnValue({ - providerEndpoint: '', - index: 'test-index' - }); + test.each([ + ['search.providerEndpoint', ''], + ['search.providerEndpoint', undefined], + ['search.index', ''], + ['search.index', ' '], + ['search.index', 123] + ])('throws when %s is missing or invalid (%s)', (key, value) => { + mockConfig[key] = value; expect(() => searchProvider.getSearchProviderConfig()).toThrow( 'OpenSearch providerEndpoint and index must be configured.' ); }); - test('normalizes string FALSE from AppConfig before constructing managers', () => { - mockGetDefaultSearchProviderSpec.mockReturnValue({ - providerEndpoint: 'https://search.example', - index: 'test-index', - allowUnknownSslCerts: true - }); - mockAppConfigHas.mockReturnValue(true); - mockAppConfigGet.mockReturnValue('FALSE'); + test('normalizes string FALSE before constructing a client', () => { + mockConfig['search.allowUnknownSslCerts'] = 'FALSE'; expect(searchProvider.getSearchProviderConfig()).toStrictEqual({ providerEndpoint: 'https://search.example', index: 'test-index', allowUnknownSslCerts: false }); - expect(mockSearchProviderSpec._sDefaultSearchProviderSpec.allowUnknownSslCerts).toBe(false); + searchProvider.getSearchClient(); + expect(mockClient.mock.calls[0][0].ssl.rejectUnauthorized).toBe(true); }); - test('normalizes string TRUE from AppConfig for local certificate bypasses', () => { - mockAppConfigHas.mockReturnValue(true); - mockAppConfigGet.mockReturnValue('TRUE'); + test('normalizes string TRUE for local certificate bypasses', () => { + mockConfig['search.allowUnknownSslCerts'] = 'TRUE'; expect(searchProvider.getSearchProviderConfig().allowUnknownSslCerts).toBe(true); + searchProvider.getSearchClient(); + expect(mockClient.mock.calls[0][0].ssl.rejectUnauthorized).toBe(false); }); test('reads search timeout runtime settings from config', () => { @@ -416,4 +427,393 @@ describe('searchProvider', () => { expect(searchProvider.getOpenSearchReadinessTimeoutMs()).toBe(5000); } ); + test('provides PIT tuning defaults without constructing a search client', () => { + expect(searchProvider.getSearchPitBatchSize()).toBe(1000); + expect(searchProvider.getSearchPitKeepAliveMs()).toBe(60000); + expect(mockClient).not.toHaveBeenCalled(); + }); + + test.each([ + ['search.pitBatchSize', 'getSearchPitBatchSize', '500', 500], + ['search.pitBatchSize', 'getSearchPitBatchSize', 1, 1], + ['search.pitBatchSize', 'getSearchPitBatchSize', '10000', 10000], + ['search.pitBatchSize', 'getSearchPitBatchSize', '10001', 1000], + ['search.pitKeepAliveMs', 'getSearchPitKeepAliveMs', '30000', 30000], + ['search.pitKeepAliveMs', 'getSearchPitKeepAliveMs', 120000, 120000], + ['search.pitKeepAliveMs', 'getSearchPitKeepAliveMs', Number.MAX_SAFE_INTEGER + 1, 60000] + ])('reads %s through %s when configured as %p', (key, getter, configuredValue, expected) => { + mockAppConfigHas.mockImplementation((path) => path === key); + mockAppConfigGet.mockImplementation((path) => (path === key ? configuredValue : undefined)); + + expect(searchProvider[getter]()).toBe(expected); + expect(mockClient).not.toHaveBeenCalled(); + }); + + test.each(['not-a-number', '', '0', '-1', '1.5', 'Infinity'])('rejects invalid PIT tuning value %p', (value) => { + mockAppConfigHas.mockReturnValue(true); + mockAppConfigGet.mockReturnValue(value); + + expect(searchProvider.getSearchPitBatchSize()).toBe(1000); + expect(searchProvider.getSearchPitKeepAliveMs()).toBe(60000); + }); + + test.each([[true], [[1000]], [{ value: 1000 }], [null]])('does not coerce non-numeric config type %p', (value) => { + mockAppConfigHas.mockReturnValue(true); + mockAppConfigGet.mockReturnValue(value); + + expect(searchProvider.getSearchPitBatchSize()).toBe(1000); + expect(searchProvider.getSearchPitKeepAliveMs()).toBe(60000); + expect(searchProvider.getOpenSearchRequestTimeoutMs()).toBe(20000); + }); + + describe('PIT lifecycle', () => { + test('creates, searches, and deletes one snapshot through the shared client', async () => { + const body = { query: { match_all: {} }, size: 1000, sort: [{ 'cveMetadata.cveId.keyword': 'asc' }] }; + mockClientSearch.mockResolvedValue(openSearchResponse()); + + const result = await searchProvider.withOpenSearchPit((pitId) => + searchProvider.searchOpenSearchPit(pitId, body) + ); + + expect(result).toStrictEqual(openSearchResponse().body); + expect(mockClient).toHaveBeenCalledTimes(1); + expect(mockClientCreatePit).toHaveBeenCalledWith( + { index: 'test-index', keep_alive: '60000ms', allow_partial_pit_creation: false }, + { maxRetries: 0, requestTimeout: 20000 } + ); + expect(mockClientSearch).toHaveBeenCalledWith( + { + allow_partial_search_results: false, + body: { ...body, pit: { id: 'test-pit', keep_alive: '60000ms' } } + }, + { maxRetries: 0, requestTimeout: 20000 } + ); + expect(body).not.toHaveProperty('pit'); + expect(mockClientDeletePit).toHaveBeenCalledWith( + { body: { pit_id: ['test-pit'] } }, + { maxRetries: 0, requestTimeout: 5000 } + ); + }); + + test('uses configured lifetimes and operation timeouts for PIT requests', async () => { + const config = { + ...mockConfig, + 'search.pitKeepAliveMs': '30000', + 'search.openSearchRequestTimeoutMs': '2000' + }; + mockAppConfigHas.mockImplementation((key) => key in config); + mockAppConfigGet.mockImplementation((key) => config[key]); + mockClientSearch.mockResolvedValue(openSearchResponse()); + + await searchProvider.withOpenSearchPit((pitId) => searchProvider.searchOpenSearchPit(pitId, {})); + + expect(mockClientCreatePit).toHaveBeenNthCalledWith( + 1, + { index: 'test-index', keep_alive: '30000ms', allow_partial_pit_creation: false }, + { maxRetries: 0, requestTimeout: 2000 } + ); + expect(mockClientSearch.mock.calls[0][0].body.pit.keep_alive).toBe('30000ms'); + expect(mockClientSearch.mock.calls[0][1].requestTimeout).toBe(2000); + expect(mockClientDeletePit.mock.calls[0][1].requestTimeout).toBe(2000); + }); + + test.each(['createPit', 'deletePit'])( + 'rejects a missing %s method without disabling ordinary search', + async (method) => { + delete searchProvider.getSearchClient()[method]; + const operation = jest.fn(); + + await expect(searchProvider.withOpenSearchPit(operation)).rejects.toBeInstanceOf(TypeError); + expect(operation).not.toHaveBeenCalled(); + expect(mockClientCreatePit).not.toHaveBeenCalled(); + mockClientSearch.mockResolvedValue(openSearchResponse()); + await expect(searchProvider.searchOpenSearch({})).resolves.toStrictEqual(openSearchResponse().body); + } + ); + + test('cleans up a partial snapshot before rejecting creation', async () => { + mockClientCreatePit.mockResolvedValue( + pitCreationResponse({ _shards: { total: 2, successful: 1, skipped: 0, failed: 1 } }) + ); + const operation = jest.fn(); + + await expect(searchProvider.withOpenSearchPit(operation)).rejects.toMatchObject({ + code: 'INVALID_OPEN_SEARCH_RESPONSE' + }); + expect(operation).not.toHaveBeenCalled(); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + }); + + test('does not attempt deletion when creation fails without returning an identifier', async () => { + const failure = new Error('creation failed'); + mockClientCreatePit.mockRejectedValue(failure); + + await expect(searchProvider.withOpenSearchPit(jest.fn())).rejects.toBe(failure); + expect(mockClientDeletePit).not.toHaveBeenCalled(); + }); + + test('classifies timed-out PIT responses and cleans up the snapshot', async () => { + mockClientSearch.mockResolvedValue({ body: { ...openSearchResponse().body, timed_out: true } }); + + await expect( + searchProvider.withOpenSearchPit((pitId) => searchProvider.searchOpenSearchPit(pitId, {})) + ).rejects.toMatchObject({ code: 'SEARCH_TIMEOUT' }); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + }); + + test('clamps creation and each search to the remaining request budget without shrinking cleanup', async () => { + let remaining = 30000; + const options = { getRemainingTimeMs: () => remaining }; + mockClientCreatePit.mockImplementation(async () => { + remaining = 17000; + return pitCreationResponse(); + }); + mockClientSearch.mockImplementation(async () => { + remaining = 4500; + return openSearchResponse(); + }); + + await searchProvider.withOpenSearchPit(async (pitId) => { + await searchProvider.searchOpenSearchPit(pitId, {}, options); + return searchProvider.searchOpenSearchPit(pitId, {}, options); + }, options); + + expect(mockClientCreatePit.mock.calls[0][1]).toStrictEqual({ maxRetries: 0, requestTimeout: 20000 }); + expect(Array.from(mockClientSearch.mock.calls, ([, options]) => options.requestTimeout)).toStrictEqual([ + 17000, 4500 + ]); + expect(mockClientDeletePit.mock.calls[0][1]).toStrictEqual({ maxRetries: 0, requestTimeout: 5000 }); + }); + + test.each([ + ['create', 0], + ['search', 0], + ['create', 1], + ['search', 1] + ])('does not start expired PIT %s work after %s budget checks', async (phase, validChecks) => { + const getRemainingTimeMs = jest.fn().mockReturnValue(0); + if (validChecks) getRemainingTimeMs.mockReturnValueOnce(100); + const options = { getRemainingTimeMs }; + const run = + phase === 'create' + ? searchProvider.withOpenSearchPit(jest.fn(), options) + : searchProvider.searchOpenSearchPit('test-pit', {}, options); + + await expect(run).rejects.toMatchObject({ code: 'SEARCH_TIMEOUT' }); + expect(mockClientCreatePit).not.toHaveBeenCalled(); + expect(mockClientSearch).not.toHaveBeenCalled(); + }); + + test.each([ + ['create', 1200.8, 1200], + ['search', 0.5, 1] + ])( + 'uses a positive integer %s timeout for a fractional remaining budget', + async (phase, remaining, expected) => { + const options = { getRemainingTimeMs: () => remaining }; + mockClientSearch.mockResolvedValue(openSearchResponse()); + if (phase === 'create') await searchProvider.withOpenSearchPit(() => 'done', options); + else await searchProvider.searchOpenSearchPit('test-pit', {}, options); + + const clientCall = phase === 'create' ? mockClientCreatePit : mockClientSearch; + expect(clientCall.mock.calls[0][1].requestTimeout).toBe(expected); + } + ); + + test.each(['create', 'search'])( + 'rejects a late %s response and still cleans up independently', + async (phase) => { + let remaining = 100; + const options = { getRemainingTimeMs: () => remaining }; + if (phase === 'create') { + mockClientCreatePit.mockImplementation(async () => { + remaining = 0; + return pitCreationResponse(); + }); + } else { + mockClientSearch.mockImplementation(async () => { + remaining = 0; + return openSearchResponse(); + }); + } + const operation = jest.fn((pitId) => searchProvider.searchOpenSearchPit(pitId, {}, options)); + + await expect(searchProvider.withOpenSearchPit(operation, options)).rejects.toMatchObject({ + code: 'SEARCH_TIMEOUT' + }); + expect(operation).toHaveBeenCalledTimes(phase === 'create' ? 0 : 1); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + expect(mockClientDeletePit.mock.calls[0][1].requestTimeout).toBe(5000); + } + ); + + test('normalizes expired PIT errors without retrying or retaining private upstream details', async () => { + const failure = Object.assign(new Error('private upstream details'), { + meta: { + statusCode: 404, + body: { + error: { + type: 'search_phase_execution_exception', + root_cause: [{ type: 'search_context_missing_exception', reason: 'private PIT identifier' }] + } + } + } + }); + mockClientSearch.mockRejectedValue(failure); + + await expect( + searchProvider.withOpenSearchPit((pitId) => searchProvider.searchOpenSearchPit(pitId, {})) + ).rejects.toMatchObject({ + code: 'SEARCH_PIT_UNAVAILABLE', + message: 'Search snapshot is no longer available.' + }); + expect(mockClientCreatePit).toHaveBeenCalledTimes(1); + expect(mockClientSearch).toHaveBeenCalledTimes(1); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + }); + + test('does not misclassify unrelated upstream 404 errors as PIT expiration', async () => { + const failure = Object.assign(new Error('private index details'), { + meta: { statusCode: 404, body: { error: { type: 'index_not_found_exception' } } } + }); + mockClientSearch.mockRejectedValue(failure); + + await expect(searchProvider.searchOpenSearchPit('test-pit', {})).rejects.toBe(failure); + }); + + test.each(['', ' ', '_all', '*'])( + 'rejects invalid or global PIT identifier %p before client construction', + async (pitId) => { + await expect(searchProvider.searchOpenSearchPit(pitId, {})).rejects.toBeInstanceOf(TypeError); + await expect(searchProvider.deleteOpenSearchPit(pitId)).rejects.toBeInstanceOf(TypeError); + expect(mockClient).not.toHaveBeenCalled(); + } + ); + + test('does not create or search a PIT when already canceled', async () => { + const controller = new AbortController(); + const reason = new searchDeadline.SearchClientDisconnectedError(); + controller.abort(reason); + + await expect(searchProvider.withOpenSearchPit(jest.fn(), { signal: controller.signal })).rejects.toBe( + reason + ); + await expect( + searchProvider.searchOpenSearchPit('test-pit', {}, { signal: controller.signal }) + ).rejects.toBe(reason); + expect(mockClient).not.toHaveBeenCalled(); + }); + + test('cleans up even when cancellation occurs while creation completes', async () => { + const controller = new AbortController(); + const reason = new searchDeadline.SearchClientDisconnectedError(); + const request = Promise.resolve(pitCreationResponse()); + request.abort = jest.fn(); + mockClientCreatePit.mockImplementation(() => { + controller.abort(reason); + return request; + }); + const operation = jest.fn(); + + await expect(searchProvider.withOpenSearchPit(operation, { signal: controller.signal })).rejects.toBe( + reason + ); + await new Promise(setImmediate); + expect(request.abort).toHaveBeenCalledTimes(1); + expect(operation).not.toHaveBeenCalled(); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + }); + + test('aborts canceled creation and cleans up an identifier delivered later', async () => { + const pending = deferred(); + pending.promise.abort = jest.fn(); + mockClientCreatePit.mockReturnValue(pending.promise); + const controller = new AbortController(); + const operation = jest.fn(); + const run = searchProvider.withOpenSearchPit(operation, { signal: controller.signal }); + const reason = new searchDeadline.SearchDeadlineExceededError(); + + controller.abort(reason); + await expect(run).rejects.toBe(reason); + expect(pending.promise.abort).toHaveBeenCalledTimes(1); + pending.resolve(pitCreationResponse()); + await new Promise(setImmediate); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + expect(operation).not.toHaveBeenCalled(); + }); + + test.each([ + ['deadline', () => new searchDeadline.SearchDeadlineExceededError()], + ['disconnect', () => new searchDeadline.SearchClientDisconnectedError()] + ])('aborts a PIT search on %s and deletes with an independent signal', async (_label, getReason) => { + const pending = deferred(); + pending.promise.abort = jest.fn(); + mockClientSearch.mockReturnValue(pending.promise); + const started = deferred(); + const controller = new AbortController(); + const run = searchProvider.withOpenSearchPit( + (pitId) => { + const search = searchProvider.searchOpenSearchPit(pitId, {}, { signal: controller.signal }); + started.resolve(); + return search; + }, + { signal: controller.signal } + ); + await started.promise; + + const reason = getReason(); + controller.abort(reason); + await expect(run).rejects.toBe(reason); + expect(pending.promise.abort).toHaveBeenCalledTimes(1); + expect(mockClientDeletePit).toHaveBeenCalledTimes(1); + }); + + test.each([false, true])( + 'does not replace the operation result or error on cleanup failure (operation fails: %s)', + async (fails) => { + const warning = jest.spyOn(console, 'warn').mockImplementation(() => {}); + const failure = new Error('operation failed'); + mockClientDeletePit.mockRejectedValue(new Error('private backend details and PIT identifier')); + + const run = searchProvider.withOpenSearchPit(() => { + if (fails) throw failure; + return 'complete'; + }); + if (fails) await expect(run).rejects.toBe(failure); + else await expect(run).resolves.toBe('complete'); + expect(warning).toHaveBeenCalledWith( + 'OpenSearch PIT cleanup failed; the context will expire after its keep-alive period.' + ); + expect(warning).toHaveBeenCalledTimes(1); + } + ); + + test('rejects an unsuccessful deletion confirmation', async () => { + mockClientDeletePit.mockResolvedValue({ body: { pits: [{ pit_id: 'test-pit', successful: false }] } }); + + await expect(searchProvider.deleteOpenSearchPit('test-pit')).rejects.toMatchObject({ + code: 'INVALID_OPEN_SEARCH_RESPONSE' + }); + }); + + test('bounds cleanup independently and aborts its pending transport request', async () => { + jest.useFakeTimers({ doNotFake: ['performance'] }); + const warning = jest.spyOn(console, 'warn').mockImplementation(() => {}); + const pending = deferred(); + pending.promise.abort = jest.fn(); + const started = deferred(); + mockClientDeletePit.mockImplementation(() => { + started.resolve(); + return pending.promise; + }); + const run = searchProvider.withOpenSearchPit(() => 'complete'); + await started.promise; + + jest.advanceTimersByTime(5000); + await expect(run).resolves.toBe('complete'); + expect(pending.promise.abort).toHaveBeenCalledTimes(1); + expect(warning).toHaveBeenCalledTimes(1); + expect(jest.getTimerCount()).toBe(0); + }); + }); }); diff --git a/utils/searchQueryBuilder.js b/utils/searchQueryBuilder.js index 1a95324..4853a8d 100644 --- a/utils/searchQueryBuilder.js +++ b/utils/searchQueryBuilder.js @@ -3,6 +3,7 @@ const cpe = require('./cpe'); const cvss = require('./cvss'); const searchScope = require('./searchScope'); const searchPostFilters = require('./searchPostFilters'); +const { isValidPitSearchAfter } = require('./searchResponseValidator'); const ssvc = require('./ssvc'); const kev = require('./kev'); @@ -83,6 +84,30 @@ function buildPostFilterCandidateSearchBody(searchOptions, queryStringObjects = }); } +function buildPitCandidateSearchBody( + searchOptions, + queryStringObjects = [], + postFilters = {}, + { batchSize = constants.SEARCH_PIT_BATCH_SIZE, searchAfter } = {} +) { + if (!Number.isInteger(batchSize) || batchSize < 1 || batchSize > constants.SEARCH_PIT_MAX_BATCH_SIZE) { + throw new TypeError('PIT candidate batch size is outside the supported range.'); + } + if (searchAfter !== undefined && !isValidPitSearchAfter(searchAfter)) { + throw new TypeError('PIT search_after must contain one nonempty CVE ID sort value.'); + } + + const body = buildPostFilterCandidateSearchBody( + { ...searchOptions, resultsPerPage: batchSize }, + queryStringObjects, + postFilters + ); + delete body.from; + if (searchAfter !== undefined) body.search_after = [...searchAfter]; + + return body; +} + function buildCveIdsSearchRequest(cveIds) { const filter = buildCveIdsFilter(cveIds.join(','), constants.FIELDS.cveIds); @@ -696,6 +721,7 @@ function cvssV2SeverityToRange(sev) { module.exports = { buildCveIdsSearchRequest, buildOpenSearchBody, + buildPitCandidateSearchBody, buildPostFilterCandidateSearchBody, buildQueryStrings, buildSearchRequest, diff --git a/utils/searchQueryBuilder.test.js b/utils/searchQueryBuilder.test.js index fd628fc..a255ebd 100644 --- a/utils/searchQueryBuilder.test.js +++ b/utils/searchQueryBuilder.test.js @@ -732,6 +732,95 @@ describe('searchQueryBuilder getFilterFields() function', () => { }); }); +describe('PIT candidate query construction', () => { + const request = queryBuilder.buildSearchRequest({ + virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', + keywordSearch: 'kernel', + pubStartDate: '2024-01-01T00:00:00.000Z', + resultsPerPage: 10, + pageNumber: 2 + }); + + function buildBody(options) { + return queryBuilder.buildPitCandidateSearchBody( + request.searchOptions, + request.queryStringObjects, + request.postFilters, + options + ); + } + + test('starts a batch without offset or continuation and leaves PIT lifecycle to the provider', () => { + const original = JSON.stringify(request); + const body = buildBody(); + + expect(body.size).toBe(1000); + expect(body.track_total_hits).toBe(false); + expect(body.sort).toStrictEqual([{ 'cveMetadata.cveId.keyword': { order: 'asc' } }]); + expect(body).not.toHaveProperty('from'); + expect(body).not.toHaveProperty('search_after'); + expect(body).not.toHaveProperty('pit'); + expect(JSON.stringify(request)).toBe(original); + expect(request.responseOptions).toStrictEqual({ resultsPerPage: 10, from: 10 }); + }); + + test.each([1, 250, 10000])('continues with the supplied batch size %i and a copied cursor', (batchSize) => { + const searchAfter = Object.freeze(['CVE-2024-0001']); + const body = buildBody({ batchSize, searchAfter }); + + expect(body.size).toBe(batchSize); + expect(body.search_after).toStrictEqual(searchAfter); + expect(body.search_after).not.toBe(searchAfter); + expect(body.sort).toStrictEqual(buildBody().sort); + expect(body).not.toHaveProperty('from'); + }); + + test.each([0, -1, 1.5, 10001, NaN, Infinity, '1000', true, null])('rejects invalid batch size %p', (batchSize) => { + expect(() => buildBody({ batchSize })).toThrow(TypeError); + }); + + test.each( + [null, [], [''], [' '], [null], [1], ['CVE-2024-0001', 1], 'CVE-2024-0001', {}].map((searchAfter) => [ + searchAfter + ]) + )('rejects an invalid continuation %p', (searchAfter) => { + expect(() => buildBody({ searchAfter })).toThrow(TypeError); + }); + + test.each([ + ['CPE name', { cpeName: 'cpe:2.3:a:vendor:product:2.10:*:*:*:*:*:*:*' }], + ['virtual match', { virtualMatchString: 'cpe:2.3:o:linux:linux_kernel' }], + [ + 'virtual match range', + { virtualMatchString: 'cpe:2.3:o:linux:linux_kernel', versionEnd: '6.0', versionEndType: 'excluding' } + ], + ['FULL CVSS', { scope: 'FULL', cvssV3Metrics: 'AV:N/AC:L' }], + ['ADP SSVC', { scope: 'ADP_ONLY', ssvcExploitation: 'ACTIVE' }], + [ + 'CNA KEV dates', + { scope: 'CNA_ONLY', knownExploited: true, kevDateAddedStartDate: '2024-01-01T00:00:00.000Z' } + ] + ])('preserves candidate predicates and restricted source fields for %s', (_description, params) => { + const { searchOptions, queryStringObjects, postFilters } = queryBuilder.buildSearchRequest({ + ...params, + keywordSearch: 'remote code', + pubStartDate: '2024-01-01T00:00:00.000Z' + }); + const boundedBody = queryBuilder.buildPostFilterCandidateSearchBody( + searchOptions, + queryStringObjects, + postFilters + ); + const pitBody = queryBuilder.buildPitCandidateSearchBody(searchOptions, queryStringObjects, postFilters); + delete boundedBody.from; + + expect(postFilters).toBeDefined(); + expect(pitBody).toStrictEqual({ ...boundedBody, size: 1000 }); + expect(pitBody._source).toContain('cveMetadata.cveId'); + expect(pitBody._source).not.toContain('*'); + }); +}); + describe('scope-aware search filters', () => { test('FULL searches description fields in CNA and ADP containers', () => { const output = queryBuilder.buildSearchRequest({ diff --git a/utils/searchResponseBuilder.js b/utils/searchResponseBuilder.js index 04920e6..88a8832 100644 --- a/utils/searchResponseBuilder.js +++ b/utils/searchResponseBuilder.js @@ -36,7 +36,7 @@ function getCveIdWarnings(value) { ]; } -function getCpeWarnings(params = {}) { +function getCpeWarnings(params = {}, { usesPit = false } = {}) { const warnings = []; if (params.cpeName) { @@ -59,8 +59,9 @@ function getCpeWarnings(params = {}) { if (broadComponents.length) { warnings.push({ code: 'BROAD_VIRTUAL_MATCH_STRING', - message: - 'Wildcard or omitted leading CPE components broaden this search and may cause results to reach the semantic candidate limit.', + message: usesPit + ? 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.' + : 'Wildcard or omitted leading CPE components broaden this search and may cause results to reach the semantic candidate limit.', components: broadComponents }); } @@ -69,8 +70,8 @@ function getCpeWarnings(params = {}) { return warnings; } -function getRequestWarnings(params = {}) { - return [...getCpeWarnings(params), ...getCveIdWarnings(params.cveIds)]; +function getRequestWarnings(params = {}, options) { + return [...getCpeWarnings(params, options), ...getCveIdWarnings(params.cveIds)]; } function buildSearchResponse({ totalResults, hits, pagingOptions, totalIsExact = true, warnings = [] }) { diff --git a/utils/searchResponseBuilder.test.js b/utils/searchResponseBuilder.test.js index 8e51f4e..9333e47 100644 --- a/utils/searchResponseBuilder.test.js +++ b/utils/searchResponseBuilder.test.js @@ -137,6 +137,19 @@ describe('search response builder', () => { ).toStrictEqual([]); }); + test('PIT keeps broad-CPE guidance without claiming a semantic candidate cap', () => { + expect( + searchResponseBuilder.getRequestWarnings({ virtualMatchString: 'cpe:2.3:o' }, { usesPit: true }) + ).toStrictEqual([ + { + code: 'BROAD_VIRTUAL_MATCH_STRING', + message: + 'Wildcard or omitted leading CPE components broaden this search and may increase search processing time.', + components: ['vendor', 'product'] + } + ]); + }); + test('orders CPE interpretation warnings before CVE ID warnings', () => { expect( searchResponseBuilder diff --git a/utils/searchResponseValidator.js b/utils/searchResponseValidator.js index bc5d17e..369c6fe 100644 --- a/utils/searchResponseValidator.js +++ b/utils/searchResponseValidator.js @@ -1,3 +1,5 @@ +const { SearchDeadlineExceededError } = require('./searchDeadline'); + class InvalidOpenSearchResponseError extends Error { constructor(message) { super(message); @@ -6,6 +8,37 @@ class InvalidOpenSearchResponseError extends Error { } } +class SearchPitUnavailableError extends Error { + constructor() { + super('Search snapshot is no longer available.'); + this.name = 'SearchPitUnavailableError'; + this.code = 'SEARCH_PIT_UNAVAILABLE'; + } +} + +function containsMissingContext(reason, depth = 0) { + if (!reason || typeof reason !== 'object' || depth > 8) return false; + if (reason.type === 'search_context_missing_exception') return true; + + return ( + containsMissingContext(reason.caused_by, depth + 1) || + (Array.isArray(reason.root_cause) && + reason.root_cause.some((cause) => containsMissingContext(cause, depth + 1))) || + (Array.isArray(reason.failed_shards) && + reason.failed_shards.some((shard) => containsMissingContext(shard?.reason, depth + 1))) + ); +} + +function throwIfPitContextMissing(body) { + if ( + containsMissingContext(body?.error) || + (Array.isArray(body?._shards?.failures) && + body._shards.failures.some((shard) => containsMissingContext(shard?.reason))) + ) { + throw new SearchPitUnavailableError(); + } +} + function fail(message) { throw new InvalidOpenSearchResponseError(message); } @@ -45,7 +78,106 @@ function validateOpenSearchResponse(body) { return body; } +function validateCompletePitShards(shards, allowSkipped) { + validateShardMetadata(shards); + const { total, successful, skipped } = shards; + // Skipped search shards are included in the successful count, not added to it. + if ( + !Number.isSafeInteger(total) || + total < 1 || + successful !== total || + !Number.isSafeInteger(skipped) || + skipped < 0 || + skipped > successful || + (!allowSkipped && skipped !== 0) + ) { + fail('OpenSearch PIT response did not account for every requested shard.'); + } + if (shards.failures !== undefined && (!Array.isArray(shards.failures) || shards.failures.length !== 0)) { + fail('OpenSearch PIT response contains inconsistent shard failures.'); + } +} + +function validatePitSearchResponse(body) { + if (body?.timed_out === true) throw new SearchDeadlineExceededError(); + throwIfPitContextMissing(body); + validateOpenSearchResponse(body); + if (body.error !== undefined) fail('OpenSearch PIT response contains an error.'); + validateCompletePitShards(body._shards, true); + if (body.terminated_early !== undefined && body.terminated_early !== false) { + fail('OpenSearch PIT search did not complete.'); + } + return body; +} + +function isValidPitSearchAfter(searchAfter) { + return ( + Array.isArray(searchAfter) && + searchAfter.length === 1 && + typeof searchAfter[0] === 'string' && + searchAfter[0].trim() !== '' + ); +} + +function validatePitCandidateResponse(body, searchAfter) { + if (searchAfter !== undefined && !isValidPitSearchAfter(searchAfter)) { + throw new TypeError('PIT search_after must contain one nonempty CVE ID sort value.'); + } + validatePitSearchResponse(body); + + let previous = searchAfter?.[0]; + for (const hit of body.hits.hits) { + if (!isValidPitSearchAfter(hit.sort) || hit.sort[0] !== hit._source.cveMetadata?.cveId) { + fail('OpenSearch PIT candidate is missing a matching CVE ID sort value.'); + } + + const current = hit.sort[0]; + // Keyword sorting uses byte order, not numeric or locale-aware CVE ID ordering. + if (previous !== undefined && Buffer.compare(Buffer.from(current), Buffer.from(previous)) <= 0) { + fail('OpenSearch PIT candidate sort values did not advance.'); + } + previous = current; + } + + return body; +} + +function isValidPitId(pitId) { + return typeof pitId === 'string' && pitId.trim() !== '' && pitId !== '_all' && pitId !== '*'; +} + +function validatePitCreationResponse(body) { + if (!isValidPitId(body?.pit_id)) { + fail('OpenSearch PIT creation response is missing a valid identifier.'); + } + + validateCompletePitShards(body._shards, false); + + return body; +} + +function validatePitDeletionResponse(body, pitId) { + if ( + !Array.isArray(body?.pits) || + body.pits.length !== 1 || + body.pits[0]?.pit_id !== pitId || + body.pits[0]?.successful !== true + ) { + fail('OpenSearch did not confirm deletion of the requested PIT.'); + } + + return body; +} + module.exports = { InvalidOpenSearchResponseError, - validateOpenSearchResponse + SearchPitUnavailableError, + isValidPitId, + isValidPitSearchAfter, + throwIfPitContextMissing, + validateOpenSearchResponse, + validatePitCandidateResponse, + validatePitCreationResponse, + validatePitDeletionResponse, + validatePitSearchResponse }; diff --git a/utils/searchResponseValidator.test.js b/utils/searchResponseValidator.test.js index 5562b9d..09f80b6 100644 --- a/utils/searchResponseValidator.test.js +++ b/utils/searchResponseValidator.test.js @@ -1,4 +1,5 @@ const searchResponseValidator = require('./searchResponseValidator'); +const { SearchDeadlineExceededError } = require('./searchDeadline'); function validResponse(overrides = {}) { return { @@ -37,3 +38,212 @@ describe('searchResponseValidator', () => { ); }); }); + +describe('PIT candidate response validation', () => { + function candidateResponse(ids) { + return validResponse({ + hits: { + hits: ids.map((cveId) => ({ _source: { cveMetadata: { cveId } }, sort: [cveId] })) + } + }); + } + + test('accepts keyword-ordered candidates, including lexically rather than numerically ordered IDs', () => { + const body = candidateResponse(['CVE-2024-10000', 'CVE-2024-9999']); + + expect(searchResponseValidator.validatePitCandidateResponse(body)).toBe(body); + }); + + test('accepts a subsequent batch only beyond the previous continuation', () => { + const body = candidateResponse(['CVE-2024-0002', 'CVE-2024-0003']); + const searchAfter = Object.freeze(['CVE-2024-0001']); + + expect(searchResponseValidator.validatePitCandidateResponse(body, searchAfter)).toBe(body); + expect(body.hits.hits[1].sort).toStrictEqual(['CVE-2024-0003']); + }); + + test.each([undefined, ['CVE-2024-0001']].map((searchAfter) => [searchAfter]))( + 'accepts an empty final batch after %p', + (searchAfter) => { + const body = candidateResponse([]); + + expect(searchResponseValidator.validatePitCandidateResponse(body, searchAfter)).toBe(body); + } + ); + + test.each([undefined, [], [''], [' '], [null], [1], ['CVE-2024-0001', 1], ['CVE-2024-0002']].map((sort) => [sort]))( + 'rejects missing, malformed, or mismatched hit sort values %p', + (sort) => { + const body = candidateResponse(['CVE-2024-0001']); + body.hits.hits[0].sort = sort; + + expect(() => searchResponseValidator.validatePitCandidateResponse(body)).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + } + ); + + test('rejects candidates without a source CVE ID needed for subsequent page retrieval', () => { + const body = candidateResponse(['CVE-2024-0001']); + body.hits.hits[0]._source = {}; + + expect(() => searchResponseValidator.validatePitCandidateResponse(body)).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + }); + + test.each([ + [['CVE-2024-0001', 'CVE-2024-0001'], undefined], + [['CVE-2024-0002', 'CVE-2024-0001'], undefined], + [['CVE-2024-0001'], ['CVE-2024-0001']], + [['CVE-2024-0001'], ['CVE-2024-0002']], + [['CVE-2024-0001', 'CVE-2024-0003'], ['CVE-2024-0002']] + ])('rejects non-advancing candidates %p after %p', (ids, searchAfter) => { + expect(() => searchResponseValidator.validatePitCandidateResponse(candidateResponse(ids), searchAfter)).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + }); + + test.each([null, [], [1], ['CVE-2024-0001', 1]].map((searchAfter) => [searchAfter]))( + 'rejects invalid prior continuation %p', + (searchAfter) => { + expect(() => + searchResponseValidator.validatePitCandidateResponse(candidateResponse([]), searchAfter) + ).toThrow(TypeError); + } + ); + + test.each([{ _shards: { failed: 1 } }, { hits: { hits: null } }])( + 'retains ordinary response validation for %p', + (override) => { + expect(() => searchResponseValidator.validatePitCandidateResponse(validResponse(override))).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + } + ); +}); + +describe('PIT response validation', () => { + function pitCreationResponse(overrides = {}) { + return { pit_id: 'test-pit', _shards: validResponse()._shards, ...overrides }; + } + + test.each([0, 1, 2])('accepts skipped search shards as a subset of successful shards: %s', (skipped) => { + const response = validResponse({ + _shards: { total: 2, successful: 2, skipped, failed: 0 }, + hits: { hits: [] } + }); + + expect(searchResponseValidator.validatePitCandidateResponse(response)).toBe(response); + }); + + test.each([ + ['missing total', { successful: 1, skipped: 0, failed: 0 }], + ['missing successful count', { total: 1, skipped: 0, failed: 0 }], + ['missing skipped count', { total: 1, successful: 1, failed: 0 }], + ['unaccounted shard', { total: 2, successful: 1, skipped: 0, failed: 0 }], + ['skipped added to successful', { total: 2, successful: 1, skipped: 1, failed: 0 }], + ['too many skipped', { total: 1, successful: 1, skipped: 2, failed: 0 }], + ['negative skipped', { total: 1, successful: 1, skipped: -1, failed: 0 }], + ['fractional skipped', { total: 1, successful: 1, skipped: 0.5, failed: 0 }], + ['string total', { total: '1', successful: '1', skipped: 0, failed: 0 }], + ['no shards', { total: 0, successful: 0, skipped: 0, failed: 0 }], + ['contradictory failures', { total: 1, successful: 1, skipped: 0, failed: 0, failures: [{}] }], + ['malformed failures', { total: 1, successful: 1, skipped: 0, failed: 0, failures: 'none' }] + ])('rejects an empty PIT batch with %s', (_label, shards) => { + expect(() => + searchResponseValidator.validatePitCandidateResponse(validResponse({ _shards: shards, hits: { hits: [] } })) + ).toThrow(searchResponseValidator.InvalidOpenSearchResponseError); + }); + + test('classifies a timed-out PIT response as a timeout rather than a malformed response', () => { + expect(() => searchResponseValidator.validatePitCandidateResponse(validResponse({ timed_out: true }))).toThrow( + SearchDeadlineExceededError + ); + }); + + test('rejects an error payload even if other completion fields claim success', () => { + expect(() => + searchResponseValidator.validatePitSearchResponse(validResponse({ error: { type: 'unknown' } })) + ).toThrow(searchResponseValidator.InvalidOpenSearchResponseError); + }); + + test.each([true, 'false', null])('rejects incomplete or malformed termination status: %p', (terminated_early) => { + expect(() => searchResponseValidator.validatePitSearchResponse(validResponse({ terminated_early }))).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + }); + + test.each([ + { error: { type: 'search_context_missing_exception' } }, + { + error: { + type: 'search_phase_execution_exception', + root_cause: [{ type: 'search_context_missing_exception' }] + } + }, + { error: { caused_by: { type: 'search_context_missing_exception' } } }, + { error: { failed_shards: [{ reason: { type: 'search_context_missing_exception' } }] } }, + { _shards: { failures: [{ reason: { caused_by: { type: 'search_context_missing_exception' } } }] } } + ])('recognizes missing contexts in structured PIT errors: %p', (body) => { + expect(() => searchResponseValidator.validatePitSearchResponse(body)).toThrow( + searchResponseValidator.SearchPitUnavailableError + ); + }); + + test('does not infer PIT expiration from unstructured messages or unrelated error types', () => { + expect(() => + searchResponseValidator.throwIfPitContextMissing({ + error: { type: 'index_not_found_exception', reason: 'search_context_missing_exception' } + }) + ).not.toThrow(); + }); + + test('accepts a complete PIT snapshot', () => { + const response = pitCreationResponse(); + + expect(searchResponseValidator.validatePitCreationResponse(response)).toBe(response); + }); + + test.each([ + ['a missing identifier', { pit_id: undefined }], + ['an empty identifier', { pit_id: ' ' }], + ['a global identifier', { pit_id: '_all' }], + ['a wildcard identifier', { pit_id: '*' }], + ['missing shards', { _shards: undefined }], + ['failed shards', { _shards: { total: 2, successful: 1, skipped: 0, failed: 1 } }], + ['unaccounted shards', { _shards: { total: 2, successful: 1, skipped: 0, failed: 0 } }], + ['skipped shards', { _shards: { total: 2, successful: 2, skipped: 1, failed: 0 } }], + ['an empty snapshot', { _shards: { total: 0, successful: 0, skipped: 0, failed: 0 } }] + ])('rejects PIT creation with %s', (_description, overrides) => { + expect(() => searchResponseValidator.validatePitCreationResponse(pitCreationResponse(overrides))).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + }); + + test('requires confirmation of the specific PIT deletion', () => { + const response = { pits: [{ pit_id: 'test-pit', successful: true }] }; + + expect(searchResponseValidator.validatePitDeletionResponse(response, 'test-pit')).toBe(response); + }); + + test.each([ + undefined, + {}, + { pits: [] }, + { pits: [null] }, + { pits: [{ pit_id: 'other-pit', successful: true }] }, + { pits: [{ pit_id: 'test-pit', successful: false }] }, + { pits: [{ pit_id: 'test-pit', successful: 'true' }] }, + { + pits: [ + { pit_id: 'test-pit', successful: true }, + { pit_id: 'other-pit', successful: true } + ] + } + ])('rejects an invalid PIT deletion response %p', (response) => { + expect(() => searchResponseValidator.validatePitDeletionResponse(response, 'test-pit')).toThrow( + searchResponseValidator.InvalidOpenSearchResponseError + ); + }); +}); diff --git a/utils/webSearchQueryBuilder.js b/utils/webSearchQueryBuilder.js new file mode 100644 index 0000000..ecf8a3b --- /dev/null +++ b/utils/webSearchQueryBuilder.js @@ -0,0 +1,142 @@ +const path = require('path'); +const validator = require('validator'); +const { WEB_SEARCH_RESULTS_PER_PAGE } = require('./constants'); + +const DATE_FIELDS = Object.freeze([ + 'cveMetadata.datePublished', + 'cveMetadata.dateRejected', + 'cveMetadata.dateReserved', + 'cveMetadata.dateUpdated', + 'containers.cna.datePublic', + 'containers.cna.providerMetadata.dateUpdated', + 'containers.cna.timeline.time', + 'containers.adp.metrics.other.content.dateAdded', + 'containers.adp.metrics.other.content.timestamp', + 'containers.adp.providerMetadata.dateUpdated' +]); +const URL_OPTIONS = { protocols: ['http', 'https', 'ftp', 'app', 'wss', 'ws', 'file'] }; + +function parseDate(value, allowPartial = false) { + const text = value.trim(); + const match = + /^(\d{4})(?:-(\d{2})(?:-(\d{2}))?)?(?:T(\d{2}):(\d{2}):(\d{2})(?:\.(\d+))?(Z|[+-]\d{2}:\d{2})?)?$/.exec(text); + if (!match) return null; + const [, yearText, monthText, dayText, hourText, minuteText, secondText, fraction, zone] = match; + if ((!allowPartial || hourText !== undefined) && !dayText) return null; + + const year = Number(yearText); + const month = Number(monthText ?? 1); + const day = Number(dayText ?? 1); + const hour = Number(hourText ?? 0); + const minute = Number(minuteText ?? 0); + const second = Number(secondText ?? 0); + if ( + year < 1900 || + year > 2100 || + month < 1 || + month > 12 || + day < 1 || + day > new Date(Date.UTC(year, month, 0)).getUTCDate() || + hour > 23 || + minute > 59 || + second > 59 + ) + return null; + + const milliseconds = Number((fraction ?? '0').padEnd(3, '0').slice(0, 3)); + let timestamp = Date.UTC(year, month - 1, day, hour, minute, second, milliseconds); + if (zone && zone !== 'Z') { + const [hours, minutes] = zone.slice(1).split(':').map(Number); + timestamp -= (zone[0] === '+' ? 1 : -1) * (hours * 60 + minutes) * 60000; + } + const utc = new Date(timestamp).toISOString().replace('.000Z', 'Z'); + const rounding = hourText !== undefined ? '' : !monthText ? 'y' : !dayText ? 'M' : 'd'; + return { timestamp, utc, text, rounding }; +} + +function parseDateRange(text) { + const parts = text.split('..'); + if (parts.length !== 2) return null; + const [start, end] = parts.map((part) => parseDate(part, true)); + if (!start || !end) return null; + if (start.timestamp > end.timestamp) throw new Error('Invalid web search date range.'); + return [start, end]; +} + +function dateExpression(date, rangeEndpoint) { + const value = rangeEndpoint || !date.rounding ? date.utc : date.text; + return date.rounding ? `${value}||/${date.rounding}` : value; +} + +function isUrl(text) { + return !validator.isIP(text) && validator.isURL(text, URL_OPTIONS); +} + +function describeToken(token) { + if (token.includes('{')) return null; + if (/"[^"]*"/.test(token)) return 'search text is a phrase (surrounded by double quotes)'; + if (parseDate(token)) return 'search text is a date (ISO 8601)'; + if (parseDateRange(token)) return 'search text is a date range (ISO 8601)'; + if (token.includes('*')) return "search as wildcard ('*')"; + if (token.includes('?')) return isUrl(token) ? 'search text is a URL' : "search as wildcard ('?')"; + if (token.includes('%')) return "does not support wildcard character '%'"; + if (isUrl(token)) return 'search text is a URL'; + if (token.includes('-')) { + if (/^CVE-[1,2]\d{3}-\d{4,7}$/i.test(token)) return 'search text is a CVE ID'; + if (/^CVE-\d{4}$/i.test(token)) return 'search text is a CVE ID year series'; + if (/^CWE-\d{1,4}$/i.test(token)) return 'search text is a CWE ID'; + if (/^capec-\d{1,4}$/i.test(token)) return 'search text is a CAPEC ID'; + if (path.parse(token).base) return 'search text is a version string'; + } + if (token.includes(':')) return validator.isIP(token, 6) ? 'search text is an IP v6 address' : null; + if (token.includes('.')) { + return validator.isIP(token, 4) ? 'search text is an IP v4 address' : 'search text is a version string'; + } + if (token.includes('&')) return 'search text is a phrase (surrounded by double quotes)'; + return 'general search text string'; +} + +function describeQuery(query) { + const cleaned = query.trim().replace(/([^\p{L}0-9.-])\1{2,}/gu, ''); + const tokens = (cleaned.match(/"[^"]*"|[\S]+/g) ?? []).filter((token) => token !== ','); + let note; + for (let token of tokens) { + if (token.startsWith('"') && token.endsWith('"')) { + token = token.slice(1, -1).replace(/\\"/g, '"').replace(/\\\\/g, '\\'); + } + // Preserve the legacy final-token note/acceptance behavior; the DSL uses the original query. + note = describeToken(token); + } + if (!note) throw new Error('Unsupported web search expression.'); + return [note]; +} + +function buildWebSearchRequest(query, { sort, from = 0, size = WEB_SEARCH_RESULTS_PER_PAGE } = {}) { + const notes = describeQuery(query); + const date = parseDate(query); + const range = date ? [date, date] : parseDateRange(query); + const searchQuery = range + ? { + bool: { + should: DATE_FIELDS.map((field) => ({ + range: { [field]: { gte: dateExpression(range[0], !date), lte: dateExpression(range[1], !date) } } + })), + minimum_should_match: 1 + } + } + : { query_string: { query, default_operator: 'AND' } }; + + const offset = Math.max(0, from); + return { + body: { + query: searchQuery, + track_total_hits: true, + sort: sort ?? [{ 'cveMetadata.cveId.keyword': { order: 'desc' } }], + from: offset, + size + }, + notes + }; +} + +module.exports = { buildWebSearchRequest }; diff --git a/utils/webSearchQueryBuilder.test.js b/utils/webSearchQueryBuilder.test.js new file mode 100644 index 0000000..5c85777 --- /dev/null +++ b/utils/webSearchQueryBuilder.test.js @@ -0,0 +1,117 @@ +const { buildWebSearchRequest } = require('./webSearchQueryBuilder'); + +const dateFields = [ + 'cveMetadata.datePublished', + 'cveMetadata.dateRejected', + 'cveMetadata.dateReserved', + 'cveMetadata.dateUpdated', + 'containers.cna.datePublic', + 'containers.cna.providerMetadata.dateUpdated', + 'containers.cna.timeline.time', + 'containers.adp.metrics.other.content.dateAdded', + 'containers.adp.metrics.other.content.timestamp', + 'containers.adp.providerMetadata.dateUpdated' +]; + +describe('raw web search compatibility', () => { + test.each([ + 'apache', + 'remote code', + 'remote code execution', + '"remote code execution"', + 'apache*', + 'ap?che', + 'apache AND nginx', + '(apache OR nginx)', + 'cveMetadata.cveId.keyword:CVE-2024-0001', + 'CVE-2024-0001', + 'https://example.com/a?b=1', + '192.0.2.1', + '2001:db8::1', + '2024', + '2024-01', + '2024-02-30', + 'prefix\\:value apache' + ])('preserves the original query expression: %s', (query) => { + expect(buildWebSearchRequest(query).body).toStrictEqual({ + query: { query_string: { query, default_operator: 'AND' } }, + track_total_hits: true, + sort: [{ 'cveMetadata.cveId.keyword': { order: 'desc' } }], + from: 0, + size: 25 + }); + }); + + test.each([ + ['apache', 'general search text string'], + ['"remote code"', 'general search text string'], + ['CVE-2024-0001', 'search text is a CVE ID'], + ['CVE-2024', 'search text is a CVE ID year series'], + ['CWE-79', 'search text is a CWE ID'], + ['CAPEC-1', 'search text is a CAPEC ID'], + ['apache*', "search as wildcard ('*')"], + ['ap?che', "search as wildcard ('?')"], + ['foo%bar', "does not support wildcard character '%'"], + ['https://example.com/a?b=1', 'search text is a URL'], + ['192.0.2.1', 'search text is an IP v4 address'], + ['2001:db8::1', 'search text is an IP v6 address'], + ['1.2.3', 'search text is a version string'], + ['some-file', 'search text is a version string'], + ['file.txt', 'search text is a URL'], + ['a&b', 'search text is a phrase (surrounded by double quotes)'], + ['"a\\\\b"', 'general search text string'], + ['{reserved} apache', 'general search text string'] + ])('preserves the legacy description for %s', (query, note) => { + expect(buildWebSearchRequest(query).notes).toStrictEqual([note]); + }); + + test.each([ + ['2024-02-29', '2024-02-29||/d', '2024-02-29||/d'], + ['2024-01-01T12:30:00Z', '2024-01-01T12:30:00Z', '2024-01-01T12:30:00Z'], + ['2024-01-01T12:30:00.25+02:00', '2024-01-01T10:30:00.250Z', '2024-01-01T10:30:00.250Z'], + ['2024-01-01T12:30:00', '2024-01-01T12:30:00Z', '2024-01-01T12:30:00Z'], + ['2024-01-01..2024-01-31', '2024-01-01T00:00:00Z||/d', '2024-01-31T00:00:00Z||/d'], + ['2023..2024', '2023-01-01T00:00:00Z||/y', '2024-01-01T00:00:00Z||/y'], + ['2024-01..2024-03', '2024-01-01T00:00:00Z||/M', '2024-03-01T00:00:00Z||/M'], + ['2024-01-01 .. 2024-01-02', '2024-01-01T00:00:00Z||/d', '2024-01-02T00:00:00Z||/d'] + ])('expands %s across the legacy date fields with inclusive bounds', (query, gte, lte) => { + expect(buildWebSearchRequest(query).body.query).toStrictEqual({ + bool: { + should: dateFields.map((field) => ({ range: { [field]: { gte, lte } } })), + minimum_should_match: 1 + } + }); + }); + + test.each(['', ' ', ',', '{reserved}', 'field:value', '2024-02-01..2024-01-01'])( + 'rejects unsupported input without sending a query: %s', + (query) => { + expect(() => buildWebSearchRequest(query)).toThrow(); + } + ); + + test('quoted dates remain literal query expressions', () => { + expect(buildWebSearchRequest('"2024-01-01"').body.query).toStrictEqual({ + query_string: { query: '"2024-01-01"', default_operator: 'AND' } + }); + }); + + test.each([ + [0, 25, 25], + [5, 10, 10], + [25, 25, 25], + [50, 25, 25], + [500, 1, 1], + [4999, 1, 1], + [9500, 500, 500], + [9975, undefined, 25], + [-1, 10, 10] + ])('keeps page size independent of offset %s with size %s', (from, size, effectiveSize) => { + const sort = [{ _score: { order: 'asc' } }]; + expect(buildWebSearchRequest('apache', { from, size, sort }).body).toMatchObject({ + from: Math.max(0, from), + size: effectiveSize, + sort + }); + }); +});