Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ OpenSearchDomainEndpoint=
SearchOpenSearchRequestTimeoutMs=20000
SearchRequestDeadlineMs=30000
SearchReadinessTimeoutMs=5000
SearchLogPerformanceTimings=false
SearchPerformanceTelemetryIntervalMs=10000
RequestLogSuccessSampleRate=0.1
RequestLogHealthChecks=false
WebSearchEnabled=true
Expand Down
42 changes: 25 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Running the API locally

Rename `.env.example` to `.env` and replace or check the following values.
Copy `.env.example` to an untracked `.env` and replace or check the following values. Keep the tracked example in place.

- `PORT` - port used for the search API, defaulted to 3000 for development
- `OpenSearchCveIndex` - existing CVE index name in your OpenSearch deployment
Expand All @@ -11,13 +11,13 @@ Rename `.env.example` to `.env` and replace or check the following values.
- `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000`
- `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000`
- `SearchReadinessTimeoutMs` - optional timeout for the OpenSearch index readiness check, default `5000`
- `RequestLogSuccessSampleRate` - optional fraction of successful non-health-check access logs to retain, default `0.1`
- `RequestLogHealthChecks` - optional flag to retain successful health-check access logs, default `false`
- `RequestLogSuccessSampleRate` - optional fraction of successful access logs to retain, default `0.1`; includes health checks when they are enabled for logging
- `RequestLogHealthChecks` - optional flag to include successful health checks in access-log sampling, default `false`
- `WebSearchEnabled` - optional `/webSearch` availability flag, default `true`
- `NPM_CA_CERT` - optional absolute host path to a corporate or self-signed CA certificate bundle used only during local Docker Compose builds

```bash
mv .env.example .env
cp .env.example .env
```

## Docker API Only
Expand Down Expand Up @@ -48,7 +48,9 @@ The compose file runs only the Search API with `NODE_ENV=production` and `NODE_C

## Deploying with npm

**NOTE: The current locked `cve-core` dependency requires Node.js `>= 16.20.2 < 22`.**
Use Node.js 24 for the repository's current runtime and development tooling. CI and the Docker dependency-build stage use Node.js 24.14.1; the final runtime image uses Distroless Node.js 24.

**Upstream compatibility caveat:** The locked `cve-core` package still declares Node.js `>= 16.20.2 < 22`. That metadata conflicts with this repository's Node.js 24 baseline and can cause installation warnings or failures when strict engine checks are enabled. Resolving the declaration requires coordination with the `cve-core` maintainers; it is not a recommendation to use Node.js 16 or 18 with this repository's current development tooling.

1. Set `OpenSearchDomainEndpoint` in `.env` to an existing OpenSearch deployment:

Expand All @@ -59,7 +61,7 @@ OpenSearchDomainEndpoint=https://search.example.com
2. Install dependencies:

```bash
npm install
npm ci
```

3. Run the CVE Search API in development mode:
Expand All @@ -74,13 +76,13 @@ npm run dev
npm start
```

The Search API will be served at `localhost:3000`.
The Search API uses `PORT`, defaulting to `http://localhost:3000`. The tracked Postman environment defaults to port `4000`; either set `PORT=4000` before starting the API or change Postman's `baseUrl` to match the running server.

## Health Checks

`GET /api/health-check` is a process liveness check. It returns HTTP 200 when the Node.js process is running and deliberately does not contact OpenSearch. Use it where a dependency outage should not restart otherwise healthy API tasks.

`GET /api/ready-check` first verifies that the configured CVE index exposes compatible field capabilities for each logical `/search` feature, then sends a minimal zero-result search. CNA, ADP, legacy, and CVSS-version fields that implement the same feature are treated as alternatives, while mapped fields must still use compatible searchable types. This checks mappings, connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 when search requests can be served. Configuration errors, connection failures, timeouts, incompatible `cve-core` SearchReader contracts, missing logical capabilities, incompatible mapped fields, and a missing index return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. Each OpenSearch readiness request is limited to five seconds by default and can be changed with `SearchReadinessTimeoutMs`.
`GET /api/ready-check` first verifies that the configured CVE index exposes compatible field capabilities for each logical `/search` feature, then sends a minimal zero-result search. CNA, ADP, legacy, and CVSS-version fields that implement the same feature are treated as alternatives, while mapped fields must still use compatible searchable types. This checks mappings, connectivity, index availability, and the same search permission used by the API without retrieving CVE records or counting all matches. It returns HTTP 200 when these checks pass; it does not prove data completeness or capacity for every search. Configuration errors, connection failures, timeouts, incompatible `cve-core` SearchReader contracts, missing logical capabilities, incompatible mapped fields, and a missing index return a sanitized HTTP 503 `SEARCH_NOT_READY` response with `Retry-After: 5`. `SearchReadinessTimeoutMs` defaults to five seconds for the overall readiness check, covering both field-capability and search operations. Each individual operation also uses that value as its transport timeout, and the overall deadline can abort it earlier.

An AWS load balancer can use `/api/ready-check` to stop routing search traffic to an unready task while the container orchestrator continues using `/api/health-check` for process liveness. Coordinate that probe selection with the infrastructure team before changing deployed health-check behavior.

Expand Down Expand Up @@ -108,7 +110,9 @@ Scope applies to filter matching as follows:
| `cpeName`, `isVulnerable`, `virtualMatchString`, and version range filters | CNA CPE applicability data; incompatible with `ADP_ONLY` |
| `resultsPerPage`, `pageNumber` | Response controls; they do not select record data |

If the selected containers do not contain the requested data, the request is valid and returns zero matches. The API does not remove unselected containers from a returned record.
Missing data in selected containers does not make an otherwise compatible request invalid. A positive filter cannot match a record that lacks the requested data. Exclusion filters behave differently: `knownExploited: false` and `disputed: false` can match records with no selected KEV metric or disputed tag, including records without a selected container. Other supplied filters must still match. The API does not remove unselected containers from a returned record.

`keywordSearch` searches only description text selected by `scope`. By default, the API escapes caller-supplied query syntax, adds a trailing prefix wildcard to each whitespace-delimited term, and combines terms with `AND` using `simple_query_string`. For example, `remote code` searches for `remote* AND code*`. With `keywordExactMatch: true`, it instead uses an analyzed phrase query in the selected description fields. Phrase matching follows the index's text analyzer; it is not a case-sensitive, character-for-character substring comparison.

SSVC metrics can be filtered with `ssvcExploitation` (`NONE`, `POC`, or `ACTIVE`), `ssvcAutomatable` (`YES` or `NO`), `ssvcTechnicalImpact` (`PARTIAL` or `TOTAL`), `ssvcRole`, `ssvcTimestampStartDate`, and `ssvcTimestampEndDate`. Enum and role comparisons are case-insensitive, timestamps use the API's full UTC timestamp format with inclusive boundaries, and all supplied SSVC conditions must match the same structured SSVC metric entry in a container selected by `scope`.

Expand Down Expand Up @@ -234,6 +238,8 @@ Numeric segments used in version range comparisons ignore leading zeros. A store

Future recommendation: broad `virtualMatchString` values can be expensive because they may require large candidate scans plus semantic CPE post-filtering. Requiring a concrete `part` and `vendor` for normal searches and a concrete `part`, `vendor`, and `product` for version range searches would reduce the most expensive public search shapes. Narrowing the accepted searches beyond the current valid-part requirement should be handled as a policy decision before implementation.

## Raw Query Endpoint

`POST /webSearch` is a supported raw-query endpoint. It is enabled by default and can be disabled without a code change by setting `WebSearchEnabled=false` and restarting the service. When disabled, it returns HTTP `410` with the stable error code `WEB_SEARCH_DISABLED` before validating the request or contacting OpenSearch.

While enabled, `POST /webSearch` requires a non-empty JSON `query` string. It also accepts optional `sort`, `from`, and `size` fields:
Expand All @@ -250,8 +256,8 @@ While enabled, `POST /webSearch` requires a non-empty JSON `query` string. It al
}
```

- `sort.property` may be `cveId` or `relevancy`
- `sort.order` may be `asc` or `desc`
- `sort.property` is required when `sort` is supplied and may be `cveId` or `relevancy`
- `sort.order` may be `asc` or `desc`; when omitted, the chosen property is sorted descending
- `from` must be an integer greater than or equal to `0`
- `size` must be an integer from `1` to `500`

Expand All @@ -277,7 +283,7 @@ Unit, route, OpenAPI, runtime config, and error helper tests can be run with npm
npm test
```

Static quality gates can be run with npm:
Linting, formatting, and coverage checks can be run with npm. `npm run quality` runs all three, and the coverage command executes the test suite:

```bash
npm run lint
Expand All @@ -288,15 +294,15 @@ npm run quality

An importable Postman collection for manual `/search` contract and filter testing is documented in [test/postman/README.md](test/postman/README.md). The complete collection targets a fully populated index like those used by shared managed environments; data-dependent positive assertions may fail against an incomplete local index even when the API is operating correctly.

### Operational Logging
## Operational Logging

Every request receives an `X-Request-ID` response header. A caller-provided `X-Request-ID` is preserved when it contains only letters, numbers, `.`, `_`, `:`, or `-` and is no longer than 128 characters; otherwise the API generates a UUID. The same value appears as `request-id` in access logs.

Access logs omit query strings and request bodies. By default, the API retains 10% of successful non-health-check access logs, suppresses successful `/api/health-check` and `/api/ready-check` access logs, and retains every HTTP 4xx or 5xx access log. Set `RequestLogSuccessSampleRate` from `0` through `1` to change successful-request sampling. Set `RequestLogHealthChecks=true` to retain successful health-check logs.
Access logs omit query strings and request bodies. By default, the API retains 10% of successful non-health-check access logs, suppresses successful `/api/health-check` and `/api/ready-check` access logs, and retains every HTTP 4xx or 5xx access log. Set `RequestLogSuccessSampleRate` from `0` through `1` to change successful-request sampling. `RequestLogHealthChecks=true` includes successful health checks in that same sample; it does not bypass sampling. To retain every successful health check, enable that flag and set `RequestLogSuccessSampleRate=1`, which also retains all other successful access logs.

Each `/search` request owns a lifecycle cancellation signal. When its overall deadline expires, the API aborts the active OpenSearch transport request before returning HTTP 504. When the caller disconnects, the API aborts work that is needed only by that caller and does not attempt to write a response. This prevents timed-out or abandoned standard searches from continuing until the transport timeout.

### Semantic Post-Filtering
## Semantic Post-Filtering

Two separate 10,000-record controls apply to `/search`:

Expand All @@ -307,13 +313,15 @@ CVSS vector filters, SSVC filters, date-bounded positive `knownExploited` search

The reported total and pagination are complete only within that bounded candidate set; qualifying records beyond the first 10,000 search candidates may not be represented. When a semantic search reaches that limit, the response sets `pagination.totalIsExact` to `false` and includes a `SEMANTIC_CANDIDATE_LIMIT_REACHED` warning. Semantic searches that remain below the limit and direct searches set `totalIsExact` to `true`. This preserves the existing bounded post-filter behavior until PIT-based pagination is implemented. A `knownExploited` filter without date boundaries uses a direct query and retains ordinary exact-total behavior.

`/search` orders search results by CVE ID so repeated shallow-page requests use a deterministic sort. The response `pagination.pageSize` preserves the requested `resultsPerPage` value, while `pagination.resultsReturned` reports the number of records actually returned and may be smaller for a partial or out-of-range page. A stable sort does not provide a point-in-time snapshot, so index updates between separate page requests can still shift results until PIT-based pagination is implemented.
`/search` orders search results by CVE ID so repeated shallow-page requests use a deterministic sort. The response `pagination.pageSize` preserves the requested `resultsPerPage` value, while `pagination.resultsReturned` reports the number of records actually returned and may be smaller for a partial or out-of-range page. Current searches do not use a point-in-time snapshot, so index updates between separate page requests can shift results. Request-scoped PIT traversal would improve consistency within one request, but consistency across separate page requests would additionally require reusing the same snapshot.

The API does not cache, coalesce, queue, rate-limit, or aggregate performance telemetry in process. Each request executes its own required OpenSearch operations. Public rate enforcement, admission control, shared caching, and durable metrics collection must be supplied by the deployment infrastructure.

OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed the 20-second default timeout. The same response is returned when a `/search` request exceeds the 30-second default overall deadline, and the active OpenSearch request is aborted.

Docker runtime smoke checks can be run after building the image. The default `Dockerfile` avoids BuildKit `RUN --mount` syntax for AWS builders that do not support it:
## Docker Runtime Checks

Docker runtime smoke checks can be run after building the image. The default `Dockerfile` avoids BuildKit `RUN --mount` syntax for AWS builders that do not support it. The example below checks packaging and process liveness only; replace the placeholder endpoint/index and check `/api/ready-check` separately to verify an actual OpenSearch dependency:

```bash
docker build -t cve-search-api:runtime-smoke .
Expand Down
18 changes: 14 additions & 4 deletions api-docs/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,14 @@
}
},
"400": {
"description": "Bad Request"
"description": "Malformed JSON or invalid raw-query request body, sort, or pagination",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ApiErrorResponse"
}
}
}
},
"410": {
"description": "The endpoint is disabled by service configuration",
Expand Down Expand Up @@ -327,6 +334,9 @@
},
"sort": {
"type": "object",
"required": [
"property"
],
"additionalProperties": false,
"properties": {
"property": {
Expand Down Expand Up @@ -590,7 +600,7 @@
"type": "string",
"maxLength": 512,
"example": "remote code execution",
"description": "Returns CVE records where a word or phrase is included in a description selected by scope. Multiple keywords operate like an AND statement, returning records whose descriptions include every keyword. Limited to 512 characters and 64 whitespace-delimited terms."
"description": "Searches description text selected by scope. Unless keywordExactMatch is true, caller query syntax is escaped, each whitespace-delimited term receives a trailing prefix wildcard, and terms are combined with AND using simple_query_string. For example, remote code searches for remote* AND code*. Matching follows the index text analyzer. Limited to 512 characters and 64 whitespace-delimited terms."
},
"keywordExactMatch": {
"oneOf": [
Expand All @@ -606,7 +616,7 @@
}
],
"default": false,
"description": "When true, a multi-word keywordSearch value must appear exactly and in the same order. keywordSearch is required when keywordExactMatch is true. JSON booleans and the lowercase strings \"true\" and \"false\" are accepted."
"description": "When true, uses an analyzed phrase query instead of keyword prefix matching in the scope-selected description fields. Terms must match in order according to the index text analyzer; this is not a case-sensitive, character-for-character substring comparison. keywordSearch is required when keywordExactMatch is true. JSON booleans and the lowercase strings \"true\" and \"false\" are accepted."
},
"cveIds": {
"type": "string",
Expand Down Expand Up @@ -735,7 +745,7 @@
}
],
"default": false,
"description": "When true, returns CVE records where the same CPE Match Criteria entry both matches cpeName and is marked vulnerable. False or omitted does not apply a vulnerability restriction. cpeName is required when isVulnerable is true. virtualMatchString cannot be provided with isVulnerable. JSON booleans and the lowercase strings \"true\" and \"false\" are accepted."
"description": "When true, returns CVE records where the same CPE Match Criteria entry both matches cpeName and is marked vulnerable. False or omitted does not apply a vulnerability restriction. cpeName is required when isVulnerable is true. virtualMatchString cannot be provided with isVulnerable set to true; false is allowed. JSON booleans and the lowercase strings \"true\" and \"false\" are accepted."
},
"lastModStartDate": {
"type": "string",
Expand Down
2 changes: 1 addition & 1 deletion config/custom-environment-variables.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"search": {
"providerEndpoint": "OpenSearchDomainEndpoint",
"index": "OpenSearchCveIndex",
// allows local development using containers that do not have SSL certs
// TRUE disables OpenSearch server certificate verification; isolated local development only.
"allowUnknownSslCerts": "OpenSearchAllowUnknownSslCerts",
"openSearchRequestTimeoutMs": "SearchOpenSearchRequestTimeoutMs",
"requestDeadlineMs": "SearchRequestDeadlineMs",
Expand Down
Loading
Loading