Harden Search API execution, telemetry, endpoint controls, and quality gates

.env.example

@@ -7,4 +7,13 @@ SearchPostFilterCacheTtlMs=60000
 SearchPostFilterCacheMaxEntries=100
 SearchPostFilterColdSearchConcurrency=10
 SearchPostFilterColdSearchMaxQueueDepth=500
+SearchPostFilterColdSearchMaxWaitMs=10000
+SearchOpenSearchRequestTimeoutMs=20000
+SearchRequestDeadlineMs=30000
+RequestLogSuccessSampleRate=0.1
+RequestLogHealthChecks=false
+WebSearchEnabled=true
+SearchRateLimitEnabled=false
+SearchRateLimitWindowMs=60000
+SearchRateLimitMaxRequests=0
 NPM_CA_CERT=

.github/workflows/quality.yml

@@ -0,0 +1,33 @@
+name: Search API Quality
+
+on:
+    pull_request:
+
+permissions:
+    contents: read
+
+concurrency:
+    group: search-api-quality-${{ github.event.pull_request.number }}
+    cancel-in-progress: true
+
+jobs:
+    quality:
+        name: Search API Quality
+        runs-on: ubuntu-latest
+        timeout-minutes: 15
+
+        steps:
+            - name: Check out repository
+              uses: actions/checkout@v4
+
+            - name: Set up Node.js
+              uses: actions/setup-node@v4
+              with:
+                  node-version: 24.14.1
+                  cache: npm
+
+            - name: Install dependencies
+              run: npm ci
+
+            - name: Run quality checks
+              run: npm run quality

.gitignore

@@ -81,9 +81,6 @@ web_modules/
 .env.test.local
 .env.production.local
 .env.local
-test/artillery/.env
-test/artillery/.env.*
-!test/artillery/.env.example
 
 # parcel-bundler cache (https://parceljs.org/)
 .cache

README.md

@@ -13,6 +13,15 @@ Rename `.env.example` to `.env` and replace or check the following values.
 - `SearchPostFilterCacheMaxEntries` - optional per-process CPE post-filter cache entry limit, default `100`
 - `SearchPostFilterColdSearchConcurrency` - optional per-process concurrent cold CPE candidate search limit, default `10`
 - `SearchPostFilterColdSearchMaxQueueDepth` - optional per-process waiting queue depth for cold CPE candidate searches, default `500`
+- `SearchPostFilterColdSearchMaxWaitMs` - optional maximum cold CPE candidate queue wait, default `10000`
+- `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000`
+- `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000`
+- `RequestLogSuccessSampleRate` - optional fraction of successful non-health-check access logs to retain, default `0.1`
+- `RequestLogHealthChecks` - optional flag to retain successful health-check access logs, default `false`
+- `WebSearchEnabled` - optional deprecated `/webSearch` feature flag, default `true`
+- `SearchRateLimitEnabled` - optional per-process `/search` rate-limit flag, default `false`
+- `SearchRateLimitWindowMs` - optional rate-limit window in milliseconds, default `60000`
+- `SearchRateLimitMaxRequests` - required positive per-process request limit when rate limiting is enabled
 - `NPM_CA_CERT` - optional absolute host path to a corporate or self-signed CA certificate bundle used only during local Docker Compose builds
 
 ```bash
@@ -153,7 +162,9 @@ Version ranges use overlap semantics. A stored wildcard-version CPE Match Criter
 
 Future recommendation: broad `virtualMatchString` values can be expensive because they may require large candidate scans plus semantic CPE post-filtering. Requiring concrete leading components, such as `part` and `vendor` for normal searches and `part`, `vendor`, and `product` for version range searches, would reduce the most expensive public search shapes. That would intentionally diverge from NVD behavior and should be handled as a policy decision before implementation.
 
-`POST /webSearch` requires a non-empty JSON `query` string. It also accepts optional `sort`, `from`, and `size` fields:
+`POST /webSearch` is deprecated in favor of `POST /search`. It remains enabled by default during consumer migration and can be disabled without a code change by setting `WebSearchEnabled=false` and restarting the service. When disabled, it returns HTTP `410` with the stable error code `WEB_SEARCH_DISABLED` before validating the request or contacting OpenSearch.
+
+While enabled, `POST /webSearch` requires a non-empty JSON `query` string. It also accepts optional `sort`, `from`, and `size` fields:
 
 ```json
 {
@@ -199,18 +210,23 @@ npm run coverage
 npm run quality
 ```
 
-Artillery live API checks can be run against a running API and OpenSearch deployment:
+### Operational Logging
 
-```bash
-npm run artillery:smoke
-npm run artillery:contract
-npm run artillery:load
-npm run artillery:stress
-```
+Every request receives an `X-Request-ID` response header. A caller-provided `X-Request-ID` is preserved when it contains only letters, numbers, `.`, `_`, `:`, or `-` and is no longer than 128 characters; otherwise the API generates a UUID. The same value appears as `request-id` in access logs and `requestId` in structured `/search` events.
+
+Access logs omit query strings and request bodies. By default, the API retains 10% of successful non-health-check access logs, suppresses successful `/api/health-check` access logs, and retains every HTTP 4xx or 5xx access log. Set `RequestLogSuccessSampleRate` from `0` through `1` to change successful-request sampling. Set `RequestLogHealthChecks=true` to retain successful health-check logs.
+
+The API always emits a sanitized JSON `search.performance` event for `/search` queue saturation, queue wait timeout, search timeout, OpenSearch connection errors, and upstream search errors. These events include the request ID, outcome, HTTP status, strategy, cache state, total elapsed time, and queue state when applicable. Search terms, CPE values, request bodies, credentials, and upstream error details are never included.
+
+Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. Successful requests then emit the same structured event with additional redacted timing details, including candidate counts, filtered counts, returned count, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable.
 
-See `test/artillery/README.md` for environment selection, Artillery Cloud recording, and target override examples.
+### Application Rate Limiting
 
-Set `SearchLogPerformanceTimings=true` on the API deployment when diagnosing slow `/search` requests. The timing log is redacted and includes request strategy, cache hit status, in-flight hit status, candidate counts, filtered counts, returned count, OpenSearch elapsed time, OpenSearch `took`, post-filter elapsed time, cold-search queue wait time, page fetch elapsed or wait time, and total handler elapsed time. Leave it disabled during normal operation unless the deployment logging volume is acceptable.
+Application rate limiting is disabled by default and applies only to `POST /search`. It does not apply to `/webSearch`, health checks, Swagger, OpenAPI, or static assets.
+
+The fallback limiter uses one in-memory counter per Node.js process. Set `SearchRateLimitEnabled=true`, provide a positive `SearchRateLimitMaxRequests`, and optionally change `SearchRateLimitWindowMs` to enable it. Invalid enabled configuration prevents the application from starting. Rejected requests return HTTP 429 with `SEARCH_RATE_LIMITED`, `Retry-After`, `RateLimit`, and `RateLimit-Policy` headers, and emit a sanitized `search.performance` event with the `rate_limited` outcome.
+
+This is a process-wide safety limit, not a precise distributed quota. Each ECS task has an independent counter, so total permitted traffic scales with the number of tasks and load-balancer routing. It deliberately does not use caller IP addresses or forwarded headers until the ALB proxy chain and Express `trust proxy` setting are confirmed. AWS WAF or another infrastructure-level distributed limit remains the preferred public-facing control.
 
 ### CPE Post-Filter Cache
 
@@ -227,10 +243,15 @@ The cache is intentionally small and short-lived:
 - In-flight behavior: concurrent identical cold searches share one candidate query per process
 - Cold-search concurrency: cold CPE candidate queries are limited to 10 per process before reaching OpenSearch
 - Cold-search queue depth: up to 500 unique cold searches can wait per process behind the concurrency limit
+- Cold-search queue wait: up to 10 seconds before a waiting search returns HTTP 429
+- OpenSearch request timeout: 20 seconds for each operation
+- Overall `/search` deadline: 30 seconds
 - Eviction: oldest entries are removed when the cache exceeds its max size
 
 When the cold-search queue is full, new unique CPE post-filter searches fail closed with HTTP 429 and a `SEARCH_QUEUE_FULL` response. Identical requests for a search that is already in flight still wait on the shared in-flight result and do not consume additional queue slots.
 
+When a cold search waits in the queue for more than 10 seconds, it fails closed with HTTP 429 and a `SEARCH_QUEUE_TIMEOUT` response. OpenSearch operations use one attempt and return a sanitized HTTP 504 `SEARCH_TIMEOUT` response when they exceed 20 seconds. The same response is returned when a `/search` request exceeds the 30-second overall processing deadline.
+
 Healthy dev timing logs showed the impact for repeated identical `virtualMatchString` range searches:
 
 | Metric                        | Cold Cache |               Warm Cache |
@@ -246,36 +267,7 @@ Healthy dev timing logs showed the impact for repeated identical `virtualMatchSt
 
 This helps most when multiple users submit the same expensive CPE search within the TTL. The first request is a cold cache miss; later identical requests on the same running task are warm cache hits. If identical cold requests arrive concurrently before the first one finishes, the later requests wait on the same candidate query instead of starting duplicate 10,000-candidate OpenSearch requests. Each request still fetches its requested page of full CVE records after the shared candidate result is available.
 
-The cache does not share entries across ECS tasks. With multiple tasks, each task warms independently depending on load balancer routing. Steady repeated traffic can warm every active task, but unique one-off searches still pay the cold-cache cost. The per-process TTL, entry limit, cold-search concurrency, and cold-search queue depth can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, and `SearchPostFilterColdSearchMaxQueueDepth`. Future scale improvements could add a shared cache such as Redis/ElastiCache.
-
-### Interpreting Artillery Results
-
-Artillery exits with status code `0` when the profile finishes and all expectations and SLO checks pass. A non-zero exit means at least one request expectation, virtual user run, or `ensure` SLO check failed.
-
-Expectation failures come from request-level `expect` checks. These validate HTTP response behavior such as status codes, JSON content type, and required response properties. A failed expectation usually points to a route contract issue, an unavailable upstream dependency, or a test payload that no longer matches the API contract.
-
-SLO failures come from the `ensure` gates in each profile:
-
-- `smoke.yml`: p95 response time must stay under 5000 ms and virtual user error rate must be 0.
-- `contract.yml`: p95 response time must stay under 5000 ms and virtual user error rate must be 0.
-- `load.yml`: p95 response time must stay under 3000 ms, p99 response time must stay under 8000 ms, and virtual user error rate must be at or below 1%.
-- `stress.yml`: p95 response time must stay under 8000 ms, p99 response time must stay under 15000 ms, and virtual user error rate must be at or below 5%.
-
-Latency percentiles summarize response time distribution. `p95` means 95% of measured responses completed at or below that value; `p99` means 99% completed at or below that value. A high p95 or p99 with successful status codes usually means the API is reachable but slower than the profile allows.
-
-Virtual user counters show whether Artillery completed the test flow. `vusers.created` is the number of virtual users started, and `vusers.completed` is the number that finished. A gap between those values contributes to the virtual user error rate used by `maxErrorRate`.
-
-Endpoint-level metrics are grouped by stable request names such as `POST /search keyword`, `POST /search cveIds`, and `POST /webSearch keyword`. Use these groups to identify whether a failure or latency spike is broad across the API or isolated to a specific request shape.
-
-When recording to Artillery Cloud, compare runs with the same `env` and `profile` tags. For example, compare `env:test,profile:smoke` runs to other test smoke runs rather than to local, load, or stress runs. Treat `stress.yml` failures as capacity signals rather than route contract failures unless the failing endpoint also fails smoke or contract checks.
-
-If Artillery fails with `SELF_SIGNED_CERT_IN_CHAIN` while recording to Artillery Cloud or calling an HTTPS deployment, configure Node with the same host CA bundle used for local Docker builds. `Dockerfile.local` receives that file through the `NPM_CA_CERT` environment variable and mounts it as a build secret at `/run/secrets/npm_ca`; local Artillery runs need the host path:
-
-```bash
-export NODE_EXTRA_CA_CERTS="$NPM_CA_CERT"
-```
-
-Then rerun the Artillery command with `ARTILLERY_CLOUD_API_KEY`, `--record`, and the target environment variables. Do not commit Artillery Cloud keys, deployment URLs, or certificate file paths.
+The cache does not share entries across ECS tasks. With multiple tasks, each task warms independently depending on load balancer routing. Steady repeated traffic can warm every active task, but unique one-off searches still pay the cold-cache cost. The per-process TTL, entry limit, cold-search concurrency, queue depth, queue wait, OpenSearch timeout, and overall request deadline can be tuned with `SearchPostFilterCacheTtlMs`, `SearchPostFilterCacheMaxEntries`, `SearchPostFilterColdSearchConcurrency`, `SearchPostFilterColdSearchMaxQueueDepth`, `SearchPostFilterColdSearchMaxWaitMs`, `SearchOpenSearchRequestTimeoutMs`, and `SearchRequestDeadlineMs`. Future scale improvements could add a shared cache such as Redis/ElastiCache.
 
 Docker runtime smoke checks can be run after building the image. The default `Dockerfile` avoids BuildKit `RUN --mount` syntax for AWS builders that do not support it:
 

api-docs/openapi.json

@@ -25,13 +25,47 @@
         "operationId": "search",
         "responses": {
           "200": {
-            "description": "A list of matching CVE records"
+            "description": "A list of matching CVE records",
+            "headers": {
+              "X-Request-ID": {
+                "description": "Request correlation ID",
+                "schema": {
+                  "type": "string"
+                }
+              }
+            }
           },
           "400": {
             "description": "Bad Request"
           },
           "404": {
             "description": "Invalid cpeName value"
+          },
+          "429": {
+            "description": "Application rate limit exceeded, or the cold CPE search queue is full or exceeded its maximum wait",
+            "headers": {
+              "Retry-After": {
+                "description": "Seconds until a rate-limited request can be retried",
+                "schema": {
+                  "type": "integer"
+                }
+              },
+              "RateLimit": {
+                "description": "Current application rate-limit state when application limiting is enabled",
+                "schema": {
+                  "type": "string"
+                }
+              },
+              "RateLimit-Policy": {
+                "description": "Application rate-limit policy when application limiting is enabled",
+                "schema": {
+                  "type": "string"
+                }
+              }
+            }
+          },
+          "504": {
+            "description": "Search exceeded its OpenSearch or overall processing deadline"
           }
         },
         "requestBody": {
@@ -114,7 +148,15 @@
         "operationId": "healthCheck",
         "responses": {
           "200": {
-            "description": "OK"
+            "description": "OK",
+            "headers": {
+              "X-Request-ID": {
+                "description": "Request correlation ID",
+                "schema": {
+                  "type": "string"
+                }
+              }
+            }
           }
         }
       }
@@ -124,15 +166,29 @@
         "tags": [
           "CVE Record"
         ],
-        "summary": "Retrieves all matching CVE records after applying the JSON request body",
-        "description": "  <h2>Expected Behavior</h2>  <p>Retrieves all matching CVE records after applying the JSON request body</p>",
+        "summary": "Deprecated raw query search",
+        "description": "  <h2>Expected Behavior</h2>  <p>Retrieves all matching CVE records after applying the JSON request body. This endpoint is deprecated and can be disabled by the service operator. Use POST /search instead.</p>",
         "operationId": "webSearch",
         "responses": {
           "200": {
-            "description": "A list of matching CVE records"
+            "description": "A list of matching CVE records",
+            "headers": {
+              "X-Request-ID": {
+                "description": "Request correlation ID",
+                "schema": {
+                  "type": "string"
+                }
+              }
+            }
           },
           "400": {
             "description": "Bad Request"
+          },
+          "410": {
+            "description": "The deprecated endpoint is disabled; use POST /search"
+          },
+          "504": {
+            "description": "Search exceeded its OpenSearch request deadline"
           }
         },
         "requestBody": {
@@ -183,7 +239,8 @@
               }
             }
           }
-        }
+        },
+        "deprecated": true
       }
     }
   },

api-docs/openapi.test.js

@@ -85,6 +85,8 @@ describe('OpenAPI contract', () => {
         const operation = openapi.paths['/webSearch'].post;
         const schema = operation.requestBody.content['application/json'].schema;
 
+        expect(operation.deprecated).toBe(true);
+        expect(operation.description).toContain('Use POST /search instead');
         expect(operation.parameters || []).toStrictEqual([]);
         expect(operation.requestBody.required).toBe(true);
         expect(schema.required).toStrictEqual(['query']);
@@ -93,6 +95,7 @@ describe('OpenAPI contract', () => {
         expect(schema.properties.sort.additionalProperties).toBe(false);
         expect(schema.properties.sort.properties.property.enum).toStrictEqual(['cveId', 'relevancy']);
         expect(schema.properties.size.maximum).toBe(constants.MAX_RESULTS_LIMIT);
+        expect(operation.responses['410'].description).toContain('use POST /search');
     });
 
     test('documents cpeName as a CPE Match Criteria filter with required components', () => {
@@ -105,6 +108,18 @@ describe('OpenAPI contract', () => {
         expect(cpeName.description).toContain('virtualMatchString');
         expect(cpeName.example).toBe('cpe:2.3:a:freetype:freetype:2.8.1:*:*:*:*:*:*:*');
         expect(openapi.paths['/search'].post.responses['404'].description).toBe('Invalid cpeName value');
+        expect(openapi.paths['/search'].post.responses['429'].description).toContain('rate limit');
+    });
+
+    test('documents request correlation headers', () => {
+        expect(openapi.paths['/search'].post.responses['200'].headers['X-Request-ID']).toStrictEqual({
+            description: 'Request correlation ID',
+            schema: {
+                type: 'string'
+            }
+        });
+        expect(openapi.paths['/api/health-check'].get.responses['200'].headers['X-Request-ID']).toBeDefined();
+        expect(openapi.paths['/webSearch'].post.responses['200'].headers['X-Request-ID']).toBeDefined();
     });
 
     test('committed OpenAPI artifact matches generated route documentation', async () => {