Skip to content

feat(policy): v1.2.42 built-in event retention, verified archive, size guard, hot-path indexes [t_1d99b96b] - #63

Open
maximalang wants to merge 6 commits into
codex/company-osfrom
feat/t_1d99b96b-event-retention
Open

maximalang wants to merge 6 commits into
codex/company-osfrom
feat/t_1d99b96b-event-retention

Conversation

@maximalang

Copy link
Copy Markdown
Owner

fleet-policy v1.2.42 — built-in event retention, verified archive, size guard, hot-path indexes

Task: fleet-ops t_1d99b96b (ROOT FIX of the 2026-10-07 deny-triage stall: window queries over a 2.8 GB unbounded event store timed out 3× at 240s; the emergency hand-created index cut them 14.5s→0.001s).

What changes

  • Schema v5, idempotent, applied on every plugin load: idx_events_created (time window) + idx_events_kind_created (class window) join the existing task window; maintenance_state KV ledger for cross-process throttling. Index coverage never depends on a manual hotfix again.
  • Built-in retention, not a cron: throttled tick from plugin hooks (register, kanban_task_claimed) + fleet-policy maintenance CLI. Expired events are archived to deterministic JSONL.gz + .sha256 sidecar + manifest.jsonl under .state/fleet-policy-archive/; a batch's primary keys are deleted only after read-back verification (hash + gzip decode + row count). History stays restorable (INSERT OR IGNORE replay, PK-deduped).
  • Size guard: ≥100 MB soft — one significant event per UTC day via the existing notification lane; ≥300 MB hard — forces an immediate retention run regardless of interval. Maintenance failures land in a report dict and never crash the gate.
  • Vacuum: wal_autocheckpoint=1000 per connection, auto_vacuum=INCREMENTAL on fresh stores, gradual freelist drain after runs; fleet-policy maintenance --full-vacuum for a one-time legacy-store activation.
  • status now reports db_size_bytes + maintenance_last_run_epoch.

Verification (local, Windows worktree)

  • pytest tests/ -q → 925 passed, run twice; worker-env variant (HERMES_KANBAN_TASK=t_ci_simulated) → 925 passed.
  • New battery tests/test_v1242_retention_maintenance.py (12 tests): v5 heal idempotency incl. legacy hand-patched stores, EXPLAIN QUERY PLAN proves idx_events_created, archive→verify→delete ordering, tampered archive keeps every row, interval throttle + force bypass, soft/hard guard semantics + per-day dedupe + audit-only without a binding, 30-day storm bounded well under the soft cap.
  • uv lock --check OK; compileall OK; drift-check {"ok": true}; --version → 1.2.42; build-bundle + verify-bundle OK (87 files).
  • Version claim: check_version_uniqueness → {"collision": false, "version": "1.2.42"} (1.2.37–1.2.41 are all held by open PRs 56–60).

Bans honored

  • No policy-semantics changes (deny/allow rules, protected paths, approver roles, gates untouched) — storage lifecycle only.
  • The live deployed config is untouched; the repo's config/fleet-policy.yaml was NOT modified (a write attempt was denied by the live gate as control-plane mutation — correctly; new knobs are code defaults, existing retention keys still override).
  • No secrets, no paid calls, no external network in tests.
  • Rollback: revert + redeploy previous bundle; v5 schema objects are additive and harmless to older code.

Post-install readback (consumer card): sqlite3 ... "PRAGMA index_list('events')" shows both new indexes; one forced fleet-policy maintenance run archives expired events and stamps maintenance_last_run_epoch in status; DB size trend flat afterwards.

fleet-worker added 2 commits October 7, 2026 22:06
…lure status, retry throttle [t_7733d471]

F1: PRAGMA auto_vacuum=2 staged in migrate() never persisted —
journal_mode=WAL materializes the db header first (fresh stores were
silently born in mode NONE), and a staged mode does not cross
connections. Issue the pragma BEFORE the WAL switch and re-issue it in
full_vacuum() on the vacuuming connection, so legacy auto_vacuum=0
stores actually convert to INCREMENTAL on the advertised operator full
vacuum. Proven by freelist + file-size reclamation, not a marker.

F2: fleet-policy maintenance exited 0 when the report carried per-batch
errors (fail-closed archive verification). Now exits 1 on error/errors.

F3: a run with errors no longer stamps retention_last_run_epoch, so the
documented retry-next-tick fires; failed attempts are ledgered in
retention_last_failure_epoch / retention_consecutive_failures.

Ports the independent QA assertions (t_218bc548 comment 6212) into
tests/test_v1242_retention_maintenance.py: fresh/legacy conversion +
reclamation proof, mode-NONE negative control, CLI exit assertion,
throttle preservation across failure.
@maximalang

Copy link
Copy Markdown
Owner Author

Consolidated into PR #65 (fix/t_40022daf-consolidation, v1.2.43): this branch was merged onto current codex/company-os with conflict resolution and the residual read-lane fix on top; full suite 1011 passed. Can be closed once #65 merges. (fleet-ops t_40022daf)

fleet-worker and others added 3 commits October 13, 2026 00:35
…lure status, retry throttle [t_7733d471]

F1: PRAGMA auto_vacuum=2 staged in migrate() never persisted —
journal_mode=WAL materializes the db header first (fresh stores were
silently born in mode NONE), and a staged mode does not cross
connections. Issue the pragma BEFORE the WAL switch and re-issue it in
full_vacuum() on the vacuuming connection, so legacy auto_vacuum=0
stores actually convert to INCREMENTAL on the advertised operator full
vacuum. Proven by freelist + file-size reclamation, not a marker.

F2: fleet-policy maintenance exited 0 when the report carried per-batch
errors (fail-closed archive verification). Now exits 1 on error/errors.

F3: a run with errors no longer stamps retention_last_run_epoch, so the
documented retry-next-tick fires; failed attempts are ledgered in
retention_last_failure_epoch / retention_consecutive_failures.

Ports the independent QA assertions (t_218bc548 comment 6212) into
tests/test_v1242_retention_maintenance.py: fresh/legacy conversion +
reclamation proof, mode-NONE negative control, CLI exit assertion,
throttle preservation across failure.
) [t_5857485a]

Company decision 2026-10-12: rebase/re-merge the PR #63 branch onto the
moved base now; conflict resolution preserves F1/F2 fix intent (verified
by range-diff: F1-F3 QA commit 3cfc0b2 = 977e966 byte-identical).
Full suite green on the rebase result: 953 passed (uv run pytest).
No force-push: this merge commit fast-forwards the reviewed head.

# Conflicts:
#	CHANGELOG.md
#	src/fleet_policy/runtime.py
@maximalang

Copy link
Copy Markdown
Owner Author

Rebase/re-merge на сдвинувшуюся базу выполнен [t_5857485a, company decision 2026-10-12].

Новый head: 38d21a0475b71d8ebb1ed4d12d162e48e12ffdcb — merge-коммит, родители: 3cfc0b2 (прошедший review head, нетронут) + 977e966 (rebase-результат на базу 6985cdee). Push — чистый fast-forward 3cfc0b2..38d21a0, force не использовался.

Что изменилось при разрешении конфликтов:

  • 5cbcf59 → 9acb050 (1-й коммит воспроизведён на новую базу): единственные семантические изменения — пины версии → 1.2.42 в 6 местах (plugin.yaml ×2, pyproject.toml, src/fleet_policy/__init__.py, uv.lock, тест-ассерт), interleaving CHANGELOG (обе записи сохранены: [1.2.46] базовая + [1.2.42] этого PR), runtime.py both-keep (_owner_principal_ref v2 базы + maybe_maintenance v1.2.42).
  • F1-F3 QA-коммит сохранён байт-в-байт: git range-diff показал 2: 3cfc0b2 = 2: 977e966 (legacy auto_vacuum conversion, CLI failure status, retry throttle — нетронуты).

Верификация:

  • Дерево merge-результата байт-идентично дереву rebase-результата 977e966: git diff 977e966 38d21a0 пуст.
  • Полный тестовый набор: 953 passed, rc=0 (uv run --frozen pytest tests/ -q, 56.6s) — включает 23 теста human-boundary v2 базы и retention/maintenance-тесты этого PR.
  • Readback после push: PR OPEN, mergeable=MERGEABLE, head=38d21a0, base=6985cdee.

STOP до merge — слияние по команде company/owner.

…runk [t_6fa10ff8]

Company disposition 2026-10-13 (t_6fa10ff8): PR #63 must not land at
38d21a0 because that head regresses the trunk version labels from 1.2.46
back to 1.2.42 while all ten live profiles run 1.2.46. Version-regressing
merges into the protected trunk are prohibited.

This commit is the version re-pin only. The branch already carries the
current trunk (6985cde) as an ancestor through merge 38d21a0, so no
content rebase was needed: the working tree changed version labels and
release documentation, nothing else.

- pyproject.toml, plugin.yaml, integrations/hermes/fleet-policy-plugin/plugin.yaml,
  src/fleet_policy/__init__.py, uv.lock: 1.2.42 -> 1.2.47
- tests/test_hardening_v121.py: expected pin 1.2.42 -> 1.2.47
- tests/test_v1242_retention_maintenance.py -> tests/test_v1247_retention_maintenance.py
- CHANGELOG.md: retention section heading moved to 1.2.47 with a re-pin note;
  the 1.2.42 pin it released is recorded free again
- docs/FLEET_POLICY.md: storage-maintenance section heading 1.2.47

No policy-semantics change: deny/allow rules, protected paths, approver
roles and gates are untouched. 1.2.47 was verified collision-free against
the released set (max 1.2.46) and the open-PR heads.

Verified locally: pytest 953 passed (plain, repeat, and worker-env run);
uv lock --check clean; compileall clean; drift-check ok; build-bundle +
verify-bundle ok.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant