SmplCache is a source-available, proprietary diagnostic library and incremental cache-maintenance engine (Copyright © 2026 Jeremy H. Carroll, All Rights Reserved) that tests the repairability of cached database workloads under streaming CDC (Change Data Capture) updates.
Runtime boundary: use one serialized processor owner per source stream. Redis publication now watches and commits one cache value with its receipt, epoch metadata and default watermark; it is not a transaction-wide checkpoint/lease protocol or Redis Cluster implementation. Checkpoint identity and read-freshness obligations remain with the caller. See the PR 27 review and validation contract before treating the diagnostic engine as a production cache service.
An opt-in PostgreSQL 16 / Redis 7 profile adds a durable generation authority, aligned native WAL worker, authenticated reader grants, and a Redis Cluster mirror. Its release gates track the required evidence separately from the diagnostic API.
SmplCache checks explicit evidence before maintaining admitted SQL results. Implemented profiles include row-local predicates, conditional aggregates, supplied tenant context, distinct counts, top-k, bounded joins/CTEs/membership/set operations, and partition or whole-result reread boundaries for windows and recursion. The machine-checked coverage matrix distinguishes extraction, preservation, repair, invalidation and rejection, with named tests and profile limits. SQLite oracle campaigns, native database tests and mathematical models supply different, bounded evidence.
- Technical Wiki Home
- Acquirer Technical Due Diligence Data Room
- 15-Minute Adoption Quickstart
- 8-Level Pedagogical Curriculum
- SmplCache Live Architecture & Operations
- SQL Coverage and Evidence
- Current PgCache / Readyset Comparison
- Open Production Gates and Benchmark Protocol
SmplCache explores whether data systems over-invalidate because they route state changes through machinery that does not check repair evidence.
Relational theory normalizes data at rest (smplcache.normal_forms). SmplCache evaluates the repairability of data in motion within its declared SQL fragment (smplcache.smplcache, smplcache.join).
- No silent under-invalidation.
- No repair claim without evidence.
- No unsupported SQL feature quietly accepted.
Architecture choices for a repeated SQL result. These are patterns, not measured vendor benchmarks. Product behavior varies by query, configuration and deployment.
For concrete products, PgCache documents CDC invalidation with eventual consistency and same-connection read forwarding; Readyset distinguishes incrementally maintained deep caches from TTL-based shallow caches. See PgCache's consistency contract, Readyset's deep-cache mechanism and shallow-cache mechanism. The sourced comparison includes their scope and SmplCache's open production obligations. No performance ranking has been measured.
| Question | Application-managed cache-aside | Native materialized/indexed views | Streaming/incremental query systems | SmplCache today |
|---|---|---|---|---|
| Who decides when a result changes? | Application invalidation and/or TTL policy | Database refresh/maintenance mechanism | Supported query dataflow/replication mechanism | Dependency/evidence checks for admitted shapes |
| Can this result change incrementally? | Possible with application-specific logic | Depends on database and view type | Yes for supported queries/configurations | Yes for supported aggregates/joins when evidence and state suffice |
| What state is retained? | Keys/values and application metadata | Database-managed structures | System-managed state; limits/eviction depend on product | Output groups plus auxiliary rows, indexes, tombstones and frequency metadata; auxiliary row cap is not a total RAM cap |
| What happens if required change evidence is missing? | Integration must define its policy | Database-specific | Source/product-specific | Refuse repair and invalidate; integration must withhold/rebuild at a consistent boundary |
| Schema advice | Separate tools or application work | Database tooling | Product-specific tooling | Declared-FD normalization, chase checks and advisory DDL |
| Best evaluation | Measure your app policy | Measure write/refresh cost and SQL coverage | Measure supported-query freshness/resource behavior | Replay the real workload, compare with source SQL, test refusal/recovery and memory |
Source code is available for inspection under a restricted proprietary license. Commercial or internal-business evaluation and CLI execution require the appropriate written permission.
In a licensed evaluation checkout, from the repository root:
# 1. Install package in editable mode from licensed source checkout
python -m pip install -e ".[dev,demo]"
# 2. Run the workload advisor on the example schema
python -m smplcache.cli report examples/workload.common.json
# 3. Run SQL AST & FD-redundant GROUP BY doctor diagnostics
python -m smplcache.cli doctor examples/workload.common.json
# 4. Run 3NF/BCNF schema normalization, Tableau Chase check, and PostgreSQL/MSSQL DDL generation
python -m smplcache.cli normalize examples/ecommerce_before.json --dialect postgres --format markdown
# 5. Execute streaming in-place delta repair demo
python examples/01_python_orders_revenue_demo.py
# 6. Run multi-table join benchmark under Zipfian foreign-key skew
python -m smplcache.cli ingest --benchmark --transactions 10 --zipf-s 1.2
# 7. Run unit, property, CLI, substrate, and demo acceptance tests
python -m pytest
# 8. Launch SmplCache Live interactive web demo locally
uvicorn demo.api.app:app --host 0.0.0.0 --port 8000Note: SmplCache is not published as an open-source PyPI package. pip install smplcache is not supported. For adoption guidance and terminal output logs, refer to the Adoption Guide.
The core library supports Python 3.10+. The MongoDB driver in the optional
live extra requires Python 3.11+: on Python 3.10 that extra intentionally does
not install PyMongo. The supported MongoDB installation requires dnspython
2.9.0 or later, which fixes
GHSA-45c3-73f7-pv4m.
Use Python 3.11+ to run the full live-service suite. Already-installed older
drivers must be upgraded or removed separately; changing project metadata does
not uninstall packages from an existing environment.
Use the same explicit relation identity in the SQL, seed tables, and change events.
tenant_a.orders and tenant_b.orders are distinct; a qualified name cannot be
matched to bare orders without an upstream binding. The library does not infer a
SQL search path. Scope each engine instance to one source database.
Seed aggregates with AggregateCache.from_rows(shape, rows) or
AggregateCache.from_group_state(shape, group_stats) (via ColdRebuildSubstrate for
sqlite3/duckdb), and joins with rebuild_join_cache plus complete JoinAuxState
tables. A totals-only AggregateCache(values=...) is readable but cannot authorize
repair: totals alone omit the row/non-null counts needed for deletion and SQL NULL
semantics.
After invalidate or unsupported, callers of the low-level processing functions
must discard/rebuild the cached result before serving it or processing further
changes. WalStreamProcessor remembers invalidation until its rebuild callback
succeeds; join rebuilds must replace both the cache and auxiliary state. Callbacks
must rebuild at the processed committed boundary, not an arbitrary later snapshot.
SmplCache decomposes a cached query shape into its dependency boundary (predicates, aggregates, groups, projections, security columns).
When a CDC WriteEvent occurs, SmplCache:
- Checks whether the event intersects the shape boundary.
- Checks whether the event contains sufficient old/new row evidence to repair the cache (Theorem 7).
- Emits a
Decision(preserve,repair,invalidate,unsupported) backed by a machine-readableCertificate(includingboundary_clockLSN/sequence watermarks).
Certificate for an Invalidated Cache (Missing Evidence):
{
"shape": "inventory_stock_count",
"event_id": "evt_6",
"relation": "inventory",
"decision_kind": "invalidate",
"reason_code": "missing_evidence_for_repair",
"required_evidence": ["item_id", "quantity"],
"available_evidence": ["item_id", "quantity"]
}Certificate for a Repaired Cache:
{
"shape": "revenue_by_customer_paid",
"event_id": "evt_5",
"relation": "orders",
"decision_kind": "repair",
"reason_code": "group_move",
"required_evidence": ["amount", "customer_id", "status"],
"available_evidence": ["amount", "customer_id", "status"],
"repair_program": "sum_by_group"
}SmplCache organizes developer onboarding into 8 progressive cognitive levels:
- Level 0 (The Pattern Break): Exposing the failure modes of naive cache-aside.
-
Level 1 (Coordinate Anchoring): Anchoring queries as geometric coordinate regions in
$C^1$ . - Level 2 (Refusal Assurance): Enforcing Theorem 7: fail-closed refusal when CDC evidence is incomplete.
- Level 3 (In-Place Delta Repair): Applying incremental aggregate updates with sufficient group state.
-
Level 4 (Auxiliary Semi-Join Tracking): Tracking foreign-key fanout in
JoinAuxStatewith hash-indexed primary-key and secondary foreign-key tables. - Level 5 (Capacity Bounds & Skew Eviction): Enforcing hard row and tombstone limits with W-TinyLFU retention under Zipfian skew.
- Level 6 (Multi-Tenant Isolation): Exploring tenant-aware boundaries; production isolation is not established by these synthetic tests.
-
Level 7 (Systemic Workload Diagnosis & Schema Lift): Workload pathology analysis and 3NF/BCNF synthesis with
smplcache doctorandsmplcache normalize.
See 8-Level Curriculum for full details.
- Shape-driven preserve, repair, invalidate, and unsupported decisions for the admitted SQL fragment.
- Incremental
SUM,COUNT,COUNT(*), andAVGstate (plus bounded-windowMIN/MAXviaSortedMultisetExtremumCache), including group and predicate changes and external group-state seeding viaAggregateCache.from_group_stateandColdRebuildSubstrate. - Foreign-key join maintenance with hash-indexed primary-key and secondary foreign-key tables, W-TinyLFU auxiliary row limits (
max_total_rows), bounded tombstone tracking (max_tombstones), and explicit fallback after evidence eviction. - Relational normalization engine (
smplcache.normal_forms): functional dependency closure, minimal cover, candidate keys, 1NF/2NF/3NF/EKNF/BCNF classification, Bernstein 3NF synthesis, Tsou-Fischer BCNF decomposition, Tableau Chase losslessness verification with counterexample tuple generation, FD-redundantGROUP BYreduction, and PostgreSQL 16 / SQL Server 2022 DDL generation. - Decoded Debezium/wal2json event replay, transaction buffering, LSN/txId
boundary_clockpropagation, and live containerized verification against PostgreSQL 16, SQL Server 2022, and MongoDB 7. - Interactive SmplCache Live web demonstration (
demo/) with side-by-side table-invalidation baseline comparison, interactive Schema Normalizer & Doctor tab (/api/normalize), independent native SQLite verification, and reproducible decision traces.
The adapter
consumes decoded JSON fixtures; it is not a native PostgreSQL pgoutput binary
protocol client. Live replication, durable offsets, duplicate suppression, and
production concurrency remain integration work.
max_total_rows bounds resident auxiliary rows, not total process memory. Indexes,
eviction tombstones (bounded by max_tombstones), frequency history, output groups,
and transient work also use memory. Decimal sums retain finite input digits; displayed
AVG division follows the active Decimal context and is not an exact rational or a
cross-database rounding assurance under all dialects.
Join maintenance requires each right-side join key to be that table's primary
key. The default key name is id; provide the same accurate primary_keys
mapping to SQL extraction and auxiliary-state seeding when it differs. The
extractor does not inspect the database schema, and seeding rejects duplicate
or null primary keys.
Boolean and float group and join keys, float predicate values, and float SUM/AVG evidence are rejected rather than being interpreted using Python equality or Decimal arithmetic. Use integer or exact Decimal aggregate inputs; JSON float representations of database numeric values are not exact evidence. Database collation, numeric precision, and AVG rounding must still match the integration's declared semantics.
See the PR #7 review and the SmplCache Live architecture & operations runbook.
SmplCache tunes itself from its own receipts. Every change it proposes is
tested on your recorded traffic and shown to you before it takes effect.
Learned policies change only how much work SmplCache does: what to cache, how
to partition, what to evict, what to precompute. They never decide whether a
served value is correct; that stays with certify_event, and the replay
harness checks every served value against a SQL oracle (0 mismatches on every
reported run; the TTL baseline's stale serves are counted the same way).
Each behaviour is off unless its TuningFlags field is set, and each one
leaves hash-chained receipts (ReceiptLog) for proposal, evaluation,
authorization, application, failure and learning. Nothing self-applies:
a change reaches the store only after a supported replay verdict and an
approved authorization from a declared Policy or a human; code changes
are only ever proposed as pull requests. Results and verdicts are in
docs/EXPERIMENTS.md; the fault-injection table is in
docs/MUTATIONS.md.
| Flag | What it may change | Verdict | Limit |
|---|---|---|---|
spectral_partitioning |
how shapes are grouped into store partitions (only matters for stores that flush whole partitions) | drop | never beat per-relation grouping or the per-shape store on the benchmark; kept as code behind the flag for stores with coarse flush granularity |
entropy_drift_detector |
when a repartition is proposed (von Neumann entropy of the co-invalidation density matrix over a sliding window) | flag | detects the workload phase change; the proposals it triggers were refuted by replay; diagnostic only, threshold must be > 0 bits |
config_search |
which shapes are materialized (UCT search; greedy single-drop is the comparison) | flag | gain exists only under a declared memory cost per resident group; costs database reads; UCT does not beat greedy on the held-out segment |
bandit_admission |
which shape is evicted when a shape cap is full (Thompson sampling on hit-per-epoch) | drop | a prior learned on one phase lost to LRU/LFU on the next; flat Thompson ties LFU |
reread_suggestions |
nothing live: human-reviewed suggestions (REPLICA IDENTITY FULL, admit a shape) with counterfactual replay evidence | keep | every suggestion is needs_human; with min_max_window_k > 0 (SortedMultisetExtremumCache), the extremum suggestion is replayed and supported with 0 oracle mismatches (and underdetermined when min_max_window_k = 0) |
counterfactual_estimates |
nothing live: off-policy estimates of a capped admission policy from receipts | keep | exact only for policies that depend on reads and invalidations alone (LRU, LFU) |
The replay harness (smplcache.tuning.replay) is a synthetic day (6,000
writes, 2,000 reads, Zipf skew, one phase change, one table without
before-images, one MIN/MAX shape outside the fragment). Its numbers describe
that workload, not yours. p99 decision latency is wall-clock on a shared
machine and varied 0.6-2.7 ms between runs of identical code. The mistake
memory is exact-match and does not expire.
In accordance with the project Claim Hygiene Rule:
- What It Supports: Supports boundary-quotient certification, certified in-place arithmetic repair for admitted SPJA single-block aggregates, bounded-window
MIN/MAXmaintenance, capacity-bounded join tracking under Zipfian skew, and functional dependency 3NF/BCNF schema normalization checked by Tableau Chase. - What It Does Not Support: Does not claim support for arbitrary unconstrained SQL (e.g., recursive CTEs, window functions, non-deterministic expressions), does not eliminate physical memory bounds for unbounded cardinality, and confers no operational execution authority (
AuthorityEffect: None). - Narrowest Defensible Claim: Within its declared single-block SPJA equijoin SQL fragment and given complete before-and-after CDC row images, SmplCache identifies repairable state transitions, maintains aggregate sufficient statistics and, for joins, auxiliary row/index state, and deterministically falls back to certified invalidation whenever evidence is absent or capacity bounds are exceeded.
- Next Test Needed: Multi-process Redis store contention and network-fault injection across concurrent
SharedJoinAuxPoolworkers consuming partitioned WAL streams.
- Technical Wiki Home
- Acquirer Due Diligence Data Room
- Mathematical foundations:
docs/smplcache_theory_quotient_repairability.md - Dependency fingerprinting:
docs/dependency_fingerprinting.md - Invalidation rules:
docs/invalidation_rules.md - CDC Evidence, Store Application, and Kernel Contracts
Copyright © 2026 Jeremy H. Carroll (Anomalon). All Rights Reserved.
This repository is publicly visible on GitHub solely for technical inspection, continuous integration (CI) verification, and pre-acquisition due diligence. Making this repository publicly readable does not grant an open-source license. Commercial use, production deployment, internal business use, redistribution, modification, and derivative works are strictly prohibited without a written commercial license or IP acquisition agreement executed by Jeremy H. Carroll. See LICENSE for full legal terms.