Check financial AI outputs with an open protocol, 40 control requirements and 24 synthetic conformance cases. Local demo, no API key.
A local-first MCP host with pluggable skills, multi-provider routing, and multi-channel I/O. Kotlin. Plus a Python eval harness with rule + LLM-judge scorers over a human-authored golden set.
Production-grade evaluation framework for structured LLM systems: golden sets, agreement metrics, confidence intervals and regression gates.
Evaluation integrity layer for RAG/LLM golden sets: content-addressed suites that refuse unsound comparisons, negative controls that must go red, human-calibrated item analysis, and a release gate that refuses thresholds it cannot resolve.
Agent evaluation harness: golden set regression + paired bootstrap CI + 16 scorers + 30 cases across 6 failure modes. v0.2.0 baseline: deterministic 30/30, flaky 5/30, 5 scorers regressed at 95% CI.
Build a deduped, clustered, representative LLM eval golden set from production interaction logs (JSONL). Offline, deterministic, no model/network/API key; emits a goldrun-ready case file.
Retrieval regression gate for RAG systems: paired per-query comparison against labelled relevance, an exact sign test, and an UNDERPOWERED exit code when the golden set cannot support a verdict. The naive one-point mean rule it replaces fired on 84.5% of resampled identical-system trials; the gate held 2.5% against a 5% alpha.
Baptisms: San Antonio Baptism Records 1788-1824, 1858-1898 Build and development artifact cache
Evaluation pipeline for LLM outputs: an anchored rubric, a hand-labeled golden set, an LLM judge checked for agreement against it, and preference data generation for fine-tuning.
Golden-set evaluation for a quoting agent: deterministic checks and a release gate.
Build, adjudicate, freeze and audit golden sets for LLM evaluation: stratified sampling with reported shortfalls, inter-annotator agreement, adjudication that refuses to guess, tamper-evident freezing, drift detection. Dependency-free.
Minimal Python script to evaluate your RAG pipeline against a golden set.
Evaluation dataset quality auditor for LLM and RAG applications. Checks golden sets for conflicting labels, duplicate prompts, weak reference answers, ambiguous questions, over-easy examples, and category coverage gaps.
To associate your repository with the golden-set topic, visit your repo's landing page and select "manage topics."