selective-classification

A list of papers that studies out-of-distribution (OOD) detection and misclassification detection (MisD)

PyTorch implementation of our ECCV 2022 paper "Rethinking Confidence Calibration for Failure Prediction"

A decision-safety lab for loan approval: trains a baseline classifier, calibrates probabilities (ECE/Brier), sweeps confidence thresholds to build a coverage, quality frontier and outputs a defensible abstention policy (auto-decide vs review). Includes a Streamlit dashboard for report cards, triage UI, and data quality checks.

Longform article reframing abstention (reject option / selective prediction) as product design, not model weakness. Covers coverage as a KPI, calibration as a prerequisite, threshold selection under review capacity and risk, queue/UX design for human-in-the-loop workflows, and anti-patterns that break safety in production.

Decision-safe evaluation + Streamlit dashboard for AI vs Human vs Post-Edited AI text detection. Generates a reliability report card (Accuracy, Macro F1, ECE, Brier), calibration plots, confidence histograms, and a coverage-vs-performance abstention curve. Recommends an operating threshold for human-review routing.

GenPark AI Agent Skill - Distribution-free conformal prediction sets with rigorous marginal coverage guarantees, calibrated non-conformity scores, and split conformal inference.

GenPark AI Agent Skill - Decomposes total prediction uncertainty into epistemic (model knowledge void) vs aleatoric (inherent task ambiguity) components via Shannon mutual information.

Promises and Pitfalls of Threshold-based Auto-labeling (NeurIPS 2023, Spotlight)

GenPark AI Agent Skill - Post-processing confidence calibration using temperature scaling, expected calibration error (ECE), and Brier score evaluation.

GenPark AI Agent Skill - Risk-coverage trade-off optimization; controls agent abstention, fallback escalation, and human-in-the-loop triggers under dynamic risk budgets.

GenPark AI Agent Skill - Risk-coverage trade-off optimization; controls agent abstention, fallback escalation, and human-in-the-loop triggers under dynamic risk budgets.

GenPark AI Agent Skill - Distribution-free conformal prediction sets with rigorous marginal coverage guarantees, calibrated non-conformity scores, and split conformal inference.

GenPark AI Agent Skill - Detects LLM hallucinations and confabulations by clustering sampled responses into semantic equivalence classes and computing semantic entropy.

GenPark AI Agent Skill - Decomposes total prediction uncertainty into epistemic (model knowledge void) vs aleatoric (inherent task ambiguity) components via Shannon mutual information.

GenPark AI Agent Skill - Post-processing confidence calibration using temperature scaling, expected calibration error (ECE), and Brier score evaluation.

GenPark AI Agent Skill - Detects LLM hallucinations and confabulations by clustering sampled responses into semantic equivalence classes and computing semantic entropy.

Code for "Budgeted Classification with Rejection: An Evolutionary Method with Multiple Objectives", at IEEE CEC 2022

Robust Selective Classification of Skin Lesions with Asymmetric Costs

Reliable and Efficient Selective Machine Learning

More human oversight can make an AI agent less safe. Headroom is a human-in-the-loop firewall for coding agents that measures when to trust the human: oversight as resource allocation, not just classification.