Most AI projects stop at the demo. I build the layer that makes them usable: evaluation harnesses, evidence trails, human review loops, and interfaces for engineering and operations.
I care about systems that are measurable, auditable, and useful — not just impressive in a notebook.
|
Agent benchmarks, calibration, inter-rater agreement, regression gates, drift. |
RAG over engineering documents, document extraction, SOP compliance, supplier risk. |
|
Excel and report pipelines, data cleaning, API reliability, log analysis. |
Evidence graphs, launch-risk twins, procurement exceptions, production scheduling. |
Sources → Retrieve + claims → Evidence → Human review → Decision
The goal is not just an answer. It is an answer that can be traced, challenged, and improved.
|
Continuous agent qualification.
Versioned benchmarks, isolated trials, pass@k, McNemar, and drift signals.
|
Bug-repair evaluation.
Isolated grading, anti-tampering checks, and 12 reproducible repos.
|
|
Evidence-first manufacturing audits.
Quote-level verification, RBAC, and conflict detection.
|
Quality regression and drift.
Retrieval, answers, citations, latency, and cost tracked together.
|
|
Decision intelligence for launches.
CPM, CP-SAT, Monte Carlo, and paired scenario comparison.
|
Trace decisions to evidence.
Typed evidence spans, conflict surfacing, and auditable gates.
|
New here? Start with AgentEvalOps for evaluation, NPI Launch Risk Twin for decision intelligence, or Smart File Organizer for practical automation.
- Measure the claim. If it cannot be evaluated, it is still a demo.
- Show the evidence. Every decision should trace back to a source.
- Keep humans in the loop. Automation handles scale; people handle judgment.
- Make it local-first. Reproducible, privacy-aware, and usable offline.
| Year | Theme | Representative work |
|---|---|---|
| 2023 | Python, automation, first AI experiments | Formula solver, Excel reports, API monitor, prompt lab |
| 2024 | Applied AI, LLMs, ML | RAG assistant, formula validator, defect detection, supplier risk |
| 2025 | Agents, optimization, benchmarking | Tool-calling agent, RAG drift monitor, SOP audits, CP-SAT scheduling |
| 2026 | Advanced benchmarks and agent systems | Mini SWE-Bench, multimodal verification, evidence graphs, AgentEvalOps |
$ whoami
sanjay mouraly
$ focus
evaluation-first ai systems · agents · applied automation
$ building
AgentEvalOps — continuous qualification for AI agents
The best way to reach me is here on GitHub — open an issue or start a discussion on any repository.
36 public repositories · 2023–2026 · built with Python, Markdown, and a bias toward evidence.