I build AI systems for legal work, and the measurement that decides whether a model can be trusted with it.
Most AI writing reports the systems that work. In legal practice the work that matters is the distance between a model that sounds authoritative and a model that is correct. That gap is where the risk sits, and it has to be measured rather than assumed.
🟢 published with results, 🟡 working prototype, ⚪ built but not yet measured
| Project | What it is | Where it stands |
|---|---|---|
| auslawexam‑bench | A reproducible benchmark of Australian legal reasoning for LLMs. Its headline metric is how often a model invents a citation, which is the exposure a law firm actually carries. | 🟡 16 provisional questions and 87 tests. The whole pipeline runs offline on a deterministic mock, and each commercial model slot switches on with its API key. |
| Contract‑clause‑classifier | A zero-shot LLM against a fine-tuned transformer on CUAD contract clauses, compared on precision, recall, F1, cost, and latency. It is framed as a build-or-buy decision. | ⚪ Runs end to end on CUAD with a test model, with 28 tests. No comparison run has been published yet. |
| legislation‑monitor | Change detection over Commonwealth and Victorian legislation. It fetches each Act on request, hashes the text, diffs it against the last version, and shows the amendment, so finding a change no longer means rereading the Act. | 🟡 Runs locally against both official registers, with 22 tests. |
| legal‑rag‑evaluation | Hybrid retrieval over Australian statute, with a local OpenAI-compatible backend and an evaluation that says plainly what it has and has not run. | 🟡 One two-Act run is committed with its provenance. A hand-written routing probe put the right Act first in 7 of 8 queries. |
| local‑llm‑inference‑notes | Fifteen measurement investigations from one workstation, eight of them negative. The method the rest of the work leans on. | 🟢 Published, with the written method and the five tools that enforce it. |
| Capability | In practice | Where to look |
|---|---|---|
| Evaluation before deployment | Measurement that says whether a model is safe to rely on, beyond whether it sounds right. I declare noise bands before the run, build benchmarks that reproduce, and count invented citations. | auslawexam‑bench, local‑llm‑inference‑notes |
| Grounded retrieval for legal text | Retrieval over statute that keeps each section whole and pairs vector search with keyword matching, so that an answer traces to the provision behind it. A citation that matches no known authority is counted as invented. | legal‑rag‑evaluation, auslawexam‑bench |
| Provenance a regulated firm can audit | Content-hashed questions, append-only runs, contamination canaries, and every score kept next to the raw output it came from. | auslawexam‑bench |
| Full-stack delivery | The data pipeline and the model, then the FastAPI backend and the React and TypeScript interface a practitioner uses. | legislation‑monitor, Contract‑clause‑classifier, auslawexam‑bench |
I declare noise bands and success criteria before a run. A band chosen after seeing the numbers is a rationalisation. The rules are written down in METHOD.md for the inference notes and METHODOLOGY.md for the benchmark.
Negative results get published. Eight of the fifteen inference investigations end in no, and several of them overturned a claim the repository had already made.
Every benchmark, prompt, and published result is a commit, so any number in a README traces back to the code and the run that produced it.