Skip to content
View aidless's full-sized avatar
2 followers · 0 following
  • Shandong, China

Achievements

Achievement: Pull SharkAchievement: Quickdraw

Achievements

Achievement: Pull SharkAchievement: Quickdraw

Block or report aidless

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
aidless/README.md

Liu Zewen (刘泽文)

LLM Evaluation & Reliability. I build a three-layer LLM evaluation pipeline — execution sandbox → verdict gate → rubric — that decides whether a model actually improved, and I publish the evidence when it refuses to pass. Start with eval-sandbox: one command, no API key, ~5 seconds.

📧 17353895263@163.com · China · open to relocate · available immediately

One chain, three repos

Execution → Verdict → Criteria. Each repo is independently clonable, runs offline, and fails loudly instead of passing quietly.

Layer Repository The question it answers Measured now
Execution eval-sandbox Did the task actually run isolated — network, mounts, memory, CPU, PIDs? SUMMARY: 10/10 PASS, 43 passed
Verdict eval-gate Did this change make the model worse? 4/4 verdicts as expected, 70 tests
Criteria rubric-support-qa Is the scoring rubric itself right, before anyone scores with it? 300 dev + 800 frozen items hash-verified, 114 tests 
# Execution — also needs bubblewrap + a writable cgroup v2 subtree; missing either and it refuses to run
git clone --depth 1 https://github.com/aidless/eval-sandbox && cd eval-sandbox
python3 -m eval_sandbox.cli check

# Verdict — quickstart needs no dataset and no API key
git clone --depth 1 https://github.com/aidless/eval-gate && cd eval-gate
python3 -m venv .venv && .venv/bin/pip install . && .venv/bin/eval-gate quickstart

# Criteria — hash-checks the frozen sets, then lints rubric v0
git clone --depth 1 https://github.com/aidless/rubric-support-qa && cd rubric-support-qa
python3 scripts/freeze.py --check && python3 scripts/freeze_test.py --check && python3 rubric/lint.py

The refusal is the product: eval-sandbox will not report a PASS when the isolation primitives are unavailable.

Supporting layer

  • trace-replay — replays recorded agent tool-call traces offline; byte-identical reports across repeated runs (sha256 verified), 111 tests, CI green on 3.10 / 3.11 / 3.12.
  • auditable-llm-eval — is the reported number fake-green? Naive checker 100/100 green vs authoritative 0.69 / 0.67; 88 tests.
  • mcp-guard — offline MCP tool-definition poisoning scan, with hash pinning against rug pulls.
  • agent-reliability — joint verdict over the tool surface and the output surface; 60 passed, 16 subtests passed.
  • llmtriage — one-command triage of a broken inference deployment; 57 passed.
  • tmaudit — TMLR self-audit CLI on PyPI; 264 passed, 3 skipped.
  • arx — refuses to rank a leaderboard whose labels cannot resolve it; 371 passed, 3 skipped after pip install -e ..
  • agent-redteam — OWASP-aligned red-team harness for LLM agents.

What I don't claim

No peer-reviewed acceptance. No external users yet. No live demo. eval-sandbox, eval-gate and rubric-support-qa are not on PyPI. rubric-support-qa has no model run and no inter-annotator κ yet — its README says so instead of quoting a number. Where I could not reproduce a figure, I removed it rather than hedge.

The longer lines — preprints, agent-evaluation instruments, and the portfolio MCP server — are in AGENT_EVAL.md.

Pinned Loading

  1. arx arx Public

    An autonomous research loop refusing to write any claim its statistical gate cannot resolve, logging every refusal in a verifiable ledger

    Python

  2. auditable-llm-eval auditable-llm-eval Public

    LLM evaluation pipeline where every number is auditable: verifier separate from scorer, programmatic reference checks, documented rollback

    Python

  3. flowgrid-aml-retriever flowgrid-aml-retriever Public

    Deterministic evidence-retrieval entry for the Agent Memory Leaderboard: Add and Search only, no answer generation

    Python

  4. tmaudit tmaudit Public

    TMLR self-audit CLI: C1–C10 rulepack over your paper directory, shipped as a zero-dependency pyz

    Python

  5. wse-bench-public wse-bench-public Public

    Web-search evaluation benchmark for self-evolving LLM loops: 156 tasks, an append-only result ledger, and paired significance testing

    Python

  6. agent-redteam agent-redteam Public

    Red-team harness for LLM agents: multi-attacker evaluation with an OWASP-aligned playbook and a documented fake-version-pin incident

    Python