Skip to content

Home

Jump to bottom
Harish edited this page Aug 16, 2026 · 1 revision

Welcome to the Wiki_textgraph wiki!

TextGraph

Turn a pile of case documents into a queryable, byte-cited knowledge graph — deterministic & local-first by default. Explore it in an interactive graph studio, derive new relations with link prediction & Datalog rules, and layer opt-in LLM extraction, grounded answer synthesis, and semantic search on top (always GENERATED-quarantined, never masquerading as ground truth).

TextGraph helps investigators make sense of financial-crime and technical-crime evidence: filings, contracts, wire-transfer logs, SARs, memos, chat/email exports, reports. It ingests that corpus and emits a structured, versioned graph that shows who is connected to whom, through what, and — crucially — why — with every edge carrying the exact source span that supports it, so a finding can be re-verified and stands up to audit.

It is the natural-language successor to llm-wiki: where llm-wiki gave an agent cited, streaming answers, TextGraph generalizes that to any textual corpus and produces a graph an agent can traverse.

Why it's different (vs "GraphRAG")

Most GraphRAG tools build the graph with an LLM: extraction is non-deterministic, edges arrive without verifiable sources, and you can't reproduce or audit the result. TextGraph inverts that — the edge is trust, not just recall:

  • 🔁 Deterministic by construction — the same corpus always produces a byte-identical graph.json, gated in CI. Diff two runs, reproduce any finding.
  • 🔍 Byte-level provenance on every edge — each non-generated claim carries the exact [doc:start-end] span and re-hashes against source bytes; 100% re-verification is gated.
  • 🚫 Zero LLM calls by default — ingest → IE → resolution → claims → analytics → retrieval runs locally, CPU-only, no API key. LLMs are opt-in, quarantined, and GENERATED-tagged.
  • 🕓 Bi-temporal versioning — claims carry [t_valid, t_invalid) windows; a later fact invalidates an earlier one (cited SUPERSEDES) rather than overwriting it.
  • 📏 Honest about quality — publishes the hallucinated-edge rate, a number most systems don't report.

Works out of the box (zero config)

pip install textgraph-kg → textgraph build ./docs already does the investigator happy path — no flags, no API key, no server:

  • 📄 PDFs ingest by default (pypdf is a core dep); .docx/.html/.epub/logs/chat too.
  • 🔗 Entity resolution on by default — Acme Corporation / Acme Corp / ACME collapse to one canonical identity via cited SAME_AS.
  • 🔎 Hybrid retrieval by default — BM25 + Personalized PageRank + RRF, every hit cited.
  • 🧠 LLM & embeddings opt-in — --llm-extract, --narrate, --embed, always GENERATED-quarantined.

Architecture — a bottom-up layer stack

Each layer is a pure function of the layer below it plus a pinned config hash — the property that makes determinism (G1) and incrementality (G5) achievable.

flowchart TD
    IN["📄 Case corpus<br/>PDF · DOCX · ODT · RTF · HTML · EPUB · JSON/YAML · logs · chat"]
    subgraph SPINE["🟢 Structural spine (zero LLM, deterministic)"]
        L0["L0 · Ingest & Normalize<br/>CanonicalDoc = UTF-8 + offset map + block tree + chunks"]
        L1["L1 · Deterministic Structure<br/>sections, links, definitions, citations, Rationale & Requirement"]
        L0 --> L1
    end
    subgraph SEM["⚪ Extraction (deterministic core + opt-in enrichers)"]
        L2["L2 · coref · temporal · negation"]
        L3["L3 · Encoder IE · entities + typed relations (EXTRACTED)"]
        XLLM["🧠 --llm-extract · chunk → LLM → triples (GENERATED)"]
        XCO["🔗 --co-occurrence · co-mentioned entities (STRUCTURAL)"]
        L2 --> L3 --> XLLM
        L3 --> XCO
    end
    subgraph GRAPH["⚪ Resolution & graph model (deterministic)"]
        L5["L5 · Entity resolution · SAME_AS lattice, non-destructive"]
        L6["L6 · Claim reification · reified Claims + t_valid + invalidation"]
    end
    subgraph RETR["🟢 Analytics & retrieval (zero LLM, deterministic)"]
        L7["L7 · PageRank · communities · layout · bridges · contradictions"]
        L8["L8 · Retrieval · dual-node graph · BM25 + PPR + RRF"]
        L7 --> L8
    end
    L4["🧠 L4 · --llm synthesis (opt-in) · GENERATED summaries"]
    L9["📦 L9 · Artifacts + MCP · graph.json · graph.html · GRAPH_REPORT.md"]
    IN --> L0
    L1 --> L2
    L3 --> L5
    XLLM -.-> L5
    XCO -.-> L5
    L5 --> L6 --> L7
    L7 -.-> L4 -.-> L9
    L8 --> L9
Loading

The deterministic core — L0 → L1 → L2/L3 → L5 → L6 → L7 → L8 — never calls a model. The LLM appears at exactly two opt-in, GENERATED-quarantined touchpoints: relation extraction (input enricher, runs before resolution/analytics) and synthesis/narration (output pass). Turn both off and the build is byte-identical.

Confidence stratification

Every edge is tagged STRUCTURAL, EXTRACTED, INFERRED, or GENERATED. Merging node identity never launders an edge's tag — a model-authored relation between two EXTRACTED entities stays GENERATED and filterable (one-click in the console, tag: "GENERATED" in graph.json).

Quickstart

pip install textgraph-kg

# Deterministic, offline build:
textgraph build ./case-files -o textgraph-out
# → graph.json, GRAPH_REPORT.md, graph.html, schema.yaml, manifest.json

# Query the graph directly — bounded, byte-cited answers, no LLM:
textgraph query   ./case-files "who transferred funds to whom"
textgraph path    ./case-files "Acme Corp" "Gamma Holdings"
textgraph explain ./case-files "Acme Corp"
textgraph timeline ./case-files "Acme Corp"
textgraph contradictions ./case-files

# Interactive graph console (canvas viewer + all eight tools):
textgraph console ./case-files          # → http://127.0.0.1:8765

# Standard GQL (ISO/IEC 39075 / Cypher subset):
textgraph gql ./case-files "MATCH (a:Organization)-[:CONTROLS*1..3]->(b) RETURN a.name, b.name"

Python API — build once, then call the eight typed, cited tools:

from textgraph.pipeline import build
from textgraph.l8_retrieval import QueryEngine

result = build("./case-files")  # deterministic; byte-identical graph.json
engine = QueryEngine(result.nodes, result.edges)
hits = engine.search("who transferred funds to whom", k=5)
engine.path("Acme Corp", "Gamma Holdings")   # max-likelihood cited path
engine.why("Acme Corp")                        # cited claims + validity windows
engine.conflicts()                             # single-truth conflicts, surfaced not merged

Design goals (non-negotiable)

G1 determinism · G2 local-first · G3 provenance (no assertion without a re-verifiable byte-range citation) · G4 confidence stratification · G5 incrementality · G6 agent-legible output (bounded, ranked, cited context — never raw Cypher) · G7 bounded, auditable cost.

Status

The full L0–L9 stack is shipped (deterministic ingest → IE → resolution → bi-temporal claims → analytics → hybrid retrieval → optional LLM) with CLI, MCP, and a dependency-free local web console. Recent releases add openCypher export → Neo4j, forward-chaining Datalog rules, structural link prediction, GQL, vision-native retrieval, fine-grained access control (ReBAC + ABAC), and Graph-of-Thoughts reasoning. See CHANGELOG.md and PLAN.md.

Learn more

  • ARCHITECTURE.md, docs/how-it-works.md, docs/retrieval-routing.md
  • BENCHMARKS.md — recall@k / MRR with tokens-per-query and latency ("no number without its cost")
  • docs/COMPARISON_SEMANTICA.md — honest comparison with the closest peer
  • docs/RUNNING.md — step-by-step getting-started guide

Clone this wiki locally