A self-evolving librarian agent for document collections.
Important
Looking for the original AutoRAG (RAG AutoML / pipeline optimization tool)?
This repository now hosts AutoRAG 2.0, a complete reimagining of AutoRAG as a self-evolving librarian agent. The original Python-based AutoRAG — the RAG AutoML tool for automatically finding an optimal RAG pipeline for your data — now lives in the legacy/ directory of this repository.
The legacy AutoRAG is NOT abandoned. It continues to be maintained (bug fixes, dependency updates, and PyPI releases via pip install AutoRAG) in maintenance mode. Existing users can keep using it exactly as before — see the legacy README for its documentation, and file issues in this repository as usual. New feature development is focused on AutoRAG 2.0.
AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units. No raw grep dumps. Just answers.
AutoRAG is a customized Pi agent — the Pi agent loop configured into a librarian. The AutoRAG librarian retrieves candidates, reads source files directly, judges the evidence, and curates the structured answer. Its model and provider come from the user's authenticated runtime; AutoRAG does not ship a private provider default.
AutoRAG itself is the specialized search agent, not a coordinator for other model roles. You configure one model, and that model owns the complete retrieval, reading, judgment, and curation loop.
Every search tool gives you the same thing: a list of file paths and matching lines. Then you have to:
- Open each file
- Read the surrounding context
- Decide what's relevant
- Synthesize an answer
- Remember what worked for next time
That's the human doing all the hard work. The tool just points.
AutoRAG is not a search tool. It's a librarian — it searches, reads, thinks, and reports back:
You ask: "What were the key findings in the Q3 report?"
AutoRAG:
[1] Revenue grew 23% YoY to $4.2M, driven by enterprise contracts. (pages 3-5)
[2] Three new risk factors: supply chain, regulatory, talent retention. (pages 12-14)
[3] Headcount target missed by 12 — engineering hiring bottleneck. (page 8)
No file paths. No line numbers. Just curated knowledge you can act on.
AutoRAG has a self-evolving memory system. Every search teaches it something:
- Which retrieval methods work for which types of queries
- Which document areas are most productive
- What the caller found useful (via explicit feedback)
A fresh AutoRAG tries everything. A seasoned one knows exactly where to look. This is not a static configuration — it's learned behavior from real usage.
Different documents need different search strategies:
| Your documents | Best method | Why |
|---|---|---|
| Plain text, config files | grep (pattern matching) | Fast, precise, literal |
| Research papers, dense prose | Vector search (semantic) | Understands meaning, not just keywords |
| Legal documents, specifications | BM25 (keyword ranking) | Handles domain terminology well |
| Mixed collections | Hybrid (vector + BM25) | Combines precision and recall |
AutoRAG supports pluggable retrieval methods. Local lexical BM25, semantic vector, and hybrid retrieval all go through MinSync over one shared CDC chunk lifecycle, wired through the RetrievalMethodRegistry. The librarian invokes retrieval tools, reads the underlying documents directly through bash, and curates one unified result set after ResultMerger score normalization and deduplication. External datasources keep their own archive/index lifecycle.
BM25, vector, and hybrid are enabled by default whenever MinSync is enabled. Disable local indexing with "minSync": false, or disable only lexical search with "bm25": false. MinSync uses a pre-installed binary (autoInstall: false); configure minSync.embedder via autorag init --embedder-* flags for remote embedding endpoints. AutoRAG never forces TEI or any external embedding service.
AutoRAG reads configured source directories directly through its built-in bash tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured SearchDocumentsResponse; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. MinSync indexes parsed markdown mirrors under .autorag for BM25, vector, and hybrid retrieval.
AutoRAG can opt into Jikji as a local CLI-backed find-first discovery and indexing layer. Jikji is optional: AutoRAG does not vendor it, install it, or register it as a retrieval backend when enabled.
When Jikji is configured, AutoRAG calls jikji find ROOT "query" --json via the jikji_find tool. The tool parses and validates the upstream answer pack and exposes its handoff_action, tool_call_policy, and agent_should_not_rerank to the librarian. Direct file reading remains available for source verification. prepare/refresh remain for indexing only and do not answer queries directly.
Programmatic use:
const agent = new AutoRAGAgent({
searchPaths: ["/path/to/documents"],
jikji: { binaryPath: "jikji" },
});
await agent.prepareJikji();AutoRAG can use the external dupey CLI
to detect exact, near, and containment document families.
autorag duplicates /path/to/documents
autorag duplicates --jsonThe command is read-only: it reports exact duplicate groups and review
guidance, but never moves or deletes source files. The scan_duplicate_documents
Agent tool exposes the same read-only scan to the orchestrator.
Exact duplicate exclusion is enabled by default during parsed-mirror refresh.
For each exact canonical-text hash, the newest filesystem copy is indexed and
older copies are omitted. Disable it in config.json when both copies must be
searchable:
{
"dupey": { "enabled": true },
"excludeExactDuplicates": false
}If dupey is not installed or fails, refresh continues without exclusion and
reports no destructive action; install it with cargo install dupey.
The same .autorag/jikji.json shape configures Jikji when present:
{
"enabled": true,
"binaryPath": "jikji",
"timeoutMs": 10000,
"maxBufferBytes": 1048576,
"includeHidden": false,
"includeSensitive": false,
"maxFiles": 0,
"writeAgentRules": false,
"enableMediaIndex": false,
"exclude": []
}Call agent.prepareJikji() (or agent.refresh()) to prepare configured source roots. Hidden files, sensitive files, and media indexing are disabled by default; AutoRAG does not pass --include-hidden, --include-sensitive, or --enable-media-index unless the corresponding option is true. AutoRAG-managed prepare runs with --no-agent-rules by default, so it never rewrites the consumer repo's AGENTS.md/CLAUDE.md/.cursorrules; an explicit writeAgentRules: true opt-in re-enables upstream routing-block injection. AutoRAG passes --enable-media-index only when enableMediaIndex: true.
The upstream Rust PrepareArgs defines reference defaults that AutoRAG does not override unless explicitly configured: parse timeout 5.0, max hash bytes 512 MiB, doc text max chars 2,000,000, doc text chunk chars 1,000,000, and media index max MB 25.0. AutoRAG emits --parse-timeout, --max-hash-bytes, --doc-text-max-chars, --doc-text-chunk-chars, and --media-index-max-mb only when the matching option is set, so the upstream defaults apply otherwise. AutoRAG answers queries through jikji find (find-first) plus the Pi agent loop and its registered retrieval methods; prepare/refresh are indexing-only.
Datasource skills let AutoRAG search external, server-configured data sources through the same retrieval pipeline as local documents. A skill describes what it indexes, how it should be refreshed, what source instances exist, and which permission tags/scopes bound access. Retrieval still flows through RetrievalMethodRegistry → ParallelRetriever → datasource result filtering → ResultMerger; datasource skills do not create a parallel search path.
Every datasource can be registered multiple times through a connection alias:
use the config key as the unique name and set type to the reusable backend
(gmail, github, slack, discord, kakao, cloud-drive, and so on).
Each alias becomes an independently loadable agent skill with its own source
scope and workspace namespace. Chat aliases search all channels by default;
trusted channels.ids / channels.names allowlists can expose a particular
channel or group chat as its own datasource.
Security defaults are intentionally strict:
- datasource access is default-deny unless trusted server/API configuration supplies
datasourceAccess.allowedTagsanddatasourceAccess.allowedScopes; - model/tool arguments never grant datasource tags or scopes;
search_datasource_documentsaccepts only{ query, topK?, scope? }, andscopecan only narrow trusted access.
| Datasource | Skill | Connects via | Notes |
|---|---|---|---|
| KakaoTalk | katok |
external katok CLI |
first datasource skill; AutoRAG never reads KakaoTalk databases directly |
whatsapp |
external wacrawl CLI |
local-first incremental archive + FTS5 search; live desktop ingestion is macOS-only | |
| Telegram | telegram |
external telecrawl CLI |
local-first archive + FTS5 search; live desktop ingestion is macOS-only |
| Slack | slack |
external slacrawl CLI |
local-first workspace/channel/thread archive + FTS5 search |
| Discord | discord |
external discrawl CLI |
guild/channel/thread/DM archive; FTS5 + semantic + hybrid retrieval, incremental sync |
| Notion | notion |
external notcrawl CLI |
local-first page/database/block archive + FTS5 search |
| GitHub Issues/PRs | github |
GitHub REST (token optional) | issues + PR bodies per owner/repo; public repos work unauthenticated |
| Cloud drives | cloud-drive |
rclone CLI |
Incremental Google Drive Tier-1; OneDrive/network remotes; iCloud experimental |
| Gmail / IMAP | gmail |
Gmail REST v1, or himalaya CLI (backend: "himalaya") |
the himalaya backend indexes any IMAP/Maildir account it has configured — no OAuth plumbing |
| Local mail exports | mail-export |
filesystem (.mbox / .eml) |
classic From_ splitting, mailparser-based; count-only warnings |
| Obsidian vault | obsidian |
external qmd CLI |
incremental qmd update, BM25 qmd search, semantic qmd vsearch; vault path via connector.vaultPath |
| RSS / news | rss |
HTTP feed polling | RSS 2.0 + Atom, feed/category hierarchy, 24h dedupe window |
Connector-backed skills fetch documents into AutoRAG's local chunk store. External-crawler skills such as KakaoTalk, WhatsApp, Telegram, Slack, and Notion leave incremental archive and FTS ownership with their CLI and map query results into the same retrieval pipeline. Obsidian uses the external qmd CLI (incremental update + BM25 + semantic). Tokens are referenced by environment variable name only, never stored in config. Process/API failures surface as path/PII-opaque diagnostics. See docs/manual-qa-datasources.md for the QA harnesses.
Configure them in config.json (CLI) or pass datasourceSkills programmatically:
Install wacrawl with brew install openclaw/tap/wacrawl. AutoRAG invokes wacrawl sync during datasource refresh and wacrawl --json --sync never search during retrieval. Optional trusted connector fields are binaryPath, databasePath, and sourcePath. The child process receives only a restricted environment; unrelated model/provider secrets are not forwarded. Live WhatsApp Desktop discovery requires macOS and the permissions documented by wacrawl, while an existing portable archive can be queried on other supported platforms.
Install telecrawl with brew install openclaw/tap/telecrawl. AutoRAG invokes telecrawl import during datasource refresh and telecrawl --json search during retrieval. It uses the same optional trusted connector fields and restricted child environment as wacrawl. Live Telegram Desktop discovery requires macOS and the permissions documented by telecrawl, while an existing portable archive can be queried on other supported platforms.
Install slacrawl with brew install openclaw/tap/slacrawl. AutoRAG invokes slacrawl sync during datasource refresh and slacrawl --json search during retrieval. Optional trusted connector fields are binaryPath, configPath, and syncSource. Slack credentials and source definitions remain in slacrawl's own configuration rather than AutoRAG.
Install notcrawl with brew install openclaw/tap/notcrawl. AutoRAG invokes notcrawl sync during datasource refresh and notcrawl search --json during retrieval. Optional trusted connector fields are binaryPath and configPath. Notion credentials and workspace definitions remain in notcrawl's own configuration rather than AutoRAG.
Install and authenticate rclone separately (brew install rclone && rclone config on macOS), then configure the provider-neutral cloud-drive skill.
cloud-drive is a reusable type: each datasource config key is a connection
alias and becomes a separate agent skill and scope. Multiple Google accounts,
or Google Drive plus OneDrive/iCloud, can therefore be loaded and searched
independently.
AutoRAG inventories with rclone lsjson, keeps a workspace-local manifest and
managed mirror, and downloads only added or changed indexable files. Google
Drive is Tier-1. OneDrive, Dropbox, SMB/SFTP/WebDAV, and mounted drives share
the same manifest contract. iCloud Drive is experimental because rclone marks
that backend Tier 4 and Apple ID/2FA sessions periodically require
reauthentication. See docs/datasource-skills.md
for filtering, size, concurrency, bandwidth, dry-run, and agent tool-calling
details.
KakaoTalk was the first datasource skill. It uses the external katok CLI only — AutoRAG never reads KakaoTalk databases directly. katok failures return diagnostics instead of throwing, and remote embedding egress configuration is rejected before the CLI is spawned.
import { AutoRAGAgent, KatokSkill } from "@autorag/librarian";
const kakao = new KatokSkill({
instanceId: "personal",
tags: ["kakaotalk", "personal", "pii"],
// Optional: client: new KatokClient({ binaryPath: "katok" })
});
const agent = new AutoRAGAgent({
searchPaths: ["/path/to/documents"],
datasourceSkills: [kakao],
datasourceAccess: {
allowedTags: ["kakaotalk"],
allowedScopes: ["/kakao/personal/**"],
},
});
await agent.refresh(); // refreshes parsed mirrors, BM25/MinSync, and datasource indexes
const results = await agent.searchDatasourceDocuments("meeting with Mina", { topK: 5 });Discord uses the external discrawl CLI, which owns the SQLite archive, the FTS5 index, and the message vectors. AutoRAG never calls the Discord API itself.
brew install openclaw/tap/discrawlTwo archive sources are supported. wiretap (the default) reads the local Discord Desktop cache and needs no token at all; discord uses a bot token, which is the ToS-sanctioned automation path. AutoRAG refuses to spawn the CLI when a Discord user token is present in the environment — automating a user account violates Discord's Community Guidelines and can get the account terminated.
import { AutoRAGAgent, DiscrawlClient, DiscrawlSkill } from "@autorag/librarian";
const discord = new DiscrawlSkill({
client: new DiscrawlClient({ source: "wiretap", root: process.cwd() }),
instanceId: "community",
});
const agent = new AutoRAGAgent({
searchPaths: ["/path/to/documents"],
datasourceSkills: [discord],
datasourceAccess: {
allowedTags: ["discord"],
allowedScopes: ["/discord/community/**"],
},
});Or through the trusted config factory:
{
"datasources": {
"discord": {
"instanceId": "community",
"connector": {
"source": "wiretap",
"embeddingProvider": "ollama",
"embeddingModel": "embeddinggemma",
"defaultMode": "hybrid"
}
}
}
}Two defaults are deliberate and worth keeping:
defaultMode: "hybrid"— discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See #1413.embeddingProvider: "ollama"+embeddingModel: "embeddinggemma"— semantic search requires an embedding provider (ollama serve && ollama pull embeddinggemma). For workspace-managed discrawl state, AutoRAG writes these values to.autorag/datasources/discrawl/config.tomlwhile preserving unrelated discrawl settings. An explicitconnector.configPathremains operator-owned and is never rewritten. EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources share one local embedder. Do not usenomic-embed-text: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See #1414.
A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as /kakao/personal/chunks/<chunk-id>, and permission tags that match your server-side access policy.
AutoRAG is built for non-code document retrieval: manuals, legal docs, internal wikis, meeting notes, research literature, knowledge bases, PDFs.
Code repositories work too (direct grep is useful), but AutoRAG's real value shows on unstructured text where simple pattern matching isn't enough.
The default home state is kept outside the workspace:
~/.autorag/
├── config.json
├── memory.json
└── logs/
└── runs.jsonl
config.json selects sources, the workspace, memory path, retrieval settings, and the agent model. Provider and model IDs must refer to a model available in the user's authenticated runtime:
{
"searchPaths": ["/path/to/documents"],
"workspacePath": "/path/to/workspace",
"memoryPath": "/Users/you/.autorag/memory.json",
"model": { "provider": "provider-name", "id": "reasoning-model" }
}autorag init leaves model unset when no model flags are supplied. At search time AutoRAG resolves an authenticated local provider when possible; otherwise configure the model explicitly.
For fast interactive search, prefer a model with reliable tool calling, high output TPS, and low first-token latency. A query can require several short model turns while AutoRAG alternates between retrieval tools and direct source reading, so model throughput has a visible effect on end-to-end response time. It does not accelerate BM25, MinSync, Jikji, filesystem access, or indexing itself. Larger reasoning models remain useful for difficult synthesis, conflicting evidence, and specialized domain judgment, but they are not a requirement for ordinary retrieval.
Config path precedence is --config > AUTORAG_CONFIG > ~/.autorag/config.json. When the home config is absent and <cwd>/autorag.config.json exists, AutoRAG copies the legacy file to ~/.autorag/config.json without deleting or modifying the legacy file. The legacy cwd file is a migration source, not the default location.
memory.json stores retrieval memory and logs/runs.jsonl records run events. Model authentication remains with the user's configured provider or authenticated local runtime. Corpus indexes remain workspace-local: refresh keeps parsed mirrors and BM25/MinSync indexes under <workspace>/.autorag.
autorag refresh and autorag index reset|rebuild accept --method <csv> (e.g. --method bm25,minsync,parsed) to scope which indexing methods run or which index directories are removed. When omitted, all methods run. autorag init accepts --embedder-* flags to configure the MinSync embedder endpoint in the config file.
autorag health checks model/provider auth before a search — it resolves the model, verifies credential presence, and optionally probes one completion call. Use it to diagnose model, provider, auth, or timeout failures. autorag status remains the model-free index-health command (corpus freshness and BM25/MinSync readiness). When autorag search fails for a model/provider reason, the error output includes a hint pointing to autorag health.
Published as @autorag/librarian (dist bundled with Bun, runtime Node ≥ 24 or Bun):
bun add @autorag/librarian # library
bun install -g @autorag/librarian # autorag CLI
# or run directly from the repo:
bun add github:NomaDamas/AutoRAG-2.0Git-based installs build dist/ via the prepare script and require Bun on the installing machine.
External tool binaries auto-install on first use into <workspace>/.autorag/bin: MinSync downloads a verified GitHub release asset (on by default; minSync.autoInstall: false to opt out), and Jikji compiles the jikji-cli crate via cargo (requires the Rust toolchain; jikji.autoInstall: false to opt out). New autorag init configs enable Jikji find-first discovery by default. KakaoTalk (katok) and Discord (discrawl) stay manual, optional installs. All of them degrade gracefully when missing — core BM25 search works without any of them.
import { AutoRAGAgent } from "@autorag/librarian";
const agent = new AutoRAGAgent({
searchPaths: ["/path/to/documents"],
});
const response = await agent.searchDocuments("summarize the compliance requirements");
console.log(response.answer);
for (const result of response.results) {
console.log(`[${result.number}] ${result.title} — ${result.summary}`);
}
// Mark which results were useful — AutoRAG remembers for next time
agent.recordFeedbackByNumbers(response.sessionId, [1, 3], [2]);searchDocuments() runs the Pi agent loop — it searches, reads, consults memory, curates, and finalizes through the emit_autorag_results structured tool — then returns a typed SearchDocumentsResponse. The caller consumes the structured payload directly; no assistant text parsing.
You ask a question
│
▼
┌──────────────┐
│ Plan + search│ ← check_memory, Jikji, and retrieval tools
└──────┬───────┘
▼
┌──────────────┐
│ Direct read │ ← bash find/grep/cat
└──────┬──────┘
▼
┌─────────────┐
│ Curate │ ← Extract key insights, not raw lines
└──────┬──────┘
▼
[1] Finding A — summary (location)
[2] Finding B — summary (location)
│
Caller says "1 was useful, 2 wasn't"
│
▼
┌─────────────┐
│ Memory │ ← Remembers what worked, adapts next time
└─────────────┘
MIT
{ "datasources": { "whatsapp": { "instanceId": "personal", "connector": { "binaryPath": "wacrawl" } }, "telegram": { "instanceId": "personal", "connector": { "binaryPath": "telecrawl" } }, "slack": { "connector": { "binaryPath": "slacrawl", "configPath": "/path/to/slacrawl.yaml", "syncSource": "primary" } }, "notion": { "connector": { "binaryPath": "notcrawl", "configPath": "/path/to/notcrawl.yaml" } }, "github": { "connector": { "repos": ["owner/repo"] } }, "gmail": { "connector": { "backend": "himalaya", "account": "gmail", "folder": "INBOX" } }, "personal-google-drive": { "type": "cloud-drive", "instanceId": "personal", "connector": { "provider": "google-drive", "remote": "personal-gdrive:", "include": ["**/*.md"] } }, "company-onedrive": { "type": "cloud-drive", "instanceId": "work", "connector": { "provider": "onedrive", "remote": "company-onedrive:Documents" } }, "obsidian": { "connector": { "vaultPath": "/path/to/vault" } }, "rss": { "connector": { "feeds": [{ "url": "https://example.com/feed.xml" }] } } }, "datasourceAccess": { "allowedTags": ["whatsapp", "telegram", "slack", "notion", "github", "gmail", "cloud-drive", "obsidian", "rss"], "allowedScopes": ["/whatsapp/**", "/telegram/**", "/slack/**", "/notion/**", "/github/**", "/gmail/**", "/personal-google-drive/**", "/company-onedrive/**", "/obsidian/**", "/rss/**"] } }