-wsearch

wsearch is a self-hosted web research tool. It provides search discovery across several sources, reading of web pages as markdown, news and Hacker News lookups, a staged research command, multi-phrasing search, image search and image fetching, and a street-route tool. It is served as a command-line program and as a Model Context Protocol (MCP) server over stdio and over HTTP.

The tool is built for agents that need dated, attributable evidence. Results carry their engine, source and publication date where it can be determined, and failures are reported with a reason instead of an empty success.

Components

• wsearch.py — the command-line tool and the search, read and research logic.
• wsearch_tools.py — the single tool layer shared by both MCP transports: tool list, argument schemas, child-process execution, time caps, error mapping and credential redaction.
• mcp_stdio.py — MCP server over stdio.
• mcp_http.py — MCP server over streamable HTTP (standard library only), with GET /healthz.
• escalation.py — the research page-selection and sufficiency rules.
• newspool.py and news_feeds.tsv — a local news pool built from RSS feeds.
• recency.py and recency_terms.tsv — time decay for dated results.
• bsearch.py — an optional browser leg for the msearch tool.
• bin/wsearch — a launcher that runs wsearch.py in the locked environment.
• ref_data/ and scripts/search_result_contract.py — the evidence contract for search results (see ref_data/search-result.v1.md).

Requirements

• Python 3.12.

• The locked environment in pyproject.toml and uv.lock. Create it with:

  uv sync --frozen --python 3.12

• For the browser-based features, a Chromium build for Playwright (playwright install chromium inside the environment) and a pbrowse command that provides render and script subcommands with the interface described in the Browser section. Without it, those features report themselves as unavailable.

• A SearXNG instance that you run yourself, if you want the aggregated search leg. Without one, search uses the DuckDuckGo lite leg and the other configured sources.

Quick start

export WSEARCH_VENV="$PWD/.venv"             # the locked environment
export WSEARCH_SEARXNG="http://127.0.0.1:8080"  # your SearXNG instance, or "none"
bin/wsearch deps                              # check dependencies
bin/wsearch search "example query"
bin/wsearch read "https://example.org/"

bin/wsearch uses $WSEARCH_VENV/bin/python when the variable is set and python3 otherwise. Run bin/wsearch deps first; it lists every dependency with ok, missing, degraded or optional and a repair hint.

Command line

wsearch search  "<query>"              web discovery (SearXNG and DuckDuckGo lite, merged by URL)
wsearch research "<topic>" [--no-read] staged research with page selection and sufficiency checks
wsearch news    "<query>"              local news pool first, then live Google News results
wsearch news-pool refresh|stats|search  maintain and query the local news pool
wsearch hn      "<query>"              Hacker News via the Algolia API
wsearch read    "<url>" [n]            page as markdown; --find "<regex>" returns matching lines only
wsearch read-m  "<u1>" "<u2>" ...     several pages
wsearch msearch "<question>" --queries '<json>' [--local '["xx"]']
                                       one question in several phrasings, fused by rank
wsearch images  "<query>" [--min-side N]
wsearch image-fetch "<img-url>|-" [--page "<page>"] [--hint "<name>"] [--out f]
wsearch page-images "<page>" [--find "<text>"] [--out f.jpg]
wsearch place-photos "<place>" [--city "<city>"]
wsearch route   "<lon,lat;lon,lat>" [--profile foot|car|bike]
wsearch deps    [--json]
wsearch health  [--json]
wsearch --version

Each command has a total time budget (search and news 45 s, read 90 s, research 300 s); WSEARCH_BUDGET overrides it. A command that runs out of time returns partial results with a [部分结果] marker.

Exit codes

Code	Meaning
0	Success, including zero results and tagged fallback results.
65	Request refused (for example a loopback, private, link-local or CGNAT target in read).
69	Backend unavailable: every source failed. stderr names the backend and the reason.
78	Missing required dependency or configuration. stderr lists the item and the fix.

MCP servers

Both transports expose the same tools. The tool list is web_search, news, hn, read_url, image_search, image_fetch, page_images, place_photos, route, msearch, research.

Stdio:

{
  "mcpServers": {
    "wsearch": {
      "command": "/path/to/wsearch/.venv/bin/python",
      "args": ["/path/to/wsearch/mcp_stdio.py"]
    }
  }
}

Use the Python interpreter of the locked environment so that the browser and crawler subprocesses see the same packages.

HTTP (mcp_http.py, standard library only): POST /mcp accepts JSON-RPC requests, and GET /healthz returns a cached engine probe, capacity and call counters without secrets. Bind to loopback or to a specific interface address. Wildcard binds (0.0.0.0, ::) are refused. Set a bearer token file for any non-loopback bind.

Configuration

All settings are environment variables. None is required for a basic local run, but the values shown are examples and must be set to match your own machine.

Variable	Example	Purpose
WSEARCH_VENV	/opt/wsearch/.venv	The locked Python environment for the CLI and browser subprocesses.
WSEARCH_SEARXNG	http://127.0.0.1:8080	Your SearXNG base URL, or none to disable this leg.
WSEARCH_PROXY_FILE	~/.config/wsearch/proxy.env	Mode-0600 env-format file that may define WSEARCH_DDG_PROXY for the DuckDuckGo leg. Credentials stay in this file.
WSEARCH_PROXY / WSEARCH_PROXY_LIST	http://192.0.2.10:3128	Additional exit candidates; none disables them.
WSEARCH_PROXY_FALLBACK	http://127.0.0.1:18080	Opt-in local proxy, used only when no other proxy is configured. It is probed by TCP first. Unset means no local proxy is used.
WSEARCH_JINA_KEY_FILE	~/.config/wsearch/jina.key	Mode-0600 file holding a Jina API key. Without it the Jina fallbacks make no request.
WSEARCH_YOUTUBE_CLI	/usr/local/bin/yt-cli	Executable that implements yt_video <json> for YouTube reads. Unset means YouTube pages are read as ordinary pages.
PBROWSE_BIN	/usr/local/bin/pbrowse	Path to the pbrowse command. Default: found on PATH.
WSEARCH_BUDGET	60	Overrides the per-command time budget in seconds.
WSEARCH_UPSTREAM_RATE_PER_MIN / WSEARCH_UPSTREAM_BURST	12 / 3	Token bucket for SearXNG queries, shared by all local processes. 0 or less disables it.
WSEARCH_UPSTREAM_STATE	~/.local/state/wsearch/upstream-bucket	State file for the bucket (mode 0600).
WSEARCH_DEPS_TTL	60	Cache time in seconds for passing dependency checks. 0 disables the cache.
WSEARCH_PREFLIGHT	0	Skips the startup dependency preflight (troubleshooting only).
WSEARCH_EGRESS_PEERS	peer=198.51.100.7:8932	Optional peer wsearch used as a second egress when a local call fails.
WSEARCH_EGRESS_TOKEN_FILE	~/.config/wsearch/peer.token	Bearer token file for the peer.
WSEARCH_MCP_BIND	127.0.0.1	Addresses the HTTP server binds to (comma list).
WSEARCH_MCP_PORT	8932	HTTP server port.
WSEARCH_MCP_TOKEN_FILE	~/.config/wsearch/mcp.token	Optional bearer token for the HTTP server (mode 0600).
WSEARCH_MCP_CONCURRENCY	3	Concurrent tool calls on this host.
WSEARCH_MCP_ALLOW_ORIGIN	https://app.example	Browser origins allowed to call the HTTP server.

Proxy and key files must be mode 0600, or a symbolic link to such a file. Credentials are redacted from every output, label and error message.

Sources and behaviour

• Search: SearXNG aggregation (when configured) in parallel with a DuckDuckGo lite leg; hits are merged by URL and show every engine that returned them. Bing appears only as a member of the SearXNG aggregation.
• Freshness: general searches add a last-month pass when the rate bucket has a token free. Dated results are ranked by time decay; undated results carry a lower weight and are never presented as dated.
• Reading: read and read-m check every redirect hop against the SSRF rules before a request is sent. A page that cannot be rendered falls back to the exit-group retry, then Jina (with a key), then the newest Wayback snapshot, each labelled with its source.
• Research: research ranks candidates, reads the top two, and reports resolved or unresolved with the evidence it used. Decisions are logged as JSON lines under $XDG_STATE_HOME/wsearch/ for calibration.
• News: a local pool of RSS feeds (news_feeds.tsv) with conditional requests and retry backoff, plus live Google News results.
• Images: candidates are fetched and their pixel sizes read from the file header. Usable images meet min_side (default 1080). HEIC and HEIF originals are transcoded with pillow-heif when that optional package is installed.
• Routes: OSRM-based routing with a Valhalla fallback, following the public services' usage rules (an identifying User-Agent, one request at a time, and Retry-After honoured).
• Reddit: post and subreddit reads come from the Arctic Shift archive, which is paced by its documented limits. Posts younger than about 36 hours are labelled because their scores are not yet updated.

SSRF and safety rules

• read, read-m, the image tools and the browser legs refuse loopback, private, link-local, CGNAT (100.64.0.0/10), unique-local IPv6 and non-http(s) targets.
• Redirects are followed hop by hop with the same check at every hop. A downgrade from https to http is refused.
• Output never contains proxy credentials, tokens or keys.

Browser (optional external command: pbrowse)

pbrowse is an optional browser-rendering command. It is not shipped with wsearch: install it separately, then point PBROWSE_BIN at it or put it on PATH. wsearch never imports its code; the two sides only exchange the command line, stdin and stdout described here.

Without pbrowse, the browser features report themselves as unavailable and wsearch uses its other sources. Run wsearch deps to see whether it is found.

Interface wsearch expects

1. Version check: pbrowse --version prints a line that starts with pbrowse followed by the version.
2. Render one page: pbrowse render <url> --net public --wait dom --settle 0.5 --timeout <seconds> --user-agent <ua> --persona wsearch, plus --owner-pid <pid> when a resident service owns the browser, and --egress-stdin when a proxy is used.
   • stdin: {"egress": "<proxy URL>"} when --egress-stdin is given, otherwise empty. A proxy URL can carry credentials, so it is never placed in argv.
   • stdout: the last line is one JSON object. Success: {"ok": true, "url": "...", "status": 200, "html": "..."}. Failure: {"ok": false, "error": "..."}. wsearch reads url, status (default 200; 400 and above count as a failed page) and html. Other fields are ignored. A missing or unparseable last line is reported as a failure.
3. Run a browser job: pbrowse script --net public --timeout <seconds> --persona wsearch, plus --owner-pid <pid> as above.
   • stdin: {"script": "<job script>", "argv": [...]}.
   • stdout: the last line is one JSON object. Success: {"ok": true, "out": "<line>"}, where out starts with OK , FAIL , SMALL or SRC . Failure: {"ok": false, "error": "..."}. Any other out value is reported as an unrecognised reply.

--net public means the browser tool must refuse non-public destinations on every connection and redirect hop. The browser tool owns its own memory limits, pool and network policy.

What degrades without it

• read and read-m cannot render pages, so they use the other sources: Jina (only with a key) and then the newest Wayback snapshot, each labelled with its source.
• image_fetch still fetches a direct image URL. Its screenshot fallback, page_images and place_photos are unavailable.
• The msearch browser leg stays off (it is also off unless WSEARCH_BROWSER=1 is set).

Tests

Tests run offline in the locked environment:

uv run --python 3.12 --frozen python -m unittest discover -s tests
uv run --python 3.12 --frozen python scripts/search_result_contract.py --check

The tests use temporary directories and mock network access. Some tests read this checkout's own source files. wsearch --version takes its revision from git; outside a git checkout it reports unknown and exits with status 1, which is what the symlink test sees when it runs from an exported copy.

Evidence contract

Search results follow the contract in ref_data/search-result.v1.md, with the machine-readable form in ref_data/search-result.v1.json and the manifest in ref_data/contracts.manifest.json. scripts/search_result_contract.py --check verifies that the generated sections match.

Status

Interfaces may change between versions; see CHANGELOG.md.

Related projects

These four projects were built together and can be used together. Each one also runs on its own.

Project	What it does	How it connects
-agents-msger	Messages between agent sessions on different machines. Sender identity is detected from the process tree.	Its hub lists which sessions are alive on each host (/who, /status).
-d-browser	A browser pool for agents. One Chromium per caller, verified by the kernel, closed when the caller exits. Provides the pbrowse command.	For a call forwarded from another machine, it asks the agents-msger hub whether that caller is still alive and closes its Chromium when it is gone.
-wsearch	Search service. The calling model writes the query in several languages; fixed code runs the searches and merges the results.	Its browser features (page rendering, scripts) use d-browser's pbrowse.
-verdict	Decision recall. Before an agent decides, a hook gives it similar earlier decisions and the general design rules. Recall is done by code (BM25).	No dependency on the others.

Using them together, in this order:

1. agents-msger: run one hub, and one shim on every machine (see the agents-msger README, Install). Its installer writes ~/.config/mcp-msgers/config.json, which holds the hub address and the token file.
2. d-browser: install it on the machines that run browsers. It reads that same agents-msger config to find the hub; nothing else needs to be set. Its pbrowse command is installed with it.
3. wsearch: install it, and point PBROWSE_BIN at d-browser's pbrowse (or put pbrowse on PATH).
4. verdict: install the hook in Claude Code (see the verdict README, Hook setup). It needs none of the others.

Without pbrowse (see Browser above), wsearch still works and reports its browser features as unavailable.

License

MIT. See LICENSE.