TorchNode Ethereum node crawler

TorchNode discovers Ethereum nodes, inspects their public services, and stores the results in SQLite.

Discovery uses a provider boundary with independent discv4 and discv5 providers. Cryptographic node identities are separate from endpoint observations, whose source, provenance and timestamps are retained in SQLite. See the discovery architecture decision.

Completed deep inspections and background API scans are now retained as measurement occurrences, with bounded identity history in Deep Inspection. Repeated identical evidence shares stored payloads while each attempt keeps its own time and outcome. Existing node rows remain the latest projection; history records factual observations. A separate, bounded Deep Inspection section shows conservative derived comparisons between compatible evidence; the observations remain authoritative. See the historical observations ADR and change detection ADR.

TorchNode acquires and validates real ENRs over discv4 and discv5, preserving raw records, sequence, unknown fields and advertised endpoints independently of discovery and P2P evidence. Trusted IPv4 and IPv6 endpoints support active discovery and inspection. See ENR support and Active IPv6 support.

discv5 uses the pinned geth discovery engine through a separate helper. Build it before scanning with both providers:

cd p2p-helper
GOCACHE=/private/tmp/torchnode-go-build go build -o ../target/torchnode-discovery-helper ./cmd/discovery
cd ..

Set TORCHNODE_DISCV5_HELPER to override its path and TORCHNODE_DISCV5_BOOTSTRAPS to an ENR configuration file. Missing or failed helpers leave discv4 available. Observatory counts cryptographic identities; endpoint observations remain preserved. See discv5 architecture. Both endpoint families retain their provenance under one cryptographic identity. IPv6 discovery, TCP/RLPx, and HTTP transport have deterministic loopback coverage; public IPv6 success depends on observer connectivity. See the IPv6 validation report.

Build and run

mvn package
java -jar target/torchnode-1.0-SNAPSHOT-jar-with-dependencies.jar

Open http://localhost:8080. Scanning, stopping, inspection, filtering, statistics, and CSV export are available in the dashboard.

The main Dashboard v2 shows bounded observation coverage, a local country-level endpoint-context map, client evidence, discovery/address-family views, protocol attempts, independent RPC/Beacon probes, and derived changes. Its 24-hour, 7-day and 30-day historical windows are separate from the latest-projection node table. No peer addresses are sent to a map service. See Dashboard v2.

Deep Inspection includes bounded, expandable discovery, ENR, enrichment, run, and derived-change history. The Network Measurement Report prints a selected Network Analytics window with units, denominators, unknowns, and methodology; browser Print can save it as a PDF. Reports are live views of stored evidence, not immutable snapshots. See History UX and reports.

/analytics provides a bounded UTC measurement window over recorded observations, with explicit counting units, denominators, and unknown evidence. /analytics.json exposes the same internal report; /analytics/snapshot.json is a separate latest-projection view. These observer-local counts are not Ethereum population estimates. See Network Analytics.

Set TORCHNODE_PORT or TORCHNODE_DB to override the default port and database path.

Node Inspect measures discovery, P2P TCP, RLPx Auth, devp2p Hello, ETH Status, JSON-RPC and Beacon API independently. The optional RLPx inspector is an isolated Go helper using pinned go-ethereum; the Java dashboard still builds and runs without it. See the architecture decision.

Enable authenticated P2P inspection

From the repository root, build the Java jar and the companion helper with Go 1.25 or newer (or Go's automatic toolchain download):

mvn package
(cd p2p-helper && GOTOOLCHAIN=auto go build -o ../target/torchnode-p2p-helper .)
java -jar target/torchnode-1.0-SNAPSHOT-jar-with-dependencies.jar

The application discovers an executable torchnode-p2p-helper beside its jar (or in target/ / p2p-helper/ when running from the repository root). mvn package alone does not build the Go helper. For an installed layout or an explicit override, set TORCHNODE_P2P_HELPER before starting Java:

export TORCHNODE_P2P_HELPER="$PWD/target/torchnode-p2p-helper"

An explicit path may be absolute or relative to the process working directory; if it is wrong or not executable, TorchNode does not silently fall back to another binary. Without a runnable helper, TCP is still tested while Auth, Hello and ETH Status remain NOT_TESTED. The helper connects only to the discovered node's advertised TCP port and authenticates its discovered secp256k1 node ID. UDP and TCP ports are never assumed equal.

ETH Status requires a real local chain context. Set TORCHNODE_P2P_STATUS_RPC_URL to an operator-trusted execution node's HTTP(S) JSON-RPC endpoint (prefer loopback), for example only if you run and trust a synced local execution client:

export TORCHNODE_P2P_STATUS_RPC_URL=http://127.0.0.1:8545

Do not point this variable at the node being inspected: its RPC is not independent local Status context. TorchNode never fills the variable from a discovered peer's RPC endpoint. If you have no trusted local execution RPC, leave the variable unset; real Auth and Hello can still run, with ETH Status NOT_TESTED. The helper makes only read-only chain, network and block calls. It verifies canonical genesis and a current head; currently Mainnet, Sepolia and Hoodi are supported. Without valid context, Auth and Hello can still pass, but ETH Status remains NOT_TESTED with a reason. The helper supports devp2p 4/5 Hello and negotiates the highest common ETH version among 69–72. It does not advertise SNAP or serve chain data; it only records a peer's advertised snap/1 capability. ETH/69–72 Status provides network ID, execution genesis hash, fork ID, and the available full-block range (earliest, latest, latestHash), but no total difficulty. Authenticated Hello and Status observations are retained in the p2p_observations latest SQLite projection and in completed inspection history. The pinned go-ethereum release exposes ETH/69–72, not ETH/68. An ETH/68-only peer can complete Auth and Hello, but ETH Status remains NOT_TESTED with NO_COMPATIBLE_ETH_CAPABILITY; TorchNode does not fabricate an ETH/68 Status. The local Hello is devp2p/5 with client ID TorchNode Observatory/1.0, the pinned geth ETH capabilities, listen port 0 (no inbound listener), and the ephemeral authenticated secp256k1 public key. Its RLP fields and encrypted message code 0 are exercised against a local geth p2p.Server in Go tests. For failed Hello exchanges, diagnostics distinguish decoded Disconnect reason codes, clean EOF, TCP reset, timeout, malformed frame and invalid MAC when the transport actually exposes those conditions. A decoded frame count of zero does not establish whether the peer sent an incomplete encrypted frame.

Default helper deadlines are TCP 2 s, Auth 3 s, Hello 2 s, Status 3 s and 12 s total. They can be bounded via TORCHNODE_P2P_TCP_TIMEOUT_MS, TORCHNODE_P2P_AUTH_TIMEOUT_MS, TORCHNODE_P2P_HELLO_TIMEOUT_MS, TORCHNODE_P2P_STATUS_TIMEOUT_MS, and TORCHNODE_P2P_TOTAL_TIMEOUT_MS. Java permits at most four concurrent helper processes and kills a child after 15 s or when its inspection task is interrupted.

PASS means that specific protocol exchange completed; TCP reachability does not imply Auth/Hello/Status success. A single network value is OBSERVED; independent agreement is MATCH, disagreement is MISMATCH, and only consistent independent evidence makes a network VERIFIED. A failed P2P interface does not invalidate independently reachable RPC or Beacon services. Local deterministic RLPx peers and a pinned geth p2p.Server exercise Auth, Hello, Status, malformed input, timeouts and cancellation with go test ./...; no public peer is required. Geth, Reth, Nethermind, Besu and Erigon binaries are not bundled, so live multi-client interoperability must be run separately before claiming coverage. The Status latestHash is the hash of the latest available full block, not necessarily the canonical chain head. A first non-fixture bidirectional ETH/72 Status exchange was captured on Sepolia on 2026-09-25 using a verified local light-client context and a separate unmodified geth 1.17.6 peer. The peer was still at genesis, and Reth, Nethermind and Besu Status interoperability remains unevaluated. The P2P milestone therefore remains open. See the inspection ADR for the evidence and limits.

Optional offline network enrichment

Deep Inspection and CSV expose address-scoped country and ASN evidence for IPv4 and IPv6. They never assign one location/provider to a cryptographic identity. Hosting classification is NOT_AVAILABLE: ASN organization is not a hosting, cloud or residential classification. Endpoint/NAT conclusions are unchanged.

Obtain GeoLite2 Country and GeoLite2 ASN MMDB files separately under MaxMind's download/setup terms and GeoLite EULA. Configure the local files before starting TorchNode:

export TORCHNODE_GEOIP_COUNTRY_DB=/path/to/GeoLite2-Country.mmdb
export TORCHNODE_GEOIP_ASN_DB=/path/to/GeoLite2-ASN.mmdb

GeoIP2 Country files are also accepted. TorchNode does not download datasets or accept/download license keys. Keep databases and credentials outside the repo; follow the dataset terms, including updates and attribution. This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com when those optional files are configured. The Java MMDB reader is Apache 2.0; the datasets have separate terms.

Without either file, that lookup reports DATASET_UNAVAILABLE; Ethereum observation/inspection continues. Private, local, documentation and other excluded special-purpose addresses report NOT_APPLICABLE. Missing records and runtime failures remain distinct. Country is approximate IP-network location, not a peer's physical location; no city/region precision is claimed.

Lookups are local, asynchronous and bounded. Rendering/export only read stored evidence; no observed IP is sent to a geolocation service. Results retain separate country/ASN source, SHA-256/build version, prefix and lookup timestamp. Replace datasets and restart the scanner/application to change the active version. Same-version stored results (including misses/failures) are reused; there are no automatic retries or background dataset updates. Prior-version records remain evidence, and UI/export show the latest known lookup, explicitly separate from the endpoint observation time. See the ADR and validation.