Repository navigation
Bugfix/tmosley/network scanner fixes 2 - #66
Merged
Merged
Conversation
recordConnectionAttempts persisted an observation and a candidate update per (attempt x candidate-at-that-address), each in its own transaction. That is fine against the data the feature was designed for -- a validator identity plus a couple of DNS-resolved children -- but migration 1785360000000 backfilled one candidate per node ever seen, and recycled hosting IPs accumulated thousands of dead identities each. One address carried 31,894 candidates. A pubnet crawl reports ~1400 attempts, so the inner loop ran ~48,000 times and issued ~96,000 serial round trips, taking roughly fifteen minutes. Production scans went from a steady 5.0 minutes to 20+ the hour that migration ran. The loop now only reads and mutates in memory; the writes happen afterwards in bulk. Observations are append-only, so they go through insert() rather than save(), which would first SELECT to decide insert-vs-update. Candidates go through a single save() so TypeORM loads the existing rows in one SELECT and writes inside one transaction. A gossiped address that fails validation now skips that attempt instead of throwing; previously one bad address cost the whole crawl's diagnostics, since the caller only logs the error. Pruning the backfilled rows is what fixes the current numbers (23.6 -> 6.7 min in production). This keeps a future backfill, or a genuinely busy address, from recreating the collapse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjRxs5zPEq6Q7GatFf17VQ
SatoshiPay stopped validating in June, but their three archive hostnames no longer resolve and every scan logged three ENOTFOUND failures against them. Read as archive defects, they are really just a departed operator. getHistoryArchiveUrls now requires isActive() alongside a historyUrl. The nodes themselves stay in the trust graph deliberately. InactiveNodesArchiver declines to archive a node still trusted by active nodes -- 42 active validators still list each SatoshiPay node in their quorum sets -- so archival is working as designed and will not retire them until those operators update. The archive poller is the right place to draw the line; the trust graph still needs the nodes to exist. One trade-off: a node unreachable during a single crawl drops out of archive polling for that scan, so its archive status goes stale rather than refreshing. Stale-but-known beats a reachability error misattributed to the archive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjRxs5zPEq6Q7GatFf17VQ
An archive Radar cannot read was reported the same way as one that is genuinely lagging: isFullValidator went false and the UI said the archive was behind. MoneyGram's three archives are the clearest case -- their CloudFront WAF returns 403 to the production egress IP (159.223.128.81), verified from inside the network-scanner container, while the identical request with the identical User-Agent returns 200 from elsewhere. Nothing is behind; we are blocked. Same for the SatoshiPay hostnames that no longer resolve and the LOBSTR archives returning 410. historyArchiveUnreachable and historyArchiveCacheMaxAge are now carried on the node DTO, and the warning detector reports unreachable ahead of the cache-misconfigured and behind branches. Both fields are optional in NodeV1Schema so older payloads still validate. This only changes how the condition is displayed. A WAF block is not something Radar can fix in code; getting the egress IP allowlisted is the actual remedy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjRxs5zPEq6Q7GatFf17VQ
undici reports connection-level failures as a bare "fetch failed" and hides the reason on error.cause. Production logged the fallback to the Rust scanner with no way to tell a refused connection from a DNS failure from a timeout. Flatten the cause chain into the message, with each link's code where it has one, so the log says what actually happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjRxs5zPEq6Q7GatFf17VQ
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
DNS refresh races, incorrect archive-check handling, and retained cache entries remain unresolved.
Review effort: Balanced
Findings: 4
Open (4)
What changed in this PR
This PR aims to bring network scans back below five minutes while preserving validator measurements and improving archive diagnostics.
Changes:
- Batch endpoint persistence, parallelize network work, cache TOML responses, and suppress recently failed gossip addresses.
- Preserve crawl validation evidence during endpoint probes and distinguish unreachable archives.
- Improve scan timing logs and Python analysis handling.
| File | Description |
|---|---|
| python-fbas-service/app.py | Run synchronous handlers outside the event loop. |
| packages/shared/test/node.test.ts | Update serialized-node expectations. |
| packages/shared/src/node.ts | Expose archive reachability status. |
| packages/shared/src/network.ts | Distinguish archive warning types. |
| packages/shared/src/dto/node-v1.ts | Extend the node response schema. |
| packages/crawler/src/crawler.ts | Exclude suppressed gossip addresses. |
| packages/crawler/src/crawl.ts | Store suppressed addresses. |
| packages/crawler/src/crawl-factory.ts | Pass suppression into crawls. |
| packages/crawler/src/__tests__/crawler.test.ts | Test gossip suppression. |
| apps/frontend/src/services/NodeWarningDetector.ts | Display archive reachability warnings. |
| apps/backend/src/network-scan/mappers/NodeV1DTOMapper.ts | Map archive reachability. |
| apps/backend/src/network-scan/mappers/__tests__/NodeV1DTOMapper.test.ts | Update mapping expectations. |
| apps/backend/src/network-scan/infrastructure/database/repositories/TypeOrmEndpointCandidateRepository.ts | Add bulk persistence operations. |
| apps/backend/src/network-scan/domain/Scanner.ts | Log scan-stage durations. |
| apps/backend/src/network-scan/domain/node/scan/NodeScannerTomlStep.ts | Merge probes without replacing crawl measurements. |
| apps/backend/src/network-scan/domain/node/scan/NodeScannerCrawlStep.ts | Prepare suppression and log crawl timings. |
| apps/backend/src/network-scan/domain/node/scan/NodeScan.ts | Merge probes and filter archive checks. |
| apps/backend/src/network-scan/domain/node/scan/node-crawl/PeerNodeToNodeMapper.ts | Separate identity updates from measurements. |
| apps/backend/src/network-scan/domain/node/scan/node-crawl/CrawlerService.ts | Forward suppressed addresses. |
| apps/backend/src/network-scan/domain/node/scan/__tests__/NodeScannerTomlStep.test.ts | Verify probe-specific merging. |
| apps/backend/src/network-scan/domain/node/scan/__tests__/NodeScan.test.ts | Test measurement preservation and archive filtering. |
| apps/backend/src/network-scan/domain/node/endpoint/ValidatorEndpointCandidate.ts | Assign UUIDs before persistence. |
| apps/backend/src/network-scan/domain/node/endpoint/EndpointCandidateRepository.ts | Define bulk repository methods. |
| apps/backend/src/network-scan/domain/node/endpoint/EndpointCandidateManager.ts | Batch writes and parallelize endpoint work. |
| apps/backend/src/network-scan/domain/node/endpoint/__tests__/EndpointCandidateManager.test.ts | Test suppression and batch persistence. |
| apps/backend/src/network-scan/domain/network/scan/TomlService.ts | Cache TOML requests temporarily. |
| apps/backend/src/network-scan/domain/network/scan/python-fbas/PythonFbasHttpClient.ts | Include underlying connection errors. |
| apps/backend/src/network-scan/domain/network/scan/python-fbas/PythonFbasAdapter.ts | Trigger fallback when aggregated analysis fails. |
| apps/backend/src/network-scan/domain/network/scan/__tests__/TomlService.test.ts | Test cached response reuse. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
October 5, 2026 00:28 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
tmosleyIII
had a problem deploying
to
obsrvr-radar-production
October 5, 2026 00:28 — with
GitHub Actions
Failure
GitHub Actions
Failure
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
October 5, 2026 00:44 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
October 5, 2026 00:44 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
October 5, 2026 12:50 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
October 5, 2026 12:51 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
November 10, 2026 10:31 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
tmosleyIII
temporarily deployed
to
obsrvr-radar-staging
November 10, 2026 10:32 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previous network scanner updates pushed the scan time above our normal 5 minute threshold. This PR is to address those issues.