Repository navigation
Ingester loops have no upper bound on a single cycle #351
Description
Background
Each ingester is an infinite loop restarted on crash by restartIngester. There
is no limit on how long one iteration may take. With L162's missing timeouts, a
single slow upstream can stretch one cycle indefinitely; the loop has not
crashed, so nothing restarts it, and the venue goes quiet without any error
being raised anywhere.
This is the most dangerous shape in this batch, because every monitoring signal
the service currently has reports it as healthy.
What to build
- A per-cycle deadline. A cycle exceeding it is abandoned and the next one
starts, rather than waiting forever. - Count abandoned cycles in a metric, labelled by venue and network.
- Make sure abandoning a cycle cannot leave a partial write committed — check
how each ingester writes before changing the control flow. - Overlap must remain impossible: abandoning a cycle must not start a second
concurrent one against the same venue.
Acceptance criteria
- A cycle that exceeds its deadline is abandoned, not awaited indefinitely
- The deadline is configurable per ingester
- Abandoned cycles are counted in a metric
- No partial write survives an abandoned cycle
- A test with a slow upstream proves the loop keeps making progress
Drips Wave · Complexity: Intermediate · 150 points
Repo: Divinity-Labs/Lens. Node + TypeScript, Fastify, Prisma on Postgres,
Redis for queues, Prometheus via prom-client.
Run the tests: npm test. The tests are vitest, not jest. Running jest
against this repo reports "66 suites, 0 tests" and means nothing. A single file
is npx vitest run src/__tests__/yourfile.test.ts.
Load test (prices) is already failing on main and has been for some
time. It is not your PR that broke it. Every other check should be green.
This service prices real assets on Stellar mainnet. A number that is wrong
is worse than a number that is missing, because a caller cannot tell the
difference. Prefer failing loudly over returning a plausible value.
Never elide an address, a contract id or a key in code, a comment or a test
fixture. A previous batch wrote a shortened contract id in an issue text and
five contributors invented plausible-looking ids to fill the gap. Write the
whole thing or do not write it at all.
The house pattern for an outbound deadline already exists — see
src/alerts/delivery.ts, which passes signal: AbortSignal.timeout(10_000) to
fetch. Where an issue below asks for a timeout, copy that, do not invent a
second convention.
Required: Before submitting, join the contributor Telegram so your work can be tracked and counted toward the Stellar Wave: https://t.me/+fxHXq8f1SwlkZDBk
Activity
felixumeh commentedon Oct 16, 2026
@felixumeh has applied to work on this issue as part of the Stellar Wave Program's 10th wave.
ℹ️ Repo Maintainers: To accept this application, review their application or assign @felixumeh to this issue.
moon-code12 commentedon Oct 16, 2026
@moon-code12 has applied to work on this issue as part of the Stellar Wave Program's 10th wave.
ℹ️ Repo Maintainers: To accept this application, review their application or assign @moon-code12 to this issue.
Oluoma123 commentedon Oct 16, 2026
@Oluoma123 has applied to work on this issue as part of the Stellar Wave Program's 10th wave.
ℹ️ Repo Maintainers: To accept this application, review their application or assign @Oluoma123 to this issue.
drips-wave commentedon Oct 16, 2026
Congratulations, @felixumeh! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on October 22, 2026.
🧑💻 @felixumeh: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.
Warning
When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.
🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.
🌊 Happy Wave 🌊
drips-wave commentedon Oct 21, 2026
This issue has been completed by @felixumeh as part of the Stellar Wave Program's 10th Wave 🥳
😎 @felixumeh: You earned 200 Points for completing this issue! Once the Wave's rewards are finalized, you'll be eligible for a percentage of the Wave's reward pool based on the percentage of total points you've earned. Learn more here. You can also Leave a review to share your experience working on this issue.
🧑💻 Repo maintainers: How'd the contributor do? Leave a review to share your experience working with them.
Merge PR Divinity-Labs#479: back off restarting a crashing ingester, …
Miracle656 commentedon Oct 23, 2026
Reopening — I closed this in error when merging #479. That PR added exponential backoff and crash reporting for an ingester that crashes, which is a different failure: this issue is about a cycle that never finishes and therefore never crashes, so nothing restarts it and the venue goes quiet with no error raised.
src/ingest/venues/runner.tsonmainstill has no per-cycle deadline — only a classifier that labels an error astimeoutonce something else has already thrown. #471 addresses the actual issue and is under review.feat(ingest): bound every ingester cycle with a deadline (closes Divi…
Merge PR Divinity-Labs#471: bound every ingester cycle with a deadline (
Merge PR #453: ingest Blend lending pool state, rates and utilisation (…
fix: distinguish failed venue fetches from empty venues in aggregate (D…