Repository navigation
fix: read comments again after Reddit's new comment IDs, and stop Phase 2 crashing - #105
Merged
Merged
Conversation
When Reddit returned no data for a stored comment, the "comment unavailable" log line read its permalink, which fetched it again and raised ClientException (getattr's default only covers AttributeError). Every Phase 2 pass since then crashed on the same deleted comment, so none of the 1483 waiting comments were checked. The log line now falls back to the comment ID. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019J1tUq4uEQ6i91Nag6bzaa
Toldry
marked this pull request as ready for review
October 25, 2026 22:22
Since about 2026-10-21 Reddit gives every new comment a 64-bit, 13-character ID that is no longer sequential (announced on r/redditdev), so the reader that fetched blocks of 100 consecutive IDs read nothing for days. CommentReader now polls the r/all comment listing on STREAM_READERS threads (12 by default, 3 per account, each at most every 3 s), skips comments any thread saw before, and counts polls that may have missed comments as gaps in the heartbeat. The heartbeat warns when no comment was read in its interval. STREAM_SOURCE=ids is gone, the single PRAW stream is now STREAM_SOURCE=stream, and the ID-reader script and live tests are replaced by listing ones. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019J1tUq4uEQ6i91Nag6bzaa
Toldry
changed the title
Toldry
added a commit
that referenced
this pull request
Oct 25, 2026
The master run after #105 failed because the test gallery was still unprocessed after 30 s, so the deploy was skipped and the new comment reader never went live. Wait up to 90 s and say what Reddit returned when it times out. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019J1tUq4uEQ6i91Nag6bzaa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Querent · project thread
Two problems found in the logs from 2026-10-25 20:57 to 22:17 UTC.
1. Comment reader read nothing since about 2026-10-21
Every heartbeat said
scanned 0 comments, and every head poll was rejected. The r/all listing now returns IDs like19rnei2yv8zk1(about 6 × 10^18). Reddit announced on r/redditdev ("Upcoming changes to the comment ID endpoint", rolled out from the week of 2026-09-28) that new comment IDs are 64-bit, up to 13 base-36 characters, and "will not be monotonically increasing anymore". Two comments seconds apart are about 2 × 10^9 IDs apart, so reading blocks of 100 consecutive IDs can't work anymore. Third-party summary: https://www.redditapis.com/blogs/reddit-comment-ids-13-characters-2026Change:
CommentReadernow polls the r/all comment listing.STREAM_READERSthreads, 12 by default (3 per account). Each thread asks for the newest 100 comments at most every 3 s, and the threads start staggered.SeenComments(the last 20,000 fullnames) makes sure each comment is handed on once.new/requestandgaps, which counts requests whose comments were all new, meaning comments in between were missed. It also warns whenever no comment was read during the interval.STREAM_SOURCE=idsis removed and falls back tolistingwith a warning. The one-request PRAW stream is nowSTREAM_SOURCE=stream.scripts/verify_comment_ids.pyand the ID live tests are removed. New live tests cover the listing and a 45 s run of the real reader.stream_cursorstable and DB functions are left in place, unused.Rate and egress: about 120 requests a minute with 12 readers, compared with about 75 for the ID reader. That stays within Reddit's 100 per minute per account. If the VM's
.envstill setsSTREAM_READERS=4, that overrides the new default of 12.2. Phase 2 crashed on every pass
Each pass ended with
ClientException('No data returned for comment t1_phkdvpc'). The cause: the "comment unavailable" log line readpermalink, which re-fetched a deleted comment and raised again (getattr's default only coversAttributeError). As a result, none of the 1,483 waiting comments got checked.permalink_safenow falls back to the ID.Tests
uv run pytest: 1771 passed. The live tests run in CI after merge.🤖 Generated with Claude Code
https://claude.ai/code/session_019J1tUq4uEQ6i91Nag6bzaa