Repository navigation
fix: deduplicate DSH usage across chunks, messages and v3 snapshots - #1
Merged
Merged
Conversation
luwill
pushed a commit
that referenced
this pull request
Sep 28, 2026
The request identity merged in #1 (session + turn + step) held on both datasets checked, 21k requests with every copy agreeing on usage. If it ever fails, for example a step retried after an error, the collector would keep only the record with more output and silently drop a request. The key now carries the input-side usage. Copies of one request always agree on input because streaming only grows output, so differing input means two requests. The key depends on the record alone: resolving conflicts as they are met would produce different keys when the two snapshots list the records in a different order. Rows stored by the pre-merge branch under the shorter key are replaced during the same rescan. On real logs the result is unchanged: 11,787 requests, 1.998B tokens. Release 1.8.2 with the dsh fix from #1.
Owner
|
感谢这个 PR，问题定位准确，测试和迁移都考虑得很周全。 合并前我在自己的数据上独立核对过（138 个文件 / 137 个会话）：逐行统计 12,117 条、20.60 亿，按请求去重后 11,787 条、19.98 亿，与你的核对脚本和采集器结果完全一致。CI 在 macOS / Ubuntu / Windows 上全部通过，已合并。 合并后补了一处改动（c5a84fc）：请求键里加上了输入侧用量。目前两边的数据都没有出现 turn/step 相同而用量不同的情况，但万一出现（比如某一步失败后重试），原来的写法会只保留一条。你本地用这个分支跑出来的旧键会在对应文件下次重扫时自动替换，不会重复计数。 这个修复会随 1.8.2 发布，更新日志里已注明出处。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DSH can write the same API request as an
assistant/chunkusage record and anassistant/message, then copy it intosession.v3.jsonl.zstdwith a differentseq. The current collector keys these by log row/file, so one 1,100-token request can become three events totaling 3,300 tokens. The timestamps can differ by 1 ms, so timestamp-based deduplication would also miss it.This change uses session +
data.turn+data.stepas the request identity across both record types and snapshot formats. Both files are still read so requests found only in either snapshot remain counted. Records without a valid request identity retain their legacy keys instead of being heuristically merged. The existing token formula is unchanged: reasoning is already included in output.For existing installations, DSH collector version 4 forces a rescan even when file size/mtime is unchanged. Each decoded request replaces its observed legacy keys inside the scanner's existing file transaction. This repairs stored duplicates without clearing entire sessions; failed writes roll back both events and the file cursor. Legacy rows whose source records are no longer available are preserved, so this cannot guarantee repair of unavailable historical sources.
Validation
npm test: 568 checks passed on macOS / Node 26.0.0. Seven of the nine new DSH checks fail against unmodified98ad4c1.seqvalues, non-overlapping history, equal usage from distinct requests, separate sessions, missing/invalid request identities, partial output completion, stale snapshots, migration, rollback/retry, and decompression failure.python3 test/dsh-audit.py <sessions-root>.All copies of each request agreed on every usage component; all declared
totalTokensmatched input + cache read + cache write + output. The fix removes 9,374 duplicate events / 2,720,040,424 duplicated tokens. Both an unchanged incremental scan and a forced full collector replay inserted 0 events. Unrelated database rows and source-file hashes remained unchanged.