Skip to content

Fix stdin hashing, upgrade ed2k, add --absolute and hashdeep mtime fixes - #29

Merged
boyter merged 3 commits into
masterfrom
issue-fixes
Oct 29, 2026
Merged

boyter merged 3 commits into
masterfrom
issue-fixes

Conversation

@boyter

@boyter boyter commented Oct 29, 2026

Copy link
Copy Markdown
Owner

Closes #19, closes #20, closes #25, closes #27

Stdin hashing was wrong for input over 4KB

Not reported in an issue, found while testing. Hashes of piped input larger than 4KB were wrong on every run and for every algorithm, as the read
buffer was reused for the next read while the hash goroutines were still processing it. For example 50MB of random data gave a sha256 of 2ef527…
where sha256sum gives 348464…. Each chunk is now copied before being sent to the hashes.

This affects the released 1.4.0. The existing tests only piped hello so never caught it. Added a 20MB stdin test compared against sha256sum.

#25 ed2k

  • Upgraded go.felesatra.moe/hash/ed2k to v1.2.0 which has the upstream fix, replacing the local vendor patch. The upstream fix also does not modify
    the hash state when Sum is called.
  • Added processor/ed2k_test.go comparing against a reference implementation at sizes either side of the chunk boundaries, through the library, read
    file and scanner paths, and checking repeated Sum calls.
  • Verified against rhash for file, stream and stdin input. All sizes match except exact multiples of 9,728,000 bytes, where rhash implements the
    variant which appends an empty chunk. We keep the correct variant as the library implements.
  • Also brings golang.org/x/crypto to v0.46.0, and re-vendoring restores gocodewalker to its released v1.5.1 source as the vendored copy had
    drifted from it.

#27 requested hashes ignored for large files

Already fixed on master, confirmed only requested hashes are output for large files and values match sha256sum. Also fixes the hashdeep side: -f hashdeep -c sha256 previously forced md5 back in. It now writes a sha256 only file as hashdeep -c sha256 does, and only falls back to md5 and
sha256 when neither is requested.

#19 absolute paths

Added --absolute to output absolute paths, matching hashdeep's default output without -l. Relative remains the default as it makes a manifest
which can be shipped and verified elsewhere. Output matches hashdeep for ., d, ./d, ../p/d, single files and no arguments, and hashdeep audits
it.

#20 mtime with commas in filenames

Real hashdeep rejected any -f hashdeep --mtime file as Badly formatted file, as it does not know the mtime column, and the column after the
filename was ambiguous with commas. mtime is now written as a comment line after each file which hashdeep ignores:

2,b026324c6904b2a9cb4b88d6d61c81d1,4355a4…,./comma,file
## mtime 2026-10-29 10:54:36

The header is now the standard size,md5,sha256,filename, so filenames with commas are unambiguous and hashdeep can audit the file. Older files with
the mtime column still audit with hashit.

Other fixes

  • -i and --audit never read stdin. Previously when stdin was a pipe, such as in scripts, CI or over ssh, stdin was hashed instead of the input
    file list or the audited directory.

Behaviour changes for release notes

  • Stdin hashes are now correct for input over 4KB, so will differ from those produced by 1.4.0.
  • -f hashdeep --mtime writes ## mtime comment lines instead of an mtime column.
  • -f hashdeep -c sha256 (or -c md5) writes only that hash, as hashdeep does.
  • New --absolute flag.

Testing

test-all.sh passes in full, with new tests for large stdin, -i with a piped stdin, hashdeep auditing --absolute, sha256 only and --mtime
output, and hashit auditing --mtime output.

boyter and others added 3 commits October 29, 2026 10:55
Replaces the local vendor patch with the upstream fix, which also does
not modify the hash state when Sum is called. Adds tests comparing
against a reference implementation across chunk boundaries, through the
library, read file and scanner paths.

This also brings golang.org/x/crypto to v0.46.0, and re-vendoring
restores gocodewalker to its released v1.5.1 source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hashes of stdin larger than 4KB were wrong as the read buffer was
  reused while the hash goroutines were still processing it. Each chunk
  is now copied. This affects the released 1.4.0
- Add --absolute to output absolute paths as hashdeep does without -l (#19)
- The hashdeep format only falls back to md5 and sha256 when neither is
  requested, so -c sha256 gives a sha256 only file as hashdeep -c does (#27)
- Never read stdin when -i or --audit is used, previously a piped stdin
  was hashed instead of the input file list or audited directory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hashdeep rejects files with an mtime column as badly formatted, and the
column after the filename was ambiguous when filenames contain commas.
mtime is now written as a "## mtime" comment line after each file,
which hashdeep ignores. Older files with the mtime column still audit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pr-insights pr-insights Bot added L/complexity Low complexity XL/size Extra large change labels Oct 29, 2026
@boyter
boyter merged commit 9929c61 into master Oct 29, 2026
1 check passed
@boyter
boyter deleted the issue-fixes branch October 29, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L/complexity Low complexity XL/size Extra large change

Projects

None yet

1 participant