Repository navigation
Fix stdin hashing, upgrade ed2k, add --absolute and hashdeep mtime fixes - #29
Merged
Merged
Conversation
Replaces the local vendor patch with the upstream fix, which also does not modify the hash state when Sum is called. Adds tests comparing against a reference implementation across chunk boundaries, through the library, read file and scanner paths. This also brings golang.org/x/crypto to v0.46.0, and re-vendoring restores gocodewalker to its released v1.5.1 source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hashes of stdin larger than 4KB were wrong as the read buffer was reused while the hash goroutines were still processing it. Each chunk is now copied. This affects the released 1.4.0 - Add --absolute to output absolute paths as hashdeep does without -l (#19) - The hashdeep format only falls back to md5 and sha256 when neither is requested, so -c sha256 gives a sha256 only file as hashdeep -c does (#27) - Never read stdin when -i or --audit is used, previously a piped stdin was hashed instead of the input file list or audited directory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hashdeep rejects files with an mtime column as badly formatted, and the column after the filename was ambiguous when filenames contain commas. mtime is now written as a "## mtime" comment line after each file, which hashdeep ignores. Older files with the mtime column still audit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #19, closes #20, closes #25, closes #27
Stdin hashing was wrong for input over 4KB
Not reported in an issue, found while testing. Hashes of piped input larger than 4KB were wrong on every run and for every algorithm, as the read
buffer was reused for the next read while the hash goroutines were still processing it. For example 50MB of random data gave a sha256 of
2ef527…where
sha256sumgives348464…. Each chunk is now copied before being sent to the hashes.This affects the released 1.4.0. The existing tests only piped
helloso never caught it. Added a 20MB stdin test compared againstsha256sum.#25 ed2k
go.felesatra.moe/hash/ed2kto v1.2.0 which has the upstream fix, replacing the local vendor patch. The upstream fix also does not modifythe hash state when
Sumis called.processor/ed2k_test.gocomparing against a reference implementation at sizes either side of the chunk boundaries, through the library, readfile and scanner paths, and checking repeated
Sumcalls.rhashfor file, stream and stdin input. All sizes match except exact multiples of 9,728,000 bytes, where rhash implements thevariant which appends an empty chunk. We keep the correct variant as the library implements.
golang.org/x/cryptoto v0.46.0, and re-vendoring restoresgocodewalkerto its released v1.5.1 source as the vendored copy haddrifted from it.
#27 requested hashes ignored for large files
Already fixed on master, confirmed only requested hashes are output for large files and values match
sha256sum. Also fixes the hashdeep side:-f hashdeep -c sha256previously forced md5 back in. It now writes a sha256 only file ashashdeep -c sha256does, and only falls back to md5 andsha256 when neither is requested.
#19 absolute paths
Added
--absoluteto output absolute paths, matching hashdeep's default output without-l. Relative remains the default as it makes a manifestwhich can be shipped and verified elsewhere. Output matches hashdeep for
.,d,./d,../p/d, single files and no arguments, and hashdeep auditsit.
#20 mtime with commas in filenames
Real hashdeep rejected any
-f hashdeep --mtimefile asBadly formatted file, as it does not know the mtime column, and the column after thefilename was ambiguous with commas. mtime is now written as a comment line after each file which hashdeep ignores:
The header is now the standard
size,md5,sha256,filename, so filenames with commas are unambiguous and hashdeep can audit the file. Older files withthe mtime column still audit with hashit.
Other fixes
-iand--auditnever read stdin. Previously when stdin was a pipe, such as in scripts, CI or over ssh, stdin was hashed instead of the inputfile list or the audited directory.
Behaviour changes for release notes
-f hashdeep --mtimewrites## mtimecomment lines instead of an mtime column.-f hashdeep -c sha256(or-c md5) writes only that hash, as hashdeep does.--absoluteflag.Testing
test-all.shpasses in full, with new tests for large stdin,-iwith a piped stdin, hashdeep auditing--absolute, sha256 only and--mtimeoutput, and hashit auditing
--mtimeoutput.