[boyter]

Fixes found by checking every claim in the README and help text against the code.

Blake3 missing for files between 200KB and the stream size

Files over 200KB up to --stream-size (1MB by default) are hashed by processReadFileParallel when not using --hash all. It had no blake3 branch, so the hash was silently empty: hashit -c blake3 printed
no hash and JSON omitted the field. Audits were also weakened, as an empty hash is skipped when comparing.

This affects the released 1.4.0.

Also fixes a data race in the same function, where every goroutine wrote to a shared start time used for trace output. The race detector reported 7 races before the change and none after.

Adds a unit test checking every hash is produced and agrees across the serial, parallel and scanner paths. It fails on the old code. test-all.sh now checks every hash on a 300KB file against streamed
processing.

Error handling and validation

• Files which cannot be opened, read or walked are reported and the remaining files still processed, then hashit exits non zero, as md5sum does. Previously missing and unreadable files exited 0.
• Directories supplied using --input are reported, rather than hashed as an empty file (d41d8cd9…).
• Unknown --format and --hash values are an error listing the valid options, rather than silently falling back to text or skipping the hash.
• --vv includes everything -v outputs. Previously an audit with only --vv listed matched files but not modified, moved, new or missing ones.
• Debug, trace and verbose output goes to stderr so it does not mix with sum, hashdeep and other output.
• Messages where processing continues are labelled WARN rather than ERROR.
• A .b2 or .blake2b sum file can contain blake2b-256, as written by b2sum -l 256, which previously never verified.

README and help text

• Install with go install github.com/boyter/hashit@latest, as go get no longer works outside a module.
• The sqlc generated files are in processor/database, not processor/db.
• The --help output is pasted from the current build.
• Examples use files which exist, and run as written:
  • hashdeep needs -l (and -v for the counts) to audit hashit's relative paths.
  • Checksum files are written outside the directory being checked, as hashdeep and md5sum otherwise see the half written file as changed.
• Documents:
  • exit codes
  • stdin being hashed when piped
  • -r only mattering with multiple paths
  • md5sum * not including subdirectories
  • all sum extensions
• SQLite memory use is described accurately: it does not grow with the database, but new files are held in memory.
• The Python example works with paths stored with a leading ./, opens the database read-only, hashes in chunks, and handles a missing sha256 column. Previously it could crash on a bad database path and
  created an empty database file for a mistyped path.
• Help text for -r, -v, --vv, --no-stream, --stream-size and -M describes what they actually do, and fixes a typo in -i.
• Stale comments in test-hashdeep.sh are removed, as all of its tests now pass.

Behaviour changes for release notes

• hashit exits non zero when any file cannot be processed, for example an unreadable file or broken symlink in a tree.
• Unknown --format and --hash values are an error.
• Debug, trace and verbose output is written to stderr.
• Non fatal messages are prefixed WARN.

Testing

test-all.sh and test-hashdeep.sh pass in full, and go test -race ./processor is clean. New tests cover the blake3 paths, the input file with a directory and a missing file, unknown hashes and formats,
debug output on stderr, --vv audits, and b2sum -l 256. The SQLite tests in test-all.sh now clear their directories first, so an aborted run no longer breaks the next.