[boyter]

Fixes from a third round of checking every README and help text claim against the code, and checking the README documents everything a user needs.

Bugs fixed

• --overwrite could delete an empty directory. An empty directory reported size 0, passed the "SQLite database or empty" check, and was replaced with a database. --output now never replaces a directory
  or other non regular file, for any format.
• Symlink cycles walked one level too far. Only walk roots and followed links were recorded, so a link to a parent directory hashed a duplicate before stopping. Cycles are now detected as hashdeep does, by
  a link leading into a directory on its own path. The output now matches hashdeep for parent links, mutually linked directories, sibling links, links to /, and ./ or absolute roots.
• Empty audits could never pass. The audit file of an empty directory always failed with "contains no records". It now passes against an empty directory, and any file present is still reported as new.
• "Audit passed" with exit code 1. When a path was missing or a file could not be read, the audit said passed but exited 1. It now reports failed.
• Broken symlinks found while walking are skipped with a warning, as hashdeep ignores them, rather than failing the run.
• Filenames containing [ ] * ? could not be hashed by name, because hashit expanded them as wildcards. Existing paths are now used as is before being expanded.
• -vv meant -v. The flag library read it as -v twice. It is now the same as --vv, as hashdeep users expect.
• BSD style lines padded by rhash (MD4 (file) = …) are accepted.
• A record sharing its name with an outside audit file was dropped. With an audit file ../a.txt, the record ./a.txt also resolved to the audit file through its directory and was dropped as its own
  entry. A record is now only dropped when it refers to no other file.
• test-hashdeep.sh uses temporary directories, rather than deleting audit.txt from the working directory.
• examples/performance_tests/create_folders_with_files.py requires a target directory, rather than writing around 72,000 files into the current one.

README

Corrections:

• @latest installs the last tagged release, which lacks newer features.
• hashdeep's -X is a matching mode, not "display failed hashes".
• Stdin is read when data is piped or redirected from a file, and never by audits or --input.

New sections:

• Paths: no path, multiple paths and -r, wildcards, --input, symlinks, special files, hidden files, the ignore flags, --exclude-dir and --not-match.
• Output Formats: every format including json and hashonly, --mtime in each, --output, --stream-size, --no-stream, and debug output going to stderr.
• Errors and Exit Codes.

Additions to existing sections:

• Audits:
  • the -v and -vv output lines
  • which formats can be audited
  • auditing a subset of files reports the rest as missing
  • copies are reported as new, where hashdeep reports them as moved
  • how unreadable files are reported
  • mtime is not compared
  • empty audits pass
  • sum file hash guessing, including b2sum, and which directory to run the audit from
• SQLite:
  • hashit.db is used when --output is not given
  • mtime storage
  • the temporary table directory and SQLITE_TMPDIR
  • subsets and --allow-moved
  • rejected databases from older development builds
• The --hash names to use, and further hashdeep differences.

Behaviour changes for release notes

• --output refuses to replace a directory or other non regular file.
• An audit reports failed when any file could not be processed.
• Broken symlinks are skipped with a warning.
• Existing paths are used as is before wildcard expansion.
• -vv is the same as --vv.

Testing

test-all.sh and test-hashdeep.sh pass in full, and go test -race ./processor is clean. New tests cover:

• symlink cycles compared against hashdeep
• broken symlinks
• --overwrite on a directory
• empty audits
• glob characters in filenames
• -vv
• an audit with a missing path
• an audit file sharing a name with an audited file