Fighting bitrot by improving repo health checks

[Skyedra]

What problem would this solve?

Hello,

I would like to suggest some improvements to gitea's health check.

Bitrot (files silently corrupting on disk due to hardware faults) is something that can happen on any system, and I have even observed it on large, name brand cloud providers in past. Gitea provides some assistance via health checks, but this system could be further improved to help keep repos safe.

What do you propose?

1) Warning e-mail

An administrator needs to be alerted promptly of a bitrot situation so repos can be restored from backup. As far as I can tell (but I could be wrong, please correct me if so), check.go only reports to an error log. Without surfacing this more visibly, it is likely an administrator would miss this. In a worst case scenario, backups might rotate and the good copy of the repo might be lost before an administrator becomes aware of the problem.

Sending an e-mail notification on failed fsck would help alert an administrator to take action urgently and restore from backup.

2) git lfs fsck

Looking at check.go, it does not appear to check lfs objects for consistency by running git lfs fsck LFS objects should be checked for consistency and alerted on errors.

3) Healthchecks integration

Healthchecks is a popular self-hostable tool that allows administrators to be pinged when a script either fails or is not run. This second case is particularly useful in case there is corruption so bad that the entire gitea instance is affected to the point where warning e-mails would not trigger (or perhaps cron gets disabled accidentally).

[Skyedra]

Temporary workaround: Here is a bash script that can be run via cron to send healthchecks alerts on either a git lfs file corruption or repo fsck error. It requires git-lfs be installed. Only supports local file LFS. Very likely it might be missing some checks.

#!/usr/bin/env bash

set -u
shopt -s nullglob

BASE_DIR="${HOME}/data/git/repositories"
LFS_DIR="${HOME}/data/git/lfs"
HEALTHCHECKS_URL="${HEALTHCHECKS_URL:?Set HEALTHCHECKS_URL first}"

failure_url="${HEALTHCHECKS_URL%/}/fail"

notify_failure() {
    local repo="$1"
    local message="$2"

    curl --fail --silent --show-error \
        --max-time 30 \
        --data-binary "git fsck failed for ${repo}

${message}" \
        "$failure_url" || true
}

# Check Git repositories.
user_dirs=("${BASE_DIR}"/*/)

for user_dir in "${user_dirs[@]}"; do
    repo_dirs=("${user_dir}"/*/)

    for repo_dir in "${repo_dirs[@]}"; do
        repo_dir="${repo_dir%/}"

        if ! cd "$repo_dir"; then
            notify_failure "$repo_dir" "Could not enter repository directory."
            exit 1
        fi

        fsck_output="$(git fsck 2>&1)"
        fsck_status=$?

        if (( fsck_status != 0 )); then
            notify_failure "$repo_dir" "$fsck_output"
            exit 1
        fi
    done
done

# Check Git LFS object files.
#
# Each object is expected at:
#   ~/data/git/lfs/XX/YY/REMAINDER
#
# Its complete SHA-256 is:
#   XX + YY + REMAINDER
#
# The glob only matches two-character directories, so ~/data/git/lfs/tmp
# is ignored.

first_level_dirs=("${LFS_DIR}"/??/)

for first_dir in "${first_level_dirs[@]}"; do
    first_component="$(basename "$first_dir")"

    # Ensure this is a directory whose name is exactly two characters.
    [[ -d "$first_dir" && "${#first_component}" -eq 2 ]] || continue

    second_level_dirs=("${first_dir}"??/)

    for second_dir in "${second_level_dirs[@]}"; do
        second_component="$(basename "$second_dir")"

        [[ -d "$second_dir" && "${#second_component}" -eq 2 ]] || continue

        lfs_files=("${second_dir}"*)

        for lfs_file in "${lfs_files[@]}"; do
            [[ -f "$lfs_file" ]] || continue

            remainder="$(basename "$lfs_file")"
            expected_hash="${first_component}${second_component}${remainder}"

            # Hash the file and capture any hashing error.
            if ! hash_output="$(sha256sum -- "$lfs_file" 2>&1)"; then
                notify_failure "$lfs_file" "$hash_output"
                exit 1
            fi

            actual_hash="${hash_output%% *}"

            if [[ "$actual_hash" != "$expected_hash" ]]; then
                notify_failure "$lfs_file" \
                    "SHA-256 mismatch.
Expected: ${expected_hash}
Actual:   ${actual_hash}"
                exit 1
            fi
        done
    done
done

# Verify that every Git LFS object referenced by every repository exists
# in the shared LFS object directory.

for user_dir in "${BASE_DIR}"/*/; do
    repo_dirs=("${user_dir}"/*/)

    for repo_dir in "${repo_dirs[@]}"; do
        repo_dir="${repo_dir%/}"

        if ! cd "$repo_dir"; then
            notify_failure "$repo_dir" \
                "Could not enter repository directory."
            exit 1
        fi

        if ! lfs_output="$(git lfs ls-files -l 2>&1)"; then
            notify_failure "$repo_dir" "$lfs_output"
            exit 1
        fi

        # git lfs ls-files -l returns an empty string when the repository
        # contains no LFS files. The loop consequently does nothing.
        while IFS= read -r lfs_line; do
            [[ -n "$lfs_line" ]] || continue

            # The hash is the first whitespace-delimited field.
            read -r object_hash _ <<< "$lfs_line"

            # Construct the expected shared-object path from the SHA-256 hash.
            lfs_object="${LFS_DIR}/${object_hash:0:2}/${object_hash:2:2}/${object_hash:4}"

            if [[ ! -f "$lfs_object" ]]; then
                notify_failure "$repo_dir" \
                    "Missing Git LFS object.

Referenced by: ${lfs_line}
Expected path: ${lfs_object}"
                exit 1
            fi
        done <<< "$lfs_output"
    done
done


# Notify success
curl -fsS -m 10 --retry 5 -o /dev/null ${HEALTHCHECKS_URL}

exit 0