feat(cli): default 'add' to raw/ with summary + cleanup, auto_delete_added_files config

[sebastianbraun25]

Note

This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.

Problem

The openkb add command supports adding directories recursively, but users who frequently
download documents to raw/ had to manually invoke openkb add raw/ and then manually
delete processed files. There was no batch-processing shortcut for the raw/ staging
directory, no summary of the outcome, and no configuration option to auto-delete successfully
ingested files and duplicates.

An earlier version of this PR addressed this by adding a separate openkb add-all command.
On further review this duplicated almost the entire add directory-processing code path
(same file walk, same filtering, same per-file ingestion call). This PR now consolidates that
functionality directly into add instead of introducing a second, largely redundant command.

Solution / Changes

openkb add (no changes for existing usage):

• add <path> behaves exactly as before for files, directories, and URLs.
• add with no PATH argument (and no --from-pageindex-cloud) now defaults to the
  KB's raw/ directory, walks it recursively, and ingests all supported file types
  (PDF, Markdown, DOCX, PPTX, XLSX, XLS, HTML, TXT, CSV).
• When processing a directory (default raw/ or an explicit path), add now prints an
  aggregated summary at the end: Added: X, Skipped: Y, Failed: Z, Deleted: W.
• If auto_delete_added_files is enabled, now-empty subdirectories are cleaned up after a
  directory run (in addition to per-file deletion, which already existed).

New configuration parameter:

• auto_delete_added_files (boolean, default: false)
• When enabled, add automatically deletes files after processing
• Deletes on both "added" (successful ingestion) and "skipped" (duplicate/already in KB)
• Preserves "failed" files to allow user retries
• Applies to all ingest methods: direct files, directories (including the default raw/), and URLs

Implementation:

• Added _delete_if_auto_cleanup_enabled() helper function to handle cleanup logic
• Added _cleanup_empty_directories() for post-run empty-subdirectory cleanup
• Removed the standalone add-all command entirely (unreleased, no deprecation/alias needed)
  in favor of add with no PATH argument
• Updated add's docstring to document the default-raw/, summary, and cleanup behavior
• Updated tests and README accordingly

Issues

• Resolves feat(cli): add-all command and auto_delete_added_files config #231

[sebastianbraun25]

Rewrote this branch's history to remove unrelated commits.

The branch had accidentally picked up commits from the (now-withdrawn, see
#229/#230) retry-timeout work as ancestors — likely from being built on top
of a local integration branch at the time rather than directly off
main. That made this PR's diff show unrelated changes to
openkb/agent/compiler.py and a new tests/test_compiler_retry.py
that have nothing to do with add-all/auto-delete.

Rebuilt cleanly from current main with only the two commits that
actually belong to this feature (add-all command + the
auto-delete-on-skipped follow-up fix). Diff now only touches
openkb/cli.py and openkb/config.py as expected. No functional
changes to the add-all feature itself — force-pushed the same branch name so
this PR stays open with its history/comments intact.