Skip to content

Add audit hooks for inter-node user data transfers - #18569

Merged
HTHou merged 9 commits into
apache:masterfrom
HTHou:codex/audit-user-data-transfer
Sep 3, 2026
Merged

Add audit hooks for inter-node user data transfers#18569
HTHou merged 9 commits into
apache:masterfrom
HTHou:codex/audit-user-data-transfer

Conversation

@HTHou

@HTHou HTHou commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description

Add sender-side audit hooks for inter-node user-data transfers required by the FDP_ITT.1 audit requirements from GB/T 20273-2019.

This PR only adds audit event modeling, audit hooks, and routing through the existing audit logger. It does not introduce a new audit sink, persistence mechanism, protection mechanism, or policy implementation.

Audit record

Each transfer attempt records only:

  • timestamp
  • subject/initiator
  • source and destination
  • protection method
  • result
  • one error value when the attempt fails

The event is routed as AuditEventType.USER_DATA_TRANSFER. Payload contents are never retained.

Instrumented transfer paths

  • cross-DataNode MPP TsBlock transfer, recorded by the pull initiator
  • cross-DataNode write dispatch when the plan tree contains an InsertNode
  • remote Load TsFile piece transfer
  • IoTConsensus user-data log batches and snapshot fragment transfer
  • IoTConsensusV2 tablet and TsFile piece transfer

Review follow-ups

  • classify IoTConsensus requests and emit only for batches containing user data
  • exclude audit-origin and audit-database writes to prevent recursive audit events
  • isolate handler lookup, enablement checks, and event delivery from transfer and replication paths
  • record MPP success only after exact response validation and local buffer acceptance
  • reduce the event model to the minimum audit fields and add focused behavior tests
  • classify transfer audit records as CONTROL so concrete audit filters can retain them
  • preserve MPP retry and terminal failure behavior for short non-empty responses
  • skip IoTConsensus user-data classification entirely while auditing is disabled
  • use the documented protection_method values TLS and NONE

Explicitly out of scope

  • Ratis transfer paths
  • AINode
  • ordinary query FragmentInstance or query-plan dispatch
  • SchemaRegion and ConfigRegion consensus transfer
  • heartbeats, control commands, delete events, and file seal signals
  • local in-process calls and general Pipe transfers

This PR has:

  • been self-reviewed.
  • added Javadocs for the audit event model and non-blocking handler contract.
  • added comments explaining audit isolation and payload exclusion.
  • added focused unit tests for event fields, user-data classification, recursion prevention, handler isolation, and MPP response acceptance.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Verification

  • mvn spotless:apply -pl iotdb-core/node-commons,iotdb-core/consensus,iotdb-core/datanode
  • targeted unit tests for the changed audit and transfer paths
  • mvn test-compile -DskipTests
  • mvn test-compile -DskipTests -P with-zh-locale
  • git diff --check

All commands above pass locally.

@HTHou HTHou left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main findings are inline. Please also add behavior-level tests that capture emitted events for success, remote-status failure, exception/retry, and source/destination direction, plus tests proving that a throwing audit handler never changes the transfer outcome and that audit-log writes do not recursively generate USER_DATA_TRANSFER events. The current tests only cover the event value object and the PlanNode classifier.

@HTHou HTHou left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second-pass findings after 43a7104 are inline. The earlier recursion, batch-classification, handler-isolation, and event-field issues are otherwise addressed.

@HTHou

HTHou commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the latest audit review feedback in 85ff1e2:

  • MPP PlanNode classification now short-circuits on DataNodeUserDataTransferAuditor.isEnabled() before inspecting the plan tree; the regression test verifies getChildren() is never called while audit logging is disabled.
  • Added consensus-group-level user-data classification for snapshot transfers. DataRegion groups resolve their database once per snapshot, and __audit / root.__audit snapshots are excluded before any fragment event is emitted.

Validation passed: focused consensus/DataNode tests (10 tests), full English reactor test-compile, and full Chinese-locale reactor test-compile.

…ta-transfer

# Conflicts:
#	iotdb-core/node-commons/src/main/i18n/en/org/apache/iotdb/commons/i18n/CommonMessages.java
#	iotdb-core/node-commons/src/main/i18n/zh/org/apache/iotdb/commons/i18n/CommonMessages.java
@HTHou HTHou changed the title [Feature] Add audit hooks for inter-node user data transfers Add audit hooks for inter-node user data transfers Sep 2, 2026
@HTHou
HTHou marked this pull request as ready for review September 2, 2026 10:05
@HTHou
HTHou merged commit d6b3afb into apache:master Sep 3, 2026
55 of 56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants