docs(kafka): deploy Kafka on node-local disks with pre-bound PVs - #845
Merged
Conversation
Adds a deployment guide for running Kafka on node-local storage under the Strimzi 0.48 operator line (ACP 4.3, KRaft, KafkaNodePool). Covers the two field failures with local storage: - A broker reschedules and comes up on an empty or foreign log directory. Root cause is a hostPath PV without nodeAffinity; local PVs cannot do this because the API server requires nodeAffinity on them. - PVs cross-bind so broker-0 lands on broker-1's disk. This is not caused by restarting pods -- a bound PVC never changes its PV -- but by PVCs being recreated (uninstall/reinstall, PVC deletion, pool or cluster rename, node-ID reassignment), where matching is first-come-first-serve on class + capacity + accessMode. The fix is PV-side claimRef pre-binding, since the two configuration-level alternatives are gone on this operator line: Kafka.spec.kafka.storage is ignored in favour of KafkaNodePool.spec.storage, and per-broker storage.overrides has been ignored since Strimzi 0.46.0. Verification commands in the guide have not been run against a live cluster; the doc says so.
Tested the procedure end to end on an ACP 4.3 cluster (operator 0.48.0 / v4.3.3, Kafka 4.2.0, three worker nodes). Several claims were wrong. Confirmed as written: - Deleting all broker pods simultaneously does not reshuffle storage. Every pod returned to its original node, bindings unchanged, all 1000 test messages intact with full ISR. - Cross-binding on PVC recreation is real. Recreating the PVCs swapped brokers 1 and 2 onto each other's disks. - The claimRef pre-binding fix works, and the released-PV recovery command in the guide works verbatim. Corrected: - The wrong-disk failure had its emphasis inverted. log.dirs is <mount>/kafka-log<ownNodeId>, so a broker on a foreign disk never reads the other broker's meta.properties -- it formats an empty log beside the orphaned data and reports Running 1/1. Two of three brokers failed silently; the topic went from 1000 messages to 0 while every byte was still on disk. The Invalid cluster.id crash is the exception, hitting only a broker that kept its own disk. - Verification check 4 could not detect a wrong binding: a reformatted broker writes a node.id matching itself. Replaced with a count of kafka-log* directories per disk, which is the real signal. - deleteClaim:false does not make uninstall/reinstall safe. The KRaft cluster ID lives only in Kafka.status.clusterId, so deleting the Kafka resource loses it and every broker rejects its retained disk. Added the record-and-restore procedure, including the reconcile trigger the status patch needs to survive. - Wrong-binding recovery must identify each disk's true owner by directory size, not meta.properties. - hostPath PVs get no fsGroup treatment from kubelet, so Kafka's UID 1001 cannot write a kubelet-created root:root 0755 directory. Another concrete reason to prefer local. - Pod template settings must go on the KafkaNodePool; a pool template replaces Kafka.spec.kafka.template.pod entirely and a securityContext placed there is silently dropped. Single-node/non-HA mode and node-failure/disk-replacement remain untested.
- Drop "with no replication from peers" from the recovery result. Only the final 1000/1000 count was measured; whether partitions re-replicated while the mis-bound brokers ran empty was not checked. - The KRaft cluster ID also falls back to each pool's status.clusterId, so "lives only in Kafka.status.clusterId" was wrong. The operational conclusion is unchanged: deleting those resources loses it. - Say plainly that the exercise ran on hostPath PVs. Binding, claimRef and recovery behaviour carries over to local; the local-specific claims (API-enforced nodeAffinity, fsGroup applied) are from the Kubernetes docs and were not under test.
…claims Repeated the full test on the same ACP 4.3 cluster using local PVs on /cpaas/kafka-lpv, with directories created on the nodes beforehand. Results matched the hostPath run everywhere it matters, and the two local-specific claims that were previously only cited from the Kubernetes docs are now measured: - The API server rejects a local PV with no nodeAffinity: "spec.nodeAffinity: Required value: Local volume requires node affinity". - kubelet applies fsGroup to local volumes. Starting from the same root:root 0755 directory that made hostPath crash-loop, it became drwxrwsr-x on mount and the cluster came up first time at default privileges -- no runAsUser override. Repeated on local, unchanged from the hostPath run: deleting all broker pods at once preserves bindings and data; PVC recreation cross-binds (brokers 1 and 2 swapped again, two of three silently formatted empty logs beside the real data, topic 1000 -> 0); claimRef recovery restores all 1000 messages; and the same destructive cycle with claimRef in place holds the mapping with one kafka-log* directory per disk. Also, from doing the recovery twice: restoring status.clusterId is a race against the operator and can need several attempts, so the procedure is now a loop that verifies the ConfigMap before restarting brokers. Added the cleanup step for the orphaned empty log directories. Initial binding was not name-ordered in either run (c/b/a then c/a/b), which is the clearest evidence that a PVC name has no bearing on which PV it receives.
Covers the last outstanding requirement. Verified on the same ACP 4.3 cluster with two single-replica instances in separate namespaces, each holding different data. - States plainly that RF=1 has no fault tolerance: one copy, one disk, one node. Node down means the cluster is down and its data unreachable; disk lost means data lost. Worth being blunt about for a log-queue use case. - Gives the manifest: one pool, one replica, combined roles, every replication factor at 1, no anti-affinity needed. - The cross-binding risk does not vanish, it moves between instances. Any number of single-replica instances sharing one no-provisioner StorageClass draw from the same pool of unreserved PVs, and a PVC has no notion of which instance a PV belongs to. Demonstrated: with s1's pod scheduled onto s2's node, s1's PVC bound s2's PV. - The failure mode differs from the multi-broker case, and is safer. Every single-replica instance is node 0, so its log directory is always kafka-log0 -- a broker landing on another instance's disk does find a kafka-log0, reads a foreign cluster.id and crash-loops instead of formatting. The victim's data survived intact. Multi-broker cross-binding loses data quietly; single-replica theft is an outage that preserves data. - Pod restart, cluster-ID loss on CR deletion, and claimRef recovery all behave identically; both instances recovered their own data (500 and 300 messages).
- Applicable versions are v4.1, v4.2, v4.3 and v4.4, not 4.3 alone. Updated the frontmatter ProductsVersion and the applicability note. Testing was done on v4.3, and the doc now says that rather than implying all four were exercised. - Use the product name "Alauda Streaming Service for Kafka" throughout, replacing "Alauda Kafka Operator". - Remove upstream community version numbers (0.48 / 0.46 / 0.25). Where a behaviour was described as changing at a community release, it is now stated as applying to every version this document covers. - Since the doc now spans four operator versions, note that spec.kafka.version must be one the installed operator supports rather than presenting 4.2.0 as fixed. - Drop the internal Java class names from the naming table. The naming rule is the actionable part; a reader cannot verify the implementation. Remaining strimzi.io references are API identifiers -- CRD group, labels and annotations that appear verbatim in user-authored YAML.
Previous commit conflated the two. Alauda Streaming Service for Kafka is 4.3; v4.1 through v4.4 are the Alauda Container Platform versions it runs on. - Applicability now reads: Alauda Streaming Service for Kafka 4.3, running on ACP v4.1, v4.2, v4.3 or v4.4. - ProductsVersion frontmatter back to 4.3, which is the product this document is about. - Test-environment statements say ACP v4.3, so they cannot be read as a service version distinction. The verified cluster was ACP v4.3.2. - Dropped the note telling readers to pick a Kafka version their operator supports; with a single service version, the 4.2.0 in the examples is simply correct. - Replaced the remaining "this operator line" phrasing with the product name.
The applicability note now names only the product this document is about: Alauda Streaming Service for Kafka 4.3. Which ACP versions it runs on is not listed here. Test-environment statements in the body still say which cluster a given result came from; those describe where a measurement was taken, not a compatibility range.
Two problems with the previous version. It was written against the raw Kafka and KafkaNodePool resources rather than the RdsKafka instance API that users actually create. Retested through that API, which changes the procedure materially: - A highly available instance is three brokers AND three controllers, in separate pools with separate storage. Six volumes, not three. - spec.storage is broker storage; spec.controller.storage is controller storage. Brokers take node IDs 0-2, controllers 3-5. - Claim names embed a hash generated per instance -- two instances on the same cluster carried cb42e1 and a28da4. It cannot be predicted, so volumes cannot be created up front. The procedure is now two-phase: create the instance, read the claim names, then create volumes reserved for exactly those names. Verified: claims sit Pending, then bind to the reserved volume as soon as it appears, and pods start on the right nodes. - The instance storage block exposes only class, size and deleteClaim. There is no selector, so that fallback is gone entirely. New prerequisite, found the hard way. A StorageClass must be granted to the project or the pvc-validator.cpaas.io webhook denies every claim: StorageClass "kafka-local" is not allowed in project "demo" The failure is indirect -- no pods, no claims, nothing on the instance -- and visible only in the Kafka resource conditions. Earlier testing missed it because those namespaces had no project label. Documented with the project.cpaas.io label and a troubleshooting entry. Second problem: this is a solution document that led with several screens of failure analysis. Reordered so the deployment procedure comes first and the reasoning follows in "Why This Design". Added the manual cleanup section after the procedure, since Retain means volumes and on-disk data are never reclaimed and the host directories are the step people forget. Also notes that bash and zsh index arrays differently, which silently produced an off-by-one node mapping while preparing this.
Matches every other document in the kafka solutions directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a deployment guide for running Kafka on node-local storage (hostPath / local disks).
docs/en/solutions/ecosystem/kafka/Deploy_Kafka_with_Local_HostPath_Storage.mdApplies to Alauda Streaming Service for Kafka 4.3. KRaft mode with
KafkaNodePool.Why
Two failures reported from the field with local-disk Kafka:
1. A broker reschedules and comes up on an empty or foreign log directory.
Root cause is a
hostPathPV withoutnodeAffinity— kubelet resolves the same path on whatever node the pod landed on.localPVs cannot do this, because the API server requiresnodeAffinityon them.2. PVs cross-bind, so
broker-0lands onbroker-1's disk.Not caused by restarting pods — a bound PVC never changes its PV. It happens when PVCs are recreated: uninstall/reinstall, a PVC deleted by hand, a pool or cluster rename, or node-ID reassignment. Three fresh PVCs then race three Retained PVs, matched only on storage class + capacity + access mode.
The fix
PV-side
claimRefpre-binding. The two configuration-level alternatives have no effect:Kafka.spec.kafka.storageis ignored in favour ofKafkaNodePool.spec.storage, and per-brokerstorage.overridesis ignored.storage.selectorapplies one selector to the whole pool, so it narrows candidates rather than assigning them.Tested on a live cluster
Run end to end on ACP v4.3 with three worker nodes, twice — once with
hostPathPVs, once withlocalPVs — plus a separate two-instance single-replica test.claimRefrecoveryclaimRefkafka-log*per diskTesting corrected several things the first draft got wrong:
log.dirsis<mount>/kafka-log<ownNodeId>, so a broker on a foreign disk never reads the other broker'smeta.properties— it formats an empty log beside the orphaned data and reportsRunning 1/1. Two of three brokers failed silently.node.idmatching itself. Replaced with countingkafka-log*directories per disk.deleteClaim: falsedoes not make reinstall safe. The KRaft cluster ID lives inKafka.status.clusterId; deleting the resource loses it and every broker rejects its retained disk. Added a record-and-restore procedure, including the reconcile trigger the status patch needs to survive.hostPathgets nofsGroupfrom kubelet, so Kafka's UID 1001 cannot write aroot:root 0755directory. Withlocalit works at default privileges. Concrete reason to preferlocal.KafkaNodePool— a pool template replacesKafka.spec.kafka.template.podentirely.Single-replica mode is covered too: RF=1 has no fault tolerance, and the cross-binding risk moves to the boundary between instances sharing a StorageClass. That case fails loudly (every single-replica instance is node 0, so it finds a foreign
kafka-log0and rejects the cluster ID) and the victim's data survives.Notes for reviewers
class/selectoron the node pool. Flagged as an open item.id:anddocs/zh/are produced by CI on push tomain.check-translation-integrity.mjs --sincepasses; the file is 44KB, under the 60KB chunking threshold.strimzi.io/*strings are API identifiers (CRD group, labels, annotations) that appear verbatim in user-authored YAML.Tracked by ECO-933.
🤖 Generated with Claude Code