An asynchronous Apache Kafka client with producer, consumer, group, transaction, and Admin APIs. This repository also contains an experimental broker and a Schema Registry companion.
[dependencies]
partitionline = "0.2"See CHANGELOG.md for what changed in 0.2.0 since
0.1.0, and
docs/RELEASE.md for publication gates on
crates.io. Until 0.2.0 finishes
publishing—or when testing unreleased main commits—you can depend on a local
checkout (adjust the relative path as needed):
[dependencies]
partitionline = { path = "../partitionline" }partitionline is an 0.x crate under active qualification (Suite HOLD
remains unsigned, and production rankings have not been established). See
current status for validation results and open work, and the
support matrix for tested Kafka versions and platforms.
Build with Rust 1.99 or newer (latest stable recommended). Building the TLS
dependency ring requires a C toolchain. The examples assume a running Kafka
broker and existing topics.
# async fn example() -> partitionline::Result<()> {
use partitionline::{ProduceRecord, Producer};
let producer = Producer::connect("127.0.0.1:9092").await?;
let md = producer
.send(ProduceRecord::to("events").value(&b"hello"[..]))
.await?;
println!("{}-{}@{}", md.topic, md.partition, md.offset);
producer.close().await?;
# Ok(())
# }send waits for that record's delivery result. A successful send_all returns
the results for multiple records; on error, other accepted records may still be
delivered. try_send queues without waiting; flush can return before some
retries finish or miss delivery timeouts. Use send for
an individual delivery result. With acks=0, completion means the request was
written locally; the broker provides no acknowledgement or offset.
# async fn example() -> partitionline::Result<()> {
use partitionline::Consumer;
let mut consumer = Consumer::connect("127.0.0.1:9092").await?;
consumer.assign("events", 0, 0).await?;
let recs = consumer.fetch().await?;
# let _ = recs;
# Ok(())
# }fetch and group poll return ConsumerRecords, with records grouped by
partition and the next offsets to commit. Share poll returns ShareRecords.
assign_topic assigns every partition. Use seek, pause, resume, and
wakeup to control fetching. Fetch talks to every partition leader, and
leader-epoch recovery uses OffsetForLeaderEpoch.
# async fn example() -> partitionline::Result<()> {
use partitionline::{ConsumerConfig, ConsumerGroup};
let mut group = ConsumerGroup::join_topics(
ConsumerConfig::bootstrap(["127.0.0.1:9092"]),
"workers",
["orders", "payments"],
)
.await?;
let recs = group.poll().await?;
group.commit_with_metadata(recs.next_offsets()).await?;
group.leave().await?;
# Ok(())
# }Classic range, sticky, cooperative-sticky (KIP-429), KIP-848
(join_consumer), and KIP-932 share groups (ShareGroup::join / poll /
accept / release / reject). auto_offset_reset is used when the group
has no committed offset (Earliest by default, unlike Java's latest).
# async fn example() -> partitionline::Result<()> {
use partitionline::{ProduceRecord, Producer, ProducerConfig};
let producer = Producer::new(
ProducerConfig::bootstrap(["127.0.0.1:9092"]).transactional_id("orders-txn"),
)
.await?;
producer.init_transactions().await?;
producer.begin_transaction().await?;
producer
.send(ProduceRecord::to("events").value(&b"hello"[..]))
.await?;
producer.commit_transaction().await?;
producer.close().await?;
# Ok(())
# }transactional.id implies idempotence. send_offsets_to_transaction takes
TopicPartition. send_offsets_for_group uses the group's
ConsumerGroupMetadata (see examples/eos.rs).
# async fn example() -> partitionline::Result<()> {
use partitionline::{Admin, NewTopic};
let mut admin = Admin::connect("127.0.0.1:9092").await?;
admin
.create_topics(&[NewTopic::new("events", 1, 1)], 30_000, false)
.await?;
let topics = admin.list_topics_with(false).await?;
# let _ = topics;
admin.close().await?;
# Ok(())
# }Admin includes topic, configuration, ACL, group, transaction, log-directory,
and quota operations. Supported versions and remaining gaps are listed in
protocol support.
Use typed configuration builders:
use std::time::Duration;
use partitionline::{Acks, Compression, IsolationLevel, ProducerConfig, Sasl};
let _cfg = ProducerConfig::bootstrap(["127.0.0.1:9092"])
.acks(Acks::All)
.linger(Duration::from_millis(5))
.compression(Compression::Lz4)
.sasl(Sasl::scram_sha256("alice", "secret"));
let _iso = IsolationLevel::ReadCommitted;Use TlsConfig for TLS and optional client certificates. SASL supports PLAIN,
SCRAM-SHA-256/512, OAUTHBEARER, and OIDC token acquisition. Automatic
reauthentication on existing client connections is still in progress; see
token and session lifecycle. Compression supports gzip
(zlib-rs by default; see gzip backends for the
default-features = false miniz_oxide fallback), snappy, lz4, and optional
zstd (features = ["zstd"]). The default partitioner hashes keys with murmur2
and uses round-robin for records without keys. Opt-in StickyPartitioner is
available via ProducerConfig::partitioner while independent qualification
continues.
Defaults that differ from Java:
auto.offset.resetisEarliest(Javalatest)allow.auto.create.topicsisfalse(Java consumertrue)delivery.timeout.msis 30s (Java 120s)max.block.msis 30s (Java 60s)
buffer.memory (32 MiB) and max.request.size (1 MiB) match Java.
retry.backoff.ms / reconnect.backoff.ms / connections.max.idle.ms /
metadata.max.age.ms / transaction.timeout.ms match Java.
Authoritative defaults reference: operator guide.
See the operator guide for recipes and troubleshooting, published API docs for methods and options, and the migration guide for porting from rust-rdkafka.
| Area | Support |
|---|---|
| Client APIs | Produce, fetch, groups, transactions, Admin, share groups |
| Compression | gzip, snappy, lz4; optional zstd |
| Security | TLS, mTLS, SASL PLAIN, SCRAM, OAUTHBEARER, OIDC |
| Diagnostics | Metrics snapshots and optional tracing spans |
| Schema formats | Separate companion adapters for Protobuf, Avro, and JSON Schema |
Kerberos is not implemented. The companion requires an application-selected serializer. See protocol support for version ranges and remaining gaps.
With a broker on 127.0.0.1:9092 and an existing topic named partitionline:
cargo run --release --example roundtrip
Also: produce, consume, group, txn, admin, sasl, oauth, tls, eos,
offsets, share, wakeup, pause, metrics, cooperative, intercept,
consume_intercept.
The repository includes produce, fetch, and latency benchmarks. For a local produce run:
COUNT=100000 WARMUP_SECS=0 PAYLOAD_BYTES=100 ACKS=1 LINGER_MS=5 KAFKA_TOPIC=plbench \
cargo run --release --example bench_produceCreate a dedicated topic and verify its record history before comparing results. This local sample uses seeded 100-byte payloads, 16-byte ID keys, and no warmup. Benchmark instructions and results describe the settings and limits of recorded runs. They do not establish a fastest-client or fastest-broker claim; qualification remains in progress.
The documentation index links to guides, reference material, benchmarks, and implementation plans.
MIT OR Apache-2.0