Skip to content

About

An Apache Kafka client and protocol implementation

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

partitionline

An asynchronous Apache Kafka client with producer, consumer, group, transaction, and Admin APIs. This repository also contains an experimental broker and a Schema Registry companion.

ci crates.io docs.rs

[dependencies]
partitionline = "0.2"

See CHANGELOG.md for what changed in 0.2.0 since 0.1.0, and docs/RELEASE.md for publication gates on crates.io. Until 0.2.0 finishes publishing—or when testing unreleased main commits—you can depend on a local checkout (adjust the relative path as needed):

[dependencies]
partitionline = { path = "../partitionline" }

partitionline is an 0.x crate under active qualification (Suite HOLD remains unsigned, and production rankings have not been established). See current status for validation results and open work, and the support matrix for tested Kafka versions and platforms.

Build with Rust 1.99 or newer (latest stable recommended). Building the TLS dependency ring requires a C toolchain. The examples assume a running Kafka broker and existing topics.

Produce

# async fn example() -> partitionline::Result<()> {
use partitionline::{ProduceRecord, Producer};

let producer = Producer::connect("127.0.0.1:9092").await?;
let md = producer
    .send(ProduceRecord::to("events").value(&b"hello"[..]))
    .await?;
println!("{}-{}@{}", md.topic, md.partition, md.offset);
producer.close().await?;
# Ok(())
# }

send waits for that record's delivery result. A successful send_all returns the results for multiple records; on error, other accepted records may still be delivered. try_send queues without waiting; flush can return before some retries finish or miss delivery timeouts. Use send for an individual delivery result. With acks=0, completion means the request was written locally; the broker provides no acknowledgement or offset.

Fetch

# async fn example() -> partitionline::Result<()> {
use partitionline::Consumer;

let mut consumer = Consumer::connect("127.0.0.1:9092").await?;
consumer.assign("events", 0, 0).await?;
let recs = consumer.fetch().await?;
# let _ = recs;
# Ok(())
# }

fetch and group poll return ConsumerRecords, with records grouped by partition and the next offsets to commit. Share poll returns ShareRecords. assign_topic assigns every partition. Use seek, pause, resume, and wakeup to control fetching. Fetch talks to every partition leader, and leader-epoch recovery uses OffsetForLeaderEpoch.

Groups

# async fn example() -> partitionline::Result<()> {
use partitionline::{ConsumerConfig, ConsumerGroup};

let mut group = ConsumerGroup::join_topics(
    ConsumerConfig::bootstrap(["127.0.0.1:9092"]),
    "workers",
    ["orders", "payments"],
)
.await?;
let recs = group.poll().await?;
group.commit_with_metadata(recs.next_offsets()).await?;
group.leave().await?;
# Ok(())
# }

Classic range, sticky, cooperative-sticky (KIP-429), KIP-848 (join_consumer), and KIP-932 share groups (ShareGroup::join / poll / accept / release / reject). auto_offset_reset is used when the group has no committed offset (Earliest by default, unlike Java's latest).

Transactions

# async fn example() -> partitionline::Result<()> {
use partitionline::{ProduceRecord, Producer, ProducerConfig};

let producer = Producer::new(
    ProducerConfig::bootstrap(["127.0.0.1:9092"]).transactional_id("orders-txn"),
)
.await?;
producer.init_transactions().await?;
producer.begin_transaction().await?;
producer
    .send(ProduceRecord::to("events").value(&b"hello"[..]))
    .await?;
producer.commit_transaction().await?;
producer.close().await?;
# Ok(())
# }

transactional.id implies idempotence. send_offsets_to_transaction takes TopicPartition. send_offsets_for_group uses the group's ConsumerGroupMetadata (see examples/eos.rs).

Admin

# async fn example() -> partitionline::Result<()> {
use partitionline::{Admin, NewTopic};

let mut admin = Admin::connect("127.0.0.1:9092").await?;
admin
    .create_topics(&[NewTopic::new("events", 1, 1)], 30_000, false)
    .await?;
let topics = admin.list_topics_with(false).await?;
# let _ = topics;
admin.close().await?;
# Ok(())
# }

Admin includes topic, configuration, ACL, group, transaction, log-directory, and quota operations. Supported versions and remaining gaps are listed in protocol support.

Configure

Use typed configuration builders:

use std::time::Duration;
use partitionline::{Acks, Compression, IsolationLevel, ProducerConfig, Sasl};

let _cfg = ProducerConfig::bootstrap(["127.0.0.1:9092"])
    .acks(Acks::All)
    .linger(Duration::from_millis(5))
    .compression(Compression::Lz4)
    .sasl(Sasl::scram_sha256("alice", "secret"));
let _iso = IsolationLevel::ReadCommitted;

Use TlsConfig for TLS and optional client certificates. SASL supports PLAIN, SCRAM-SHA-256/512, OAUTHBEARER, and OIDC token acquisition. Automatic reauthentication on existing client connections is still in progress; see token and session lifecycle. Compression supports gzip (zlib-rs by default; see gzip backends for the default-features = false miniz_oxide fallback), snappy, lz4, and optional zstd (features = ["zstd"]). The default partitioner hashes keys with murmur2 and uses round-robin for records without keys. Opt-in StickyPartitioner is available via ProducerConfig::partitioner while independent qualification continues.

Defaults that differ from Java:

  • auto.offset.reset is Earliest (Java latest)
  • allow.auto.create.topics is false (Java consumer true)
  • delivery.timeout.ms is 30s (Java 120s)
  • max.block.ms is 30s (Java 60s)

buffer.memory (32 MiB) and max.request.size (1 MiB) match Java. retry.backoff.ms / reconnect.backoff.ms / connections.max.idle.ms / metadata.max.age.ms / transaction.timeout.ms match Java. Authoritative defaults reference: operator guide.

See the operator guide for recipes and troubleshooting, published API docs for methods and options, and the migration guide for porting from rust-rdkafka.

Capabilities

Area Support
Client APIs Produce, fetch, groups, transactions, Admin, share groups
Compression gzip, snappy, lz4; optional zstd
Security TLS, mTLS, SASL PLAIN, SCRAM, OAUTHBEARER, OIDC
Diagnostics Metrics snapshots and optional tracing spans
Schema formats Separate companion adapters for Protobuf, Avro, and JSON Schema

Kerberos is not implemented. The companion requires an application-selected serializer. See protocol support for version ranges and remaining gaps.

Examples

With a broker on 127.0.0.1:9092 and an existing topic named partitionline:

cargo run --release --example roundtrip

Also: produce, consume, group, txn, admin, sasl, oauth, tls, eos, offsets, share, wakeup, pause, metrics, cooperative, intercept, consume_intercept.

Performance

The repository includes produce, fetch, and latency benchmarks. For a local produce run:

COUNT=100000 WARMUP_SECS=0 PAYLOAD_BYTES=100 ACKS=1 LINGER_MS=5 KAFKA_TOPIC=plbench \
  cargo run --release --example bench_produce

Create a dedicated topic and verify its record history before comparing results. This local sample uses seeded 100-byte payloads, 16-byte ID keys, and no warmup. Benchmark instructions and results describe the settings and limits of recorded runs. They do not establish a fastest-client or fastest-broker claim; qualification remains in progress.

Documentation

The documentation index links to guides, reference material, benchmarks, and implementation plans.

License

MIT OR Apache-2.0

About

An Apache Kafka client and protocol implementation

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages