High-performance Async Write-Ahead Log DB

Note: I AM BUILDING THIS PROJECT, BECAUSE I FUMBLED AND HUMILIATED MYSELF IN INTERVIEW QUESTIONS ABOUT TOKIO, AND I AM SAD N LONELY.

High-level design (last updated 09/19/2026)

Architecture more or less resembles Bitcask's design

Network protocol

• Request

#[repr(C)]
pub enum Operation {
    Stats = 0x00,
    Ping = 0x01,
    Close = 0x02,
    Create = 0x03,
    Info = 0x04,
    Drop = 0x05,
    Get = 0x06,
    Set = 0x07,
    SetIfExists = 0x08,
    SetIfNotExists = 0x09,
    CompareSet = 0x10,
    Delete = 0x11,
    CompareDelete = 0x12,
    WriteBatch = 0x13,
}

See src/lib.rs for more details on payload framing.

• Response

pub enum Response {
    Ok(Bytes) = 0x00,
    Pong(Bytes) = 0x01,
    DbNotFound(Bytes) = 0x02,
    DbAlreadyExists(Bytes) = 0x03,
    KeyValueToken(Bytes) = 0x04,
    KeyNotFound(Bytes) = 0x05,
    KeyAlreadyExists(Bytes) = 0x06,
    KeyTokenMismatch(Bytes) = 0x07,
    InvalidRequest(Bytes) = 0x08,
    PayloadTooLarge(Bytes) = 0x09,
    InternalError(Bytes) = 0x10,
}

All response are framed as (1B status|4B payload_length LE|payload LE), see src/protocol.rs for more details.

For mini-client library, copy and paste src/protocol.rs and MiniClient struct from src/lib.rs into your project.

Benchmarks

• Startup performance

mini-db start
[14:14:39][INFO] Performing necessary initialization...
[14:14:39][DEBUG] Indexing WAL directories from: "./mini-data/"
[14:14:39][DEBUG] Initializing WAL/Index and spawning DbWorkers for: 'default' (size: 215.255 MB)
[14:14:39][DEBUG] Initializing WAL/Index and spawning DbWorkers for: 'foobar' (size: 116.830 MB)
[14:14:39][DEBUG] WAL index build for: 'foobar' in 324.21ms (1 segments, 780288 entries)
[14:14:39][DEBUG] WAL index build for: 'default' in 558.97ms (2 segments, 1437650 entries)
[14:14:39][DEBUG] Indexed 2 databases in 560.35ms
[14:14:39][INFO] Server listening on 127.0.0.1:8787 in 562.1671ms
[14:14:48][INFO] Shutdown signal received, stopping server...
[14:14:48][INFO] Waiting for 0 active connections to complete...
[14:14:48][INFO] Server has stopped

• Compaction performance

mini-db compact
Compacting segmented WAL files...
Compaction completed for database: foobar
Compaction completed for database: default
Compaction for 2 completed in 595.79ms.
Disk freed: 307.085 MB (before: 332.085 MB, after: 25.000 MB)

mini-db start
[14:14:55][INFO] Performing necessary initialization...
[14:14:55][DEBUG] Indexing WAL directories from: "./mini-data/"
[14:14:55][DEBUG] Initializing WAL/Index and spawning DbWorkers for: 'default' (size: 17.500 MB)
[14:14:55][DEBUG] Initializing WAL/Index and spawning DbWorkers for: 'foobar' (size: 7.500 MB)
[14:14:55][DEBUG] Compact index loaded for database: 'foobar' (size: 7.500 MB, KV count: 49152)
[14:14:55][DEBUG] Compact index loaded for database: 'default' (size: 17.500 MB, KV count: 114688)
[14:14:55][DEBUG] Indexed 2 databases in 21.95ms
[14:14:55][INFO] Server listening on 127.0.0.1:8787 in 23.8502ms

• Read/Write performance (E2E)

mini-bench --workload read-overwrite

Mini-db e2e benchmark
  server:    127.0.0.1:8787
  db:        default
  workload:  ReadOverwrite
  clients:   32
  ops:       256000
  keyspace:  16384
  key/value: 16/128 bytes
  mix:       80% GET / 20% SET
  warmup:    4096

Prefilling 16384 keys... done in 1.39s
Warming up with 4096 operations... done in 35.66ms
Running 256000 sampled operations

Results
  completed:  256000
  elapsed:    2.214 s
  throughput: 115634 ops/s
  min:        22.000 us
  mean:       275.779 us
  p50:        242.000 us
  p90:        465.500 us
  p95:        558.900 us
  p99:        717.600 us
  p99.9:      966.500 us
  max:        1.540 ms
  
mini-bench --workload write

Mini-db e2e benchmark
  server:    127.0.0.1:8787
  db:        default
  workload:  Write
  clients:   32
  ops:       256000
  keyspace:  16384
  key/value: 16/128 bytes
  warmup:    4096

Warming up with 4096 operations... done in 75.14ms
Running 256000 sampled operations

Results
  completed:  256000
  elapsed:    4.693 s
  throughput: 54546 ops/s
  min:        60.900 us
  mean:       586.256 us
  p50:        610.400 us
  p90:        814.900 us
  p95:        882.400 us
  p99:        1.099 ms
  p99.9:      1.309 ms
  max:        2.118 ms

# last updated 10/24/2026

TODO

• Multiple DB workers (Sharding and Synchronization)
• Faster path to memory index for read operations (Rwlock)
• Multiple segmented WALs & Compaction [DONE]
• Group (Batch) commits and Transaction support [DONE]
• Solve Infinite RAM growth
• Rollback and backup support
• Better & safer data compaction/checkpointing
• More Analytics and more read optimizations
• Multi-tenancy and proper userspace [DONE]
• Implement data types (int, float, bool, string, list, bytes)
• OLAP features expansion (e.g., aggregation, filtering, sorting) alongside OLTP features

Features stealing

• Postgres: Mvcc, page caching
• LevelDB: Lsm tree, compaction
• Redis: Per-key TTL, eviction policies, command & operation pipelining
• ClickHouse/DuckDB: Columnar storage (MergeTree), vectorized execution, aggregation, filtering, pruning, sorting, etc
• MySQL: Query parser + planner