TidesDB Rust API Reference

If you want to download the source of this document, you can find it here.

---

Getting Started

Prerequisites

You must have the TidesDB shared C library installed on your system. You can find the installation instructions here.

Building from GitHub

To build the library directly from the GitHub repository:

# Clone the repository
git clone https://github.com/tidesdb/tidesdb-rust.git
cd tidesdb-rust

# Build the library
cargo build --release

# Run tests
cargo test -- --test-threads=1

# Install locally (optional)
cargo install --path .

Using as a local dependency

You can reference the local build in your project’s Cargo.toml:

[dependencies]
tidesdb = { path = "/path/to/tidesdb-rust" }

Using directly from GitHub

You can also add the dependency directly from GitHub:

[dependencies]
tidesdb = { git = "https://github.com/tidesdb/tidesdb-rust.git" }

# Or pin to a specific branch
tidesdb = { git = "https://github.com/tidesdb/tidesdb-rust.git", branch = "main" }

# Or pin to a specific tag/version
tidesdb = { git = "https://github.com/tidesdb/tidesdb-rust.git", tag = "v0.2.0" }

# Or pin to a specific commit
tidesdb = { git = "https://github.com/tidesdb/tidesdb-rust.git", rev = "abc123" }

Custom Installation Paths

If you installed TidesDB to a non-standard location, you can specify custom paths using environment variables:

# Set custom library path
export LIBRARY_PATH="/custom/path/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="/custom/path/lib:$LD_LIBRARY_PATH"  # Linux
# or
export DYLD_LIBRARY_PATH="/custom/path/lib:$DYLD_LIBRARY_PATH"  # macOS

# Then build
cargo build

Using pkg-config

# If TidesDB was installed with pkg-config support
export PKG_CONFIG_PATH="/custom/path/lib/pkgconfig:$PKG_CONFIG_PATH"
cargo build

Custom prefix installation

# Install TidesDB to custom location
cd tidesdb
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/opt/tidesdb
cmake --build build
sudo cmake --install build

# Configure environment
export LIBRARY_PATH="/opt/tidesdb/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="/opt/tidesdb/lib:$LD_LIBRARY_PATH"  # Linux
# or
export DYLD_LIBRARY_PATH="/opt/tidesdb/lib:$DYLD_LIBRARY_PATH"  # macOS

cargo build

Initialization

TidesDB supports optional custom memory allocators for integration with custom memory managers (e.g., jemalloc, mimalloc).

init

Initializes TidesDB with the system allocator. Must be called exactly once before any other TidesDB function when using the explicit initialization path.

use tidesdb;

fn main() -> tidesdb::Result<()> {
    tidesdb::init()?;

    // ... use TidesDB ...

    tidesdb::finalize();
    Ok(())
}

init_with_allocator

Initializes TidesDB with custom C-level memory allocator functions. This is an unsafe function for advanced use cases.

use tidesdb;

// Example with custom allocator function pointers
unsafe {
    tidesdb::init_with_allocator(
        Some(my_malloc),
        Some(my_calloc),
        Some(my_realloc),
        Some(my_free),
    )?;
}

finalize

Finalizes TidesDB and resets the allocator. Should be called after all TidesDB operations are complete (all databases closed). After calling this, init() or init_with_allocator() can be called again.

tidesdb::finalize();

Auto-initialization

If init() is not called, TidesDB will auto-initialize with the system allocator on the first call to TidesDB::open().

Usage

Opening and Closing a Database

use tidesdb::{TidesDB, Config, LogLevel};

fn main() -> tidesdb::Result<()> {
    let config = Config::new("./mydb")
        .num_flush_threads(2)
        .num_compaction_threads(2)
        .log_level(LogLevel::Info)
        .block_cache_size(64 * 1024 * 1024)
        .max_open_sstables(256)
        .max_memory_usage(0)                   // 0 = auto (50% of system RAM)
        .log_to_file(false)                    // Write logs to file instead of stderr
        .log_truncation_at(24 * 1024 * 1024);  // Log file truncation threshold (24MB)

    let db = TidesDB::open(config)?;

    println!("Database opened successfully");

    Ok(())
}

Creating and Dropping Column Families

Column families are isolated key-value stores with independent configuration.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, CompressionAlgorithm, SyncMode, IsolationLevel};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    let cf_config = ColumnFamilyConfig::default();
    db.create_column_family("my_cf", cf_config)?;

    let cf_config = ColumnFamilyConfig::new()
        .write_buffer_size(128 * 1024 * 1024)
        .level_size_ratio(10)
        .min_levels(5)
        .compression_algorithm(CompressionAlgorithm::Lz4)
        .enable_bloom_filter(true)
        .bloom_fpr(0.01)
        .enable_block_indexes(true)
        .sync_mode(SyncMode::Interval)
        .sync_interval_us(128000)
        .default_isolation_level(IsolationLevel::ReadCommitted)
        .use_btree(false); // Use block-based format (default)

    db.create_column_family("custom_cf", cf_config)?;

    db.drop_column_family("my_cf")?;

    Ok(())
}

Dropping by Pointer

When you already hold a ColumnFamily, you can skip the name lookup:

let cf = db.get_column_family("my_cf")?;
db.delete_column_family(cf)?; // cf is consumed and cannot be used after this

Which to use

• drop_column_family(name) · Convenient when you only have the name
• delete_column_family(cf) · Faster when you already hold a ColumnFamily, avoids a redundant linear scan

CRUD Operations

All operations in TidesDB are performed through transactions for ACID guarantees.

Writing Data

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;

    txn.put(&cf, b"key", b"value", -1)?;

    txn.commit()?;

    Ok(())
}

Writing with TTL

use std::time::{SystemTime, UNIX_EPOCH};
use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;

    let ttl = SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .unwrap()
        .as_secs() as i64 + 10; // Expires in 10 seconds

    txn.put(&cf, b"temp_key", b"temp_value", ttl)?;

    txn.commit()?;

    Ok(())
}

TTL Examples

use std::time::{SystemTime, UNIX_EPOCH, Duration};

let ttl: i64 = -1;

let ttl = SystemTime::now()
    .duration_since(UNIX_EPOCH)
    .unwrap()
    .as_secs() as i64 + 5 * 60;

let ttl = SystemTime::now()
    .duration_since(UNIX_EPOCH)
    .unwrap()
    .as_secs() as i64 + 60 * 60;

let ttl: i64 = 1798761599;

Reading Data

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let txn = db.begin_transaction()?;

    let value = txn.get(&cf, b"key")?;
    println!("Value: {:?}", String::from_utf8_lossy(&value));

    Ok(())
}

Deleting Data

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;
    txn.delete(&cf, b"key")?;
    txn.commit()?;

    Ok(())
}

Multi-Operation Transactions

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;

    txn.put(&cf, b"key1", b"value1", -1)?;
    txn.put(&cf, b"key2", b"value2", -1)?;
    txn.delete(&cf, b"old_key")?;

    txn.commit()?;

    Ok(())
}

Transaction Rollback

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;
    txn.put(&cf, b"key", b"value", -1)?;

    txn.rollback()?;

    Ok(())
}

Multi-Column-Family Transactions

TidesDB supports atomic transactions across multiple column families with true all-or-nothing semantics.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("users", ColumnFamilyConfig::default())?;
    db.create_column_family("orders", ColumnFamilyConfig::default())?;

    let users_cf = db.get_column_family("users")?;
    let orders_cf = db.get_column_family("orders")?;

    let mut txn = db.begin_transaction()?;

    txn.put(&users_cf, b"user:1000", b"John Doe", -1)?;
    txn.put(&orders_cf, b"order:5000", b"user:1000|product:A", -1)?;

    // Commit atomically -- all or nothing
    txn.commit()?;

    Ok(())
}

Multi-CF guarantees

• Either all CFs commit or none do (atomic)
• Automatically detected when operations span multiple CFs
• Uses global sequence numbers for atomic ordering
• Each CF’s WAL receives operations with the same commit sequence number

Transaction Reset

Transaction::reset resets a committed or aborted transaction for reuse with a new isolation level. This avoids the overhead of freeing and reallocating transaction resources in hot loops.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, IsolationLevel};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;

    txn.put(&cf, b"key1", b"value1", -1)?;
    txn.commit()?;

    txn.reset(IsolationLevel::ReadCommitted)?;

    txn.put(&cf, b"key2", b"value2", -1)?;
    txn.commit()?;

    Ok(())
}

Behavior

• The transaction must be committed or aborted before reset; resetting an active transaction returns an error
• Internal buffers are retained to avoid reallocation
• A fresh transaction ID and snapshot sequence are assigned based on the new isolation level
• The isolation level can be changed on each reset (e.g., ReadCommitted → RepeatableRead)

When to use

• Batch processing · Reuse a single transaction across many commit cycles in a loop
• Connection pooling · Reset a transaction for a new request without reallocation
• High-throughput ingestion · Reduce allocation overhead in tight write loops

Reset vs Drop + Begin

For a single transaction, reset is functionally equivalent to dropping the transaction and calling begin_transaction_with_isolation. The difference is performance: reset retains allocated buffers and avoids repeated allocation overhead. This matters most in loops that commit and restart thousands of transactions.

Iterating Over Data

Iterators provide efficient bidirectional traversal over key-value pairs.

Forward Iteration

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let txn = db.begin_transaction()?;
    let mut iter = txn.new_iterator(&cf)?;

    iter.seek_to_first()?;

    while iter.is_valid() {
        let key = iter.key()?;
        let value = iter.value()?;

        println!("Key: {:?}, Value: {:?}",
            String::from_utf8_lossy(&key),
            String::from_utf8_lossy(&value));

        iter.next()?;
    }

    Ok(())
}

Backward Iteration

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let txn = db.begin_transaction()?;
    let mut iter = txn.new_iterator(&cf)?;

    iter.seek_to_last()?;

    while iter.is_valid() {
        let key = iter.key()?;
        let value = iter.value()?;

        println!("Key: {:?}, Value: {:?}",
            String::from_utf8_lossy(&key),
            String::from_utf8_lossy(&value));

        iter.prev()?;
    }

    Ok(())
}

Seek Operations

seek(key) positions the iterator at the first key >= target key:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let txn = db.begin_transaction()?;
    let mut iter = txn.new_iterator(&cf)?;

    // Seek to prefix and iterate all matching keys
    iter.seek(b"user:")?;

    while iter.is_valid() {
        let key = iter.key()?;

        if !key.starts_with(b"user:") {
            break;
        }

        let value = iter.value()?;
        println!("Key: {:?}, Value: {:?}",
            String::from_utf8_lossy(&key),
            String::from_utf8_lossy(&value));

        iter.next()?;
    }

    Ok(())
}

seek_for_prev(key) positions the iterator at the last key <= target key:

let txn = db.begin_transaction()?;
let mut iter = txn.new_iterator(&cf)?;

// Seek for reverse iteration from a specific key
iter.seek_for_prev(b"user:2000")?;

while iter.is_valid() {
    let key = iter.key()?;
    let value = iter.value()?;
    println!("Key: {:?}, Value: {:?}",
        String::from_utf8_lossy(&key),
        String::from_utf8_lossy(&value));
    iter.prev()?;
}

Getting Column Family Statistics

Retrieve detailed statistics about a column family.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let stats = cf.get_stats()?;

    println!("Number of Levels: {}", stats.num_levels);
    println!("Memtable Size: {} bytes", stats.memtable_size);
    println!("Total Keys: {}", stats.total_keys);
    println!("Total Data Size: {} bytes", stats.total_data_size);
    println!("Average Key Size: {:.2} bytes", stats.avg_key_size);
    println!("Average Value Size: {:.2} bytes", stats.avg_value_size);
    println!("Read Amplification: {:.2}", stats.read_amp);
    println!("Hit Rate: {:.1}%", stats.hit_rate * 100.0);

    for (i, (size, count)) in stats.level_sizes.iter()
        .zip(stats.level_num_sstables.iter())
        .enumerate()
    {
        println!("Level {}: {} SSTables, {} bytes", i + 1, count, size);
    }

    // B+tree stats (only populated if use_btree=true)
    if stats.use_btree {
        println!("B+tree Total Nodes: {}", stats.btree_total_nodes);
        println!("B+tree Max Height: {}", stats.btree_max_height);
        println!("B+tree Avg Height: {:.2}", stats.btree_avg_height);
    }

    Ok(())
}

Statistics Fields

Field	Type	Description
num_levels	i32	Number of LSM levels
memtable_size	usize	Current memtable size in bytes
level_sizes	Vec<usize>	Array of per-level total sizes
level_num_sstables	Vec<i32>	Array of per-level SSTable counts
level_key_counts	Vec<u64>	Array of per-level key counts
config	Option<ColumnFamilyConfig>	Column family configuration
total_keys	u64	Total keys across memtable and all SSTables
total_data_size	u64	Total data size (klog + vlog) in bytes
avg_key_size	f64	Estimated average key size in bytes
avg_value_size	f64	Estimated average value size in bytes
read_amp	f64	Read amplification factor (point lookup cost)
hit_rate	f64	Block cache hit rate (0.0 to 1.0)
use_btree	bool	Whether column family uses B+tree KLog format
btree_total_nodes	u64	Total B+tree nodes across all SSTables
btree_max_height	u32	Maximum tree height across all SSTables
btree_avg_height	f64	Average tree height across all SSTables

Getting Cache Statistics

Retrieve statistics about the global block cache.

use tidesdb::{TidesDB, Config};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    let stats = db.get_cache_stats()?;

    if stats.enabled {
        println!("Cache enabled: yes");
        println!("Total entries: {}", stats.total_entries);
        println!("Total bytes: {:.2} MB", stats.total_bytes as f64 / (1024.0 * 1024.0));
        println!("Hits: {}", stats.hits);
        println!("Misses: {}", stats.misses);
        println!("Hit rate: {:.1}%", stats.hit_rate * 100.0);
        println!("Partitions: {}", stats.num_partitions);
    } else {
        println!("Cache enabled: no");
    }

    Ok(())
}

Range Cost Estimation

Estimate the computational cost of iterating between two keys in a column family. The returned value is an opaque double - meaningful only for comparison with other range_cost results. It uses only in-memory metadata and performs no disk I/O.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let cost_a = cf.range_cost(b"user:0000", b"user:0999")?;
    let cost_b = cf.range_cost(b"user:1000", b"user:1099")?;

    if cost_a < cost_b {
        println!("Range A is cheaper to iterate");
    }

    Ok(())
}

Behavior

• Key order does not matter - the function normalizes the range so key_a > key_b produces the same result as key_b > key_a
• A cost of 0.0 means no overlapping SSTables or memtable entries were found for the range
• With block indexes enabled, uses O(log B) binary search per overlapping SSTable
• Without block indexes, falls back to byte-level key interpolation
• B+tree SSTables use key interpolation against tree node counts plus tree height as seek cost
• Compressed SSTables receive a 1.5× weight multiplier for decompression overhead

Use cases

• Query planning · Compare candidate key ranges to find the cheapest one to scan
• Load balancing · Distribute range scan work across threads by estimating per-range cost
• Adaptive prefetching · Decide how aggressively to prefetch based on range size
• Monitoring · Track how data distribution changes across key ranges over time

Cost Values

The returned cost is not an absolute measure (it does not represent milliseconds, bytes, or entry counts). It is a relative scalar - only meaningful when compared with other range_cost results.

Listing Column Families

use tidesdb::{TidesDB, Config};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    let cf_list = db.list_column_families()?;

    println!("Available column families:");
    for name in cf_list {
        println!("  - {}", name);
    }

    Ok(())
}

Renaming Column Families

Atomically rename a column family and its underlying directory:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("old_name", ColumnFamilyConfig::default())?;

    // Rename column family (waits for flush/compaction to complete)
    db.rename_column_family("old_name", "new_name")?;

    // Access with new name
    let cf = db.get_column_family("new_name")?;

    Ok(())
}

Cloning Column Families

Create a complete copy of an existing column family with a new name. The clone contains all the data from the source at the time of cloning.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("source_cf", ColumnFamilyConfig::default())?;

    // Insert some data into source
    let cf = db.get_column_family("source_cf")?;
    let mut txn = db.begin_transaction()?;
    txn.put(&cf, b"key1", b"value1", -1)?;
    txn.put(&cf, b"key2", b"value2", -1)?;
    txn.commit()?;

    // Clone the column family
    db.clone_column_family("source_cf", "cloned_cf")?;

    // Both column families now exist independently
    let cloned_cf = db.get_column_family("cloned_cf")?;
    let txn = db.begin_transaction()?;
    let value = txn.get(&cloned_cf, b"key1")?;
    println!("Cloned value: {:?}", String::from_utf8_lossy(&value));

    Ok(())
}

Behavior

• Flushes the source column family’s memtable to ensure all data is on disk
• Waits for any in-progress flush or compaction to complete
• Copies all SSTable files to the new directory
• The clone is completely independent — modifications to one do not affect the other

Use cases

• Testing · Create a copy of production data for testing without affecting the original
• Branching · Create a snapshot of data before making experimental changes
• Migration · Clone data before schema or configuration changes

Return values

• Ok(()) · Clone completed successfully
• ErrorCode::NotFound · Source column family doesn’t exist
• ErrorCode::Exists · Destination column family already exists
• ErrorCode::InvalidArgs · Invalid arguments (same source/destination name)
• ErrorCode::Io · Failed to copy files or create directory

Compaction

Manual Compaction

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    // Manually trigger compaction (queues compaction from L1+)
    cf.compact()?;

    Ok(())
}

Manual Memtable Flush

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    // Manually trigger memtable flush (Queues sorted run for L1)
    cf.flush_memtable()?;

    Ok(())
}

Checking Flush/Compaction Status

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    // Check if operations are in progress
    if cf.is_flushing() {
        println!("Flush operation in progress");
    }

    if cf.is_compacting() {
        println!("Compaction operation in progress");
    }

    Ok(())
}

Database Backup

use tidesdb::{TidesDB, Config};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    // Create a backup to the specified directory
    db.backup("./mydb_backup")?;

    Ok(())
}

Database Checkpoint

checkpoint creates a lightweight, near-instant snapshot of an open database using hard links instead of copying SSTable data.

use tidesdb::{TidesDB, Config};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    // Create a checkpoint to the specified directory
    db.checkpoint("./mydb_checkpoint")?;

    Ok(())
}

Behavior

• Requires checkpoint_dir to be a non-existent or empty directory
• For each column family:
  • Flushes the active memtable so all data is in SSTables
  • Halts compactions to ensure a consistent view of live SSTable files
  • Hard links all SSTable files (.klog and .vlog) into the checkpoint directory
  • Copies small metadata files (manifest, config) into the checkpoint directory
  • Resumes compactions
• Falls back to file copy if hard linking fails (e.g., cross-filesystem)
• Database stays open and usable during checkpoint

Checkpoint vs Backup

	backup	checkpoint
Speed	Copies every SSTable byte-by-byte	Near-instant (hard links, O(1) per file)
Disk usage	Full independent copy	No extra disk until compaction removes old SSTables
Portability	Can be moved to another filesystem or machine	Same filesystem only (hard link requirement)
Use case	Archival, disaster recovery, remote shipping	Fast local snapshots, point-in-time reads, streaming backups

Notes

• The checkpoint represents the database state at the point all memtables are flushed and compactions are halted
• Hard-linked files share storage with the live database. Deleting the original database does not affect the checkpoint (hard link semantics)
• The checkpoint can be opened as a normal TidesDB database with TidesDB::open

Runtime Configuration Updates

Update column family configuration at runtime:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, CompressionAlgorithm};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    // Create new configuration
    let new_config = ColumnFamilyConfig::new()
        .write_buffer_size(256 * 1024 * 1024)
        .compression_algorithm(CompressionAlgorithm::Zstd);

    // Update runtime config (persist_to_disk = true to save changes)
    cf.update_runtime_config(&new_config, true)?;

    Ok(())
}

Commit Hook (Change Data Capture)

ColumnFamily::set_commit_hook registers a callback that fires synchronously after every transaction commit on a column family. The hook receives the full batch of committed operations atomically, enabling real-time change data capture without WAL parsing or external log consumers.

use std::sync::{Arc, Mutex};
use tidesdb::{TidesDB, Config, ColumnFamilyConfig, CommitOp};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let mut cf = db.get_column_family("my_cf")?;

    // Track all committed operations
    let log: Arc<Mutex<Vec<CommitOp>>> = Arc::new(Mutex::new(Vec::new()));
    let log_clone = log.clone();

    cf.set_commit_hook(move |ops, commit_seq| {
        println!("Commit seq {}: {} ops", commit_seq, ops.len());
        let mut l = log_clone.lock().unwrap();
        for op in ops {
            l.push(op.clone());
        }
        0 // return 0 on success
    })?;

    // Normal writes now trigger the hook automatically
    let mut txn = db.begin_transaction()?;
    txn.put(&cf, b"key1", b"value1", -1)?;
    txn.put(&cf, b"key2", b"value2", -1)?;
    txn.commit()?; // hook fires here

    // Detach hook
    cf.clear_commit_hook()?;

    Ok(())
}

CommitOp fields

Field	Type	Description
key	Vec<u8>	The key
value	Option<Vec<u8>>	The value (None for deletes)
ttl	i64	TTL as Unix timestamp (0 = no expiry)
is_delete	bool	Whether this is a delete operation

Behavior

• The hook fires after WAL write, memtable apply, and commit status marking are complete - the data is fully durable before the callback runs
• Hook failure (non-zero return) is logged but does not affect the commit result
• Each column family has its own independent hook; a multi-CF transaction fires the hook once per CF with only that CF’s operations
• commit_seq is monotonically increasing across commits and can be used as a replication cursor
• The hook executes synchronously on the committing thread; keep the callback fast to avoid stalling writers
• Calling set_commit_hook again replaces the previous hook (the old callback is freed automatically)
• Calling clear_commit_hook or dropping the ColumnFamily disables the hook immediately

Use cases

• Replication · Ship committed batches to replicas in commit order
• Event streaming · Publish mutations to Kafka, NATS, or any message broker
• Secondary indexing · Maintain a reverse index or materialized view
• Audit logging · Record every mutation with key, value, TTL, and sequence number
• Debugging · Attach a temporary hook in production to inspect live writes

Runtime-Only

Commit hooks are not persisted to config.ini. After a database restart, hooks must be re-registered by the application. This is by design - closures cannot be serialized.

INI Configuration Files

Load and save column family configurations from/to INI files:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    // Load configuration from INI file
    let cf_config = ColumnFamilyConfig::load_from_ini("config.ini", "my_column_family")?;

    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", cf_config.clone())?;

    // Save configuration to INI file
    cf_config.save_to_ini("config_backup.ini", "my_column_family")?;

    Ok(())
}

Sync Modes

Control the durability vs performance tradeoff.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, SyncMode};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    // SyncMode::None -- Fastest, least durable (OS handles flushing)
    let cf_config = ColumnFamilyConfig::new()
        .sync_mode(SyncMode::None);
    db.create_column_family("fast_cf", cf_config)?;

    // SyncMode::Interval -- Balanced (periodic background syncing)
    let cf_config = ColumnFamilyConfig::new()
        .sync_mode(SyncMode::Interval)
        .sync_interval_us(128000); // Sync every 128ms
    db.create_column_family("balanced_cf", cf_config)?;

    // SyncMode::Full -- Most durable (fsync on every write)
    let cf_config = ColumnFamilyConfig::new()
        .sync_mode(SyncMode::Full);
    db.create_column_family("durable_cf", cf_config)?;

    Ok(())
}

Compression Algorithms

TidesDB supports multiple compression algorithms:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, CompressionAlgorithm};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    // No compression
    let cf_config = ColumnFamilyConfig::new()
        .compression_algorithm(CompressionAlgorithm::None);

    // LZ4 compression (default, balanced)
    let cf_config = ColumnFamilyConfig::new()
        .compression_algorithm(CompressionAlgorithm::Lz4);

    // LZ4 fast compression (faster, slightly lower ratio)
    let cf_config = ColumnFamilyConfig::new()
        .compression_algorithm(CompressionAlgorithm::Lz4Fast);

    // Zstandard compression (best ratio)
    let cf_config = ColumnFamilyConfig::new()
        .compression_algorithm(CompressionAlgorithm::Zstd);

    // Snappy compression
    let cf_config = ColumnFamilyConfig::new()
        .compression_algorithm(CompressionAlgorithm::Snappy);

    db.create_column_family("my_cf", cf_config)?;

    Ok(())
}

B+tree KLog Format

Column families can optionally use a B+tree structure for the key log instead of the default block-based format. The B+tree klog format offers faster point lookups through O(log N) tree traversal.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, CompressionAlgorithm};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    // Create column family with B+tree klog format
    let cf_config = ColumnFamilyConfig::new()
        .use_btree(true)
        .compression_algorithm(CompressionAlgorithm::Lz4);

    db.create_column_family("btree_cf", cf_config)?;

    // Create column family with block-based format (default)
    let cf_config = ColumnFamilyConfig::new()
        .use_btree(false);

    db.create_column_family("block_cf", cf_config)?;

    Ok(())
}

B+tree Characteristics

• Point lookups · O(log N) tree traversal with binary search at each node
• Range scans · Doubly-linked leaf nodes enable efficient bidirectional iteration
• Immutable · Tree is bulk-loaded from sorted memtable data during flush
• Compression · Nodes compress independently using the same algorithms

When to use B+tree klog format

• Read-heavy workloads with frequent point lookups
• Workloads where read latency is more important than write throughput
• Large SSTables where block scanning becomes expensive

Tradeoffs

• Slightly higher write amplification during flush (building tree structure)
• Larger metadata overhead per node compared to block-based format
• Block-based format may be faster for sequential scans of entire SSTables

Note

use_btree cannot be changed after column family creation. Different column families can use different formats.

Custom Comparators

A comparator defines the sort order of keys throughout the entire system: memtables, SSTables, block indexes, and iterators. TidesDB ships with six built-in comparators and supports registering custom ones.

Built-in comparators: memcmp (default), lexicographic, uint64, int64, reverse, case_insensitive

Registering a Custom Comparator

Register a custom comparator after opening the database but before creating any column family that uses it. Once a comparator is set for a column family, it cannot be changed without corrupting data.

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;

    // Register a reverse byte comparator
    db.register_comparator("my_reverse", |key1, key2| {
        let min_len = key1.len().min(key2.len());
        for i in 0..min_len {
            if key1[i] != key2[i] {
                return key2[i] as i32 - key1[i] as i32;
            }
        }
        key2.len() as i32 - key1.len() as i32
    })?;

    // Use the custom comparator in a column family
    let cf_config = ColumnFamilyConfig::new()
        .comparator_name("my_reverse");
    db.create_column_family("reverse_cf", cf_config)?;

    Ok(())
}

The comparator function receives two key byte slices and must return:

• < 0 if key1 < key2
• 0 if key1 == key2
• > 0 if key1 > key2

Checking if a Comparator Exists

if db.has_comparator("my_reverse") {
    println!("Comparator is registered");
}

// Built-in comparators are always available
assert!(db.has_comparator("memcmp"));
assert!(db.has_comparator("reverse"));

Comparator Permanence

Once a column family is created with a specific comparator, it must always be opened with the same comparator. Changing or removing a comparator after column family creation will corrupt the data ordering.

Utility Functions

tidesdb::free

Frees memory allocated by TidesDB. This is primarily useful for advanced FFI scenarios. For normal Rust usage, the safe wrappers handle memory management automatically.

// Safety: ptr must have been allocated by TidesDB
unsafe { tidesdb::free(ptr); }

Database Configuration Reference

All available Config builder methods:

Method	Type	Default	Description
num_flush_threads(n)	i32	2	Number of background flush threads
num_compaction_threads(n)	i32	2	Number of background compaction threads
log_level(level)	LogLevel	Info	Minimum log level (Debug, Info, Warn, Error, Fatal, None)
block_cache_size(size)	usize	64MB	Global block cache size in bytes
max_open_sstables(n)	usize	256	Maximum number of open SSTable file descriptors
max_memory_usage(size)	usize	0	Global memory limit in bytes (0 = auto, 50% of system RAM)
log_to_file(enable)	bool	false	Write logs to file instead of stderr
log_truncation_at(size)	usize	24MB	Log file truncation threshold in bytes (0 = no truncation)

Column Family Configuration Reference

All available ColumnFamilyConfig builder methods:

Method	Type	Default	Description
write_buffer_size(size)	usize	64MB	Memtable flush threshold in bytes
level_size_ratio(ratio)	usize	10	Level size multiplier
min_levels(levels)	i32	5	Minimum LSM levels
dividing_level_offset(offset)	i32	2	Compaction dividing level offset
klog_value_threshold(threshold)	usize	512	Values > threshold go to vlog
compression_algorithm(algo)	CompressionAlgorithm	Lz4	Compression algorithm
enable_bloom_filter(enable)	bool	true	Enable bloom filters
bloom_fpr(fpr)	f64	0.01	Bloom filter false positive rate
enable_block_indexes(enable)	bool	true	Enable compact block indexes
index_sample_ratio(ratio)	i32	1	Sample every N blocks for index
block_index_prefix_len(len)	i32	16	Block index prefix length
sync_mode(mode)	SyncMode	Full	Durability mode
sync_interval_us(interval)	u64	128000	Sync interval (for Interval mode)
comparator_name(name)	&str	”memcmp”	Key comparator name
skip_list_max_level(level)	i32	12	Skip list max level
skip_list_probability(prob)	f32	0.25	Skip list probability
default_isolation_level(level)	IsolationLevel	ReadCommitted	Default transaction isolation
min_disk_space(space)	u64	100MB	Minimum disk space required
l1_file_count_trigger(trigger)	i32	4	L1 file count trigger for compaction
l0_queue_stall_threshold(threshold)	i32	20	L0 queue stall threshold
use_btree(enable)	bool	false	Use B+tree format for klog

Updatable at runtime (via update_runtime_config):

• write_buffer_size, skip_list_max_level, skip_list_probability
• bloom_fpr, enable_bloom_filter, enable_block_indexes, block_index_prefix_len
• index_sample_ratio, compression_algorithm, klog_value_threshold
• sync_mode, sync_interval_us, level_size_ratio, min_levels
• dividing_level_offset, l1_file_count_trigger, l0_queue_stall_threshold
• default_isolation_level, min_disk_space

Non-updatable settings (cannot be changed after column family creation):

• comparator_name · Cannot change sort order after creation (would corrupt key ordering in existing SSTables)
• use_btree · Cannot change klog format after creation (existing SSTables use the original format)

Error Handling

TidesDB uses a custom Result type with detailed error information:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig, Error, ErrorCode};

fn main() {
    let db = match TidesDB::open(Config::new("./mydb")) {
        Ok(db) => db,
        Err(e) => {
            eprintln!("Failed to open database: {}", e);
            return;
        }
    };

    db.create_column_family("my_cf", ColumnFamilyConfig::default()).unwrap();
    let cf = db.get_column_family("my_cf").unwrap();

    let txn = db.begin_transaction().unwrap();

    match txn.get(&cf, b"nonexistent_key") {
        Ok(value) => println!("Value: {:?}", value),
        Err(Error::TidesDB { code, context }) => {
            match code {
                ErrorCode::NotFound => println!("Key not found"),
                ErrorCode::Memory => println!("Memory allocation failed"),
                ErrorCode::Io => println!("I/O error"),
                _ => println!("Error ({}): {}", code as i32, context),
            }
        }
        Err(e) => println!("Other error: {}", e),
    }
}

Error Codes

• ErrorCode::Success (0) · Operation successful
• ErrorCode::Memory (-1) · Memory allocation failed
• ErrorCode::InvalidArgs (-2) · Invalid arguments
• ErrorCode::NotFound (-3) · Key not found
• ErrorCode::Io (-4) · I/O error
• ErrorCode::Corruption (-5) · Data corruption
• ErrorCode::Exists (-6) · Resource already exists
• ErrorCode::Conflict (-7) · Transaction conflict
• ErrorCode::TooLarge (-8) · Key or value too large
• ErrorCode::MemoryLimit (-9) · Memory limit exceeded
• ErrorCode::InvalidDb (-10) · Invalid database handle
• ErrorCode::Unknown (-11) · Unknown error
• ErrorCode::Locked (-12) · Database is locked

Complete Example

use std::time::{SystemTime, UNIX_EPOCH};
use tidesdb::{
    TidesDB, Config, ColumnFamilyConfig, CompressionAlgorithm,
    SyncMode, IsolationLevel, LogLevel,
};

fn main() -> tidesdb::Result<()> {
    let config = Config::new("./example_db")
        .num_flush_threads(1)
        .num_compaction_threads(1)
        .log_level(LogLevel::Info)
        .block_cache_size(64 * 1024 * 1024)
        .max_open_sstables(256)
        .max_memory_usage(0); // 0 = auto (50% of system RAM)

    let db = TidesDB::open(config)?;

    // Create column family with custom configuration
    let cf_config = ColumnFamilyConfig::new()
        .write_buffer_size(64 * 1024 * 1024)
        .compression_algorithm(CompressionAlgorithm::Lz4)
        .enable_bloom_filter(true)
        .bloom_fpr(0.01)
        .sync_mode(SyncMode::Interval)
        .sync_interval_us(128000);

    db.create_column_family("users", cf_config)?;

    let cf = db.get_column_family("users")?;

    {
        let mut txn = db.begin_transaction()?;

        txn.put(&cf, b"user:1", b"Alice", -1)?;
        txn.put(&cf, b"user:2", b"Bob", -1)?;

        // Write with TTL (expires in 30 seconds)
        let ttl = SystemTime::now()
            .duration_since(UNIX_EPOCH)
            .unwrap()
            .as_secs() as i64 + 30;
        txn.put(&cf, b"session:abc", b"temp_data", ttl)?;

        txn.commit()?;
    }

    {
        let txn = db.begin_transaction()?;

        let value = txn.get(&cf, b"user:1")?;
        println!("user:1 = {}", String::from_utf8_lossy(&value));
    }

    {
        let txn = db.begin_transaction()?;
        let mut iter = txn.new_iterator(&cf)?;

        println!("\nAll entries:");
        iter.seek_to_first()?;
        while iter.is_valid() {
            let key = iter.key()?;
            let value = iter.value()?;
            println!("  {} = {}",
                String::from_utf8_lossy(&key),
                String::from_utf8_lossy(&value));
            iter.next()?;
        }
    }

    let stats = cf.get_stats()?;
    println!("\nColumn Family Statistics:");
    println!("  Number of Levels: {}", stats.num_levels);
    println!("  Memtable Size: {} bytes", stats.memtable_size);

    db.drop_column_family("users")?;

    Ok(())
}

Isolation Levels

TidesDB supports five MVCC isolation levels:

use tidesdb::{TidesDB, Config, IsolationLevel};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    let txn = db.begin_transaction_with_isolation(IsolationLevel::ReadCommitted)?;

    // Use transaction...

    Ok(())
}

Available Isolation Levels

• IsolationLevel::ReadUncommitted · Sees all data including uncommitted changes
• IsolationLevel::ReadCommitted · Sees only committed data (default)
• IsolationLevel::RepeatableRead · Consistent snapshot, phantom reads possible
• IsolationLevel::Snapshot · Write-write conflict detection
• IsolationLevel::Serializable · Full read-write conflict detection (SSI)

Savepoints

Savepoints allow partial rollback within a transaction:

use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = TidesDB::open(Config::new("./mydb"))?;
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let mut txn = db.begin_transaction()?;

    txn.put(&cf, b"key1", b"value1", -1)?;

    txn.savepoint("sp1")?;

    txn.put(&cf, b"key2", b"value2", -1)?;

    // Rollback to savepoint -- key2 is discarded, key1 remains
    txn.rollback_to_savepoint("sp1")?;

    // Add different data after rollback
    txn.put(&cf, b"key3", b"value3", -1)?;

    // Commit -- only key1 and key3 are written
    txn.commit()?;

    Ok(())
}

Thread Safety

TidesDB is thread-safe. The TidesDB and ColumnFamily types implement Send and Sync, allowing them to be shared across threads:

use std::sync::Arc;
use std::thread;
use tidesdb::{TidesDB, Config, ColumnFamilyConfig};

fn main() -> tidesdb::Result<()> {
    let db = Arc::new(TidesDB::open(Config::new("./mydb"))?);
    db.create_column_family("my_cf", ColumnFamilyConfig::default())?;

    let cf = db.get_column_family("my_cf")?;

    let handles: Vec<_> = (0..4).map(|i| {
        let db = Arc::clone(&db);
        let cf_name = "my_cf".to_string();

        thread::spawn(move || {
            let cf = db.get_column_family(&cf_name).unwrap();
            let mut txn = db.begin_transaction().unwrap();

            let key = format!("key:{}", i);
            let value = format!("value:{}", i);
            txn.put(&cf, key.as_bytes(), value.as_bytes(), -1).unwrap();
            txn.commit().unwrap();
        })
    }).collect();

    for handle in handles {
        handle.join().unwrap();
    }

    Ok(())
}

Testing

cargo test

# Run tests with output
cargo test -- --nocapture

# Run specific test
cargo test test_open_close

# Run with release optimizations
cargo test --release

# Run single-threaded (recommended for TidesDB tests)
cargo test -- --test-threads=1