Invariants
Invariants
The preceding chapters each end with the rules their subsystem depends on. This chapter collects them in one place, and adds the column that matters most: what would catch a violation.
An invariant is only as good as the check that enforces it. A rule stated in a comment and guarded by nothing is not an invariant — it is a hope, and it will be broken by the next person who has a good reason to restructure the code. So the third column here is not decoration. It is the audit, and where it says unguarded, that is a real gap and a place to add a test.
How to read this
Each entry names the rule, links to the chapter that motivates it, and states what would fail if the rule were broken. The enforcement column distinguishes three cases:
- A named test or fuzzer — a violation makes something go red.
- Indirect — no test targets the rule, but a violation would likely surface through a broader test that happens to depend on it. Better than nothing, and worse than it sounds: indirect failures point at the symptom, not the rule.
- Unguarded — nothing would catch it. These are the entries to act on.
Durability and ordering
| Invariant | From | Enforcement |
|---|---|---|
| The write-ahead log must contain no gaps — only the contiguous completed run is written | Block manager | fuzz_crash (exact-prefix oracle); block_manager tests cover the ring, not the gap property directly |
| A range tombstone lives exactly as long as a table carrying it, and is honoured by every read that reads that table | Compaction | test_engine_range_tombstone_lives_as_long_as_a_table_carries_it; test_flush_output_carries_its_memtable_intervals; test_cf_source_compaction_carries_input_intervals |
| A merge leaves an interval behind only when it has finished its work — largest level, sequence below the reclamation floor, and no sstable outside the merge reaching into its range | Compaction | test_cf_source_compaction_drops_an_interval_it_has_finished — the bound on what a table accumulates. Every uncertainty in the check counts as reaching in, so it keeps intervals it could have dropped rather than dropping one it could not |
| Two committers deciding neighbouring sequences at once publish each other’s decision | Transactions and MVCC | test_mvcc_two_neighbours_publish_each_other — two threads draw, mark at the same instant, and check; before the barrier between a committer’s own store and its loads of its neighbours’, a run of two hundred thousand rounds left the watermark below the higher sequence in some of them, with no third commit to carry it |
| No sequence is drawn a ring’s width above the watermark, however many committers race for the last slot | Transactions and MVCC | test_mvcc_draws_racing_for_the_last_slot_never_overshoot_the_ring — eight drawers released onto one free slot. with the room checked apart from the take, all of them drew, the ring was overshot, and the slot of the lowest undecided sequence was recycled under one a ring above it, so the watermark never passed it. a sixteen-thread run of a retry loop on two hot keys wedged that way within seconds and committed at a twentieth of its rate for the rest of the run |
| A commit that finds the ring full waits for room before it says it is drawing | Transactions and MVCC | test_mvcc_draw_waits_for_room_before_it_announces — a validator holding the lowest undecided sequence spins on a claimant’s announcement to learn where it lands, and its own decision is what makes the room. announced first, the two waited on each other for the spin budget, fifteen to forty seconds, which a sixteen-thread run of a retry loop on two hot keys hit as a thirty-seven second commit once a stalled durable write had let the ring fill with refused attempts |
| An iterator may outlive its transaction and touches nothing of it once detached | Iterators | test_engine_iterator_outlives_its_transaction — freeing or resetting a transaction detaches the iterators still open under it. before it did, a handler freeing an iterator it had cached across statements read the freed transaction to release the iterator’s hold, which a contended run under the storage engine plugin hit as a crash, and a scan’s footprint from before a reset was carried into the transaction the reset produced and refused there as a phantom |
| A range delete is a version inside every interval it overlaps, for write and scan validation alike | Transactions and MVCC | test_engine_range_delete_conflicts_with_a_committed_overlapping_range_delete, test_engine_serializable_scan_conflicts_with_a_range_delete_under_it, test_isolation_history_ops_snapshot_forbids_g_single — two snapshot or serializable range deletes of one range both committed, and a serializable scan committed over rows deleted under it, until the interval probe asked range tombstones in memtables and tables as well as keys |
| A transaction’s scan hides what its own interval deletes cover, rows committed at its snapshot included | Iterators | test_engine_scan_honours_own_interval_delete_at_the_snapshot_sequence — rows committed by the last commit before the transaction began tied the overlay’s interval at the snapshot sequence and showed through its own range or prefix delete |
| Validation asks the claims in flight before the store, for reads, scanned intervals, writes and interval deletes | Transactions and MVCC | test_isolation_history_repeatable_read_is_acyclic_over_point_access — concurrent read-modify-write appends at repeatable read, checked as an elle-style history against adya’s definitions. asked store first, a writer one sequence below applied after the store check and dropped its claims before the claim check, and both appends committed over the same version, a lost update in about a third of runs; claims first, none in thirty |
| Each isolation level excludes the anomalies it promises to | Transactions and MVCC | test_isolation_history_read_committed_reads_only_committed_states, test_isolation_history_snapshot_forbids_g_single, test_isolation_history_repeatable_read_is_acyclic_over_point_access, test_isolation_history_serializable_is_acyclic — recorded histories searched for aborted and intermediate reads, lost appends, non-prefix reads, and dependency cycles by kind. the checker is shown able to fail on a hand-built history for each anomaly. test_isolation_history_ops_repeatable_read_is_acyclic, test_isolation_history_ops_serializable_is_acyclic and test_isolation_history_ops_serializable_across_a_restart_with_prepares_in_doubt run the same checks over scans, range deletes and two-phase commits, with prepares left in doubt through a reopen; repeatable read lost four to seven appends in every run before prepares held their scans, still failed about one run in ten with them held, and failed none of twenty-eight runs once phase two drew through the prepare’s record. test_isolation_history_mixed_levels_each_keep_their_own and test_isolation_history_mixed_levels_across_flushes_and_compactions mix read committed, repeatable read and serializable in one history over two families, with prepares lingering in doubt and, in the second, a memtable small enough to flush throughout and compact where the runner keeps up; each attempt is held to its own level, read-write edges leaving only strict attempts and lost updates counted only between strict ones |
| A commit’s validation asks only about writers sequenced below it | Transactions and MVCC | test_txn_validation_ignores_a_writer_sequenced_above_the_commit — a version one sequence above the commit’s own, already applied, refused a repeatable-read commit whose read it did not stale and a snapshot commit whose write it did not contest, until the probe took the commit’s own sequence as its ceiling |
| A reclamation floor taken during a read-committed read is at or below that read’s ceiling | Transactions and MVCC | test_txn_read_committed_read_holds_the_floor — the floor is taken from inside the read, where a flush or a merge starting mid-read stands; before read-committed transactions registered and published the ceiling of the read in flight, the floor came back unbounded, and a collection landing there would have kept one version per key and dropped the one the read resolves to. also pins that the hold is released between reads, that a scan’s ceiling outlives a point read inside it, and that the high-water mark records the floor |
| A transaction’s scan resolves its own buffered writes exactly as its point reads do, interval deletes included | Iterators | test_engine_scan_honours_own_prefix_delete, test_engine_scan_reads_own_writes, test_writeset_source_overlay — the overlay answered the merge’s covering question with nothing, so a prefix the transaction had deleted still scanned back the committed keys under it while a point read of any of them said absent |
| A read answers, with a version, with absence or with a covering interval, only if the level shape was settled when it started and stood still while it looked | Life of a read | test_engine_read_survives_a_compaction_moving_levels, test_cf_source_a_move_into_an_empty_level_never_hides_the_newer_version, test_cf_source_a_move_into_an_empty_level_never_lets_an_old_interval_answer, test_level_set_publish_brackets_the_generation — the second moves the newest version’s table between a level the read scans and one its mask called empty, without pause, and asks twenty thousand times; before the hit was guarded, hundreds of those answers were the older version from the level below. the third does the same under an old interval covering the key; before the interval’s answer was guarded, a walk that missed the key only because its table had moved reported it deleted at that interval, which the concurrency fuzzer on two cores saw as a committed key read as absent about once in three hundred runs, always at the first merge into a new level. the fourth pins the sequence-lock shape of the generation, odd inside a publish and even between them, which is what turns a read that began inside a publish into a retry |
| Data is durable before the manifest names it | Manifest | fuzz_crash; test_crash_recovery_torn_flush, and test_crash_recovery_torn_wal_append for the writes a commit itself issues |
| A write-ahead log is unlinked only after its data reaches L1 | Memtable and WAL | test_engine_write_recovery, test_engine_prepared_survives_flush |
| The batch reaches the log before the memtable | Life of a write | fuzz_crash |
| A separated value reaches the value log before the record naming it | Life of a write | fuzz_crash — the other order would let a crash leave a record pointing at bytes that are not there, where this one only leaves reclaimable garbage |
| Every holder of a value log reference is either counted or covered by a floor | Value log | test_engine_prepared_batch_keeps_its_separated_values, test_engine_separated_value_survives_reopen_and_reclaim, and the pinned fuzz_standalone corpus. There are three holders besides the installed tables — a memtable, an in-flight build, and an undecided prepared batch — and each was found by losing data |
| Manifest rollover is write-temp-then-rename | Manifest | manifest tests |
| A file whose write or sync failed is never written or synced again | Block manager | test_block_manager_failed_sync_refuses_later_writes — points the descriptor back at the real file after the failure, where the kernel would accept both, so only the recorded failure refuses them; a second sync after a real failure can report clean for pages the kernel already dropped |
| The manifest and the value log step around a failed file instead of failing until a restart | Durability | test_manifest_failed_sync_rolls_over_to_a_fresh_log, test_vlog_failed_write_rolls_to_a_fresh_segment — without the step around, every flush install, or every commit of a large value, fails from the first failure on |
| A rename the manifest could not make durable is never built on | Durability | test_manifest_unsynced_rename_fails_later_commits — a directory that cannot be opened for reading fails the sync after the rename, and every commit after it must fail, since a later sync of that directory can report clean while the rename is still not on the device |
| A backup’s copies, and a clone’s under a syncing mode, are synced before they report success | Block manager | test_engine_copy_file_reports_a_failed_sync — a destination that refuses the sync fails a durable copy and not one the caller did not ask to be durable; unsynced, a clone’s durable catalogue named tables whose bytes were still only in memory |
| An explicit sync is skipped on the descriptor’s own flag, never on the platform’s capability | Block manager | test_block_manager_full_open_tracks_descriptor_sync — asserts the recorded flag matches what F_GETFL reports for the live handle. The two questions differ only after a runtime mode change: a file switched from NONE to FULL asks for the new mode without the open descriptor gaining O_DSYNC, so a platform-level test skips a sync nothing else performs, and the write is durable in name only |
ftruncate is synced explicitly — O_DSYNC does not cover it | Block manager | Unguarded — no test kills a process between a truncation and its sync |
The value log is durable via O_DSYNC, not via a flush-time barrier | Value log | Unguarded — a change to that open would not fail any test, and the comment warning against it is the only guard |
| Closing a block manager drains its staging ring before closing the descriptor | Memtable and WAL | test_l0_wal_ack_on_stage_survives_close — under TDB_SYNC_NONE nothing else makes a staged record durable, so breaking this loses data on a clean shutdown, which no crash test would catch |
Only TDB_SYNC_NONE acknowledges a commit before its record leaves the process | Durability | fuzz_crash runs at TDB_SYNC_FULL, so its exact-durable-prefix oracle guards the modes that promise this and deliberately does not exercise the one that does not |
Visibility and ordering of state
| Invariant | From | Enforcement |
|---|---|---|
| A flush retains a key’s version chain against the reclamation floor | Memtable and WAL | test_engine_frozen_snapshot_survives_a_flush — a memtable holds the whole chain while a flush writes one sstable, so a flush taking only the newest version dropped everything under it whatever the floor said. A repeatable-read reader below an overwrite then read a live key as absent, which the default isolation level never sees because it re-reads |
| A read at a bare sequence refuses below the oldest readable point | Transactions and MVCC | test_engine_read_at_a_collected_sequence_is_refused, test_engine_oldest_readable_seq_resets_on_open — below it a merge has kept one version per key, so answering would report a key that existed as absent. The watermark rises where a floor is taken rather than where the collection ends, so a merge already running is visible to the check |
| A named snapshot holds the floor by being registered | Transactions and MVCC | test_engine_snapshot_holds_versions_a_compaction_would_have_collected, test_engine_snapshot_holds_and_releases_the_reclamation_floor — a sequence remembered outside the registry protects nothing, and a read there answers from what a merge happened to leave rather than from what was true |
| A directory is held by one handle at a time | Database | test_engine_second_open_is_refused, test_open_refused_across_processes_after_a_same_process_refusal — an exclusive lock on the LOCK file owned by the open file rather than the process (F_OFD_SETLK, flock, LockFileEx), so a same-process reopen is refused by the lock itself and no close elsewhere in the process can drop it. A second open is refused with TDB_ERR_LOCKED rather than being allowed to share the store |
| A rotation takes the prepared log’s generation under the same claim the preparer wrote it under | Memtable and WAL | Unguarded — taking the spare empties the slot, which lets the next preparer overwrite the generation field before the taker has read it. The memtable would then be named for a log it does not hold, and the flush that asks whether that generation is pinned would ask about the wrong file. Found by ThreadSanitizer under fuzz_conc, not by any test asserting it |
| The log a rotation installs is opened before the rotation, off the lock | Memtable and WAL | Unguarded — nothing asserts a rotation does no file creation, and opening one inline costs a file creation and a flush thread with every committer stopped — a cost set by the filesystem, not by the engine, and on a slow one it dwarfs the rest of the rotation. The engine logs a slow prepare separately from a slow rotation, and times the log open on its own inside both, so a rotation that overran can be read as the open or as the rest rather than as one number covering both |
| The rotation lock is never held across a wait or a device barrier | Memtable and WAL | Unguarded — no test measures lock hold time. Two of the three violations found were caught only by capturing stalled commits’ stacks under load; the third was the fullness re-check itself, which measured the memtable while holding the lock and so took the memtable’s pin and its range tombstone lock, both of which wait. That one was found from a CI stall breakdown showing a rotation far longer than every phase timed inside it, and is why the re-check now compares a rotation marker instead of measuring anything. A reviewer adding a wait under this lock would still see nothing fail |
| Rotation enqueues the sealed memtable before swapping the active slot | Memtable and WAL | fuzz_conc; test_l0_rotate_and_read covers rotation but not the ordering window |
| Shallower is newer — flush installs are strictly ordered | Life of a read | Indirect: flush, test_engine_overwrite_across_recovery |
| A drawn but undecided sequence is invisible, and a committed batch is atomic to readers | Transactions and MVCC | test_mvcc_watermark_follows_decisions, test_txn_begun_during_an_apply_sees_nothing_of_it, test_txn_api_batch_is_atomic_to_readers — every ceiling is the watermark, below which every sequence is decided; a writer committing a two-key batch under racing snapshot readers never has the pair seen from two batches |
| An sstable holds only decided versions | Life of a write | test_flush_waits_for_the_watermark_and_drops_an_abandoned_batch — a flush waits for the watermark to pass the memtable’s highest sequence before it builds, and a batch abandoned after rotation is dropped rather than written |
| An abandoned sequence is remembered exactly as long as a memtable could hold it | Memtable and WAL | test_l0_forgets_abandoned_sequences_once_their_memtables_retire — the record carries the generation active at the abandon and is forgotten when that generation retires, so the table cannot fill under a working flush |
| A commit returns only once the watermark has passed its sequence | Transactions and MVCC | fuzz_conc — a worker’s read-committed read right after its own commit finds the key; test_txn_failed_commit_leaves_the_watermark_advancing, test_txn_prepared_sequence_does_not_hold_the_watermark for the paths that decide without committing |
| A column family configuration is published whole, never written through | Column family | column_family tests; TSan across the suite — a reader takes an immutable snapshot, so no flush or plan ever sees half of one configuration and half of another |
| The memtable’s reported size counts every resident version, not every key | Memtable and WAL | test_skip_list_size_counts_every_version |
| The rotation threshold reads memory occupied, not key and value bytes | Memtable and WAL | test_skip_list_memory_bytes_counts_structural_overhead — a node, its pointer arrays and a version struct are about 100 bytes an entry, so counting only the data lets a memtable of small entries hold several times the memory its configuration asks for |
| A version costs its structure plus its value, so a tombstone is not free | Memtable and WAL | test_skip_list_size_counts_tombstone_versions |
BUSY short-circuits the source stack rather than falling through | Life of a read | source tests |
| A level’s candidates come from one layout load, not a count then a collect | Life of a read | Indirect — fuzz_conc would surface the stale answer. Two loads let a flush install between them, leaving the collected set a subset of what the count promised, so the highest sequence among the candidates was not the highest that existed. The collect reports the true overlap count even when more overlap than fit, which is what lets one load do both. A caller that keeps the two loads must read only what the collect wrote, never the array it sized from the count — a level set that shrank between them leaves the tail of that array unwritten, and ce_resolve_inputs dereferenced it as an sstable and then unreferenced it, which is a garbage pointer rather than the stale answer this entry used to describe |
| The deep-level candidate window agrees with a scan of the level | Life of a read | test_level_set_deep_level_window_matches_a_full_scan — L2+ runs are non-overlapping and sorted by min_key, so two binary searches bound the files meeting a range. A window starting one file late drops a file that holds the key and the lookup reports it absent; the test checks every boundary and gap differentially against a brute-force overlap |
| A tombstone or expired entry is a definitive absence, not a fall-through | Life of a read | test_l0_tombstone, ttl tests |
| Phase two draws a fresh sequence | Transactions and MVCC | test_engine_recovered_phase_two_durable_and_ordered, test_engine_settled_prepare_lands_above_writes_it_was_in_doubt_through |
| Phase two draws through the prepare’s own record | Transactions and MVCC | test_txn_2pc_decided_batch_orders_a_later_draw_after_it — a commit drawn above the batch’s phase two, validating from inside the COMMIT record’s append, before the batch is applied. With the draw taken outside the record the batch read as prepared, above that commit, and its read of a key the batch writes passed; the history test found it at repeatable read as write skew between a prepare and a plain commit |
| A prepare holds the intervals it scanned, live and across a restart | Transactions and MVCC | test_mvcc_prepared_scans_refuse_writers_inside_them, test_txn_prepared_scan_refuses_an_insert_inside_it, test_txn_adopted_prepare_holds_its_scans, test_engine_recovered_prepare_keeps_its_scanned_range, test_mvcc_orphaned_scans_keep_holding — each fails with the scans left off the record, and the restart test fails with them left out of the prepare-reads record |
| A range delete is refused by a prepare’s reads at every level | Transactions and MVCC | test_txn_prepared_read_refuses_a_range_delete_over_it, test_mvcc_prepared_read_claim_refuses_a_range_delete_over_it — at repeatable read the delete asked nothing of the claims in flight and went through over a key a prepare had read |
| A read-committed write is refused by an undecided prepare’s read | Transactions and MVCC | test_txn_read_committed_writer_refused_by_a_prepared_read — a point write and a range delete over a key a serializable prepare read. Read committed took no claims and validated nothing, so both committed below the prepare’s phase two; fails with the check after the draw removed |
| A prepare yields to another prepare’s write on what it read | Transactions and MVCC | test_txn_prepare_refused_by_a_prepared_write_to_what_it_read — the writer prepared at read committed and at repeatable read, the reader at serializable; a plain reader passes. Fails with a prepared owner read as future, and with read-committed prepares left unclaimed. test_isolation_history_mixed_levels_each_keep_their_own found it as a four-transaction cycle through two serializable reads |
| Every writing commit claims, at every level | Transactions and MVCC | test_txn_validation_finds_a_read_committed_writer_below_it_in_the_claims — a read-committed writer held inside its append after its draw, and a repeatable-read commit above it that read the old version, refused at once from the claim. Without claims it committed on a read already invalidated; test_isolation_history_mixed_levels_across_flushes_and_compactions found it as a cycle |
| First-committer-wins counts a write claim below it | Transactions and MVCC | test_mvcc_write_claim_below_blocks_a_first_committer — a snapshot writer claims first, a writer without the promise joins and draws below it; the write check looked only at read claims and let the snapshot writer overwrite a version it never saw |
| A memtable’s high sequence bounds every version and interval applied into it | Transactions and MVCC | test_l0_has_newer_finds_a_version_above_the_floor_in_an_immutable, test_l0_has_newer_counts_a_range_delete_above_the_floor — a bound one short of the newest version lets the conflict scan pass over the memtable holding it, and the write it should have rejected is accepted |
| An sstable’s footer sequence bounds every version it holds | Transactions and MVCC | Indirect — fuzz_standalone and fuzz_conc would surface a missed conflict, but nothing asserts the footer bound itself. A footer understating its contents would make the commit conflict scan skip an sstable that does hold a newer version, and the write it should have rejected would be accepted |
Reclamation and lifetime
| Invariant | From | Enforcement |
|---|---|---|
| A memtable is freed only after readers drain | Memtable and WAL | test_l0_retire_is_observable_to_a_bracketed_read; TSan across the suite |
| A lookup’s level-0 walk reports “not yet known”, never “absent” | Memtable and WAL | Unguarded — the skip list’s descent settles on a predecessor and then walks level 0 to the key, because an insert can splice a smaller key in between. Stopping that walk on a key below the target reports a present key as missing, and so does running its budget out and calling the result absent; the walk therefore continues while it is below, is bounded by the list’s own entry count plus slack rather than by a flat constant, and hands back a retry that re-descends rather than an absence. test_skip_list_lookup_never_misses_a_present_key_under_splicing guards the coarse form of this, but it does not reach the narrow window — reinstating a single-load hop still passes it, because landing an insert between a descent finishing and its next load needs an interleaving a test cannot force. fuzz_conc is the only other cover |
| A reclaim never retires the segment taking appends | Value log | test_retire_refuses_the_segment_taking_appends — calls retire on the active slot directly, which is the same question the race asks, and asserts the store keeps writing and reading afterwards. A reclaim reads the active slot per segment rather than once per pass, and the retire re-reads it after claiming the eviction window; deciding against one stale reading would let a roll mid-pass hand it the segment taking appends, whose file it would then unlink. test_reclaim_never_retires_a_segment_a_roll_is_publishing holds a roll between opening its successor and publishing it while a reclaim runs; read without the roll lock, the successor looked sealed and empty, was retired, and the roll published the dead slot, after which every append reported the store full. a 32-bit CI run hit it as seventy-five failed writes |
| No engine-internal error code reaches a public caller | Error codes | Unguarded — tdb_public_rc in db.c is the single choke point that maps TDB_ERR_BUSY to TDB_ERR_LOCKED, and nothing fails if a new public function forwards an engine result without it; the code db.h does not define would reach a caller’s switch and tidesdb_strerror would call it unknown |
| The plan memo is keyed on the shape and the reclamation floor together | Compaction | test_engine_scheduler_replans_when_the_gc_floor_moves — opens a snapshot transaction and asserts the family is planned again with no write having landed |
| A subdivided merge writes the same files as an undivided one | Compaction | test_compaction_subdivided_merge_matches_one_thread — runs the same merge with and without permission to split, and requires the same output count and the same value for every key. the split is at the boundaries the sink already rolls on, so the files are the same files; if that stopped holding, a merge’s result would depend on how many threads were free, which is not a thing a storage engine may do |
| A plan that produced jobs is never memoized | Compaction | Unguarded — proving it needs a compaction job to fail while the layout stays put, which the suite has no way to force without fault injection. The failure is silent: the family simply stops being scheduled, and on an idle database that is permanent. Emptiness is read from the plan’s job count and from nothing else — deriving it from whether some allocation succeeded would let a failed malloc memoize a family that did have work |
| An idle non-empty memtable is rotated on a timer | Memtable and WAL | test_engine_idle_flush_rotates_an_unwritten_memtable — writes under the rotation threshold and asserts the data reaches L1 with no further write |
| Only the handle that installed the log sink closes it | Troubleshooting | test_engine_log_to_file_writes_into_the_database_directory — opens a second database without log_to_file and asserts the first one’s file stopped growing rather than being truncated or reopened |
| An undecided prepare pins its own log generation and no other | Transactions and MVCC | test_engine_undecided_prepare_pins_only_its_own_generation — one prepare left in doubt across six flushed generations; keyed on a database-wide count instead it retained all seven |
| A batch recovered in doubt keeps its log across later flushes | Transactions and MVCC | test_engine_recovered_in_doubt_batch_survives_later_flushes — reopens, writes and flushes without ever asking for the in-doubt list, then decides the batch |
| A question asked of every table in a family is asked of all of them, however many there are | Life of a read | test_cf_source_a_range_delete_is_seen_past_any_table_count — a range delete riding in one table among 512, 513 and 700, asserting the point read before and after it, both of its bounds, the commit’s probe of one key and its probe of a range, then 20000 reads while another table moves between levels. A fixed-size view answered “covered by nothing” past its size, which reports a deleted key live, and released references it never took |
| A snapshot’s references are dropped on every path out, including the mismatch | Compaction | Unguarded — a layout that grew past the caller’s array references nothing, but one that shrank between sizing and filling is referenced in full and returns a count that does not match, so a caller treating the mismatch as a failure still owes those references. Forcing that race in a test is not currently possible. Every call site now releases across the whole zeroed array, which is the only form that cannot leak; the one that returned early instead leaked a reference per table on each attempt, pinning it for the life of the process |
| A family’s reported size counts its key logs exactly once | Statistics | test_engine_cf_data_size_counts_key_logs_once — asserts it equals both the sum of level_sizes and the real bytes on disk; adding the summed value length on top doubles it for any workload whose values stay inline |
| A borrowed root is never released by the descent that used it | Life of a read | test_get_borrows_a_root_held_across_reads — reads a tree whose root is internal through a cache too small to hold it, so the root’s frame is evicted while pinned and a descent that released the borrowed node would drop the sstable’s own reference and free it under the next reader |
| A prepare pins the log its record lands in | Memtable and WAL | test_engine_prepare_pins_the_active_log_while_a_spare_waits, test_engine_prepared_survives_flush — a spare log is prepared, then a transaction prepares, unrelated work commits and the memtable flushes. the prepare read the newest generation drawn, which a waiting spare had already moved past the active log, so it pinned the spare, the flush unlinked the log holding the undecided prepare, and the reopen did not find it. any database that had rotated once had a spare waiting |
| A rotation never installs a log whose generation is at or below the active log’s | Memtable and WAL | test_engine_rotation_never_installs_an_overtaken_spare — a preparer is held in flight while a rotation opens its own log above the spare’s generation, then released, and the next rotation takes the spare. without the guard it installed the lower generation after the higher one, which recovery would replay as the older log |
| A log that will never be written to is unlinked, not just closed | Memtable and WAL | test_engine_preparing_a_spare_log_strands_no_file — releases sixteen preparers together and asserts at most one log joined the directory; closing the loser’s descriptor without unlinking its file stranded one log per losing committer per rotation, and each one is scanned, replayed and given a memtable on every later open |
| An entry lapses wherever it lives, not only in the memtable | Life of a read | test_ttl_expires_after_reaching_an_sstable — flushes while the deadline is still ahead, so the entry reaches the sstable live and only lapses there. Expiry has to be evaluated on every source a version can be read from; confining it to the skip list and to the flush that converts a lapsed version to a tombstone would leave any lifetime longer than the flush interval never expiring at all |
| Every source ages an entry against the same published second | Life of a read | test_get_ages_entries_against_the_injected_clock — publishes a clock, reads the entry live one second short of its deadline, then advances only that clock and asserts the same read now answers as a tombstone. The memtable’s list and every sstable read one value a ticker publishes rather than calling time(NULL) per entry; two clocks a second apart would let a key read live from one source and lapsed from another, and the read would fall through to whichever answered |
| A tombstone is the only thing that means absent | Data model | test_engine_an_empty_value_reads_back_as_present — stores a zero-length value and requires it present out of the memtable, out of an iterator, out of an sstable after a flush and out of the replayed log after a reopen, while a real delete still reads as not-found. the write path once refused it, and refused it only at apply, after the batch was durable — so the caller saw an I/O error from commit for what was a legitimate write. the model fuzzers now generate empty values about one time in sixteen |
| A lapsed entry is collected by a merge, not by the read that hides it | Compaction | test_ttl_expired_entry_is_collected_by_compaction — asserts the family’s live key total falls across a forced compaction, so the bytes go rather than the key merely reading as absent forever |
| A lapsed entry shadows an older version rather than reading as absent | Life of a read | Unguarded — reading it as absent would let an older version beneath it surface, but the fall-through is not reachable through the public api: a flush triggers a compaction that merges the two versions into one run before any read can observe it, and both mutations of this rule leave the suite green. The sibling case it generalises is guarded at the unit level by test_compaction_keeps_tombstone_with_sibling |
| A scan costs its range, not its family | Iterators | test_cf_iter_narrow_range_scan_does_not_open_every_sstable — 32 disjoint sstables, a four-key scan, and an assertion that fewer than one cache lookup per sstable was taken; unbounded it took 128 for those four keys, which is how concurrent range scans saturated the cores while each wanted almost nothing |
| Pruning sources never drops one a scan needs | Iterators | test_cf_iter_full_scan_spans_every_sstable — an unbounded scan over the same 32 sstables still returns every key, so a selection rule that dropped a live source would fail rather than silently shorten the stream |
| A level below L1 is read as one source, and a seek descends one of its tables | Iterators | test_cf_iter_level_seek_descends_one_table — 32 disjoint tables moved to L2 and one seek through an unbounded iterator, asserted to take at most a quarter as many cache lookups as there are tables. With a source per table it fails, since every table is descended; test_cf_iter_level_reads_as_one_source_in_both_directions, test_cf_iter_level_seeks_land_in_the_right_table and test_cf_iter_level_interval_in_one_table_hides_keys_in_another pass either way, so the run changes cost and not answers, and the seek test catches an off-by-one in the run’s binary search |
| A merge never targets the flush tier | Compaction | test_planner_merge_always_leaves_the_flush_tier — a two-level family with its tier over the file trigger, asserting both the chosen target and the emitted job’s target sit below L1; targeting L1 consolidates the tier in place and leaves it growing one run per flush |
| The tree gains a level when its largest is full | Compaction | test_plan_grows_a_level_when_the_largest_is_full — without it a two-level family stops deepening, the dividing level collapses onto the flush tier, and the tier grows a run per flush with nowhere to drain to |
| A tree of any depth gains a level when the dividing merge would overflow its largest | Compaction | test_plan_partitioned_merge_grows_a_level_when_the_largest_is_full — a three-level tree with a full largest level and a due dividing merge, asserting the merge lands in a new fourth level, and that the same merge with room to spare lands in the third. Without it a tree stops at three levels and every dividing merge rewrites the whole of the largest |
| A level just added is not shed by the next plan | Compaction | test_plan_does_not_shed_a_level_it_just_added — a new largest level holding a little under the capacity that grew it plans nothing. Shedding at a T-th of the largest level’s own capacity sheds at the size the level was added at, so a growth merge that dropped garbage is undone at once |
| A partitioned merge moves a lone file into an empty level | Compaction | test_plan_partitioned_merge_moves_a_lone_file_into_an_empty_level — a four-level tree whose dividing level holds one file per partition over an empty third level, asserting one job per partition. Skipping partitions of fewer than two files plans nothing there, and the flush tier grows until admission stops every writer |
| A merge that density alone made due moves its table deeper, never back into its own level | Compaction | test_plan_tombstone_density_moves_the_dense_table_deeper asserts the target and the exact inputs; test_plan_tombstone_density_at_the_largest_level_waits asserts a dense largest-level table plans nothing; test_plan_tombstone_density_moves_disjoint_tables_together asserts one plan moves every dense table it can without a shared input. Writing a dense table back into its own level keeps every tombstone and plans the same merge on every pass |
| A partitioned merge fans out only when no input spans a boundary | Compaction | test_plan_partitioned covers the aligned case fanning out into disjoint jobs; test_plan_partitioned_spanning_input covers the fallback — sharing a spanning input across per-partition jobs lets only the first run, dropping a tombstone without the versions it shadows |
| Ingestion is paced against merge progress as well as flush progress | Memtable and WAL | test_tier_band_paces_against_merge_progress — asserts the tier band admits below its slow mark, dwells further past it, blocks at the stall mark, and is inert when told to weigh no tier; without it a burst outruns compaction and leaves a tier every later read merges across |
| A plan’s jobs run concurrently and the last one out tears the plan down | Compaction | Indirect — ASan and TSan across the suite. The claim is released only when the final job finishes, which is what a family drop waits on, and a queue drain pays the same count through engine_job_group_release |
| A chunk is sized to the object it holds | Block cache | Indirect — ENGINE_NODE_ARENA_CHUNK_SIZE is defined from TDB_DEFAULT_CF_BTREE_BLOCK_SIZE so the two cannot drift, but nothing fails if they do. they had already drifted sixteen-fold, and the only symptom was a cache holding a thirty-second of its budget in useful data — no test covers the ratio of a frame’s chunk to the node inside it |
| The cache charges a frame what it reserves, not what it uses | Block cache | test_arena_reserved_counts_whole_chunks_not_bytes_used — asserts one small object still reserves a whole chunk and that reserved never reads below allocated. a cached node is decoded into its own arena, so the memory a frame holds is the chunk it took; charging the bytes the node used let a 64 MiB cache sit resident at 908 MiB, and nothing reported the gap. worst on a small budget, which is why the capacity test — entries near the nominal size, nothing to round — kept passing |
| The block cache’s frame count follows its byte budget | Block cache | test_cache_reaches_its_configured_capacity — fills past the budget and asserts the cache settles within a few percent of it; a frame ceiling that binds first leaves it at a fraction, silently |
| Each class of file accounts its own device work | Statistics | test_engine_io_stats_attribute_device_work — asserts a fresh database reports zero, that committing moves the log class and flushing moves the sstable class, and that no class reports fewer bytes than writes |
| Every point a writer can wait at reports count, total and longest | Statistics | test_engine_stall_stats_attribute_writer_waits — asserts a fresh database reports zero, that the log wait counts after writes, and that no total is less than the longest wait inside it |
| A finished sstable occupies its data, not its preallocated extent | Block manager | test_engine_finished_sstable_is_trimmed_to_its_data — measures the klog while the database is open, since closing trims it either way and hides the defect |
| A key log the manifest does not name is swept at open | Recovery | test_engine_open_sweeps_orphaned_sstables — plants an unnamed klog and a staging file, then asserts both are gone and every named klog survived |
| The orphan sweep is skipped after a self-heal | Recovery | test_engine_open_keeps_sstables_when_manifest_self_healed — plants a klog the rebuild cannot adopt, so the guard is what keeps it; without the guard the sweep deletes it |
| A merged-away compaction input is unlinked when its last reference drops | Compaction | test_engine_compaction_unlinks_merged_inputs, test_engine_compaction_leaves_no_orphans_under_load — both compare the klog files on disk against the live sstable count, which is the only way the leak is visible; nothing else fails when the files pile up |
| A membership change never waits for a reader | Architecture | test_cf_registry_publish_under_sustained_readers — runs more readers than the machine has cores and watches the publishing thread from outside, because a publish that never returns cannot time itself. Waiting for the borrows to drain is the version that reads as correct and hangs anyway: the counts do fall, but the wait also needs an instant when no reader is inside the borrow guard, and continuously arriving readers never produce one. Under this test that version published 0 of 4; the deferred one publishes in microseconds |
| A flush waits for its install ticket before borrowing the family view, never while holding one | Architecture | test_engine_create_completes_under_sustained_flush — borrowing across the wait stretches one worker’s hold to cover every flush queued ahead of it |
| A level set a clone displaces outlives the borrows that could still read it | Architecture | ThreadSanitizer — level_set_l1_overlap_depth against the level_set_free inside a clone’s reload. The scheduler reads overlap depth off every published family’s set holding only a view borrow, so neither the compaction claim nor unpublishing the family covers it; the set has to be retired rather than freed. Nothing functional fails when it is freed inline — the suite passed, and only the race detector saw it |
| A family the compaction scheduler claimed is not freed under it | Architecture | AddressSanitizer, and only under fuzz_standalone at roughly one run in six — sstable_close on a level set cf_free had already released. The scheduler is the one reader that keeps a family after leaving the view, so the claim rather than the view is what holds it; a drop that reads the claim as clear instead of taking it leaves a window where a tick already inside its borrow claims the family afterwards |
| A dropped family, and a renamed family’s old name, are freed only once no live view names them | Architecture | AddressSanitizer — both were caught by fuzz_conc against a flush still building. Deferring them is not enough on its own: released on the borrow guard they are freed while a flush still holds the view by reference count and reads the handle and the name from it, which is a hold the guard was never sized for. They are released on the live view count instead. No functional test detects either |
| A family’s name and level set are published, never written in place | Architecture | ThreadSanitizer — a rename against a flush copying the name onto an sstable, where it is the first component of a block-cache key, so a torn copy is a wrong key rather than a cosmetic defect |
| A writer waiting on the manifest holds off arriving readers | Architecture | test_engine_create_completes_under_sustained_flush — the lock kind that does this is a glibc extension, so relying on it left the guarantee holding on one platform. The family registry no longer needs it at all: its readers take no lock, so its writers have nothing to be preferred over |
| A committer that finds another rotating does not queue behind it | Memtable and WAL | Unguarded — nothing asserts the acquire is a try rather than a wait. A mutex that hands off by barging starves one waiter for as long as the others keep arriving, and the waiter gains nothing the holder finishing does not already give it |
| A timed wait is taken on a clock the wall clock cannot move | Thread manager | Unguarded — nothing asserts the condvar’s clock. Every background worker parks in the same place, so a wall clock step leaves all of them waiting for a time that has moved away and they stop together for its length; the tell is a burst of work landing when they finally wake |
| A flush never waits out a reader to free a memtable | Memtable and WAL | test_l0_reclaim_defers_rather_than_blocking — it holds an immutable pinned across a retire and asserts the retire returns, which hangs outright if the wait is unbounded |
| A pinned memtable keeps its write-ahead log alive | Memtable and WAL | test_memtable_free_leaves_wal_open |
| Cache frames are never freed; only payloads are reclaimed | Block cache | cache tests; ASan across the suite |
| A cache payload is reclaimed exactly once, by the last reference dropped | Block cache | cache tests; ASan would catch a double free |
| A sstable is freed by whoever drops the last reference | SSTable | level_set, sstable tests; ASan |
| Sources stay pinned for an iterator’s life | Iterators | cf_iter, iter_api_tests; TSan |
| Ingest is paced before the staging ring fills, not when it is full | Memtable and WAL | backpressure tests cover the band, its graduation and its cap; test_txn_commit_paces_large_grouped_families covers the commit path consulting it — 512 writes over four families, asserting the hook fires exactly once per family in first-seen order — and test_txn_commit_backpressure_failure asserts a refusal stops the commit before either the log append or the apply. What remains unguarded is the append path’s own ring-lag signal, where a regression would still show only as a latency tail |
| A copy freezes nothing; it claims what removes files and snapshots the rest | Memtable and WAL | fuzz_standalone and fuzz_conc drive backup and clone against concurrent flush and compaction; nothing asserts the absence of a freeze, so a reviewer reintroducing one would see no failure |
| A caller that mutates the catalogue and cannot finish puts it back | Compaction, Memtable and WAL | test_manifest_an_abandoned_half_batch_lands_on_the_next_commit — buffered records are not discarded when a caller gives up, they wait for the next commit from any path. Both installs name their outputs in a loop and so both must undo a partial one: a compaction that stopped partway would disown its inputs durably with the replacement never named, and the orphan sweep would unlink their files on the next open; a flush would leave the catalogue naming outputs no level set ever took. The clone and the recovery rebuild reach the same end differently — the clone’s failure path drops the destination and the drop cascades those records away, and a failed rebuild aborts the open, whose manifest close discards the batch. The undo itself is unguarded: reaching it needs a refused allocation inside the install’s own window, which test_alloc_failure_through_a_compaction_keeps_every_commit walks the path of but cannot land in |
| An interval delete covering no key is refused where it is buffered | Transactions and MVCC | test_engine_range_delete_refuses_an_interval_covering_nothing — an interval ending where it starts, or before it, was taken when buffered and met only at the apply, which runs after the batch is durable and treats every failure as transient. So it burned the retries and cancelled the whole transaction with an I/O error, losing the writes that shared it, for a bound the caller could still have fixed. The same shape as the empty value above. The rule lives in range_tombstone_interval_valid, which both the API and the tombstone set ask, so the two cannot come to disagree about what an interval is |
| A claim names a key, never a hash of one; commits of different keys never conflict | Transactions and MVCC | test_txn_api_disjoint_commits_never_conflict, test_txn_api_concurrent_wide_commits_both_succeed — two writers commit disjoint keys at once, at a few hundred keys a commit and at a hundred thousand, without a refusal; a table of hashed slots refused most of the former and the second of the latter |
| A commit never conflicts with itself, however wide | Transactions and MVCC | test_txn_api_wide_commit_never_conflicts_with_itself — a hundred thousand keys commit in one transaction |
| Claims are taken before the sequence is drawn, so a validation sees every commit sequenced below it | Transactions and MVCC | test_txn_write_skew_inside_the_probe_yields_one, test_mvcc_read_stale_orders_by_sequence — a write-skew pair whose second commit lands inside the first’s validation of the key it read yields exactly one commit at repeatable read and at serializable; the probe of the store alone cleared both, since the second had not committed when the first looked |
| A prepare at repeatable read or above holds what it read, and an in-doubt batch holds its keys past its handle and across a restart | Transactions and MVCC | test_engine_prepared_read_refuses_a_writer_until_decided, test_mvcc_read_claim_refuses_writers, test_engine_recover_prepared, test_engine_recovered_prepare_keeps_its_read_claims, test_engine_abandoned_prepares_keep_their_keys_held — a writer of a key a prepare read or wrote is refused until the batch is decided, whether the prepare is live, adopted after a reopen, or freed undecided. The read keys are durable as a record of their own written ahead of the PREPARE, so a durable PREPARE always has them |
| An interval is given back once its batch is visible | Transactions and MVCC | test_engine_repeated_range_deletes_do_not_exhaust_the_reservations — the table has a fixed number of slots, so leaking one per commit ends with every later interval delete refused as a conflict no retry could clear; the test commits far more than there are slots |
| An interval and a point write inside it refuse each other whichever claims first | Transactions and MVCC | test_mvcc_interval_claims_meet_point_claims_both_ways, test_engine_prepared_prefix_delete_blocks_a_write_under_it — the interval is entered in the table before it is checked against the claims in flight, so a range delete and a write inside it meeting in the same instant find each other; before this the range delete ran alone under an exclusive commit gate, which every other commit at snapshot and above waited on |
| A scan’s footprint is validated like a read | Transactions and MVCC | test_engine_scan_footprint_refuses_a_phantom, test_txn_scan_footprint_refuses_a_phantom, test_mvcc_range_stale_orders_by_sequence — a key another commit put inside the interval an iterator covered, in the store or still a claim in flight below the scanner, refuses the commit at repeatable read and serializable; snapshot isolation records no footprint and commits |
| A copy measures the manifest and reads it inside one hold | Manifest | test_manifest_hold_keeps_the_log_still — a commit past the rollover bound renames a fresh snapshot over the same path, so a length measured outside the hold can be applied to a different file, leaving a catalogue that names klogs the copy does not hold. the test asserts both halves: that the hold blocks every commit, and that those commits really do replace the file once it lifts |
| A descriptor is reclaimed only at the idle reference count | fd manager | reaper, fdmanager tests |
| Every labelled descriptor open pairs with a labelled close | fd manager | test_engine_wal_descriptor_accounting_balances — the gate and the reaper both read the cross-label total, so a label that only decrements drives it negative and silently disables the budget while the reported per-label statistic still looks correct |
| A value-log segment is unlinked only after every reader inside it leaves | Value log | test_concurrent_reads_writes_and_reclaim; TSan across the suite |
| A sealed value-log segment is never modified | Value log | test_reclaim_leaves_the_active_segment_alone, test_recover_appends_to_a_fresh_segment |
| A value id names one block for its whole life | Value log | test_builder_records_vlog_segment_references — nothing is copied within the store, so an id cannot exist in two segments and a crash leaves no duplicate to resolve |
| Value-log liveness comes from what the installed tables report, not the index | Value log | test_reclaim_drops_dead_values — the index names every value whose segment has not been dropped, so counting it alone reports garbage as live and reclamation stops silently |
| A segment a builder may have written to is never reclaimed | Value log | test_reclaim_spares_what_a_builder_may_have_written — values are written before the table naming them is installed, and dropping that segment in the window loses them |
| A value in a segment being emptied is rewritten, not carried | Value log | test_builder_respills_a_value_out_of_a_draining_segment — carrying the reference leaves the output table holding the old segment, and it could never reach zero |
| A value is never split across segments | Value log | test_a_value_larger_than_a_segment_is_not_split — the segment target is a roll threshold, so an oversized value overshoots it whole rather than being divided |
| A segment with a reader inside it is never closed by the reaper | Value log | test_idle_segments_give_back_their_descriptors and the concurrent reclaim test, which evicts while readers are in flight; TSan across the suite |
Compaction correctness
| Invariant | From | Enforcement |
|---|---|---|
Versions are dropped only below min_snapshot_seq | Compaction | compaction_exec; fuzz_standalone model comparison |
| A tombstone drops only when every sstable that could hold an older version of the key is in the merge | Compaction | compaction_exec; fuzz_standalone. test_compaction_keeps_tombstone_with_sibling keeps one whose older put sits beside it at the same level, and test_compaction_drops_tombstone_below_a_newer_version drops one whose only sibling is a newer put above every input — asking the levels above refused that drop until the put merged down |
| A merge carries something into its target from above | Compaction | test_plan_never_rewrites_a_level_into_itself — a three-level family with an empty flush tier and L2 at its capacity, asserting the merge moves L2 into L3 rather than reading L2 alone and writing it back, which left it at its capacity and planned the same merge on every pass; test_plan_forced_pass_moves_the_shallowest_level_down and test_plan_forced_pass_rewrites_a_lone_largest_level cover a forced pass |
| Tombstone safety is evaluated over the whole job, not a sub-range | Compaction | compaction_pipeline_tests; this class of bug has escaped unit tests before and was found by fuzz_standalone |
| Outputs and input removals commit in one manifest batch | Compaction | compaction_pipeline_tests, manifest; test_crash_recovery_torn_compaction tears each write a merge issues and requires every key back, which is what a half-committed batch would cost |
| The planner performs no I/O and reads no clock | Compaction | compaction_planner — the tests pass it plain numbers, so an I/O dependency would not link |
| A read resolves against its snapshot; a compaction reads raw and retains for itself | Iterators | merge_iter, merge_sources — both scan the same sources twice, once resolving and once raw, and require the tombstones to appear only in the second |
Recovery
| Invariant | From | Enforcement |
|---|---|---|
| The clock resumes above the highest of four durable high-water marks | Recovery | test_engine_cf_id_not_reused_after_drop_reopen covers the family-id mark; the sstable and prepared marks are indirect |
| Replay applies at the record’s own sequence, not its file position | Recovery | test_engine_overwrite_across_recovery |
| The prepared-transaction staging map spans generations | Recovery | test_engine_prepared_survives_flush |
| Families are rebuilt before log records naming them are applied | Recovery | test_engine_cf_recovery |
| One create of a name wins, and the rest are refused as existing | Recovery | test_engine_concurrent_create_of_one_name_admits_one_and_reopens — sixteen threads create one name at once, five rounds, and the store reopens each time. Without the ddl lock every creator found the name free, each persisted a record of its own, the losers reported out of memory, and the next open refused the whole store |
| A duplicate name in the manifest keeps the record holding data | Recovery | test_engine_open_keeps_the_duplicate_name_that_holds_data — the winner’s data in the log, in a table, and nowhere, with the duplicate at the lower id. Keeping the lowest id fails it, and so does skipping the log scan |
| A damaged manifest is rebuilt from the sstables on disk | Manifest | manifest tests cover discarding the log; test_engine_rebuilds_catalogue_from_sstables covers the readopt, and asserts the data reads back rather than only that the open succeeded |
| A torn sstable is truncated to its last valid block | Recovery | test_crash_recovery_torn_flush |
Structural
| Invariant | From | Enforcement |
|---|---|---|
| Zero-length blocks are rejected at write | Block manager | block_manager tests |
| The trailing size must match the leading one | Block manager | block_manager, fuzz_decode |
| The footer records the geometry the file was written with | SSTable | sstable, fuzz_decode |
| The footer’s encoding pipeline is the one the nodes were encoded with | SSTable | fuzz_standalone — its families draw real pipelines and it reconfigures them mid-run, which is what caught the builder reading the family’s pipeline twice and straddling a reconfigure |
| Every value-log block carries the encoding chain it was written through | Value log | fuzz_standalone — a value carried forward by compaction into an sstable recording a different pipeline decodes by its own chain or not at all |
| The filter directory holds full first keys — no false negatives | SSTable | partitioned_bloom_filter tests |
A snapshot carries SEQ and CF_SEQ | Manifest | test_engine_cf_id_not_reused_after_drop_reopen |
| High-water marks are raised, never stored | Manifest | manifest tests |
No module below the engine takes a tidesdb_t | Architecture | Structural: every subsystem test constructs its module without a database, so a violation would not compile |
Process and shutdown
| Invariant | From | Enforcement |
|---|---|---|
| Background processes stop in reverse registration order | Thread manager | threadmanager tests |
| A pool never owns the queue it drains | Thread manager | bg_pool, queue |
| Every stop signals, wakes, and waits for the exit | Thread manager | bg_pool, bg_ticker — a missed wake hangs the test |
| No engine thread is ever joined | Thread manager | test_thread_finish_waits_for_the_body — a thread is detached at start and signals its own exit; a join would hand the joiner its CPU estimate unclamped on NetBSD and starve everything the joiner creates afterwards |
| Blocking on backpressure has a ceiling | Memtable and WAL | backpressure tests |
| The budget plus the untracked reserve fits the process limit | fd manager | test_fd_manager_budget_for_process — the cut is pure arithmetic so every case is asked directly rather than through whatever ceiling the machine running the suite has, including the default pairing that made it necessary: a 1024 budget against the usual 1024 POSIX ceiling. Bounds one database; several in one process cut against the same ceiling independently and can still exceed it together, which is the operator’s to size and the EMFILE retry’s to absorb |
Reader-gate rechecks and EMFILE/ENFILE retries are bounded, and to the same bound | fd manager | test_fd_manager_reader_gate_wait_is_bounded — drives the gate to its bound with a budget nothing can free, and times it: a gate that refuses without waiting and one that retries past its bound each fail a different assertion. A _Static_assert covers the two bounds matching, since when they did not, reads failed on a shortage the open path was about to clear. What is still untested is the real EMFILE path, which needs the process ceiling genuinely exhausted |
The gaps
Every entry above that is unguarded, incomplete, or only indirectly guarded says so in its own enforcement column — that is what the column is for, and counting them here only produces a number that goes stale the next time a test is added. The ones worth stating plainly rather than leaving in a table:
- The value log’s
O_DSYNCopen. Nothing fails if someone “optimises” it away. The result would be a durable key log referencing value bytes that never landed — visible only as a rare wrong-value read after a crash. Today the only guard is a comment. - Truncation syncing. No test kills a process between an
ftruncateand its sync. - The genuine
EMFILEpath. The reader gate’s bound is now tested directly, but by exhausting the engine’s own budget rather than the process ceiling. Nothing drives the process out of descriptors, so theEMFILE/ENFILEretry in the open path is still only covered by sharing its bound with the gate. - The sstable and prepared high-water marks in the clock reseed. The family-id mark has a direct test; the other two are covered only by broader recovery tests that would fail for many reasons.
- The two rotation rules — that the log is opened before the rotation and off the lock, and that the lock is never held across a wait or a device barrier. Neither has a test; a breach of either surfaces as stalled commits’ stacks under load, which is evidence you have to go looking for rather than a failure that reports itself.
- That the append path consults the backpressure policy at all. The commit path’s call site is now covered, but the append path’s own ring-lag signal is not, and a regression there shows only as a latency tail.
- That a copy freezes nothing. Backup and clone run against concurrent flush and compaction in the fuzzers, but nothing asserts the absence of a freeze, so reintroducing one would fail nothing.
The first is the most serious, because the failure is silent, arrives long after the change that caused it, and looks like data corruption rather than a durability bug.