Building a multi-model database: the decisions before the code

Somewhere in your stack there is a user record in Postgres, a month of events in ClickHouse, a few million embeddings in a vector database, and a graph of relationships in Neo4j.

When someone asks you to delete their account, that delete has to reach all four. This is a solved problem. You write to an outbox table in the same transaction as the primary delete, a change-data-capture process drains it, the handlers are idempotent so retries are safe. It works.

What it isn't is free, and the writes were never the hard part. The hard part is reading. There is no point in time at which you can ask all four systems a question and get answers that agree. No consistent snapshot, so no consistent backup: your point-in-time restore is four restores to four slightly different moments, and you hope. Any query that spans two of them is an application-level join with no isolation, which means every "join" in your codebase is a small distributed systems problem that someone wrote in an afternoon.

None of that is impossible. All of it is a tax, and you pay it forever.

MeridianDB is my attempt at the other answer: one system with four storage engines instead of four systems with one each. Almost none of it is built yet. This post isn't about what I've written; it's about the decisions I made before writing a storage engine, and what each one cost.

What this is, and where it is

MeridianDB unifies row, columnar, vector, and graph storage under a single transaction, security, and metadata plane. It's C++20. The governing constraint, written down before anything else:

Distributed-first in design, single-node correct in implementation.

Every interface assumes partitions, failures, and asynchrony. Correctness gets proven on one node before a second exists. You'll see that line decide something concrete later in this post; it isn't decoration.

It's a learning and portfolio project, not a product, and saying so plainly changes what "success" means: depth of understanding, correctness under adversarial conditions, and a written record of the reasoning.

Status. Phase 0 of 7. About 1,450 lines of implementation: error handling, logging, and the value/key type system. 52 tests, green across four compiler and sanitizer configurations. No storage engine yet. Twelve of sixteen architecture decisions recorded.

Yes: there are more words of design here than lines of code, and this post adds to the imbalance. That ratio is the normal state of a project like this and it's also precisely where projects like this fail, because you accumulate documents that quietly contradict each other until the first real code discovers it. So before writing the storage engine I did a design review: reading my own architecture against how PostgreSQL and Oracle actually behave, and forcing every open question into a written decision with a stated cost.

This post is what came out.

Why C++, briefly

Rust was the real alternative and I don't think choosing it would have been wrong. I went with C++ for ecosystem fit and familiarity, and the cost is precise: memory safety becomes a discipline problem rather than a compiler problem. Every debug build runs AddressSanitizer and UndefinedBehaviorSanitizer with undefined behaviour set to abort rather than warn, and there's a ThreadSanitizer configuration alongside. That's not thoroughness for its own sake; it's the bill for the language choice, and it comes due on every build. It also shapes a decision later in this post that I wouldn't have needed to make in Rust.

Specialized engines, unified semantics

The thesis: workload-specialized storage engines, unified under common transaction and security semantics, beat both monolithic engines and polyglot persistence.

The specialization half is uncontroversial:

Row-oriented storage isn't slow at scans because it's poorly implemented. It's slow because summing one column of a hundred-column table means reading a hundred columns off disk. That's a layout property, not a tuning problem.

The interesting half is unification, and there were three real alternatives.

PostgreSQL plus extensions (pgvector, TimescaleDB, Citus). The pragmatic choice, and a good one. It's constrained in a specific way: the extension API binds you to Postgres's heap, its MVCC implementation, and its executor. You get vector search inside a row store's assumptions. That's a ceiling rather than a wall, and for most projects it's high enough.

Polyglot persistence: the stack from the opening, rejected for the reason in the opening.

Distributed SQL (CockroachDB, TiDB, YugabyteDB). Partially adopted: their distributed consistency model is the reference design and I've taken it wholesale. What I didn't take is the single-engine core, which is where their analytical and vector performance goes to die.

The part where I have to be honest

My thesis is not novel, and pretending otherwise is the fastest way to lose a reader who knows this space.

The industry is converging on multi-model from three directions. Oracle got there by accretion: Database In-Memory, shipped in 12.1.0.2 in 2014, maintains the same table simultaneously as row-format blocks on disk and a compressed columnar store in memory. Both stay transactionally consistent, and the optimizer chooses per query. That's "specialized engines, unified semantics," in production, for over a decade. Oracle 23ai then added a native VECTOR type with HNSW and IVF indexes inside the same transaction and security planes.

PostgreSQL got there by extension. I'm proposing to get there by design.

So the claim I can defend is narrower than "nobody has done this": the question isn't whether one system can hold several data models (Oracle proved it can) but whether the unification layer can be designed up front rather than accreted, and what that buys in optimizer quality and operational simplicity.

That's a thesis I can defend for a year of posts. The other one dies in the first comment.

What "unified" has to actually mean

Five things could be shared. They are not equally load-bearing, and pretending they are weakens the two that matter.

Non-negotiable: one write-ahead log. A transaction touching the row store and the vector index has, with two logs, two durability points, and can therefore survive a crash half-committed. There is no application-level fix. If you share nothing else, share this.

Non-negotiable: one visibility rule. A version visible to the row store must be visible to the vector index at the same read timestamp. Otherwise "transactional vector updates" is a phrase, not a property.

Strongly preferred: one catalog. One place that knows the schema, the statistics, and the access control. You could federate it. You'd regret it.

A choice, not a requirement: one buffer pool. I want a single pool so memory is reasonable about as a whole. But Oracle runs KEEP, RECYCLE, and default pools, plus separate pools per block size, and is not less unified for it. I'm choosing this; I'm not claiming it's forced.

The hard one: one optimizer. The cost model has to compare a B-tree seek against a vector index probe in a common currency. This is the least solved problem in the design and I'm not going to pretend otherwise.

And the cost, which matters more than the list: every engine now pays a tax to the shared layer, and any engine can be the reason the whole system stalls. Polyglot persistence buys failure isolation. Your vector database falling over doesn't take OLTP with it. I'm giving that up deliberately, and a design that doesn't name what it gave up isn't a design.

How the decisions got made

One architecture decision record per decision, written before the code it governs: context, alternatives, choice, consequences, including the ones I don't like.

The rule that shapes everything below: every decision names what it cost. A record where all the alternatives are strawmen isn't a record, it's marketing. If I couldn't write a paragraph explaining why a reasonable person would choose the other thing, I didn't understand the decision well enough to make it.

Three are worth walking through, and a fourth needs its own section.

On AI assistance

Since this post is entirely about how decisions got made, you should know what part of that process involved an AI model.

No AI has written any code in this project, and none will. Not a line. That constraint is the point of the exercise; I'm building this to understand storage engines, and you don't learn a B-tree by reading someone else's. What I do plan to use it for is code review: a second reader on concurrency and lifetime issues, where a fresh pass catches things I've stopped seeing.

The architecture is a different story, and I used it heavily there. Mostly as an adversarial reviewer rather than an author: arguing the case for options I'd already rejected, surfacing prior art I didn't know about, and finding places where my own documents contradicted each other. Several things in this post came out of that: the corrected framing on pgvector, the observation that Oracle's SCN is a commit timestamp, and more than one instance of "your stated reason and your actual design disagree." The decisions are mine. The pressure-testing frequently wasn't.

For the writing: used for proofreading and editing, and it will be used that way on the posts that follow.

I'm stating this because a post arguing that reasoning should be written down shouldn't be vague about how the reasoning was produced. Judge the decisions on whether they hold up. Every one of them is a file in the repo, with the alternatives and the costs, and you can check my work once the repo opens.

Three decisions where the other answer was genuinely attractive

1. Single-node scale, not billion-scale

My early documents specified vector search targets of 100,000 vectors in one place, one million in another, ten million in a third, and a billion in a fourth, all at the same latency. Those aren't the same engineering problem. A billion 1536-dimensional vectors in fp32 is about 6 terabytes of raw data before any index structure exists; it isn't a single-node question at all. A hundred thousand fits in 600 megabytes of RAM.

The decision underneath the tidy-up is a real one: this system targets what fits on one machine. Fixed at one million vectors at 768 dimensions, with every latency figure paired to a recall number, because any approximate index is fast at low recall, and a latency without a recall isn't a measurement.

Cost: the billion-scale claim is gone permanently, and with it the entire class of problems that make disk-resident graph indexes interesting. That's the most impressive-sounding thing I could have worked on and I've ruled it out on purpose.

2. Commit timestamps instead of transaction ids

When a row is updated, the old version must stay visible to transactions that started earlier. The mechanism deciding what's visible to whom is the most consequential choice in a storage engine, because it lives in the tuple header (on disk) and in every read path in the system.

PostgreSQL uses transaction ids: each row records the transaction that created it and the one that deleted it. It works, it's the best-documented implementation in existence, and it brings three things along. Transaction-id wraparound, with the freezing machinery that exists to prevent it. A snapshot that is an array of in-flight transaction ids rather than a single value. And a separate durable structure recording which transactions committed.

I chose commit timestamps: the row records when it became visible and when it stopped being. Wraparound stops existing. A snapshot becomes one integer and a visibility check becomes two comparisons.

This is where that north-star line earned itself. All three distributed timestamp designs (a centralized oracle, TrueTime, hybrid logical clocks) produce a 64-bit monotonic value that compares identically, so the tuple format survives distribution unchanged; only the clock changes. Transaction ids don't generalize at all: node 3's transaction 900 has no ordering relationship to node 7's transaction 400, so the mechanism would have to be replaced rather than extended. "Distributed-first in design" isn't a slogan here. It picked the option.

Cost, and it's real: it's the only combination with no readable reference implementation. PostgreSQL's heapam_visibility.c no longer describes my system, and there's nothing to read in its place. CockroachDB is Go on a different storage layer, and the closest research systems are papers rather than source. Visibility bugs are silent and catastrophic. Losing the ability to read someone else's correct implementation at 2am is not a small thing.

That decision has its own post coming, including the two things I got wrong in my own first draft of it.

3. A heap, not an index-organized table

Where do rows physically live? In a separate heap with indexes pointing at physical addresses (PostgreSQL, Oracle by default), or inside the primary key's B-tree with secondary indexes storing the primary key (InnoDB, Oracle's index-organized tables).

I chose the heap.

Cost: index-organized is arguably the better engineering choice, and it composes better with the version storage I'd just picked: no dangling physical references to clean up during garbage collection, and non-key updates that touch no secondary index at all, by construction.

The heap won on implementation simplicity: two separate structures can each be made correct independently. That's a schedule trade-off, not a technical verdict, and I wrote it into the decision record in those words.

Why I'm not using the industry-standard vector index

Every vector database leads with HNSW. I'm not going to, and the reason has nothing to do with benchmark position.

First, a correction to something I believed for a while. I'd been treating "transactional vector index" versus "eventually consistent vector index" as an architectural choice. It isn't, and pgvector proves it isn't: its HNSW index is WAL-logged, its entries are heap tuple identifiers, and visibility is resolved against the row after the index returns candidates. That's real transactional behaviour in the most widely deployed vector index in the world. It requires no new mechanism, because a vector index is just an index.

So the honest framing isn't "nobody does this." pgvector does. What doesn't do it is the standalone vector database (Pinecone, Qdrant, Weaviate), and not because they can't, but because they have no row to resolve visibility against and no transaction to resolve it in.

Which leaves a better question: given that it's mechanically free, why is asynchronous index maintenance the default everywhere it's an option?

Because inserting into an HNSW graph costs one to two orders of magnitude more than inserting into a B-tree. An HNSW insert runs a construction-parameter number of graph searches, then rewires the neighbour lists of its new neighbours, each with a heuristic pruning pass. A B-tree insert finds a leaf and writes into a slot array. Asynchronous maintenance is the consequence of that cost, not a design goal.

So the real question is whether the index can be maintained synchronously at an acceptable write latency, and that depends entirely on the family:

All three favour IVF, and all three are consequences of being a transactional database rather than a vector store. Insert cost is what makes synchronous maintenance affordable. Delete cost matters because in an MVCC system deletes aren't an occasional maintenance event: every update produces one, as a matter of course. Filtered search matters because "vector similarity with a SQL predicate" is the entire use case.

That's my actual differentiation, and it's narrower than "we have transactions": pgvector has those. It's that the index family was chosen for update cost rather than for benchmark position, because update cost is what a transactional system actually pays.

Cost: at high recall targets on a static corpus, a graph index wins on query latency, the thing users feel most directly. I'm trading that for write and delete cost, because in a system where every update generates a delete, an index that can't delete is an index that needs rebuilding on a schedule, and a database that needs rebuilding on a schedule isn't transactional in any sense that matters.

The decisions are a system, not a list

What I didn't expect is how often a decision was made somewhere else entirely.

Page size was decided by torn-page protection, two layers away. Physiological write-ahead logging (where a record says "insert this tuple into page 42's slot array" rather than storing raw bytes) can't repair a page that was half-written when the machine lost power. The standard fix is to write a full image of each page into the log the first time it's touched after a checkpoint.

Now do the arithmetic for uniformly random updates, which is the OLTP worst case. If the table has far more pages than the transaction touches rows, then modifying K rows dirties roughly K distinct pages regardless of how big those pages are; you just get fewer, larger pages to choose from. So the full-page-image cost is K × page size, and 16 KB pages cost twice the log volume of 8 KB for identical work. (For sequential or clustered writes the arithmetic reverses, which is why the qualifier matters.) I'd been assuming 16 KB for no recorded reason. It's 8 KB, and a durability mechanism decided it.

Phase one needs no lock manager at all. I'd specified a hierarchical lock manager with shared, exclusive, and intention modes, and wait-for-graph deadlock detection. Then the commit-timestamp decision made it unnecessary: when a transaction updates a row it marks the old version provisionally with its own identity, so a second transaction finding that marker on a still-running transaction is the write-write conflict. First updater wins, abort immediately. No lock table, no lock modes, no deadlock detector. An entire subsystem deleted by a decision made for unrelated reasons.

Transactional DDL became nearly free. If catalog rows carry visibility timestamps like every other row, rolling back a CREATE TABLE requires no mechanism: the catalog row simply never becomes visible. PostgreSQL has this and it's a genuine operational advantage; Oracle doesn't. I get it as a side effect.

Four things built before they were needed

None of these is clever. They're all the same category (cheap now, effectively impossible later), and the only skill involved is noticing which category a decision is in.

ARM continuous integration, before writing any concurrent code. x86 has a strong memory model: loads aren't reordered with loads, stores aren't reordered with stores. ARM's is weak and reorders both. The consequence is specific and nasty. A missing acquire/release pairing in lock-free code frequently works perfectly on x86 and silently corrupts data on ARM. Not crashes. Corrupts.

My next three milestones are a buffer pool with pinning, a B-link tree with optimistic lock coupling, and version-chain traversal. All three are built on that pairing; optimistic lock coupling is literally read-a-version-counter, read-the-node, re-read-the-counter.

The obvious objection is that I could just review my memory orderings carefully. Two problems. Review doesn't scale past the first few hundred lines of concurrent code, and ThreadSanitizer, which is the tool for this, finds races a test actually exercises and doesn't model ARM's ordering at all.

I'll be honest that with 52 tests and zero atomics, these jobs currently prove nothing. They're insurance, bought while the premium is a config file. The Release build matters most: -O3 reorders far more aggressively, which is exactly when this class of bug surfaces.

Wait-event instrumentation, before there's anything to wait on. Oracle's entire performance methodology rests on one question: what is this session waiting on, right now? Every latch, every I/O, every lock wait, tagged with a class and a duration. PostgreSQL added wait events in 9.6 (roughly two decades in), and coverage is still less complete as a direct result, because retrofitting instrumentation means finding every call site that ever blocks and you never find them all. It's a field and a macro now. It's an archaeology project later.

One I/O seam, so crash tests can be unit tests. All I/O goes through a single submit-and-complete interface; no direct pread anywhere else, ever. I chose that shape so an io_uring backend could be added without touching a caller. The bigger payoff was different: a seeded fault injector sits in that one seam and tears writes, reorders them, returns short reads, and fails fsync on demand. Crash-consistency testing becomes an ordinary unit test: no root, no device mapper, milliseconds per case, and a failure is a seed number I can replay in a debugger forever.

Invariant checkers as free functions returning bool. Never embedded assertions. Small shape decision; it means they become C++26 contract predicates with a one-line change instead of a rewrite.

The code that does exist

The least glamorous part of a database: error handling, logging, and the type system. I started there because these are the decisions you can't undo: everything written afterwards depends on them, and getting them wrong isn't a bug, it's a rewrite.

Capturing source location without a macro. Errors record where they came from. In C++20 you'd reach for std::source_location as a defaulted parameter, but a defaulted parameter can't follow a variadic pack, and error constructors take format arguments. So you smuggle the location in alongside the format string:

template<typename... Args>
struct FormatWithLocation {
    fmt::format_string<Args...> format;
    std::source_location location;
 
    template<typename S>
    consteval FormatWithLocation(
        const S& fmt,
        std::source_location loc = std::source_location::current())
        : format(fmt), location(loc) {}
};

The implicit conversion happens at the call site, so the location is the caller's. No macro, no extra argument.

Why the result type isn't std::expected. C++23 standardised exactly this type, my compiler has it, and it comes with monadic combinators I'd use. I kept my own, and the reason is narrower than "not invented here": mine is marked [[nodiscard]] at class scope, so every function returning one is diagnosed if a caller ignores it. std::expected isn't, which moves that guarantee onto per-function annotation and discipline.

The obvious rebuttal is that I could wrap std::expected in a [[nodiscard]] class and have both, and that's correct, I could. It's churn against a benefit I don't currently need, with 16 call sites and no monadic chains yet. When that changes, the wrapper is the answer. Until then this is a case where the non-standard choice is the cheaper one, and I'd rather say that than pretend the standard type is deficient.

Logging has a stance. The observability layer must never be the thing that takes down the database. Logging before initialization initializes lazily. An unwritable log file degrades to console-only. Total failure installs a sink-less no-op logger and the database keeps running.

And the smallest bug, which I like most. Equal values must hash equally. In IEEE floating point -0.0 == 0.0 is true, but the bit patterns differ, so the obvious hash gives them different hashes, and a hash-partitioned lookup would put them in different buckets while insisting they're the same value.

Two lines to fix. It's the whole character of this layer: the bugs at the bottom of a database are tiny, boring, and catastrophic.

Where this goes

About 1,450 lines, 52 tests across four configurations, twelve of sixteen decisions recorded. Phase one (page layout, buffer pool, write-ahead log, B-tree, multi-version concurrency control, catalog) is estimated at 375 to 560 hours, solo, at a pace that isn't full time. I'd rather publish that number than a date.

Next is the page layout, then the buffer pool, then the log.

Planned:

2. Choosing how a row remembers its past: version storage in full, including the two things I got wrong in my own first draft
3. The buffer pool: page replacement when several engines share one pool
4. Write-ahead logging and ARIES: making a B-tree survive kill -9
5. The B+-tree: concurrency without a global latch
6. Why the vector index is an IVF and not a graph
7. The architecture I'm not building: shared-nothing versus disaggregated storage

Post 2 is the one I'd read first. The decision looked settled when I started writing it down, and it wasn't.

The code, and how to argue with me

The repository isn't public yet. It opens at github.com/haytham-ichahbane/MeridianDB when post 2 goes up. Post 2 is the version storage decision in full, and it's the one that makes the decision records worth reading; I'd rather open the repo with it in place than without.

The part worth looking at won't be the 1,450 lines; it's docs/ADR/. Every decision in this post is a file there, with the alternatives I rejected, the reasoning, and an explicit statement of what the choice cost. Two of them record positions I reversed, including the original framing that this post corrects.

If you think one of them is wrong, I'd genuinely like to hear it, and you don't need the repo for that: the decisions are all above, with what each one cost. Name the option I dismissed too quickly and tell me what I missed. That's more useful to me than agreement, and several of them are close enough that a good argument could still move them.

Reach me on LinkedIn or through the contact page. Corrections and "you've misread how PostgreSQL does X" are especially welcome; I'd rather be wrong in a comment than wrong on disk.