Four Ways a Zero-Downtime Reindex Can Betray You

At Ironclad, we build the platform legal, sales, and procurement teams use to manage their contracts. Most contracts contain obligations: commitments someone has to act on, like paying an invoice within 30 days or renewing an insurance certificate every year. Customers track them in lists, filters, and dashboards in our product. Those pages read from a search index, a separate copy of the data arranged so searches come back fast, rather than from the main database. This summer we moved obligations into a search index of their own for each customer, which meant rebuilding indexes and switching customers onto the new copies while they kept using the product. This post covers four ways that switch can go wrong, and how we handled each one.

The one-sentence problem

A search index is a second copy of your data, derived (often asynchronously) from the database and shaped so read queries come back fast. Everywhere below, “index” refers to this definition, not that of a database. Because it’s a copy, it eventually has to be rebuilt — that’s called a reindex, and changing the index’s schema, resizing it, or splitting part of its data into a separate index all require a reindex. The recipe for doing it without downtime is well established: build a fresh copy of the index, flip a pointer to it, delete the old one. The unwritten discipline between building and flipping is: how do you know the fresh copy is good enough to put in front of users?

We started taking that seriously the day our pipeline refused to promote an index that was, in fact, completely correct. The rebuild had all it should have had, but the validation gate blocked the flip anyway, because a single stale database row three tables away threw its count off by one. Inconvenient, but expected: better to refuse a good flip before you risk a bad one.

That’s the whole game in one anecdote: a reindex looks like a data-copying problem, but it’s really a trust problem — and trust is never exact or free, so every step is a bet on which way you’re willing to be wrong.

So what can actually make a flip untrustworthy? A promotion has four primary failure modes**:** the copy is wrong (fidelity), the switch isn’t clean (atomicity), the wrong actor flips it (authority), or the copy rots between switches (freshness). These are the four ways the copy can betray you; operational failures (such as capacity, availability, observability, schema compatibility) can sink a promotion too, but they’re typically loud, glaring problems that are trivial to detect and address.

What we were actually building

These four failure modes came out of production work. We were moving contractual obligations — commitments inside a contract (“send the report by the 15th”; “renew 60 days before term”) that someone has to track and act on — out of a shared, general-purpose record store (a Postgres database, plus a shared Elasticsearch index derived from DB that user-facing pages read from) into a home of their own: a dedicated, per-tenant (i.e. customer company) Elasticsearch index. We also lifted the fields that lists, filters, and dashboards read most out of a loosely nested property bag and into first-class columns. An obligation has a status, a due date, an assignee, and a cadence, and those are now top level columns that search and filters can use directly.

One bigger move was making recurring obligation instances first-class citizens too. A one-time contractual obligation is a single row. A recurring one carries one instance for every time it comes due (Q1, Q2, Q3, …). Each of those becomes its own searchable document. Concretely: “provide an audit report every quarter” is one recurring obligation (the head) that expands into Q1–Q4 instances — four separately-trackable, separately-searchable due dates.

That’s where the volume comes from, and why the reindex has to be trustworthy. At Ironclad we extract about ****25 obligations ****per contract on average, so obligations outnumber contracts roughly 25:1 in the search index, and each recurring obligation (as part of a schedule) expands into ~30× obligation instances. A single contract immediately becomes hundreds of independently searchable documents, so the new index had to be sized, sharded, and rebuilt around contractual obligations, not the contracts they hang off.

Keeping that index correct while its schema and volume keep moving is what the reindex has to guarantee. Here is the rebuild-and-promote sequence, with each failure mode marked at the step where it applies:

1. Fidelity — the copy is wrong

To trust a rebuilt index, compare it to the source: does the new index have a document for every DB row that should have one? The naive check is count(rows) == count(docs), where two parts of the pipeline have to agree: the writer, which is the indexer that decides which rows become index documents; the validator, which counts what should be there and blocks the promotion if the index comes up short.

But the rows that “should be indexed” in reality are rarely the whole database table. It’s that table filtered by whatever rules your writer applies — this status is excluded, that related record has to exist, deleted things don’t count, etc. If your validator’s definition of “should be indexed” and your writer’s definition drift apart by even one rule, the count check turns into a trap. It blocks good rebuilds (the validator counts rows the writer intentionally skips) or, worse, waves through incomplete ones (the validator applies a rule the writer doesn’t enforce, so it under-counts and a short index clears the bar).

We hit this in its sharpest form. The validator counted by joining through a normalized relationships table (which obligation instances belong to which recurring obligations). The writer decided what to emit by reading a denormalized **copy of that same relationship — the same answer copied directly onto the row, so you don’t have to look it up in the other table. That’s two ways of answering the same question, and they agreed until the denormalized copy went stale for some rows and hence diverged from reality.

Result: the validator counted N, the writer emitted N−1, and the validation gate refused an actually complete rebuild by the writer’s own rules.

Validation only works if the two counts come from two genuinely separate sources of truth, and in the above diagram they do, which is why the counts disagreed at all. Break the rule and the gate passes a short index: either the validator leans on the same denormalized data copy the writer referred to (they’d “agree” while both use the wrong source), or the thing everyone called validation was really just a “did the writer finish running?” signal, one wearing a validation costume.

One fact (“which obligation head does this obligation instance belong to?”), stored two ways:

Which exposes the real culprit:

Denormalization is a cache that nobody admits is a cache.

It’s a cached answer to a join, sitting in your schema where nobody treats it like a cache — no invalidation contract, no fallback to the source when it’s empty. And the worst thing about a cache mistaken for truth is that a cache miss returns nothing instead of the real answer. An empty value in that column doesn’t raise an error; it just means “index zero documents for this row,” and the pipeline moves on reporting success. For a tenant that means an obligation that exists in Postgres, with a real due date and a real assignee, missing from every dashboard, filter, and overdue view that reads the index. A pipeline can report green while writing zero documents for thousands of live rows, because it read an empty cache and believed it.

Finding the gap and closing it

We caught it in pre-launch verification, before any customer saw it, reconciling Postgres row counts against Elasticsearch doc counts by hand while validating a backfill. The watcher, the process that monitors the Postgres table’s change “feed” and enqueues indexing work for each one, had advanced its change-id watermark past every row, so the pipeline’s metrics reported “success” while an empty denormalized column had turned each row into a no-op.

Every reindex asks “what belongs here”: in a rebuild that’s bookkeeping; in an index split it’s the entire **feature we’re shipping. We were splitting a subset of rows (obligations, plus their recurring instances) out into an index of their own; get it wrong and you don’t just miscount, you put the wrong rows in front of the wrong tenant. So an index split makes any fidelity failure even worse and more user visible, by taking away its hiding place.

The fix is upstream of the count: make “indexable” a single shared definition that both the writer and the validator derive from, then let each side compute it independently (from its own data). Share the definition to avoid false alarms from the two sides defining “indexable” differently; keep the computation independent so a real disagreement still surfaces.

We made two deliberate trade-offs:

• Count, not checksums. We validate the number of obligation index documents rebuilt, not whether each one is byte-for-byte right, because the cost scales with the volume: at ~25 obligations per contract and ~30 instances per recurring one, a checksum means re-reading and digesting every document on every rebuild. So we may accept a weaker guarantee on purpose: a count proves cardinality, not identity.
  • A rebuild can pass the validation by hitting the right counts with a combination of duplicate, missing, or stale docs. So counts are just the cheapest first gate, run by default; where correctness demands more (deletes, tenant isolation, compliance), you’d escalate to stricter reconciliation gates or sampled/partitioned checksums.
• A tolerance, not equality. The gate isn’t always count == count; it passes anything “within a small margin”, say 10 documents or 1%, whichever is smaller. A busy table keeps changing through a rebuild (from an old snapshot) that takes several minutes or hours, so demanding exact equality may keep a busy tenant from ever promoting.
  • The bar isn’t arbitrary: it sits below the write-churn we expect over a rebuild window, and the count is taken at the point the rebuild caught up to, not against the live table, so ordinary traffic doesn’t trip it.
  • The margin can also be one-sided: the gate blocks when the new index has too few documents and passes when it has too many. Missing data shows up as an empty screen for some users, while a few extra stale documents are generally harmless and cheap to catch later. That statement flips when the index gates tenant visibility, deletion, or compliance, where an extra document is the worse failure, and the margin should be one-sided the other way.

2. Atomicity — the switch isn’t clean

A promotion has to be atomic: no reader should ever see a half-built index missing half its rows, or a mix of the old and new copies like the dashboard below:

The obvious way to switch instantaneously is to make it “all-or-nothing” — to repoint a search engine alias from the old index to the new. The switch itself is atomic inside Elasticsearch, but it isn’t transactional with the rest of your system outside of it, such as your database:

• Elasticsearch, as a distributed search engine, propagates cluster state to its data nodes asynchronously, so for a moment after the index swap, a request routed to a lagging node can still resolve to the old index. Two queries in the same page load can then read different index copies, which is what the diagram above shows.
• A crash between the index swap and a matching Postgres write leaves the two disagreeing about which index is live.

So you make the switch in the database instead, with four guiding principles:

• A rebuild never writes into the copy that’s serving traffic. Every rebuild produces a brand-new generation (a complete, numbered copy of the index), built off to the side and caught up to the present through the same change stream as the live generation, so promoting it doesn’t hand readers a stale copy.
  • This carries a separate, promotion-time concern about the rebuild **window as there should be no dual-write path or reconciliation between the live index and the new one being rebuilt: both the live and new copies are fed by the same ordered change stream, one keeps taking ordinary writes, the other starts empty and replays forward the whole change stream until it has nothing left, then validates and flips — and live indexing resumes from here.
    • One interesting fork: the “rebuild” skips the deletes from the change stream while the live path applies them, because a rebuild from scratch directly reads the source database table that doesn’t contain those deleted rows to begin with.
• The one and only “which generation is live” pointer lives in Postgres, as a row, making the promotion a single transactional update of that row.
• Readers resolve the live generation from that row at query time.
• The old generation is deleted only after the pointer has moved and deleted only after a grace period: a short lease keeps it alive until in-flight reads arrive and retries drain*.*

If that sounds like a blue-green deploy, or like how a database does MVCC, that’s the same trick under the hood: you get atomicity across two systems (Postgres and the search engine in our case) without a distributed transaction: the search engine only ever gains new generations and never has a live one rewritten, and the single meaningful mutation that decides “what’s live” happens in a relational database, which already gives you atomicity.

Besides atomicity, the same design makes a failed rebuild cheap: it never touches the live generation and just leaves an orphaned one for cleanup. That buys you a few extra wins: retries during a promotion are cheap, and rollback after a promotion is cheap while the old generation still exists during that grace period.

3. Authority — the wrong actor flips it

A big index rebuild is slow since we build a new one from scratch, replaying the whole database change stream from day zero, so you parallelize it: split the tenant’s rows into batches and rebuild them concurrently. Each batch is handled by a worker; a single parent job fans them out and waits for all of them to finish. The natural next step, as with any parallel job, is to let each worker validate its own batch and report done. Here, that produces false alarms.

The better rule: workers build, and only the parent promotes**.** The completeness check from section 1 “Fidelity” is meaningless without a whole-tenant view. A single worker has seen only its own slice, so if it compares that slice against the whole tenant, it reports the generation incomplete while the other workers are still filling it. Only the parent knows when every worker has drained (i.e. caught up to the “present” and no rows remaining), so only the parent validates and flips the “promotion pointer.”

Then there’s the case of two parents. For instance, a stalled parent gets replaced, a retry starts a second copy of the parent job while the first is still running, then the first parent heals, and as a result two processes both believe they are the coordinator. Our obligations pipeline takes a lease on the pointer row before writing to it, such that only one parent can claim it at any time.

However, if writing to that pointer row is a plain overwrite, such leasing mechanism leaves a gap: it stops and fails a second parent only while the first one holds it, so once a lease expires (e.g. a “zombie” parent hung past the lease window of the second parent as its retry, then woke up to claim the lease as well), whichever parent writes last wins, and nothing stops the index pointer from going backwards onto an older index generation. A sibling pipeline in our codebase closes this gap: it reads the live generation, commits with where generation = <the one I read>, and fails any result other than “one affected row” without retries. It also mints each new generation strictly above the last, so the pointer can only ever move forward. Partition the work; centralize the commit; make the commit unable to go backwards.

Here is the same completeness check at both scopes (worker vs. parent):

One related hazard that bites anyone running multi-tenant on shared infrastructure:

Some operations that look per-tenant are secretly global.

If tenants (e.g. customer companies) share one physical index and are separated only by a routing key, then “recreate the index” is not a per-tenant action; it touches every tenant on the index, even though the bookkeeping that triggered it was scoped to a single tenant.

4. Freshness — the copy rots between switches

The index only matches the database because the watcher from section 1 “Fidelity” keeps copying changes into it, which walks down the database table in change-id order, passes each changed row to the indexer, and records how far it has got — that position is the “change-id watermark,” and everything below this watermark is supposed to be done.

Three things can drift out of date here.

The data is missing. A rotted index is missing a database row, and the gap never closes on it own because a change below the watermark never got indexed. One way that happens is a race condition where Postgres hands out a change-id before the watcher runs the downstream indexing work, not after its transaction commits the change-id row; in the meantime, the watcher can only see committed rows:

A reconciliation job can patch this by periodically hunting down suspected-missed rows and bumping their change id above the current watermark so the watcher picks them up again. That’s a safety net, while a reindex is the fundamental repair that catches everything missed so far. A robust fix in the code to root out the race means holding the watermark behind the oldest open transaction, or ordering the change feed by commit position rather than pre-commit sequence, however both at the cost of making the index slower to update.

The data is late. This is the milder and more common case (of drifting the index away from the database underneath it): the watcher did its job, the change is picked up by the indexer, but the change hasn’t reached the index and Elasticsearch hasn’t made it searchable and user visible. This is a “freshness” problem, and freshness is tunable.

How much such “staleness” is acceptable from a product lens depends on the kind of change:

• A newly created record showing up in search a second late is usually fine.
• A deleted record still showing up is not, as that looks like a correctness or trust issue.
• So the “delete*”* path warrants something the “create*”* path doesn’t: the caller blocks until the indexer confirms it has written the removal.
  • Worth being precise: both paths write to Elasticsearch with “the index refresh” off, which means a delete is confirmed written, not searchable — the document can still come back from a query until the next refresh. We kept it as a soft commitment rather than a hard guarantee of removal; the latter means forcing an Elasticsearch refresh on every delete call, so we preferred the “bounded wait.”

Take our obligations panel for example: before freshness tuning, you could edit or delete an obligation and watch the panel stay unchanged for about 15 seconds: up to 10s waiting for the watcher’s idle poll to pick up the change, then up to 5s more for Elasticsearch to make the write searchable (this is tunable via an index refresh interval). We dropped the watcher poll to 3s and the refresh interval to 1s to cut down the worst case from 15 seconds to 4.

Elasticsearch lets you change that refresh interval on a running index, but our in-house index config deliberately doesn’t — the interval is baked into the mapping hash that identifies an index generation, so lowering it meant rebuilding every tenant’s index, and this latency fix turned out to be one of the first customers of the reindex promotion machinery.

The routing is stale. Here what’s out of date is the system’s own answer to “which copy should I read?“ An index split (as one type of reindex) opens this window that a same-shape reindex doesn’t. We flip each tenant index over with a flag and no dual-write: flag off and everything reads the old shared index, flag on and everything reads the newly split one. But flipping the flag and minting that tenant’s first index generation are two separate events, seconds or minutes apart, so a read landing between them still has to land somewhere — we let ours fall back to the old shared index during this interim (between “flag is flipped” and “the index is not minted yet”), but it means the same tenant can be served out of two different indexes inside the same minute, which is correct but not ideal.

That fallback resolution carries two smaller lessons about readiness rather than freshness:

1. Cache the readiness signal only on “yes,” never on “not yet.” Reads resolve “is there a dedicated index for this tenant?” constantly, so memoizing the answer looks free. But that caching is a dead latch: the memoization happens once per tenant and stays there forever — so if the cached answer happens to be “nope not yet”, the cache effectively pins a tenant to “no new index” (which then falls back to the old shared index) for the rest of the read request even if a new index may just come online mid-flight. In our product, that means anything created/edited after the flag flip is missing, since the live index is short circuited.
   1. So cache only the “present” answer; re-check a “miss”, up to three times, then fall back to old shared index, because unbounded re-checking sends a tenant with no index back to Postgres on every read.
2. Provision new indexes ahead of the flag flip, not upon it. The lazy version creates a new index the first time a tenant’s indexing flag turns on, but that does not immediately make the index “built and warm”; so any page that reads only the new index shows an empty screen during that window in between.
   1. So when the index is slow to build, eagerly provision it before you flip the underlying flag, and treat the flag-on as meaning “everything is ready.”

None of those three kinds of drift settle on their own; and while “the routing is stale” is a failure mode just like the first three (fidelity, atomicity and authority), “the data is missing” and “the data is late” are the common tech debts that the reindex should catch on the spot, or a failure otherwise — because a reindex is the only reliable machinery that clears both anyway.

The recipe, in four lines

• Fidelity — the copy is wrong: validate against the writer’s exact definition of “indexable”: share the predicate, count independently, tolerate a small one-sided margin, and never trust a denormalized column as if it were the source.
• Atomicity — the switch isn’t clean: keep the index append-only, put the single “what’s live” pointer in a store with ACID transactions, and promote by moving that one pointer.
• Authority — the wrong actor flips it: parallelize the build, but let only the parent with the whole-tenant view validate and promote.
• Freshness — the copy rots between switches: tune the poll and refresh for staleness, rebuild for what the watcher silently missed, never cache a “not yet,” and provision a new index before you flip the flag that points at it.

Almost none of the four is really unique to search indexes; swap the engine and most of the specifics stay with it — refresh intervals, alias semantics, shard sizing, etc. A read replica, a cache, a materialized view, a vector store, or anything holding a second copy of your truth eventually needs a reliable way to promote a new version of that copy over a live one.

Why bother, and why now

You don’t necessarily need any of this to get an index working just once, but to keep one alive under load. Ours was a live, per-tenant index migration in a load-bearing spot, on a shared cluster we could neither take down nor overfill: already ~88% full disk, with the data we were moving projected to grow 25–30×.

A search index under a high-scale product isn’t something you build and forget: its schema changes constantly (ours has changed dozens and dozens of times), and the volume behind it doesn’t hold still. On a cluster that full, “just rebuild everything, all tenants at once, and fingers crossed” is rarely an option.

That’s why you build a reindex you can trust, per tenant, with no downtime: it lets the index keep evolving safely so the next 10× growth spurt does not turn into an outage. The recipe above makes a schema change, a backfill, or a capacity move (all of which trigger a reindex) a routine run of the same trusted and reusable framework — reliably, per tenant, at any size.

---

Ironclad is not a law firm, and this post does not constitute or contain legal advice. To evaluate the accuracy, sufficiency, or reliability of the ideas and guidance reflected here, or the applicability of these materials to your business, you should consult with a licensed attorney.