RAG at Scale: From Single-Doc to Multimodal Production Pipelines

Clarix is a compliance workspace — asset managers and reviewers use it to ingest documents, validate them against regulatory rules, simulate what-if scenarios, and generate auditable, citation-backed answers, all logged through an append-only Merkle ledger for traceability. It's a larger system than any single post can cover, so this one is scoped deliberately: it's about the retrieval-and-generation layer underneath the compliance engine — not the rules validation, audit trail, or governance tooling built on top of it.

That layer started as the simplest possible version of retrieval-augmented generation: upload a document, ask a question, get an answer grounded in that document. That version works fine as a demo. It falls apart the moment you point it at a real corpus — dozens of filings, mixed formats, questions that span more than one document — because every assumption that made the single-doc version simple stops holding.

This is a write-up of what actually broke on the way from "one document, one answer" to a production-shaped multimodal RAG pipeline underneath Clarix's compliance engine, and the trade-offs I kept running into at each stage.

The single-document version is deceptively easy

A one-document RAG pipeline is almost embarrassingly simple: chunk the document, embed the chunks, embed the query, retrieve the top-k nearest chunks, stuff them into a prompt, generate. It works well precisely because the hard problems haven't shown up yet — there's no ambiguity about which document an answer should come from, no need to reconcile conflicting chunks from different sources, and the corpus is small enough that even a mediocre chunking strategy retrieves something relevant most of the time.

None of that survives contact with a real multi-document compliance corpus.

Chunking: the decision that quietly determines everything downstream

Chunking looks like a preprocessing detail. It isn't — it's the decision that most directly controls what retrieval is even capable of finding.

Fixed-size chunking (split every N tokens) is the easiest to implement and the easiest to get wrong. It cuts through sentences, tables, and section boundaries with no regard for where the actual semantic units are, which means a chunk can contain half of a relevant answer and none of the context that makes it interpretable. The failure mode is subtle: retrieval "succeeds" — it finds a chunk with the right keywords — but the chunk doesn't actually contain a complete, answerable unit of information.

Semantic or structure-aware chunking (splitting on headings, paragraphs, or embedding-similarity boundaries) fixes a lot of that, at the cost of variable chunk sizes that complicate batching and, in my experience, still need document-type-specific tuning. A regulatory filing and an internal audit memo should not be chunked the same way — a filing's meaningful unit is often a numbered clause or disclosure item; a memo's is often a full paragraph or a findings table row. Clarix ended up needing different chunking heuristics per document type rather than one universal strategy, which is a more honest answer than "we solved chunking" but a less satisfying one.

The trade-off in one sentence: smaller chunks improve retrieval precision and hurt generation coherence; larger chunks do the opposite. Every RAG system is choosing a point on that curve, whether or not the choice is explicit.

Embeddings: the model matters less than the mismatch

It's tempting to treat embedding model choice as the highest-leverage decision in a RAG pipeline. In practice, the bigger source of failure was mismatch between the embedding space and the content being embedded, not the raw quality of any single model.

Two concrete versions of this showed up repeatedly:

• Query-document asymmetry. A short, informal question like "did we flag this vendor for AML risk last quarter?" and a long, formal compliance filing don't naturally land close together in embedding space even when the filing answers the question well. Some embedding models handle this asymmetry better than others; some setups need a separate query-encoding step tuned for it. Ignoring this quietly caps retrieval quality no matter how good the underlying model is.
• Domain drift. A general-purpose embedding model trained mostly on web text does a mediocre job distinguishing between two similar regulatory clauses — say, two disclosure requirements that differ in one jurisdiction-specific detail — because the distinctions that matter in compliance work weren't well-represented in its training distribution. This is the harder problem — fine-tuning or domain-adapting an embedding model is a real project on its own, not a config change.

The pragmatic middle ground Clarix uses is picking embeddings suited to the dominant content type per corpus, rather than one embedding model for every document type uploaded — with the acknowledged cost that this makes the system less "one model fits all" and more an ensemble of decisions.

Going multimodal: text was the easy part

Extending Clarix to handle images, code files, and mixed-format documents exposed a different category of problem: there is no single shared embedding space that treats a diagram, a code snippet, and a paragraph of prose fairly.

The approach that held up reasonably well was treating each modality as its own retrieval track rather than forcing everything into one vector space:

• Text gets chunked and embedded as text.
• Code gets chunked along function/class boundaries rather than line counts, and embedded with a code-aware model — treating a function as prose loses the structural information that actually makes code retrievable.
• Images and scanned exhibits get captioned or OCR'd where relevant, and the resulting text is embedded alongside a direct image embedding, so a query can match either the described content or the visual content.

Retrieval then runs across tracks and merges results by relevance score, rather than pretending there's one universal similarity metric across modalities. This is more moving parts than a single-embedding pipeline, and it's the honest cost of "multimodal" — it's not one model doing more, it's several retrieval paths being reconciled.

The retrieval-generation tradeoff, stated plainly

The core tension in any RAG system is this: retrieve too narrowly and the model hallucinates around a gap; retrieve too broadly and the model gets lost in irrelevant context and starts blending sources it shouldn't.

This matters more than usual in a compliance context, where a plausible-sounding but ungrounded answer isn't just an inconvenience — it's a liability. A few things that helped manage this in practice:

• Grounded refusal over confident guessing. When retrieval confidence is low, the system should say so rather than let the generation step paper over the gap with a fluent-sounding but ungrounded answer. This is a UX cost — "I don't have enough information to answer that confidently" is a less satisfying response than a wrong one that reads well — but it's the difference between a tool a reviewer can actually sign off on and one that just sounds trustworthy.
• Citing the source chunk, not just the answer. Every generated answer in Clarix links back to the specific chunk(s) it drew from, which then feeds into the audit trail. This doesn't prevent hallucination, but it makes it checkable — a reviewer can verify a claim in seconds instead of having to trust the system's confidence.
• Re-ranking after retrieval, before generation. An initial vector-similarity retrieval pass is fast but imprecise; a second re-ranking pass (heavier, but only run on the much smaller candidate set) catches cases where the top-k by raw similarity isn't actually the top-k by relevance to the specific question asked.

What's still unresolved

To be direct about the current limits: cross-document reasoning — synthesizing an answer that genuinely depends on combining information from multiple filings rather than just retrieving from the best single one — remains the weakest part of the pipeline. Retrieval finds the right individual chunks reasonably often; getting generation to correctly reconcile or combine them, rather than just concatenating context and hoping, is the open problem I'd point to next.

This post covered the retrieval and generation layer. The audit trail, Merkle ledger, and compliance-specific features — rules validation, impact simulation, greenwashing detection, framework alignment — are their own story. I'll be writing about those next.