How to Build an Enterprise AI Document Search System: RAG, OCR, and Financial Data Reconciliation

Your staff know the answer exists somewhere. It may be in a 70-page PDF on a network drive, a scanned contract filed under the wrong year, a handwritten note attached to a client record, or a spreadsheet maintained by one person who understands what its columns actually mean.

With roughly 500GB of mixed documents, finding that answer is not a chatbot problem. It is a data engineering, information retrieval, security, and operational workflow problem. A large language model is only the final interface.

A production system must ingest files without losing their structure, extract usable text from scans, preserve existing permissions, retrieve the right evidence, generate answers grounded in that evidence, and show citations that a staff member can verify. A second phase that reconciles a practice or portfolio management system with QuickBooks and spreadsheets adds an entirely different class of difficulty: identifying the same real-world transaction or entity when none of the systems share reliable IDs.

This is why the realistic plan is an 18-week, two-phase engagement rather than a weekend RAG prototype.

Start With the System Boundary, Not the Chat Interface

The first architectural decision is not which language model to use. It is what the system is allowed to read, who is allowed to retrieve each record, and what constitutes an authoritative answer.

The source estate usually contains more variation than the file extensions suggest. A PDF can contain digitally generated text, page images, embedded forms, signatures, or all four. A DOCX file may include comments, headers, tracked changes, tables, and text boxes that a basic parser ignores. A spreadsheet can be a structured table, an informal report with merged cells, or a miniature application held together by formulas and color coding.

Before ingestion begins, the project needs a source inventory. For each network share or repository, record its owner, approximate volume, file types, permission model, update frequency, retention rules, and expected authority. A signed agreement and an unsigned draft may contain similar language, but they cannot be treated as equally trustworthy.

The system boundary should also define what happens when sources conflict. If the practice management system shows one balance, QuickBooks shows another, and a spreadsheet contains a manual adjustment, the AI should not silently choose one. It should identify the discrepancy, explain the evidence behind each value, and send the record into a reconciliation workflow.

A useful high-level architecture separates the platform into six layers:

1. Source connectors and ingestion
2. Parsing, OCR, and document normalization
3. Chunking, embeddings, and search indexing
4. Retrieval, reranking, and answer generation
5. Identity, permissions, audit logging, and administration
6. Financial normalization, matching, and human review

Keeping these concerns separate makes the system easier to test and maintain. It also prevents a model upgrade from forcing a rewrite of the ingestion pipeline or financial matching logic.

Ingesting 500GB of Mixed Documents

At this scale, ingestion must be resumable, observable, and incremental. A script that recursively scans a directory and sends every file to an embedding API may work for a demonstration, but it fails operationally when a connection drops, a parser hangs, or a user modifies a file halfway through processing.

The ingestion service should create a manifest for every discovered object. That manifest can include the source path, file size, content hash, modification time, MIME type, source-system identifier, document owner, access-control metadata, processing status, and parser version. A queue-based architecture then moves each object through extraction, OCR, normalization, chunking, and indexing.

Tools such as Apache Tika, PyMuPDF, pdfplumber, python-docx, and LibreOffice-based conversion can cover much of the digitally generated content. No single parser handles every enterprise document correctly, however. A robust pipeline routes files based on format and detected characteristics, then records failures for targeted reprocessing rather than quietly dropping them.

Incremental processing and duplicate control

Reprocessing 500GB after every code change is wasteful and risky. The platform should distinguish between a new file, a modified file, a renamed file, and a duplicate stored in another directory.

Cryptographic content hashes help detect exact duplicates. Near-duplicate detection may also be valuable because organizations often store “final,” “final-v2,” and “final-approved” copies with minor differences. Those files should not necessarily be discarded, since their differences may be legally or operationally significant, but the interface can group them as related versions.

Processing state should be idempotent. If a worker crashes after creating chunks but before updating the manifest, running the job again should replace or reuse those chunks rather than creating duplicates. Each indexed chunk should therefore carry a stable identifier derived from the document version, page or section location, and chunking configuration.

Incremental ingestion also needs deletion handling. If a source document is removed because of a retention request, permission change, or data correction, its chunks, embeddings, cached answers, and derived metadata must be deleted or invalidated. Adding information to a vector index is easy. Proving that removed information is no longer retrievable requires deliberate lifecycle design.

OCR for scans and handwritten material

The pipeline should first determine whether OCR is necessary. Running OCR across every PDF increases cost and can replace accurate embedded text with less accurate recognized text. A page-level classifier can check whether a page contains a meaningful text layer, appears image-only, or has suspiciously sparse text.

Printed scans can be processed with engines such as Tesseract, Azure AI Document Intelligence, Amazon Textract, or Google Cloud Document AI. The right choice depends on document layouts, handwriting requirements, cloud restrictions, expected volume, and whether tables and key-value fields need to be extracted. Evaluation should use a representative sample of the organization’s real documents, not generic invoices supplied by a vendor.

Preprocessing materially affects OCR quality. Rotation correction, deskewing, contrast adjustment, noise reduction, page boundary detection, and image resolution can determine whether an amount is read as 8,500.00 or 3,500.00. The original page image should always remain available so a user can verify the recognized text against the source.

Handwriting requires a more cautious workflow. Recognition quality changes dramatically with cursive style, abbreviations, crossed-out text, form layout, and scan quality. The system should preserve OCR confidence where the provider exposes it and mark uncertain regions rather than presenting every transcription as fact.

Low-confidence handwriting can still improve discovery. For example, an uncertain transcription may help retrieve the right scanned case note, while the generated answer links the user to the original image and labels the quoted text as OCR-derived. It should not automatically support a high-confidence financial or clinical conclusion without human verification.

Why “embed everything” breaks down

A naive approach treats every extracted page or fixed block of characters as interchangeable text. That creates several problems long before storage cost becomes the main concern.

First, source quality varies. Navigation text, repeated headers, email disclaimers, blank OCR output, corrupted characters, and duplicated templates can overwhelm useful content. Second, permissions may differ by directory, client, business unit, or individual document. Third, the same sentence can mean something different depending on its section heading, table row, document version, and effective date.

Blind embedding also creates an index with weak provenance. If a retrieved vector cannot be mapped back to a stable document version and exact page, it cannot support a trustworthy citation. If a file changes, the platform may return both old and new language unless version invalidation is implemented correctly.

The right goal is not to maximize the number of vectors. It is to create the smallest defensible set of searchable units that preserves meaning, permissions, and traceability.

Chunking and Embedding Strategy

Chunking determines what evidence the retrieval layer can return. If chunks are too large, embeddings blur unrelated subjects together and consume excessive model context. If they are too small, a relevant sentence loses the heading, qualifiers, definitions, or table labels needed to interpret it.

A practical pipeline uses document-aware rules rather than one universal character count. For a policy document, chunks may follow headings and paragraphs. For a contract, sections and clauses are stronger boundaries. For meeting notes, dates and agenda items may define the useful units.

Chunk overlap can preserve context across boundaries, but excessive overlap produces nearly identical search results. Instead of applying a large overlap to every chunk, the platform can attach hierarchical context such as the document title, section heading, subsection heading, date, and named organization. That gives a short passage useful context without repeating entire pages.

Tables need separate treatment. Flattening a table into unlabelled text can detach values from their row and column headings. A better representation might store the table title, headers, row labels, values, page number, and a text serialization designed for retrieval. Complex financial tables may need both row-level chunks for matching and a full-table representation for contextual questions.

Choosing embeddings

Embedding models trade off retrieval quality, latency, cost, context length, language coverage, and deployment control. Hosted models simplify operations, while self-hosted models can provide stronger data-residency guarantees and predictable infrastructure control.

The choice should be made through retrieval evaluation rather than leaderboard position alone. Build a test set containing real questions, the documents that should answer them, and difficult distractors. Include organization-specific abbreviations, old names, misspellings, handwritten transcriptions, and questions where the correct result is “no evidence found.”

Model migration must also be planned. Embeddings from different models are generally not interchangeable, so changing models can require re-embedding the complete corpus. The index schema should include the embedding model name and version, allowing a new index to be built in parallel and evaluated before traffic moves to it.

For sensitive healthcare or nonprofit information, the hosting decision must include contractual and technical review. Teams operating under HIPAA and GDPR face different obligations around regulated entities, lawful processing, data-subject rights, and vendor relationships. Our detailed comparison of HIPAA and GDPR architecture requirements explains why a security control that supports one regime does not automatically satisfy the other.

Retrieval Architecture Determines Answer Quality

A powerful language model cannot repair missing evidence. If the retrieval layer returns an outdated policy, an unrelated spreadsheet row, and a low-confidence OCR fragment, the model has no reliable basis for answering correctly.

Vector similarity is useful when the user’s wording differs from the source. A question about “money still owed by clients” may need to retrieve documents that use “accounts receivable” or “outstanding balances.” Keyword search remains stronger for exact invoice numbers, names, codes, dates, and unusual phrases.

A production architecture should therefore use hybrid retrieval. A lexical engine such as OpenSearch or Elasticsearch can produce BM25 results, while a vector database or vector-capable search engine returns semantic candidates. The system combines those candidate sets, filters them by permissions and metadata, and sends the best candidates to a reranker.

Vector-store options include purpose-built products such as Pinecone, Qdrant, Weaviate, and Milvus, as well as PostgreSQL with pgvector and vector functionality in OpenSearch or Elasticsearch. The decision should account for filtered search performance, index update behavior, operational expertise, backup and recovery, regional hosting, and the ability to apply access-control constraints before results reach the model. A team already operating PostgreSQL may prefer pgvector for architectural simplicity, while a larger or more retrieval-intensive deployment may justify a specialized service.

Filtering, reranking, and query planning

Metadata filters narrow the search space using information such as department, document type, client, date range, approval status, and effective date. Those filters should be derived cautiously. If a user asks, “What did the 2023 policy say about travel expenses?”, filtering by year and policy type may improve precision, but interpreting every number as a year would break questions involving dollar amounts or account codes.

Reranking applies a more computationally expensive model to a relatively small candidate set. Unlike an embedding comparison that scores the query and passage independently, a cross-encoder reranker evaluates them together. This is particularly useful when several documents share similar language but only one contains the requested exception or date-specific provision.

Some questions require query decomposition. “Which funded programs exceeded their approved budgets, and what explanations were recorded?” cannot be answered with one nearest-neighbor lookup. The system may need to retrieve budget definitions, query normalized financial records, identify variances, and then retrieve narrative explanations associated with those programs.

Retrieval evaluation should measure whether the required evidence appears in the candidate set and how highly it is ranked. Answer evaluation alone can hide problems because an LLM may produce a plausible response despite weak evidence. A useful test suite includes direct lookups, multi-document synthesis, version-sensitive questions, exact identifiers, ambiguous language, OCR-heavy records, and intentionally unanswerable questions.

Generating Answers With Verifiable Citations

The answer-generation model should receive structured evidence packages, not a wall of loosely concatenated text. Each package can contain a source identifier, title, document version, page or section, access-safe link, extracted passage, and any OCR confidence warning.

The prompt should explicitly require the model to answer only from supplied evidence. It should distinguish statements supported by sources from interpretations or calculations. If the available passages do not answer the question, the correct behavior is to say so and optionally suggest a narrower search.

Citations should be generated from system-assigned source identifiers rather than allowing the model to invent filenames or page numbers. For example, the application can label retrieved passages [S1], [S2], and [S3]. The model cites those labels, and the application converts them into links to the precise page or document viewer location.

Citation validation should occur after generation. The system can reject references to nonexistent source labels, verify that every material claim has a citation, and check that quoted text actually appears in the cited passage. More advanced validation can compare each sentence against its cited evidence, but even deterministic source-ID checks eliminate a common class of fabricated citations.

The interface also matters. A user should be able to open the source beside the answer, see the highlighted passage, inspect the document’s date and status, and report an incorrect result. For OCR-derived content, the viewer should expose the original page image rather than only the normalized transcription.

High-risk actions should remain outside the answer-generation path. The system can identify a likely outstanding obligation, but it should not initiate a payment, update an accounting record, or change a client status based solely on generated prose. Retrieval answers support decisions. Transactional changes require explicit authorization, validation, and audit records.

Building the Financial Reconciliation Layer

Phase two is not simply “add QuickBooks to the chatbot.” It requires a normalized financial data layer that can compare records from three systems with different schemas, update schedules, and assumptions.

The practice or portfolio management system may organize data around matters, engagements, grants, clients, or projects. QuickBooks organizes records around entities such as customers, vendors, invoices, payments, accounts, and journal entries. Spreadsheets may use internal labels that appear nowhere else and may contain manual adjustments that were never posted back to either system.

The first task is source profiling. For every system, identify stable keys, optional fields, date semantics, currency handling, voided records, adjustment behavior, and extraction method. A transaction date, posting date, service date, and spreadsheet reporting month are not interchangeable.

The normalized model should retain both canonical fields and raw source values. A reconciled transaction might include normalized amount, currency, counterparty, document number, project, account, date, and status, while still preserving the exact original strings and source record IDs. This lets reviewers trace every transformation.

Matching records without shared IDs

Matching should begin with deterministic rules. Exact invoice numbers, known customer mappings, identical amounts, compatible dates, and stored cross-system references can resolve the simplest cases with transparent logic.

The difficult cases require probabilistic scoring. Names may differ because of punctuation, abbreviations, legal suffixes, former names, or transcription errors. One system may store Northside Health Services, another Northside Hlth, and a spreadsheet may use NHS Program. Amounts may differ because one system records a gross invoice while another contains partial payments or fees.

A matching model can combine several signals:

• Normalized name similarity
• Invoice or reference-number similarity
• Amount equality or explainable variance
• Date distance
• Project, client, or grant compatibility
• Description similarity
• Historical mappings
• One-to-one, one-to-many, or many-to-one transaction structure

An LLM can assist with ambiguous descriptions and proposed mappings, especially where unstructured notes explain a discrepancy. It should not be the sole arbiter of whether two financial records match. Deterministic calculations and bounded scoring remain easier to test, audit, and reproduce.

Human review as a designed workflow

Matches should be divided into operational queues. High-confidence, rule-supported matches can be automatically proposed or cleared according to the organization’s policy. Medium-confidence cases should enter a reviewer interface. Contradictory or structurally complex cases should be escalated.

The reviewer needs more than “accept” and “reject.” The screen should display both records, the signals contributing to the score, related documents, previous mappings, and any amount or date discrepancy. The reviewer may classify the issue as a timing difference, split payment, duplicate, missing posting, naming mismatch, spreadsheet override, or unresolved exception.

Every decision should create an audit record containing the reviewer, timestamp, source versions, matching-rule version, selected outcome, and optional note. Accepted decisions can improve future matching by creating controlled aliases or cross-system mapping tables. They should not silently train an opaque model without governance.

Reporting is then built on reviewed reconciliation states rather than generated narrative alone. Dashboards can show unmatched items, aging exceptions, variances by program, and changes since the previous run. The LLM can explain the report in plain English, but the totals must come from deterministic queries against normalized and reconciled data.

Deployment, Privacy, and Operational Security

Authentication is only the outermost security layer. The search system must preserve source permissions so that a staff member cannot retrieve restricted documents merely because those documents were embedded into a shared index.

A practical design synchronizes user and group identities from the organization’s identity provider, then associates documents with access-control lists or security labels. Retrieval filters must be applied before evidence is sent to the LLM. Hiding an unauthorized citation in the interface is insufficient if the model has already received the text.

Security controls should cover encryption in transit and at rest, secret management, environment separation, audit logging, backup policies, vulnerability management, and administrative access. Logs require particular care because prompts, retrieved passages, and model outputs may contain sensitive personal or financial information. Logging everything for debugging can create a second, poorly governed copy of the document estate.

Data residency and model-provider retention terms must be reviewed explicitly. The deployment may use managed cloud services, private networking, customer-managed encryption keys, self-hosted components, or a combination. The right architecture depends on contractual obligations, regional requirements, threat model, and the internal team’s ability to operate infrastructure safely.

Prompt injection must also be treated as a document-security problem. An indexed document can contain text instructing the model to ignore system rules, reveal other sources, or perform an action. Retrieved content should always be treated as untrusted data, tool access should be tightly scoped, and document text should never be allowed to redefine the system’s authorization rules.

Maintainability requires administrative tooling. Operators need to see ingestion failures, OCR warnings, stale connectors, index versions, unresolved permission mappings, and reconciliation queues. Without that layer, the platform will appear successful on launch day and deteriorate quietly as sources, schemas, and staff permissions change.

A Realistic 18-Week Delivery Plan

An 18-week engagement allows the team to build operational foundations, validate retrieval against real documents, and introduce financial reconciliation without mixing two difficult data problems into one release.

A sensible plan allocates approximately 10 weeks to document search and 8 weeks to reconciliation. Some activities overlap, especially security design, source discovery, and user-interface work. The exact week boundaries can move when source access or data quality issues emerge, but compressing the same scope into a few weeks generally means removing evaluation, permissions, review tooling, or production hardening.

Phase one: document intelligence and RAG, weeks 1–10

Weeks 1–2 focus on discovery, source inventory, security requirements, representative document sampling, and success criteria. This is when the team identifies unusual formats, permission inheritance, duplicate patterns, and authoritative document classes.

Weeks 3–5 build the ingestion pipeline, parser routing, OCR workflow, document manifest, normalized schema, and incremental processing. A controlled subset should be processed first so that extraction quality can be inspected before the full corpus enters the index.

Weeks 6–8 cover chunking, embeddings, hybrid retrieval, metadata filtering, reranking, citation generation, and the document viewer. The team also builds an evaluation set from genuine staff questions and tests failure cases, including questions the system should decline to answer.

Weeks 9–10 address bulk ingestion, performance tuning, permission validation, monitoring, acceptance testing, and deployment. Phase one delivers a usable search platform with cited answers, not merely a chat interface connected to a small sample folder.

Phase two: reconciliation and reporting, weeks 11–18

Weeks 11–12 establish connectors and profile data from the practice or portfolio management system, QuickBooks, and selected spreadsheets. The team documents field meanings, source-of-truth rules, update timing, and known accounting exceptions.

Weeks 13–15 implement the canonical financial model, deterministic normalization, initial matching rules, probabilistic scoring, and exception classification. Historical samples are used to verify that apparent matches are financially meaningful rather than textually similar.

Weeks 16–17 add human-review queues, decision auditing, reconciliation dashboards, and LLM-assisted explanations grounded in normalized records and related documents. Permissions are extended so financial access follows business roles rather than general document-search access.

Week 18 is for reconciliation acceptance testing, operational documentation, training, deployment checks, and handover. The final deliverable should include runbooks for failed imports, mapping changes, model or index upgrades, user-access issues, and disputed matches.

This Is the Kind of System We Build

A successful platform does not ask staff to trust a magical answer box. It lets them inspect the page behind an answer, distinguish approved records from drafts, see where OCR is uncertain, and understand why two financial entries were proposed as a match.

That requires AI engineering alongside conventional software disciplines: data modeling, background processing, identity integration, search evaluation, accounting workflow design, secure cloud deployment, and maintainable application development. If you are evaluating whether to hire AI developers, ask how they will test retrieval, enforce document-level permissions, validate citations, and reverse an incorrect reconciliation decision. A convincing model demonstration is not a substitute for those answers.

We build custom systems for organizations whose knowledge and operations are spread across network drives, line-of-business platforms, PDFs, scans, and spreadsheets. If that describes your document or data estate, the useful first conversation is not about choosing an LLM. It is about sampling the real files, mapping the existing permissions, and identifying the financial exceptions that consume staff time today.

Frequently Asked Questions

Does the entire 500GB corpus need to be embedded?

No. Backups, exact duplicates, unsupported binaries, obsolete exports, and noninformative pages may not belong in the searchable index. The ingestion manifest should preserve what was found and record why each item was indexed, excluded, quarantined, or sent for review.

Can the system run entirely inside our own cloud environment?

Yes, although the degree of isolation depends on the selected OCR, embedding, reranking, and language models. A private deployment can combine self-hosted models with PostgreSQL, OpenSearch, object storage, private networking, and organization-controlled keys, but it also creates ongoing GPU, patching, scaling, and monitoring responsibilities.

How do you measure whether the RAG system is accurate?

Use a versioned evaluation set of real questions with expected sources, required facts, and acceptable abstentions. Measure retrieval separately from generation so you can tell whether a wrong answer came from missing evidence, poor ranking, unsupported synthesis, or an outdated source.

What happens when documents change after launch?

The connector detects changes through timestamps, hashes, source events, or scheduled scans, then creates a new document version and updates the relevant index entries. Old chunks must be invalidated according to retention rules so the system does not answer from both the current policy and a superseded copy.

Should low-confidence financial matches be shown in AI-generated reports?

They can be shown as unresolved exceptions, but they should not be included in confirmed totals without an explicit reporting rule. The report should separate reconciled records, proposed matches, and unmatched items so a fluent narrative cannot conceal accounting uncertainty.

What should we prepare before commissioning a system like this?

Provide representative documents from every major format, read-only access to source structures, current permission rules, sample staff questions, and examples of difficult reconciliation cases. A sample containing drafts, poor scans, handwritten pages, split payments, renamed clients, and spreadsheet overrides is more valuable than a clean folder of ideal documents.