Every AI search demo — from Elastic, from any vendor — runs on clean, structured, single-domain data. It performs beautifully. Your data is the opposite of all of that. The distance between what demos on Friday and what survives in production is where most projects go wrong.

The two gaps that cause the most damage are ones nobody talks about in the sales cycle.

Data governance comes before retrieval.  The demo assumes one canonical version of every document. Your environment has three versions of the same policy document: two outdated, one partially OCR’d from a scan, none marked as authoritative. A RAG system retrieves all three and generates a confident answer that blends them. Atlan’s 2026 RAG evaluation research calls this the “context trustworthiness gap” — you can score 0.95 faithfulness and still return business-wrong answers because the index is stale or contradictory. That’s not a retrieval problem. It’s a data governance problem. No platform solves it for you.

Evaluation has to come before optimization.  This is the one most teams skip entirely. Skylar Payne, who has audited dozens of enterprise RAG implementations, found that roughly 90% of teams add architectural complexity — rerankers, multi-step retrieval, routing — without measuring whether any of it actually improves results. Our CTO Phil Lewis has a simpler version of the same point: “measure, optimize, measure, optimize.” He and our senior architect Matt Willsmore walk through exactly how to do that — retrieval metrics, generation metrics, end-to-end task success — if you want a practical framework. If you can’t measure relevance against a representative query set, every architectural decision is a guess.

Three more things the demo won’t show you:

Permission-aware retrieval.  Most pilots are built without document-level access controls wired through the retrieval pipeline. Elasticsearch has the right primitives — field-level security, document-level filtering, role-based access. But connecting those controls correctly at query time is non-trivial work that gets deferred when teams are racing to ship. The result is a system that works in a sandbox and leaks sensitive content the moment it meets real users with real permission boundaries.

Hybrid search, configured carefully.  Elasticsearch’s hybrid surface — BM25 lexical retrieval combined with dense vector kNN — is powerful but genuinely expert-grade. Elastic calls it that themselves. Doug Turnbull’s 2025 production analysis documents how naïve configurations silently degrade: pre-filter vs. post-filter behavior in kNN queries, the three-way tension between similarity thresholds, candidate pool size, and filter application. His benchmarks show a naïve hybrid setup scores around 0.71 NDCG. Getting meaningfully above that requires deliberate engineering. The platform doesn’t configure itself.

Exact-match requirements.  Vector search is significantly weaker at exact matches than BM25. In most enterprise environments — legal, financial, regulatory, technical publishing — users query by clause number, case reference, SKU, or error code. A pure semantic approach confidently returns adjacent results and misses the precise document the user needed. Invisible in demos. Costly in production. BM25 handles it natively. Don’t throw it away.