Matt Willsmore
Building a working AI search system is harder than most organisations expect. At Pureinsights we are often engaged to help our clients take an enterprise RAG prototype or pilot to production and over time we see that this consistently surfaces the same sets of challenges.
This article is drawn from a presentation I gave at the Pureinsights Special Interest Group (SIG) in June 2026 and includes short video clips from the talk at the relevant point in each section, with a link to the full recording at the end. It brings together lessons from real customer engagements in a variety of sectors, including retailing, publishing, and manufacturing.
The blog is organised around four layers: data ingestion, information retrieval, answer generation and performance at scale. They are not independent. The quality of indexing sets the ceiling for retrieval and retrieval sets the ceiling for generation. Getting one layer right while neglecting the others produces a system that may never make it to production.
1. Data Indexing and Ingestion
The starting point for most implementations is to use flat, fixed-size chunking: split the document every N tokens, embed each chunk and store it. For real-world collections like legislation, technical standards, and product documentation, this produces an index that ignores the inherent (and often very important) structure. Fixed-size chunking splits content at arbitrary boundaries. A heading can get separated from its body. A table row may be detached from its caption. A clause referencing the provisions of Article 7 is stored with no indication of what Article 7 says.
The resulting chunks are not always independently meaningful.
Structure-aware chunking
The fix is to align chunk boundaries with document structure rather than token counts. Modern parsing tools such as Docling & Llamaparse can parse using native structure: HTML heading levels, DOCX outline levels and XML schemas. When you do this, a heading and its body remain together. Another option is to use a vision model to identify text that should belong together. You can take this a step further by adding parent-child references to the index which lets the system expand context at query time by fetching sibling or parent chunks. This solves the clause-without-context problem without requiring large chunks during indexing, which would degrade embedding quality. In an agent-based system the agent can autonomously decide whether it needs to expand the context to understand if the question can be fully answered.
Non-text content is a separate issue entirely. Historically, ingestion pipelines have discarded tables and images in favour of just making plain text searchable. This was fine if the tables and images were still visible when you viewed the source document in your search results. But with a chat-based system, that may never happen! In financial and technical domains, tables carry the most precise queryable information: fee schedules, regulatory thresholds, and specification values. Indexing each table as both structured data and a natural-language summary makes it retrievable. It’s even possible to generate summary statistics across tables if indexed cleverly. Finally, vision models can generate captions for diagrams at ingest time. Losing this content is one of the most consistently underestimated sources of missed answers.
Chunk enrichment
Short clauses and definition rows embed poorly because they lack vocabulary. Generating a synthetic question or summary for sparse chunks at index time, a technique called HyDE (Hypothetical Document Embeddings) expands the retrieval surface without altering the underlying chunk. We can apply this selectively to short clauses, table rows and definition sections or globally as needed.
Watch this section of the webinar:
2. Information Retrieval
A single retrieval approach often works well for the vast majority of queries, but that’s not enough and it only takes one or two confidently asserted but totally incorrect answers to seriously shake confidence. Also, not all queries are semantic. Some need exact text: the precise wording of a legal definition, a verbatim provision. Dense vector retrieval approximates meaning and does not guarantee lexical accuracy. Some queries need the freshest version of a document: a superseded regulation or technical procedure ranking alongside a current one is not merely unhelpful, it creates professional liability or safety risks in some domains. Applying the same retrieval strategy to all of these produces plausible-looking but materially wrong answers.
Intent routing
The key architectural move is to classify query intent before retrieval and route each query to a specialist agent. Definitional queries go to a term glossary agent. Archival queries go to a version-aware agent that filters by effective date. Citation queries go to an exact-text agent. The classification costs milliseconds and adds no perceptible latency. This is consistently the change that produces the largest accuracy improvement across the deployments we work on. It also reduces cost by avoiding expensive retrieval techniques for queries that do not need them.
Authority & freshness
Authority scoring adds document freshness using ‘valid from’ and ‘valid to’ metadata and adds the importance of the source type to the retrieval ranking function, ensuring official current documents outrank superseded versions and less important data. Across multiple deployments, outdated or non-authoritative data is the top user trust issue. Every answer should surface the verbatim source passage, minimally via a link to the source document. In regulated industries it may be preferable to skip the Generation part entirely and to just use the LLM to select the most appropriate part/chunk to display.
Auto-filtering
In domains with large numbers of documents, deriving and applying a filter can substantially reduce the initial retrieval search space. For example, if the phrase ‘last year’ can be turned into ‘date > 2024 and date < 2026’ or ‘Sales deck’ can be parsed as ‘Sales deck filetype:pptx’ then we can remove a lot of noise in the result set. Our experiments have shown that asking and LLM to generate SQL as an intermediate format was the most effective in generating these sorts of filters, before transforming them into the appropriate search engine DSL format.
Graph retrieval
Where the data allows due to some inherent structure, GraphRAG can be used to extend retrieval with a graph layer, enabling multi-hop queries that flat retrieval cannot provide. Finding all obligations that apply to a particular entity under a given regulatory framework is a typical example. This can be particularly useful if the original data is backed by a knowledge graph, ontology or even just a database that models the relationship between entities. It allows an agent to traverse the graph to locate the subgraph that best answers the question while leveraging the surrounding nodes and edges to create a richer contextualization.
Watch this section of the webinar:
3. Answer Generation
We’re all familiar with the concept of hallucinations. For legal, regulatory, and safety-critical domains, a fabricated procedure or claim creates professional liability. Standard LLM training does not produce the conservatism these use cases require. Retrieval-grounded prompting addresses this. The model is instructed to answer only from the provided context passages, to cite the source by context reference number instead of returning an answer and to state uncertainty explicitly when context is insufficient. This more conservative generation approach is achievable through prompt design alone, without any model changes.
Compound queries
Real user queries frequently ask several things simultaneously. A question about reporting requirements for category B operators in Q3 covers scope, content, timing and potentially jurisdiction, all in one sentence. A single-pass retrieval cannot serve all of those sub-questions reliably. The result appears complete but is actually partial and the user has no signal that anything is missing. An intent classifier/router can be used to route the query to a decomposition step which splits compound queries into sub-queries before retrieval, processes each independently and then fuses the results into a composite response. The potential complexity of having multiple independent retrieval steps means that we’d recommend an evaluation step which prompts the user to reformulate or simplify the question if it fails. It may also ask clarification questions if there is too much ambiguity.
Watch this section of the webinar:
4. Performance and Cost at Scale
The dominant contributor to response latency is where the process is dependent on a third party hosted AI service which converts the user query into a vector and generates the answer. Reliance on public networks introduces variable latency that is not under your control.
In one engagement, the target was a 95th percentile response time under 400ms at 65 queries per second. Actual measured latency had a mean above 600ms. Switching to an open-source embedding model deployed in AWS SageMaker, moving inference physically closer to the retrieval platform, brought the 95th percentile to 144ms at the same throughput. The latency problem was solved by changing where the embedding ran, not what it ran.
Open-source embedding models now match or exceed commercial APIs on standard benchmarks. Self-hosting becomes cost-competitive at a lower threshold than you might expect.
Measurement and operational complexity
Any performance optimisation risks silently degrading quality as quantized or lower fidelity embeddings offer a reduced footprint in exchange for less precise semantic meaning. We’ve worked with several customers using a golden query to measure impact against before and after. Matryoshka compression to fewer dimensions produces a small precision loss while sometimes quartering storage costs but whether that precision loss matters for your specific corpus requires measurement, not assumptions in order to roll out safely without impacting user trust.
Operational complexity is also underestimated at scale. One engagement involved a client maintaining both a vector index and a traditional search index over the same content, with in some cases tens of thousands of metadata fields duplicated across each chunked record. Consolidating to a single document structure with unified ingestion eliminated the duplication, simplified synchronisation, and freed enough capacity to repurpose an entire search cluster.
Watch this section of the webinar:
Key Recommendations for Enterprise RAG in Production
Drawing together the lessons above, these are the recommendations I believe are most broadly applicable.
- Structure-aware chunking. Replace flat fixed-size chunking with boundaries aligned to document structure. Add parent-child linking and preserve non-text content. The quality of the index determines the ceiling of everything that follows.
- Intent routing. Route queries to specialist retrieval agents based on classified intent. This is consistently the highest-ROI change and adds little user-visible latency.
- Authority scoring. Incorporate source type and validity dates into retrieval ranking so official current documents outrank superseded versions and commentary.
- Exact text return. Surface the verbatim source passage alongside every generated answer with a link to the original document. This is the feature that converts accuracy into trust.
- Query decomposition. Multi-part queries are typically the largest gap between test-set and real-world accuracy. A decomposition step addresses this more effectively than a model upgrade.
- Measure before optimising. Build a golden query set and test every optimisation against it. Quality regressions that go unmeasured surface as user trust erosion, which is much harder to recover from than a delayed release.
Conclusion
The technology has improved dramatically and the economics are shifting. But the structural failure modes described here recur in many deployments we work on. The path from a promising prototype to a system that users trust is navigable. It requires treating each layer of the pipeline with the same seriousness and attention, making best use of the available data, and applying techniques to address issues as they are identified.
If you are working on an AI search or RAG deployment and would like to discuss any of the topics covered here, please get in touch.
You can watch the full 38-minute webinar recording from the beginning here.