Matt Willsmore
Introduction: The Promise
Many of our clients are organisations who have dived head-first into custom, bespoke Retrieval Augmented Generation (RAG) implementations to find out how it might work with their data, and found themselves out of their depth and needing a lifeline to make it work. This AI stuff is fun, easy to start, but not so easy to get finished. Sometimes some expert help is needed.
- Picture the scene: your IT team has just patched a zero‑day security hole, finished a late‑night database upgrade, and cleared 137 help‑desk tickets, only to come in the next day to be greeted with, “We need one of those RAG systems, how hard can it be?”
- The promise is irresistible: smarter search, faster insights, better bridges between knowledge silos.
- The reality: somewhere between the proof‑of‑concept and a board‑ready production launch lurks a maze of data pipelines, GPUs, compliance sign‑offs, and model versioning.
Bottom line: getting an AI to behave the way you want it to across your data is not as easy as it might look initially. The road to production might be longer and bumpier than you think.
The Temptation to DIY
So back to “How hard can it be?”. Initial impressions might be that it’s easy: we just spin up a vector database, chuck in our data and then get the AI to use it. Bish bash bosh. Obviously, it all needs to be secure and compliant too.
The thing is there are some complexities lurking there:
| Task | Unforeseen Complexity | Impact |
|---|---|---|
|
Spin up a vector database |
Data quality checks, schema drift, scalability issues, hybrid search challenges |
Ops overhead > pilot budget |
|
Chuck in our data |
Synchronisation issues, data transformations, freshness, chunking strategy, patchy data quality |
Data engineering overhead and expertise are needed |
|
Get the AI to use it |
Fine tuning, and training, GPU quotas, guard‑rails, benchmarking, hallucinations, ground truth data sets |
Rising cloud/infra bill, talent hunt |
|
Secure and compliant |
Data leakage, audit trails, compliance headaches, open source library security |
Auditor on your back |
Some of these aren’t in the traditional IT job description, yet they’re critical to make a production‑grade RAG, so someone will have to figure it out on the job OR new hires are needed. Sometimes doing it all in house from scratch is not the best idea.
The AI Project Graveyard
The risks of AI projects getting stuck before getting to Production or being abandoned due to a lack of expertise are very real. Below are some recent, publicly‑available figures that show how few AI (and by extension RAG) initiatives ever make it past the proof‑of‑concept (PoC) stage and into steady‑state production.
| Original Source | What Was Measured | Made it Past PoC |
|---|---|---|
|
Boston Consulting Group survey of 1,000 C‑suite execs across 59 countries (2024) |
Companies that “developed the capabilities to move beyond PoCs and generate tangible value” |
26 % (BCG) |
|
Gartner forecast (2024) |
Generative AI projects likely to be abandoned after PoC by end 2025 |
30 % (Informatica) |
|
IDC CIO survey (2024) |
Custom built AI apps that clear PoC and enter production |
≈10 % (CIO) |
|
S&P Global Market Intelligence (2025) |
Share of PoCs scrapped before production |
54 % on average; 42 % scrapped (CIODive) |
|
Informatica digest of analyst data (2025) |
Overall AI project failure rate across industries |
80% of AI projects fail (Informatica) |
Sources:
- CIO Dive : AI project failure rates are on the rise: report
- CIO : CIOs’ lack of success metrics dooms many AI projects
- BCG : AI Adoption in 2024
- Informatica : The Surprising Reason Most AI Projects Fail
What this means:
- The funnel is very steep. Even the rosiest view (54 %) still means almost half of initiatives haven’t (yet) reached production.
- Custom builds fare worst. IDC’s finding that only ~1 in 10 home‑grown AI apps survive PoC suggests specialist expertise and repeatable patterns matter.
- Generative‑AI & RAG aren’t immune. Gartner expects nearly a third of Gen‑AI projects to be abandoned outright, and S&P reports almost half of PoCs never graduate.
- Hidden costs hurt. S&P and IDC both link high abandonment rates to spiralling infra spend, data‑quality hurdles and talent gaps, typical pain points for RAG systems as well.
So plan for success rates in the 10-40 % range unless you already have mature data pipelines, AI‑ready infrastructure, and dedicated ML ops talent in place. Anything materially better usually comes from organisations that either a) invested early and heavily to build in-house expertise, or b) leaned on external specialists who’ve solved the production gap many times before. In many cases an expert will help you evaluate which uses cases are more likely to succeed by looking at data quality, success criteria, security challenges and choice of model right up front. This helps you back the AI horses most likely to succeed.
The RAG Decision Matrix: DIY vs. Expert Help
Given this backdrop it’s worthwhile considering whether to Do It Yourself or to Phone a Friend. RAG is new territory and requires new techniques, new uses of data and new ways of measuring success. To help you decide whether to go and seek expertise, ask yourself these questions:
- How quickly does the business need results?
- Can you afford two extra quarters of experimentation and tweaking?
- Do you have search engineers fluent in information retrieval, hybrid search, AI tuning?
- Who will own prompt‑engineering, metrics and hallucination monitoring after go‑live?
- What’s your plan for model refreshes every 6–9 months?
If any of these cause doubts it may be worth considering drawing on external expertise to provide advice and guidance or implementation services and on-going support.
The Upside of External Expertise
If you decide, even just for validation purposes, that some support would be helpful — whether to confirm you’re choosing the right use cases and technology stack or to ensure you’re not overlooking key opportunities — then it’s worth considering the value of external expertise. When a solution has the potential to transform your business and deliver a strong competitive advantage, it may deserve more than a homegrown effort. Here are some of the benefits you can expect by bringing in experienced professionals:
- Accelerated proof‑points: reference architectures and pre‑baked evaluation harnesses allow you to understand what works much more quickly.
- Use case selection: with the average organisation conducting 37 AI PoCs and only a handful succeeding, prioritisation is key, and expert experience is essential in evaluating what will work.
- Knowledge Transfer: team augmentation or sessions with specialists often upskill your staff faster than self‑study.
The Conclusion: Getting Your RAG System to the Finish Line
A successful RAG initiative isn’t just about standing up a clever chatbot to wow people with a scripted demo, it’s about sustaining it through model upgrades, shifting user expectations, and crucially ensuring it actually demonstrably enables people to get things done faster and better. Whether you expand your in‑house skill set, lean on a partner for the heavy lifting, or find a sweet spot in between, make the choice that keeps your timeline realistic, spend under control and maximizes the potential for success.
This blog is primarily about using the right expertise, but sometimes the right frameworks and tools can really make life easier. We can help with that too. The Pureinsights Discovery platform helps us create AI-powered applications for our clients. Whether it’s internal knowledge systems or external portals, we designed it to make information easier to find and more actionable.
Feel free to CONTACT US for a free demo on your website data or just for a chat about your AI and search journey.