AI Model APIs are hard to beat when you’re prototyping. Sign up, create an API key, and you’re sending requests within minutes; no infrastructure to provision, no model to deploy, no GPUs to manage. To get an idea off the ground and proving it works, this speed is exactly what you want. Discovery’s new Amazon SageMaker integration is built for what comes next.

The trouble starts when that prototype is moved into a real product with real users. The same API that felt effortless during testing begins to show its limits in production. Every prompt and every piece of context still leaves your infrastructure and travels to someone else’s servers. Your requests compete for processing capacity with every other customer hitting that same endpoint. Latency issues start to show up whenever the provider’s service is saturated. And service instability, whether a brief outage or a slowdown upstream, becomes your application’s problem too, even though it’s entirely outside your control.

Being able to manage production level traffic without these tradeoffs, while keeping the features that made the prototype a success, is the gap that Discovery, Pureinsights’ end-to-end platform to design, build and launch context-augmented search and AI applications, tries to close with its new Amazon SageMaker integration.

The Hidden Costs of API-Based AI Models

Data leaves your environment: Every prompt, every document, every piece of context gets sent to a third party. For regulated industries or any organization handling sensitive data, this is more than an inconvenience, it can be a compliance blocker.

You’re sharing capacity with everyone else: Rate limits and usage caps enforced by the provider, can throttle you during your own traffic spikes regardless of what infrastructure is behind it.

Performance depends on someone else’s infrastructure: Latency is at the mercy of the provider’s load at any given moment. A clean, well-optimized application can still feel slow simply because the underlying service is saturated.

Service instability becomes your problem: Outages, degraded performance, and unannounced changes on the provider’s side flow straight into your application. You can build retries and fallbacks around it, but you can’t fix the root cause, because it isn’t yours to fix.

Introducing the Amazon SageMaker Integration in Discovery

Discovery now supports integrating with AI models via Amazon SageMaker, AWS’s managed service for building, training, and deploying models, including LLMs and embedding models, on infrastructure that runs inside your own AWS account rather than a third party’s. Instead of routing every request through an external API, the model runs on infrastructure you own and control, dedicated entirely to your application.

Why This Matters: Dedicated vs. Shared Infrastructure

The main shift is that now you are running your model, in your infrastructure, with your rules. There’s no noisy-neighbor effect from other tenants, no rate limits to navigate, and latency becomes predictable because the endpoint exists to serve your application. Performance characteristics are now something you can reason about and plan for, not something that fluctuates based on a provider’s global traffic.

Because inference happens inside your own cloud boundary, data never has to leave it. Prompts, documents, and outputs stay within infrastructure you already control and audit. For healthcare, finance, government, and other regulated sectors, sending data to a third-party API is often simply not an option, regardless of how good the model is, and this can be the difference between a viable solution and a non-starter.

Real-World Example

Recently we had a hard requirement for a hybrid search pipeline where the end-to-end response time at P95 had to be under 200ms. That budget had to cover embedding the query, running the search, reranking, and formatting the response. The embedding step alone was supposed to be a small slice of that.

It wasn’t. Every option blew past the 200ms target on the embedding call alone, before a single search or rerank step had even run. We were not dealing with an inefficient pipeline, the embedding request itself was the bottleneck, and it was a bottleneck we had no way to fix from our side. Shared, AI model provider endpoints simply didn’t have the latency headroom the use case needed.

Moving the embeddings model to a dedicated SageMaker endpoint changed the math entirely. P95 latency dropped to 27ms, roughly 9 to 10 times faster than any of the API-based options. That freed up nearly the entire 200ms budget for the rest of the pipeline, turning a target that had been mathematically out of reach into one with comfortable margin.