Aaron Vargas
We’re thrilled to announce the new integration in the latest release of the Pureinsights Discovery Platform, support for Vespa. The open-source engine for real-time, scalable, and intelligent search applications.
This integration unlocks new possibilities for building high-performance, hybrid search experiences that are both scalable and intelligent. Whether you’re deploying on Vespa Cloud or self-hosting.
Vespa brings powerful capabilities for scalable, high throughput search, making it a great engine to power the enhanced search experience provided by the Pureinsights Discovery Platform.
What ‘s New?
To enable a seamless integration between Discovery and Vespa, we’ve introduced two new components:
- Discovery Ingestion Vespa Processor – Leverage the full power of the Discovery Ingestion infrastructure to route, transform, and enrich your data before it’s fed into your Vespa application. It bridges the gap between raw data and Vespa’s document schemas, enabling smooth and efficient data onboarding.
- Discovery QueryFlow Vespa Processor – Enables you to query Vespa-managed data using Discovery QueryFlow endpoints. Bringing powerful search within easy reach through the Discovery QueryFlow API.
Why This Matters
Vespa’s architecture handles billions of documents with real-time performance and supports hybrid search (lexical or semantic search), combined with the Discovery’s plug-and-play processors simplify integration and accelerate development, making it easy to go from prototyping to production ready solutions.
How the Pureinsights Discovery Platform Unlocks the Full Potential of Vespa
Vespa excels at delivering lightning-fast query responses, while the Pureinsights Discovery Platform enables a Google-like search experience. By combining the strengths of both platforms, you can achieve high-performance, intelligent search with just a few configurations.
Data Feeding
The Discovery Platform includes a range of built-in components designed to fetch data from various sources and ingest it into the Discovery Ingestion pipeline. This pipeline offers a variety of components with different actions that allow you to enrich, transform, and prepare your data before it is ingested into Vespa.
You can apply transformations such as data verbalization, vectorization, or structural changes like combining, adding, or removing fields to better align with your use cases. The pipeline also supports record filtering and/or classification using our routing system, ensuring that the right data is delivered to the appropriate Vespa schemas.
Data Querying
The Discovery QueryFlow API allows you to expose custom endpoints for querying your data, from traditional keyword search to modern semantic vector search. It enables the use of NLP techniques, large language models (LLMs), and knowledge graphs to enhance the quality and relevance of the user’s search experience.
Vespa includes a built-in Retrieval-Augmented Generation (RAG) system that supports the generation and enrichment of answers directly within its ecosystem. Discovery not only supports similar capabilities, but it also integrates seamlessly with Vespa’s RAG pipeline. This integration enables Discovery to dynamically enrich and refine prompts based on the user’s query before they are passed to Vespa.
As a result, you can leverage either Discovery’s or Vespa’s RAG capabilities, or combine them, using Discovery as an intelligent preprocessing layer to optimize prompts and deliver more accurate, context-aware responses.
What You Can Build with the Discovery + Vespa Integration
Let’s say we’re building a global E-Commerce platform with a catalog of millions of products. To deliver a truly intelligent and modern shopping experience, we want to support features such as:
- Faceted filtering: for a more structured search, allowing users to narrow down results by brand, size, price and other attributes.
- Hybrid relevance: combining traditional keyword search with semantic similarity to enable suggestions like “You may also like”.
- Assistive search: handling complex queries such as “Looking for a gift for a 12-year-old who likes robotics and science,” and returning a diverse, relevant set of products.
- Generative answers: providing informative responses, such as comparisons, or information based on product reviews, to help users make confident decisions.
These are just some of the capabilities unlocked by this integration. Below is an architecture diagram showing how the overall solution is structured:
We start with ingestion, where data is transformed and modeled to support the features described above. This can include:
- Enriching product data with descriptions (similar to data verbalization).
- Vectorizing specific fields for semantic search and similarity.
- Restructuring data to fill in missing information or remove ambiguity.
This processed data is ingested into Vespa, which serves as the engine behind the search experience. With its scalability, high performance, and hybrid search capabilities, Vespa is an ideal fit for powering intelligent product search.
The Discovery Staging serves as an intermediate storage. It can be used when working with Discovery Ingestion or QueryFlow components, providing a layer of storage that can be used at ingestion or query time.
Thanks to Discovery QueryFlow, we can expose custom endpoints that are purpose-built for the different search features we want to offer. QueryFlow allows us to define flexible query pipelines that can be tuned to the intent behind each type of user interaction.
For example:
- A personalized recommendations endpoint can combine vector similarity with product popularity, and recency to generate “You may also like” suggestions based on user behavior or the current product context.
- An assistive search endpoint can interpret natural-language queries, map them to semantic concepts, and retrieve relevant products even when exact keywords aren’t present—great for queries like “something fun and educational for a 10-year-old.”
- A generative answer endpoint can retrieve and synthesize product information from multiple sources (e.g., descriptions, reviews, specs) and return a rich, structured response that helps users compare options or understand key differences.
QueryFlow acts as the glue that connects user intent, application logic, and the rich data served by Vespa, making it possible to implement intelligent, high-performing features with precision and control. By leveraging Vespa’s powerful search and ranking capabilities, we ensure that every user interaction is both fast and contextually relevant.
Integrating Discovery with Vespa provides a solid foundation for building intelligent, flexible search experiences at scale. By combining Discovery’s components orchestration with Vespa’s powerful search capabilities, teams can deliver accurate, context-aware results.
Conclusion: Smarter Search Starts Here
The integration of Vespa.ai into the Pureinsights Discovery Platform unlocks a new level of scalability, speed, and intelligence for search-driven applications. Whether you’re powering an e-commerce storefront, an enterprise knowledge portal, or a content-rich platform, this release makes it easier than ever to build hybrid search experiences that deliver real business impact.
If you’re ready to explore what Discovery + Vespa can do for your use case, get in touch for a personalized demo. We’ll show you how to turn complex search challenges into intelligent, scalable solutions — faster.