/

Comparing Vector Search Solutions 2024

Comparing Vector Search Solutions 2024

Vector Search Revisited

One year ago, I wrote a post about vector databases, the main contenders and how they compare. Now is a good time to revisit this topic as Forrester has just released its first Forrester Wave report on Vector Databases, shedding light on the growing importance of these technologies in the AI and search landscape.

 The report highlights key players, their strengths, and the advancements shaping the future of vector databases. As this field rapidly evolves, with increasing use cases across industries, understanding the current market and capabilities of various solutions is crucial for staying competitive and making informed decisions.

Forrester evaluates 14 vendors compared to my 8 from last year, so I will also expand my list a little, but some considered by Forrester I would think too niche or too small in the market. So, of those considered by Forrester I will exclude SingleStore, KX, Couchbase & Redis. The big cloud players all now have (sometimes multiple) offerings, so they need to be included. We have also dropped Chroma which was included last year because it’s a slightly different beast and more of a lightweight embeddable Vector DB code library aimed at developers and doesn’t really target the high scale, enterprise readiness of those included here.

comparing vector search databases and vector search solutions 2024

Categories of Vector Databases

Search engines:

  • Solr: Launched in 2006 as an Apache project, Solr is a mature open-source search engine with a large community. It’s still popular for enterprise search and content management.
  • Elasticsearch: A distributed search engine that supports various data types, including vectors. Offers machine learning, log analytics, and alerting with both open-source and licensed options.
  • OpenSearch: Forked from Elasticsearch in 2021, OpenSearch is fully open-source, community-driven, and a great alternative for organizations wanting full transparency.

Dedicated vector Databases:

  • Pinecone: A cloud-native, fully managed vector database focused on ease of use and scalability, with strong search and filtering features.
  • Milvus: Open-source and highly scalable, supporting various vector field types and distance algorithms, ideal for specialized domains.
  • Qdrant, a native open-source vector database launched in 2021, offers flexibility and ease of use but lags in advanced database features, with a roadmap focused on boosting hybrid search, automation, and performance to enhance its competitive edge.
  • Weaviate: Open source, supporting data objects and vector embeddings with advanced features for hybrid applications like aggregations and BM25 search.

Multi-capability platforms:

  • MongoDB Atlas Search: Embedded in MongoDB, this managed search engine lets developers integrate powerful search into apps without managing separate systems. Very developer friendly, and well suited to applications requiring multiple database types and search in a one stop shop.
  • DataStax, building on its Apache Cassandra expertise, offers AstraDB with robust vector capabilities and a forward-looking vision focused on GenAI, RAG, and serverless architecture, empowering developers with enhanced tools, performance, and automation for modern AI applications.
  • Oracle – has entered the vector database space with a promising yet still maturing solution, integrating vector capabilities into its core database while rapidly building a competitive roadmap to challenge established players.
  • MS Azure – Microsoft’s vector solution is promising but still lacks advanced capabilities, with multiple Azure databases offering tailored vector support and a strong roadmap focused on automation, integration, and performance improvements.
  • Amazon Web Services (AWS) has introduced vector support in its cloud databases. As well as OpenSearch, Amazon Q, Bedrock, MemoryDB, and Neptune all feature vector search.
  • Google Cloud Platform – weirdly missing from Forrester’s list is the Google Cloud Platform which has a Vector Database as part of its Vertex AI offering as well as in relational databases and Spanner.
  • Neo4J – although quite niche and, unusually, primarily a graph database Neo4J does have a vector search capability, and therefore offers novel, interesting, and effective ways to implement RAG.

How should we compare them?

Here’s a few criteria we will consider:

  • Vector search implementation: how flexible is it? Which algorithms are supported? Are there limitations?
  • Features: What other features does the implementation offer aside from vector search?
  • Scalability: How many vectors can the database support?

So how do the vector search capabilities compare?

Please note that this field is evolving fast. All evaluations are based on the latest versions at the time of writing, but if you think we’ve got anything wrong, please let us know!

Below is a simple summary focusing on the vector search capabilities. The columns are:

  • ANN implementation – which underlying K/ANN implementations are available?
  • Distance measures – which distance measures are available?
  • Filter support – can filters be applied – how flexible are they?
  • Keyword search – is keyword search also available, how flexible is it?
  • Hybrid search – can vector and keyword search be combined, are there any limitations?
  • Max Dimensions – how many dimensions do the vector fields support?
Product ANN Implementation Distance Measures Filter Support Keyword Search Hybrid Search Max Dimensions

Lucene. 

Can also do Exact KNN

Yes – can be combined with filter query (can it be used as though?)

Yes – BM25

Yes – can combine keyword and vector results with normalized scores.

4096

Yes
Yes
Yes
Yes
Yes
Yes
Yes

What about when you consider other features?

Vector search is being used in a variety of ways to provide novel search experiences. The most common are:

  1. Semantic Search – because vectors don’t rely on keywords, they can find content based on the underlying meaning and therefore are in some ways better (or at least different to) keyword search. They do prioritize recall over precision, however, which is not always what you want.
  2. Extractive answers – (also known as Featured Snippets) this is where a semantic search is used to find a candidate block of text containing an answer to a question. Another model is used to pluck out (or highlight) the answer itself.
  3. Retrieval Augmented Generation (RAG) – this is where a model like GPT is asked to provide an answer based on the results from semantic search and not its broader training base. This removes hallucinations and allows an organization’s own data to be the basis for the answer.
  4. Image / Audio search – we are seeing more use cases where vector search is being used to search image or audio embeddings, to do and audio or image similarity match.

Importantly, methods 2 and 3 rely entirely on the quality of the semantic search. If the top results generated by semantic search—which form the context for the answer—do not contain the information you need, or perhaps include incorrect, outdated, or irrelevant data, then the generative AI component will produce incorrect answers.

Currently, for datasets larger than very small ones, a hybrid search combining keyword-based and vector-based results is likely to yield the best outcomes. This approach merges the precision of keyword search with the recall capabilities of vector search. Traditional search engines have an advantage in hybrid search because they already possess the essential tools for effective keyword searching, including language-specific synonyms, stemming, spellchecking, stopwords handling, facets, and field weightings. Pure vector databases, on the other hand, generally have only basic keyword search capabilities and face some limitations in their hybrid implementations.

Now that cloud platforms and major vendors offer some form of vector database or capability, they become the natural choice for many organizations, as it’s typically preferable to stick with the provider where the majority of your investments lie. For example, MongoDB Atlas is a comprehensive document database offering a variety of features, including graph databases, stream processing, and edge computing—making it a one-stop shop for many users. Similarly, if you are heavily invested in Azure, then AI Search might be the most logical choice.

Dedicated vector databases, conversely, are specialized exclusively in vector search. This specialization grants them certain advantages, such as support for very high-dimensional fields, billion-scale indexes, additional distance measures, and multiple vector indexing types. If you require a vector search solution with specific features or greater flexibility, a dedicated vector database might be the better option. It’s also worth noting that since I first explored this topic a year ago, vendors have introduced more features to make these databases enterprise-ready, including role-based access and enhanced cloud management capabilities.

What about Scalability and Performance?

You can find some benchmarks online, but these are often provided by the vendors themselves, who may be inclined to select test criteria that favour their own products. There’s very little in the way of systematic, unbiased comparisons, so we won’t delve into them here. However, for those interested, there is the ANN benchmark here (https://ann-benchmarks.com/), which covers some of the implementations discussed in this article.

So when should you choose a pure vector database?

This is a good question, and it essentially boils down to a few considerations:

Do you also need robust keyword search?

If you’re operating within an existing search implementation and know that traditional keyword search capabilities—along with associated features like synonyms, facets, and so on—will be required, then a pure vector database is probably not the optimal choice. However, if you’re working in a data science context where the primary objective is the storage and management of vectors, a pure vector database is well-suited, as they are more targeted toward that workload. It should be noted that even with audio and image search scenarios, we find that being able to leverage text can be beneficial and help improve relevance.

Do you require very high-dimensional vectors?

Vector dimensionality is another crucial factor. The maximum number of dimensions varies across systems, but generally, vector databases support models with higher numbers of output dimensions. For context, OpenAI’s text-embedding-ada-002 model outputs 1,536 dimensions. Elasticsearch and MongoDB support up to 2,048 dimensions, whereas vector databases like Milvus and Weaviate support up to 32,768 and 65,535 dimensions, respectively. Therefore, the ability to handle very high-dimensional data is likely to be a key differentiator.

Do you need a wide variety of vector field types and distance measures?

Milvus supports multiple vector field types, whereas others typically rely on HNSW (Hierarchical Navigable Small World graphs). If you have more specialized or exotic indexing requirements, then a specialized vector database is likely to be preferable. Additionally, both Milvus and Weaviate offer a larger number of distance measures. So, if you have specific requirements or need additional options, a dedicated vector database is the way to go.

Do you need to search across a very large number of vectors?

Since dedicated vector databases are designed with vector search as their primary workload, they are typically more efficient and scale more easily. In a comparison by one of the vendors (Zilliz’s comparison), one of the criteria is “billion-scale vector support.” Only Milvus (Zilliz’s product), Qdrant, Pinecone, KDB, and Vespa are marked as supporting this, although it’s unclear what this assessment is based on.

Of course, there are additional considerations such as market presence and maturity, supportability, manageability, resilience, and so on, but perhaps those are topics for another discussion.

Wrapping up

With vector search making waves in both traditional search engines and vector databases, and the vendor landscape changing rapidly, deciding which path to take feel like solving a Rubik’s Cube blindfolded! If you’re new to search implementation and eager to explore vector capabilities, it can be daunting to know where to begin.

That’s where we come in. Armed with decades of search experience and our Pureinsights Discovery™ platform, embracing the latest in vector search technology, Pureinsights is ready to support you every step of the way.

So, as always, feel free to CONTACT US with any comments or questions or to request a complimentary consultation to discuss your ongoing search and AI projects.

Cheers,

– Matt

Twitter
LinkedIn

Stay up to date with our latest insights!