Elasticsearch 8.8 Key Release Themes

The theme of Elasticsearch 8.8 centers around improvements to vector search capabilities that allow developers to leverage leading third-party AI-powered technologies with minimal expertise required.  The goal is to make it easier to deliver an “out-of-the-box” semantic search, though this also relies on a new proprietary transformer from Elastic.

Elasticsearch 8.8 Enhancement Details

When you dig into the pragmatic details, Elasticsearch 8.8 includes improvements in the following areas:

• Efficient Value Encoding
• Increased dimensions for vector search
• Support for Reciprocal Rank Fusion (RRF) in the Search API
• Similarity field in KNN Clause
• GA Release of JWT Realm
• Elastic Learned Sparse EncodeR (ELSER) model

We elaborate on these enhancements below.  Of note is a new, proprietary machine learning model called Elastic Learned Sparse EncodeR (ELSER), which enhances search relevance by enabling semantic search without the need for fine-tuning on specific data.

Efficient Value Encoding

Elasticsearch introduces an efficient value encoding technique that optimizes compression based on the bit size of the values. Previously, only values below 32 bits were compressed, but now values ranging from 33 to 40 bits are encoded using 40 bits per value, 41 to 48 bits use 48 bits per value, and 49 to 56 bits use 56 bits per value. This results in significant storage space savings and enhances overall performance.

Increased Dimensions for Vector Search

Lucene currently limits the max number of dimensions for vector search to 1024. Elasticsearch 8.8 increases this to 2048. This allows for larger and more complex models, enhances discrimination power, reduces collisions, improves scalability, and future-proofs the system. The expanded dimensionality enables more precise search results by capturing finer distinctions and semantic relationships, accommodating a growing number of vectors, and ensuring flexibility for evolving datasets and search requirements.  See details in: https://github.com/elastic/elasticsearch/pull/95257

Support for Reciprocal Rank Fusion (RRF) in the Search API

Reciprocal Rank Fusion (RRF) is now supported in the Search API, enabling the merging of multiple search result sets. RRF considers document position and employs a ranking constant to calculate the final score, eliminating the need for score normalization across different result sets. By leveraging RRF, you can achieve more accurate and relevant search results. The algorithm allows for merging keyword and vector-based search results, or multiple vector-based searches, providing the benefits of both precision from keywords and recall from vectors in a single query.

Similarity field in KNN Clause

In the knn (k-nearest neighbors) clause of Elasticsearch’s search functionality, a new parameter has been added to enable filtering of nearest neighbor results based on similarity. By specifying a similarity threshold, you can exclude results that fall outside the desired range. This feature enhances the accuracy and precision of knn searches, particularly when using similarity-based post-filters. It empowers you to fine-tune your search queries and obtain more refined and tailored results.

GA Release of JWT Realm

JWT (JSON Web Token) is an open standard to share security information between two parties. Support for JWT realm graduated from beta phase and is now generally available, making it a reliable choice for implementing authentication mechanisms in Elasticsearch.

Elastic Learned Sparse EncodeR (ELSER) model

With Elasticsearch 8.8, Elastic introduces ELSER, a machine learning model that Elastic claims you can use “out-of-the-box.” ELSER facilitates semantic search by considering the contextual meaning of words, rather than relying solely on literal terms. It is a pre-trained, out-of-domain sparse vector model that eliminates the need for fine-tuning on specific datasets. By leveraging ELSER, Elastic claims you can immediately enhance the relevance of search results and provide a more intuitive search experience to users.