Elasticsearch 8.8 Highlights

Elasticsearch 8.8 Highlights

Elasticsearch 8.8: Enhanced support for Vector Search, AI and Semantic Search

Elastic has recently unveiled Elasticsearch Version 8.8, delivering a wealth of advancements to their platform. In this blog:

  • We summarize the key themes and innovative features introduced in Elasticsearch 8.8, keeping you up to date with the latest developments in search engine software.
  • We will also briefly explore the broader implications of this release in the ever-evolving landscape of AI’s impact on the software industry.
  • Finally, we’ve included useful resources at the end, including links to the official Release Notes and relevant Elastic blogs.
Elasticsearch 8.8 highlights

Elasticsearch 8.8 Key Release Themes

The theme of Elasticsearch 8.8 centers around improvements to vector search capabilities that allow developers to leverage leading third-party AI-powered technologies with minimal expertise required.  The goal is to make it easier to deliver an “out-of-the-box” semantic search, though this also relies on a new proprietary transformer from Elastic.

Elasticsearch 8.8 Enhancement Details

When you dig into the pragmatic details, Elasticsearch 8.8 includes improvements in the following areas:

  • Efficient Value Encoding
  • Increased dimensions for vector search
  • Support for Reciprocal Rank Fusion (RRF) in the Search API
  • Similarity field in KNN Clause
  • GA Release of JWT Realm
  • Elastic Learned Sparse EncodeR (ELSER) model

We elaborate on these enhancements below.  Of note is a new, proprietary machine learning model called Elastic Learned Sparse EncodeR (ELSER), which enhances search relevance by enabling semantic search without the need for fine-tuning on specific data.

Efficient Value Encoding

Elasticsearch introduces an efficient value encoding technique that optimizes compression based on the bit size of the values. Previously, only values below 32 bits were compressed, but now values ranging from 33 to 40 bits are encoded using 40 bits per value, 41 to 48 bits use 48 bits per value, and 49 to 56 bits use 56 bits per value. This results in significant storage space savings and enhances overall performance.

Increased Dimensions for Vector Search

Lucene currently limits the max number of dimensions for vector search to 1024. Elasticsearch 8.8 increases this to 2048. This allows for larger and more complex models, enhances discrimination power, reduces collisions, improves scalability, and future-proofs the system. The expanded dimensionality enables more precise search results by capturing finer distinctions and semantic relationships, accommodating a growing number of vectors, and ensuring flexibility for evolving datasets and search requirements.  See details in: https://github.com/elastic/elasticsearch/pull/95257

Support for Reciprocal Rank Fusion (RRF) in the Search API

Reciprocal Rank Fusion (RRF) is now supported in the Search API, enabling the merging of multiple search result sets. RRF considers document position and employs a ranking constant to calculate the final score, eliminating the need for score normalization across different result sets. By leveraging RRF, you can achieve more accurate and relevant search results. The algorithm allows for merging keyword and vector-based search results, or multiple vector-based searches, providing the benefits of both precision from keywords and recall from vectors in a single query.

Similarity field in KNN Clause

In the knn (k-nearest neighbors) clause of Elasticsearch’s search functionality, a new parameter has been added to enable filtering of nearest neighbor results based on similarity. By specifying a similarity threshold, you can exclude results that fall outside the desired range. This feature enhances the accuracy and precision of knn searches, particularly when using similarity-based post-filters. It empowers you to fine-tune your search queries and obtain more refined and tailored results.

GA Release of JWT Realm

JWT (JSON Web Token) is an open standard to share security information between two parties. Support for JWT realm graduated from beta phase and is now generally available, making it a reliable choice for implementing authentication mechanisms in Elasticsearch.

Elastic Learned Sparse EncodeR (ELSER) model

With Elasticsearch 8.8, Elastic introduces ELSER, a machine learning model that Elastic claims you can use “out-of-the-box.” ELSER facilitates semantic search by considering the contextual meaning of words, rather than relying solely on literal terms. It is a pre-trained, out-of-domain sparse vector model that eliminates the need for fine-tuning on specific datasets. By leveraging ELSER, Elastic claims you can immediately enhance the relevance of search results and provide a more intuitive search experience to users.

Looking at the bigger picture for Search and AI

On the surface, Elasticsearch 8.8 continues Elastic’s trajectory of minor improvements in security and performance, coupled with enhancements to vector search that facility better natural language processing using machine learning and AI.  Essentially improved “semantic search.”

But there is a lot more here than meets the eye.  We are seeing that AI is beginning to disrupt many industries, from marketing to technology platforms like cloud data platforms and search engines, and Elastic are trying to take steps to “future proof” their business.

On the one hand, Elastic’s is marketing itself as a cloud data platform that just happened to have roots as an open source search engine.  So just like Elastic went after Splunk, it wants a piece of the big pie dominated by cloud-based relational and NoSQL data platforms.

On the other hand, AI is disrupting the search engine space as well.  New players like Pinecone are betting that that the future for search looks less like a traditional index and more like a vector database.  Data platform vendors are also developing search engine and vector database capabilities, so Elastic has to watch their back as they try to expand their revenue base.

To appear as a relevant player in AI, Elastic unveiled Elasticsearch Relevance Engine™ (ESRE) for building AI search applications.   In addition to incorporating its proprietary ELSER model, ESRE also provides APIs to allow developers to integrate third-party AI tools like OpenAI’s ChatGPT (and yes, all these similar sounding acronyms are unfortunate.)  ESRE is available to Elastic Cloud customers, though you can also download the entire Elastic Stack.  We assume that Elastic’s commercial solutions (Enterprise Search, Observability, and Security) will also leverage ESRE in some proprietary fashion.

In Summary: is AI eating software?

Elasticsearch 8.8 is a nice release with enhancements to security, performance, and other capabilities that facilitate semantic search.  But we are seeing that, just as Marc Andreessen warned us in 2011 that “software is eating the world,” we could be saying next that “AI is eating software.”  And the search engine market certainly isn’t immune.

OpenSearch 2.8 was also recently released, so stay tuned for more insights and updates from us at Pureinsights as we continue to explore the advancements and possibilities of Search and AI. Excited by all this? Confused? Please feel free to CONTACT US for a chat.


Stay up to date with our latest insights!