Open-Source Search and Analytics Platform Celebrates Second Milestone Conference
This blog summarizes some of the highlights of OpenSearchCon 2023 the second conference sponsored by OpenSearch, the Apache 2.0 open-source initiative that forked from Elasticsearch and Kibana in 2021. OpenSearchCon 2023 brought together users, developers, and technologists to explore real-world successes and new applications.
Pureinsights was an exhibitor and speaker at the conference held September 27-29 in Seattle, Washington. Our summary and notes from the conference focuses mainly on traditional search applications versus observability or log analytics implementations and includes highlights from a presentation by one of Pureinsights’ own engineers, Pablo Arce. We will update this blog with videos of the best search sessions when they become available on YouTube from OpenSearch. You may also want to check out the official OpenSearch post event blog.
Quick Takes on the Highlights of OpenSearchCon 2023
We’ve listed the top search related sessions at the end of the blog and hope to update them with links to presentations on YouTube when they are available. But here are our other quick takes from our staff who attended:
- The conference was again in Seattle, where it rained every day. But the venue and presentation were definitely more upscale than last year.
- Registrations were up significantly to 750. A majority of the attendees were from Amazon or AWS, who are still the biggest driving force behind OpenSearch. However, session contributors and presenters included a lot more non-Amazon people this year.
- OpenSearch stats claim 300M downloads of Opensearch, mostly in the last 2 years.
- The exhibitor space was well-organized, with prime locations near the food and drinks and there were 23 vendors at the conference. Most of the exhibitors were smaller companies, reflecting the grass-roots nature of OpenSearch. Pureinsights sponsored for the second year, this time as an AWS Select Tier Services Partner.
- Pureinsights presented for the first time. Our presentation by Senior Software Engineer, Pablo Arce was well received. The presentation featured a real-life use case of drastically improving performance for a search application with very large indices, all while migrating to OpenSearch (more below).
- The keynote featured Grant Ingersoll. Grant is a well-known search guru and former CTO and co-founder of Lucidworks, a commercial search platform built on Apache Solr.
- Use cases in AI and Observability and Log Analytics dominated the presentations. Many of the sessions on “traditional” search discuss the integration of Vector Search, Large Language Models (LLMs) and hybrid search, and semantic search. Which is the new frontier thanks to advancement in AI. We list the best sessions below in the blog.
Search Session Highlights
The following represent what we thought were the most interesting presentations focused on developments in “traditional” search. We will update this blog with relevant links to the recorded sessions if they become available on from OpenSearchCon on YouTube. The full OpenSearchCon 2023 – Opening Keynote is already available.
OpenSearchCon 2023 Session: Best Practices for Managing Extremely Large, Live Search Indices
Presenter: Pablo Arce, Senior Software Engineer, Pureinsights
Summary:
Scaling search applications with large indices can be challenging, sparking debate over the efficacy of NoSQL databases versus document storage systems. This presentation compares different approaches to store vast numbers of documents, outlining each method’s pros and cons. We demonstrate these solutions through a real-world case study of a significant search engine migration project from Elasticsearch to OpenSearch.
Watch on YouTube: Pablo Arce – Best Practices for Managing Extremely Large, Live Search Indices
OpenSearchCon 2023 Session: How to Tame Your LLM · OpenSearch
Presenter: Alexander Greaves-Tunell, Applied Scientist, AWS OpenSearch
AI models for natural language can enable fundamentally new user experiences in OpenSearch. However, converting demos into production-ready features presents challenges in ensuring AI output reliability and accuracy. This talk will address these challenges in the context of building new AI-powered features for query generation and data exploration in OpenSearch. We’ll examine some risks of language models, suggest mitigations, and share experiences from building a reliable query generator.
OpenSearchCon 2023 Session: Keyword-Based or Semantic Search? Best of Both Worlds With Haystack and OpenSearch
Presenters: Bilge Yücel, Developer Advocate at Deepset and Tuana Celik, Lead Developer Advocate at deepset
Summary:
Semantic search improves relevance and efficiency in information retrieval but struggles with new domains. Keyword search is effective and adaptable, but lacks understanding of word meanings. Can we combine them to minimize drawbacks? This talk explores hybrid search on Haystack website, leveraging pipelines and optimized query clauses. We showcase building a domain-agnostic search system with open-source tools and models.
Watch on YouTube: Tuana Celik – Keyword-Based or Semantic Search? Best of Both Worlds With Haystack and OpenSearch
Presenters: Noam Schwartz, Software Engineer at Searchium.ai and Yaniv Vaknin, Machine Learning Leader
Summary:
Discover the advantages of combining text search and vector search in a hybrid setup using OpenSearch. In this session, we explore how you can leverage both methods to enhance search accuracy and deliver comprehensive results. You’ll walk away with practical techniques for setting up a hybrid search system using OpenSearch and insights on optimizing search results.
OpenSearchCon 2023 Session: Text-Expansion Technique to Outperform Unexplainable Vector Search
Presenters: Praveen Prasad, Analytics Specialist at AWS OpenSearch
Summary:
The recent vector trend poses the question: “Is a shift from keyword to vector essential for improving relevance?” This talk discusses using the text-expansion technique in Open to enhance Keyword by overcoming its limitations. We’ll discuss how to expand documents and queries with contextual features using built-in ML models or LLMs, improving Keyword ‘s performance and explainability without the need for complete transformation to vector. Key message: Vector isn’t the only route to increased relevance.
OpenSearchCon 2023 Session: End-to-End Relevance
Presenters: Jon Handler, Senior Principal Solutions Architect, AWS
Summary:
Search engines give their users an almost-magical way to go from their goals to the information that meets those goals. When the items retrieved match the user’s intent, then we say that they are relevant for that user and that query. “Relevance” spans much more than just the function used to rank the search results. Retrieving relevant results includes the presence of the data in the catalog, the way the data is analyzed, the methods used to augment and send the user’s query, going beyond text to vector representation, feeding back user behavior, and of course, the ranking function. In this session, we’ll take a broad look at how to use OpenSearch ensure that your users find the information that they need to get to where they are going, find the product they want to buy, or retrieve the data they need to solve a problem.
Watch on YouTube: Jon Handler – End-to-End Relevance
OpenSearchCon 2023 Session: Using OpenSearch to Store Embedding Data · OpenSearch
Presenters: Pedro Cruz, Engineering Manager, Canonical and Alastair Flynn, Software Engineer
Summary:
The supporting infrastructure for large LLM jobs can be difficult and costly to set up. Storing vector data requires careful consideration of resource consumption. OpenSearch offers a straightforward way to store embeddings generated by tools like Azure OpenAI or the Natural Neural Search plugin. It also handles querying, reducing the operational overhead. This talk shows how to prepare pdf files, send them to Azure’s OpenAI API to generate embeddings, and store the resulting vectors in OpenSearch. This will be running on a low maintenance Raspberry Pi cluster and Charmed OpenSearch.
Wrapping Up
Overall, OpenSearchCon 2023 was a well-organized and informative event. There was a lot of focus on AI, Observability, and Log Analytics, which are all important areas for OpenSearch. With Amazon’s continued strong support, we think OpenSearch will continue to grow as a peer, if not heir-apparent, to Solr in the Apache open-source licensing world. It is also encouraging to see that OpenSearch is committed to working with partners and growing the OpenSearch community.
As always, please CONTACT US if you have any questions about OpenSearch, or if you need any help on a search engine migration project.