Graham Gillen
In early 2021, on the heels of a major licensing change by Elastic, Amazon announced the OpenSearch project, a code branch of Elasticsearch and Kibana under the Apache 2.0 open-source license. In the first of a three-part Elasticsearch vs OpenSearch blog series, we will examine practical implications of this development on the user and vendor communities that developed around these products. Subsequent blogs will cover the Elastic point-of-view and the Amazon point-of-view.
Wait, What Happened?
If you are a search application engineer the biggest development in 2021 was Elastic’s announced license change, stating “We are moving our Apache 2.0-licensed source code in Elasticsearch and Kibana to be dual licensed under Server Side Public License (SSPL) and the Elastic License.” Elastic posted a useful FAQ and clarification of the license change.
This news might have fallen under the radar of the broader IT-verse if not for the subsequent announcement of the OpenSearch initiative by Amazon. OpenSearch is a fork of the Elasticsearch and Kibana code base which will remain under the Apache 2.0 license. You can find the official OpenSearch project here.
This news impacts a much broader ecosystem than at first glance.
- With over 100 million downloads, Elasticsearch’s popularity has eclipsed Solr (pun intended). A huge developer community is affected.
- Elasticsearch and Kibana’s functionality has created a large business ecosystem worth billions, not just around enterprise search but also analytics (observability), and IT security applications.
Elastic is hugely popular because Elasticsearch and Kibana are great products But also because they were available under the minimally restrictive Apache 2.0 license. And while the Elastic licensing change seems to mostly target Amazon, users are scrambling to interpret how they are impacted. Independent providers of products or services based on Elasticsearch or Kibana are especially concerned about the Limitations in the Elastic 2.0 license and how it impacts their business models.
But rather than take sides as things heat up between Elastic and Amazon, you should be objectively asking yourself “what does this mean to me or my company?”
Consumers Always Win When There is Choice
While it can be daunting to decide what to watch on Netflix or Hulu, consumer choice is generally always seen as a good thing.
Before the latest Elastic and Amazon drama, there were already very good commercial search solutions in the market even with recent consolidation. OpenSearch might distinguish itself over time. Solr also remains a proven (if less trendy) option as well. The market will dictate which options survive.
If OpenSearch succeeds, it will become popular among open-source purists (at the expense of Solr). Enterprises with products or services built around Elasticsearch and Kibana may have to port to Solr or OpenSearch. The risk involves balancing OpenSearch’s long-term viability versus the reduced popularity of Solr and vendor lock-in (and possibly higher costs) from working with Elastic or their partners. The market, again, will dictate what happens.
OpenSearch also benefits from the weight of Amazon’s PR machine. It could falter if Amazon loses interest, and community support fizzles. But it is off to a strong start with roots dating back to 2019 with Open Distro, the first significant branch of Elasticsearch and Kibana. Open Distro supporters included Red Hat, Netflix, and Expedia. The OpenSearch announcement quotes heavyweights like SAP and Capital One. Just as Solr and Elasticsearch benefited from building on Lucene, OpenSearch builds on Open Distro and the established success of Elasticsearch and Kibana.
OpenSearch reached its version 1.0 milestone and promises a production-quality release by the end of 2021. It is also possible, though unlikely, that Elastic could reverse directions on licensing. In any case, the next 6-12 months will be very interesting.
Elasticsearch vs OpenSearch: How Things Got Ugly
Pureinsights has always maintained vendor neutrality in the solutions we recommend or build. This goes back to our history at Search Technologies and our belief in customers having the ultimate choice. We are not taking sides in this Elasticsearch vs OpenSearch battle, but we do see growing confusion and concern among the user and vendor community. How did things get this ugly?
It may go back to February 2019, with Amazon’s support for Open Distro, including security and management enhancements. Open Distro became the open-source community’s response to X-Pack (or ELK Stack), Elastic’s commercial bundling of Elasticsearch, Logstash, Kibana and Beats. Amazon then launched the “Amazon Elasticsearch Service” without collaborating with Elastic. Elastic’s stock took a hit and Elastic sued AWS for trademark infringement in October 2019.
Much of all this is summarized in this excellent summary blog by David Tippett, a DevOps Engineer with the PRA Group: “The Difference Between Elasticsearch, Open Distro and OpenSearch.” As Tippett notes, trying to get a new branch of Elasticsearch and Kibana in working condition is a messy affair. According to Tippett and other sources, Elastic added checks to slow down or prevent users from using Open Distro. The problems eventually plagued OpenSearch, in the form embedded “phone home” code Elastic ostensibly uses to track user and application metrics. A team at OpenSearch spent weeks gutting the code to remove mentions of Elasticsearch and Kibana and get the software to work.
Escalation and Response
Elasticsearch then made changes in its downloadable client libraries, causing more problems for OpenSearch. Tim Ryan notes in an article in The Register that “Elastic modified the official Python client for its Elasticsearch database to not work with forked versions and closed the GitHub issue to comments.” Elastic viewed the changes as enhancements to ensure that users know they are using officially supported Elasticsearch client libraries. OpenSearch users viewed this as sabotage. Amazon announced that OpenSearch will address the issue by committing to developing a whole new set of client libraries for OpenSearch.
Could this mess have been avoided before the “cold war” between Elastic and Amazon escalated? It’s hard to say. Perhaps Amazon could have worked more closely with Elastic on “Amazon for Elasticsearch”. Or perhaps as one poster joked on Twitter, Amazon should have named it “RubberBandSearch”. A pragmatic conclusion, though, is that Elastic could have found a better way than licensing to address a business dispute with Amazon.
The Death of Solr? or The Revenge of Solr?
In all of this mess we can’t forget that Apache Solr is still out there, in production in many applications. Does the advent of OpenSearch accelerate the slow death of Solr? Or is this more like a Star Wars trilogy, “The Revenge of Solr”? Is Amazon, of all people, trying to convince everyone in the open-source community that Elastic is the evil Empire everyone should be drawing their light sabers against?
With Elastic’s tremendous success and IPO, it was inevitable they would lose some of the “charm” of their early days. I still have these old ELK logo stickers on my laptop.
Nostalgia aside, reaction from the open-source community has been mixed, as documented by InfoQ.
“Elasticsearch is a fork of Elasticsearch because they’re the ones who changed the license. Amazon is just continuing the original open-source project which Elasticsearch abandoned.” – Colin Percival on Twitter
“Amazon isn’t doing this for the community. They’re doing it for themselves – they’ve made billions already from Amazon Elasticsearch service and don’t want the money to stop with the new license.” – Reddit User Proton
Solr remains a very capable search engine for basic search applications. Our engineers at Pureinsights have successfully deployed and supported Solr in many large-scaled search applications and Solr deserves consideration, at least until the viability of OpenSearch is no longer in question.
Practical Recommendations: What Should I Do?
You can trust Pureinsights to help you evaluate and choose whether Elasticsearch / Kibana, OpenSearch, Solr, or some other combination of technologies is best for your search-based applications. What we’d like to offer here is an appetizer to what your decision-making process might be like. The first thing to figure out is which of the following three “buckets” you fall into, as an Elasticsearch and Kibana user:
In a nutshell:
- If you are a regular user of Elasticsearch and Kibana and you develop, manage and support your own application using the open-source versions, the new licensing probably works for you fine and you can continue with Elastic’s official distributions.
- For users of Elastic and Kibana as a managed service, you will need to find out what the vendor roadmap looks like and decide whether that impacts your choice of cloud or service providers in the future, and what a migration path would look like. Your decision might be most complicated if you service provider is Amazon.
- Providers of managed services or products with Elasticsearch or Kibana embedded in their offerings will have a business decision to make. Migrate to OpenSearch only, or invest in giving customers a choice between the Elastic or OpenSearch distributions.
We have documented a more complete set of decision trees in a PDF that may be useful.
The Ultimate Impact on our Customers
While choice can be daunting, we believe it is generally always a good thing for our customers. Vendor-neutrality is in our DNA, and we are committed to that whether you are talking about Elasticsearch vs OpenSearch, or any commercial or open-source technologies.
Our fully managed services will always offer customers a choice when it comes to search engine technologies. That could mean using Solr, Elasticsearch, OpenSearch, or solutions from commercial vendors like Coveo, Yext, Google, Sinequa, and Lucidworks.
Customer needs always drive the technology mix we recommend for each bespoke solution, including when our own Pureinsights Discovery platform is suitable. And we will ensure that any technology we use in your search application architecture will comply to the licensing terms from the original source.
Up Next in the Series...
If you enjoyed this post, please consider sharing it using the social media buttons below.
In future blogs on Elasticsearch vs OpenSearch, we’ll cover the Elastic point-of-view and the Amazon (and OpenSearch) point-of-view. This includes the business reasons behind each company’s actions, as well as the license models at the center of the dispute.
For a free 1:1 consultation to further discussion the evaluation of Elasticsearch vs OpenSearch, please feel free to CONTACT US.
Best Regards,
The Pureinsights Team
Additional Resources
If you missed any part of the series, you can read them all here: