Ricardo León
MAPT: Automated Performance Testing for MongoDB Atlas
In the ever-evolving landscape of AI and search applications, optimizing database performance is crucial. Developed by Pureinsights in partnership with MongoDB, MAPT (MongoDB Atlas Performance Testing) is a performance testing tool that is a game-changer, providing invaluable insights into MongoDB Atlas Search performance. This blog provides some practical background and details on MAPT and its usage.
MongoDB Atlas Performance Benchmarking
Atlas Search is a fully managed, integrated search functionality within MongoDB Atlas, a cloud-based database service. It enables developers to easily build sophisticated search capabilities directly on top of their MongoDB data. Powered by Apache Lucene, a powerful search engine library, Atlas Search offers full-text search, autocomplete, geospatial search, and more, making it ideal for building applications with rich search features. With its seamless integration into MongoDB Atlas, developers can leverage the scalability and flexibility of the database while delivering fast and relevant search results to their users.
Performance is crucial for search engines as users expect fast and relevant results. Slow search responses lead to frustration and abandonment, impacting user experience and potentially decreasing engagement with the application or website.
With this in mind, MongoDB partnered with Pureinsights to develop MAPT, an automated framework for comprehensive performance benchmarking of MongoDB Atlas.
Goals of MAPT Tool
MAPT aims to achieve several objectives:
- Create a repeatable, automated test bench for current and future Atlas Search releases.
- Gather performance and scalability data associated to MongoDB Atlas
- Test Atlas Search performance and scalability under various conditions.
- Compare performance with competitors, such as Elasticsearch.
- Provide easy on-demand deployment for testing new MongoDB versions or features.
Underlying Technology: Leveraging AWS Services
MAPT leverages a serverless cloud-based architecture, primarily utilizing AWS services such as Amazon S3, Lambda Functions, Step Functions, CloudWatch, etc. Throughout the execution, MAPT utilizes AWS Step Functions to manage the workflow, ensuring a smooth and organized testing process.
Key technological aspects include:
- Infrastructure-as-code (Terraform) for resource provisioning
- Persistent storage (Prometheus) for results
- Grafana for real-time visualizations
- Cost-effective processing, incurring expenses only during active runs
The infrastructure is organized into convenient layers. This layered approach allows for targeted updates and installations as needed, simplifying deployments.
Moreover, MAPT features a Command Line Interface (CLI) that serves as a bridge between the user and the solution. The CLI is a thin layer that consists of a portable executable file, capable of interacting with AWS.
Database and Search Performance Benchmarking: Conceptual Steps
Before running MAPT, a few preparation steps are necessary:
Data Preparation:
- Define the use case (e.g., e-commerce, enterprise search)
- Generate representative data
- Define mappings/schema for Atlas and Elasticsearch
- Create test cases (queries and variable inputs)
Data Publication:
- Use the CLI to publish prepared data, mappings, queries, and values. This data is uploaded to S3 so that it is available at execution time.
Once the data is available and loaded to AWS S3, then the execution can be triggered via a CLI command:
mapt execute <config>
Execution:
Execution can include two phases that can run sequentially or in parallel:
- Indexing: Adds data to an existing or a new index that follows a specific schema or mapping.
- Performance Testing: Executes performance testing based on a configuration file. The number of concurrent users, test duration, ram-up period and other variables can be configured.
Monitoring:
Last, but not least, there’s monitoring. While the performance testing is running, several metrics are gathered and stored in Prometheus, including search engine specific metrics such as:
- CPU utilization
- Memory utilization
- I/O Ops
- JVM usage
- Indexed Docs Per Seconds
- Disk Size / # of documents
- Query latency
Additionally, performance testing metrics are captured:
- Response time per request
- Response code per request
- # of requests per second
- # of responses per second
By leveraging Grafana and Prometheus together, MAPT offers a comprehensive performance monitoring dashboard. Prometheus’s robust data collection and time-series database, combined with Grafana’s powerful visualization, provide valuable insights into system performance.
By monitoring the metrics mentioned above, we can gain the following insights:
- Resource Utilization: Understand how efficiently the search engine utilizes CPU, memory, and disk resources. This helps identify bottlenecks and optimize resource allocation.
- Search Engine Performance: Assess the indexing speed, query latency, and overall throughput of the search engine, enabling targeted performance improvements.
- User Experience: Measure response times and error rates to evaluate user experience and identify areas for enhancement.
- System Capacity: Determine the search engine’s ability to handle various loads, ensuring it can meet demand and avoid overload situations.
- Reliability and Stability: Monitor error codes and response patterns to proactively detect and resolve potential issues, ensuring system stability.
Here are some example visualizations:
Wrapping Up
The MAPT performance testing tool has already demonstrated its value in benchmarking MongoDB Atlas Search performance. By aiding customers in sizing their clusters and identifying performance bottlenecks, MAPT has become an essential asset. While the tool has already proven successful, both Pureinsights and MongoDB are committed to its ongoing enhancement and expansion. This collaboration aims to further empower customers with comprehensive and effective performance testing solutions, ultimately optimizing their MongoDB Atlas experience.
Need help optimizing your Atlas Search experience? Want to benchmark Atlas against other search engines? Our team of experts can help you fine tune Atlas Search to its fullest potential and benchmark it against other search engines. Let us help you unlock the true power of search and achieve your business goals. CONTACT US and let’s tackle your search challenges together.