Illuminating Dark Data Search Performance for District 4 Labs

“DARKSIDE is constantly growing, boasting tens of billions of records and used already by cutting-edge corporate security teams, intelligence analysts, and cyber consultants to mitigate risk, prevent fraud, and uncover critical person and company-centric intelligence”.

Mateo Tomasini, Founder and CTO, District 4 Labs

Introduction to District 4 Labs

District 4 Labs (D4) is a leader in the new field of open-source intelligence (OSINT) tools and technologies. They developed DARKSIDE, a highly searchable repository of tens of billions of compromised records and other person-of-interest data. Pureinsights helped scale the search application for DARKSIDE to meet performance requirements for the security analysts and cyber consultants that use the tool for investigations.

Performance tuning for a massively large and fast-growing search index

District 4 Lab’s Darkside database contains tens of billions of records in 20+ terabytes – and growing rapidly.  During development, users were getting unacceptable response times and query errors on the search application. Pureinsights reviewed DARKSIDE’s architecture, which included using the OpenSearch search engine on AWS infrastructure.  Through the application of sharding techniques and the selection of the right AWS tools and compute resources, the customer was able to achieve desired search performance for DARKSIDE.

Pureinsights tuning methodology and results

After only two design iterations, Pureinsights helped District 4 Labs create an application architecture greatly with improved search performance for the DARKSIDE database. Query errors were virtually eliminated. Average query response times improved from 15 seconds to sub-second (15x improvement) and max response times went from 85 seconds to 15 seconds (5.6x improvement).

Optimized search application performance

In a short amount of time, our consultants helped District 4 Labs’ development team create a plan to apply different AWS technologies and optimized search application architectures to achieve incredible application performance gains for search on their flagship DARKSIDE repository.

Cost-optimized solution

The performance gains were achieved with a thorough analysis of the trade-offs of performance gains vs. operation cost for the different available AWS compute resources analyzed for the application architecture.  The end result was a more performant solution with significantly reduced monthly infrastructure costs.

Future-proof architecture

The resulting architecture is scaling and leaves the customer with options to maintain or even enhance application performance as the DARKSIDE database continues to grow and the number of users increases.

“DARKSIDE is a critical resource for security professionals worldwide. Pureinsights’ expertise was instrumental in enhancing its search performance, allowing us to better serve our clients and ultimately contribute to a safer digital landscape.”

Mateo Tomasini, Founder and CTO, District 4 Labs