Search Relevance Scoring – EKG for Search

Search Relevance Scoring – EKG for Search

How do you know if the relevancy of your search application results have degraded?  Unfortunately, it’s usually through user complaints.  Search Relevance Dashboards and methodical relevance tuning processes can help you proactively detect and resolve these problems.

The possible causes of degraded relevancy for search results include:

  • Changing nature and composition of content
  • Changing user behavior and search habits
  • Software modifications introduced with unwanted results

Search applications can benefit from regular tuning, but where do you start?

By objectively measuring the relevancy of your search results.  And that’s what we are doing with a new capability in our Pureinsights Discovery Platform™ which we call the Search Relevance Scoring Dashboard.

Introducing the Search Relevance  Dashboard

The PDP Search Relevance Scoring Dashboard is a tool that provides search engine diagnostic capabilities complementary to the rest of the platform.

One of the goals of a search application is to deliver results that are as relevant as possible to queries submitted by users. And while search relevance tuning requires significant expertise and a good methodology, the process starts with being able to objectively measure the quality or “score” of a search engine at any given time.

Search Relevance Dashboard

EKG for Search

An analogy to this in medicine is the EKG or ECG (electrocardiogram). This diagnostic tool measures certain mechanical aspects of how the heart is performing and results are compared to an expected “norm.”  Often the EKG is the first step in diagnosing a major issue that might lead to major heart surgery. But no surgeon would even consider such drastic measures without first doing an EKG.

The PDP Search Relevance Scoring Dashboard measures the “health” of search application results. It can objectively determine when new software modifications or content changes or changing user behavior over time leads to a deterioration in the quality of search results.

Part of an Iterative Tuning Process

Tuning search application results to improve relevance is not a “one and done” job. The performance of your application will change over time due to the wide number of variables we mentioned at the beginning of this article.  Instead, relevance tuning is an iterative process that needs to be revisited regularly using monitoring tools like the Search Relevance Scoring Dashboard, as well as application diagnostic tools. 

Some problems may be easily traceable to configuration issues or recent software changes. For complex problems, application developers invariably need to be involved.  Especially developers with an understanding of user behavior, the nature of the content searched, and an end-to-end understanding of the content lifecycle from ingestion to processing, to presentation of results in a user interface.

The PDP Search Relevance Scoring Dashboard leverages user activity log files to analyze submitted queries and how far down the presented results users have to scroll to click on an answer, link or document they deemed most relevant. But engine scoring is a diagnostic and not a prescriptive tool. If the score is low, search engineers still have to determine the problem and introduce changes or software fixes. Regular or continuous monitoring does, however, provide an indication of whether or not the fixes have improved (or degraded) results.

Search Relevance Scoring and Tuning Process

Stay up to date with our latest insights!