Challenge 5: Improve search quality

Previous Challenge

Introduction

The MovieGuru Product team is evolving the app’s search capabilities. To potentially reduce operational costs, we’re moving towards a more streamlined approach. The current system utilizes a mixed search strategy, combining traditional keyword search with vector (semantic) search.

The new “V2” configuration switches to purely vector-based search.

Description

Your task is to manage the rollout of this change, observe its effect on search quality, and investigate the underlying technical reasons for any observed degradation in performance.

Note: If using the Genkit Developer UI, navigate to the chatFlow in the flows section of the left nav.

Find the relevant code

chatFlow uses several other flows to create a complex AI workflow.

  • Where does document search happen in the chatFlow?
  • Where are the results of that document search used?
  • Find the corresponding code.

Perform comparative analysis

Now you are ready to do some comparative analysis of the existing version and the new version of the prompt.

  1. Baseline Test (Hybrid Search): Establish baseline performance for the existing mixed search by running the following queries and note down the results:
    • “Show me movies with ratings greater than 3”
    • “Show me some funny films”
    • “Show me some movies with dogs in them”
  2. Examine retrieval performance: How many movies were returned and used as context in the input to the model call?
  3. Implement Search Switch and restart the app: Modify the application code to adopt the new search strategy.
    • Update the DocSearchFlow (defined in the docRetriever.ts file) to use the v2 version of the docSearch.prompt, which implements the vector-only search logic.
    • Restart the app (or the Genkit Developer UI) to allow it to pick up the updated code changes.
  4. Post-Change Test (Vector Search): Re-run the test queries above and note down the results.
  5. Analyze Impact & Diagnose: Compare the results of the two tests and their traces.
    • Review the differences between the prompt instructions to better understand what the LLM is being asked.
    • Determine why the vector-only search impacted quality for some queries and not others.
  6. Make a recommendation: Form a recommendation for the product team as to whether we should roll out this change or stick with the current mixed search.

Success Criteria

  • You have successfully update the application code to use the new variant of the search prompt.
  • You identified which types of queries resulted in degraded search quality and can articulate the likely cause.
  • You have the spotted the difference in the documents the retriever outputs for the same queries before and after the change. You can examine the retriever results in traces as the input to the model in the movieQAFlow.
  • You have evidence for the Product Team to support or nor support rolling out the change in prompt variant.

Tips

  • Focus on the docSearchFlow
  • To see the documents that are relevant in the Firebase Genkit Monitoring trace viewer, look at the input in the model interaction of movieQAFlow.

Learning Resources

Previous Challenge