Challenge 3: Troubleshoot failures
Previous Challenge Next Challenge
Introduction
The Movie Guru team just released a new version of the app, and since that release, we have gotten several customer reports that the MovieGuru app is experiencing issues. In particular, user preferences aren’t always being saved (eg: “I love horror movies”, “I hate drama films etc”).
To see how preference saving is expected to work, watch this video:
Description
Your task is to use Firebase Genkit Monitoring and the application code to pinpoint and resolve the root cause of these preference saving failures.
- Identify the feature and corresponding Genkit flow that is responsible for handling user preference updates.
- Examine the failed traces for this feature and identify common failure patterns.
- Determine the underlying cause by exploring the code for the feature you identified.
- Find the hints in the code files js/flows/src/userPreferenceFlow.ts and js/flows/src/userPreferenceTypes.ts.
- If you’re really stuck, check the Tips out.
-
Apply the necessary code fix and restart the application.
-
Stop the application. Press Ctrl+C to stop the running containers in the terminal where they are running.
chmod +x stop_app.sh # Optional: Required if first time running the script ./stop_app.sh -
Restart the application after fixing it (make sure set_env.vars.sh is up to date).
chmod +x start_app.sh # Optional: Required if first time running the script ./start_app.sh
-
Success criteria
- You have idenfified the feature with the most failures.
- You understand the root case of these failures.
- You have implemented a fix and restarted the application.
- Queries such as “I love horror movies” consistently adds “horror” to your preferences in the app.
Tips
-
The Failed paths table can help reveal common error patterns among traces.
-
Finding the error cause Our team has seemingly introduced a regression in this release. The error you’re seeing is a type mismatch error. This indicates a discrepancy between the data structure the userPreferenceFlow expects to receive from the model, and the structure the model is actually producing based on the prompt’s (userPreference.prompt) instructions.
The error you’re seeing is a type mismatch error. This indicates a discrepancy between the data structure the userPreferenceFlow expects to receive from the model, and the structure the model is actually producing based on the prompt’s (userPreference.prompt) instructions.
The UserPreferenceFlow uses the output schema definition found in userPreferenceTypes.ts. Compare the output schema defined in UserPreferenceTypes.ts with the output specified in userPreference.prompt and its variants. Pay close attention to both the format schema definitions of the prompt file and the system instructions given to the model in the prompt text.
To fix the issue, you can do one of the following:
- Fix the prompt and add an output schema definition to the prompt (userPreference.experimental.prompt).
- OR, Downgrade the flow to use userPreference.prompt.
If you are unsure of the difference between a flow and a prompt, check out the section Prompts and Flows in Genkit.
-
Prompts and Flows in Genkit A flow is the executable unit – it defines and orchestrates a sequence of steps (the process) and can contain multiple sub-steps like other flows, prompts, retrievers, tools, etc.
A prompt is data – the input sent to an AI model plugin within a step of that flow, and the output received from the model in that step.
So, you execute a flow, and within the flow’s steps, prompts are used to interact with AI models. They differ because the flow is the action that runs, while the prompt is the content exchanged during a specific step involving an AI model.
Both flows and prompts can have their own input and output data schemas.
-
Useful docker compose commands The application runs inside docker containers. The start_app.sh and stop_app.sh scripts wrap docker compose commands (and other boilerplate code):