Challenge 5: Keeping It Real (Time)

Previous Challenge

Introduction

Our agent is now protected against malicious vectors and hosted on a scalable infrastructre, but we’re not done yet. In real world, customers can phrase their requests in unexpected ways and underlying foundation models or external APIs can introduce subtle behavior changes or hallucinations.

In order to make sure that our agent keeps working properly we need to continously monitor its behavior. Online Monitors sample conversations from Cloud Trace / Cloud Logging and score them against multi-turn AutoRater metrics. This way we can proactively catch quality drift before customers are impacted.

Description

Create a new Online Monitor for the agent and enable only the following metrics to track: Final Response Quality and Tool Use Quality. Set sampling percentage to 100% and configure a sample cap of 50.

Run the simulate-traffic.sh script for 20 sessions and 1.0 second of delay and wait for the evaluation to be triggered. Find the metric with the lowest value and investigate its root cause. Fix that and redeploy the agent with the fix. Run the script again to verify if the impacted metric is improved.

 Note
Make sure to either set environment variables for AGENT_RESOURCE_NAME and GOOGLE_CLOUD_LOCATION or pass them as parameters to the simulate-traffic.sh script

Success Criteria

  • An Online Monitor is created and active on Agent Platform.
  • The metrics Final Response Quality and Tool Use Quality are configured.
  • The (subtle) issue with the agent is detected through investigation of evaluations, and it’s fixed.
  • The Online Monitor metrics are above 90% after the fix.

Learning Resources

Tips

  • You can filter the telemetry using the following conditions:

      labels."gen_ai.evaluation.name" = xxxx
      labels."gen_ai.evaluation.score.value" < 1
    

    And look for a rubric around SEMANTIC_CORRECTNESS.

Previous Challenge