Challenge 3: Formula E RAG-ing
Previous Challenge Next Challenge
Introduction
In order to find semantically similar items we need to measure the distance between vectors in the embedding space. We could implement that ourselves by calculating the distance between each embedding, but BigQuery already provides a function, VECTOR_SEARCH, that simplifies this process.
Once we have determined the correct segment, we’ll use that for RAG. Retrieval augmented generation (RAG) is a popular approach for enabling LLMs to access external data and provides a mechanism to mitigate against hallucinations. The main idea is to provide the LLM more context to get reliable answers. This is typically done by looking up relevant information from a (vector) database and adding that information to the prompt of the LLM.
Description
Design a SQL query that retrieves the top result from the embeddings table given the phrase car crash. Once you have found the correct video segment, you’ll use Vertex AI Studio to extract the exact timestamp of the crash from that video segment.
Navivate to Vertex AI Studio, Freeform option, and design a prompt to get the exact timestamp of the crash, using the video segment in your prompt.
Success Criteria
- The SQL query returns the uri for
cam_15_07.mp4. - Vertex AI Studio outputs the exact timestamp for the crash covered in the video segment.
Learning Resources
Tips
- In Vertex AI Studio you can use different words to describe the crash (e.g. collision), experiment with those as well as different models/settings. And in case you need additional help with your prompt design, consider the AI-powered prompt writing.