Challenge 3: Formula E RAG-ing
Previous Challenge Next Challenge
Introduction
In order to find semantically similar items we need to measure the distance between vectors in the embedding space. We could implement that ourselves by calculating the distance between each embedding, but BigQuery already provides a function, VECTOR_SEARCH, that simplifies this process.
Once we have determined the correct segment, we’ll use that for RAG. Retrieval augmented generation (RAG) is a popular approach for enabling LLMs to access external data and provides a mechanism to mitigate against hallucinations. The main idea is to provide the LLM more context to get reliable answers. This is typically done by looking up relevant information from a (vector) database and adding that information to the prompt of the LLM.
Description
Design a SQL query that retrieves the top result from the embeddings table given the phrase car crash. Once you have found the correct video segment, you’ll use Vertex AI Studio to extract the exact timestamp of the crash from that video segment.
Navigate to Vertex AI Studio, Create Prompt section, and design a prompt to get the exact timestamp of the crash, using the video segment in your prompt.
Note Have a look at the video segment to confirm when the crash happened. You can preview the videos in the Console, just browse to the specific video through Cloud Storage Bucket browser and there will be a button for Preview.
Success Criteria
- The SQL query returns the uri for
cam_15_07.mp4. - Vertex AI Studio outputs the exact timestamp of the crash covered in the video segment in
dd/mm/yyyy * HH:MM:SSformat.
Learning Resources
Tips
- In Vertex AI Studio you can use different words to describe the crash (e.g. collision), experiment with those as well as different models/settings. And in case you need additional help with your prompt design, consider the AI-powered prompt writing.
- The video has the timestamp information in the right format in the top left corner of each frame.