Gemini Embedding 2 Multimodal RAG: Build Cross-Modal Search

Your RAG pipeline has a preprocessing tax. CLIP for images. Whisper for audio transcription. A PDF parser that chokes on scanned documents. Custom glue code holding it all together. On March 10, 2026, Google launched Gemini Embedding 2 — Google’s first natively multimodal embedding model — and that entire stack became optional.

The model, available as gemini-embedding-2-preview, maps text, images, video, audio, and PDFs into a single 3,072-dimensional vector space. One API call, five modalities, no transcription step. This gemini embedding 2 multimodal RAG tutorial walks through building a cross-modal retrieval system from scratch — including the non-text chunking strategies for audio and PDFs that no other guide covers yet, because the model is nine days old.

The Preprocessing Stack You’re About to Delete

The old multimodal RAG architecture looked like a Rube Goldberg machine: CLIP generated image embeddings, Whisper transcribed audio to text, a PDF parser extracted what it could (and silently dropped charts, diagrams, and handwritten notes), and a text embedding model like OpenAI’s text-embedding-3-large handled the actual retrieval. Each component had its own failure modes, latency profile, and version dependencies.

As Philipp Schmid, Google Developer Advocate, put it: “RAG is mostly done on text. For rich multimedia files like PDFs it often requires complex pre-processing steps and dropping relevant graphics from your context.” Gemini Embedding 2 replaces that entire pipeline with a single model. A text query can now retrieve relevant images, audio clips, and PDF pages directly — without ever converting them to text first.

The benchmark numbers back the quality claim: 68.32 on MTEB Multilingual (+5.09 over second place), 68.8 on video retrieval versus Amazon Nova 2 at 60.3. Early adopter Sparkonomy reported a 70% latency reduction — not from faster embedding calls, but from eliminating pipeline stages entirely. Your architecture diagram just lost three boxes. Pair this with Gemini 3.1 Pro as the generation model and you have a full-stack Gemini RAG system.

Setup: SDK, Client, and Your First Embedding

Install the SDK (make sure it’s google-genai, not the legacy google-generativeai package, which was deprecated as of November 30, 2025):

pip install google-genai

Initialize the client:

from google import genai
from google.genai import types

client = genai.Client(api_key='YOUR_API_KEY')

Here’s the core embedding call. Three parameters matter — and one of them will silently break your retrieval if you get it wrong:

result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents='Your document text here',
    config=types.EmbedContentConfig(
        task_type='RETRIEVAL_DOCUMENT',
        output_dimensionality=768
    )
)
embedding = result.embeddings[0].values

The task_type parameter is not optional flavor. Use RETRIEVAL_DOCUMENT when indexing content and RETRIEVAL_QUERY when searching. The model optimizes the embedding geometry differently for each — using the same type for both degrades retrieval quality. The official Gemini embeddings documentation lists eight task types total, but these two are the ones you need for RAG.

Now the gotcha that will cost you hours of debugging if you miss it. When you set output_dimensionality to anything below 3,072, the returned embeddings are not normalized. Cosine similarity will return distorted rankings — silently, with no error. The fix is one line:

import numpy as np
normed = embedding / np.linalg.norm(embedding)

Always normalize. If you’re storing embeddings at 768 dimensions (and you should be — more on that below), L2 normalization is mandatory, not a footnote. See Google’s official embeddings notebook for the complete reference implementation.

Embedding Each Modality — and Chunking Strategies the Docs Skip

Text is straightforward — pass strings directly, no special handling. But the other modalities each have constraints that the model documentation lists without explaining how to work around.

Images: Pass PIL.Image objects, up to six per request (PNG or JPEG). You can also combine text and an image into a single content entry to produce a fused embedding — useful for product search where the image and description should be one retrievable unit.

from PIL import Image

img = Image.open('product.jpg')
result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents=[img],
    config=types.EmbedContentConfig(
        task_type='RETRIEVAL_DOCUMENT',
        output_dimensionality=768
    )
)

Audio and video can be passed as inline data (bytes/base64) or uploaded via the Files API. For small clips, inline is simplest. For anything larger — or when the same file is reused across multiple requests — upload first and poll for processing completion before embedding. The Files API path looks like this:

import time

audio_file = client.files.upload(file='clip.mp3')
while audio_file.state.name == 'PROCESSING':
    time.sleep(2)
    audio_file = client.files.get(name=audio_file.name)

result = client.models.embed_content(
    model='gemini-embedding-2-preview',
    contents=[audio_file],
    config=types.EmbedContentConfig(
        task_type='RETRIEVAL_DOCUMENT',
        output_dimensionality=768
    )
)

Here’s the chunking problem nobody has documented yet. Audio input maxes at 80 seconds. A 30-minute podcast episode needs to be split into windows — and splitting naively at 80-second boundaries will cut mid-sentence. Use 70-second windows with 10-second overlap.

PDFs hit the same wall at a different scale: six pages per request. A 50-page technical report needs six-page segments with one-page overlap to preserve context at boundaries. Same chunking principles you apply to text, just on a time or page axis.

One critical distinction the API enforces but doesn’t explain well: a single content entry with multiple parts (text + image) produces one fused embedding. Multiple separate entries produce separate embeddings. For RAG indexing, you want each document as its own entry. For a product catalog where an image and caption are one concept, combine them as parts of a single entry.

Getting this wrong means your entire index is conceptually broken — queries will match against fused representations when they should match individual documents, or vice versa. For teams wanting agentic retrieval on top of this pipeline, see the Gemini Deep Research API for agentic retrieval layers.

Building the Gemini Embedding 2 Multimodal RAG Pipeline End-to-End

Here’s the complete retrieval architecture in five steps:

• Ingest and chunk by modality: text into passages, audio into 70-second windows, PDFs into 6-page segments
• Embed with RETRIEVAL_DOCUMENT at 768 dimensions: every chunk gets a vector
• L2-normalize every vector before storage
• Store in a vector database: ChromaDB for prototyping, Qdrant or Weaviate for production
• Query: embed the text query with RETRIEVAL_QUERY at 768 dimensions, normalize, cosine similarity top-k

Why 768 dimensions and not the full 3,072? The model uses Matryoshka Representation Learning (MRL), which packs the most important semantic information into the earliest dimensions. The quality penalty for cutting to 768 dims is less than half a point on MTEB benchmarks — a negligible drop that saves 75% storage. For a million-document index, that’s roughly 3 GB versus 12 GB in float32. The quality tradeoff is negligible; the infrastructure savings are not.

If you’re using LangChain, the GoogleGenerativeAIEmbeddings class wraps this pattern with built-in task_type support. LlamaIndex offers similar integration. For teams building more complex agentic pipelines, see our LangGraph tutorial and Google Cloud’s multimodal RAG notebook for reference implementations.

One migration warning: embeddings from different models live in incompatible coordinate spaces. Moving from gemini-embedding-001 (or any other model) to gemini-embedding-2-preview requires re-embedding your entire index. You cannot mix old and new vectors and expect meaningful similarity scores.

Cost — and When to Use Something Else

Text embedding at $0.20 per million tokens looks painful next to OpenAI’s text-embedding-3-small at $0.02 — a 10x gap. But that comparison only makes sense if your RAG pipeline is text-only. The real cost of the old multimodal stack was Whisper at $0.006/minute, separate OCR services, CLIP inference costs, and the engineering time maintaining four pipelines that break independently. A team spending $100/month on text embeddings was often spending $500+ on adjacent infrastructure. Consult the Gemini API pricing page for exact per-modality rates.

For prototyping, the free tier runs at roughly 60 requests per minute as of March 2026 — enough to index thousands of documents before hitting limits. Note that batch API pricing, which Google offers for gemini-embedding-001 at a discount, is not currently available for gemini-embedding-2-preview — check the pricing page as availability may expand when the model reaches GA.

When should you use something else entirely? Three scenarios: text-only RAG where cost dominates (OpenAI wins on price, period); documents exceeding 8,192 tokens (Voyage offers 32K context, Cohere offers 128K); and production workloads needing guaranteed SLAs. As of March 19, 2026, gemini-embedding-2-preview is still in public preview, and Vertex AI deployment has been reported as limited to the us-central1 region — verify current region availability in the Vertex AI documentation before planning a production deployment.

What Comes Next

The 80-second audio cap and 6-page PDF limit feel like preview-era constraints, not permanent design decisions. What happens to the chunking strategies in this tutorial when Google raises those limits — and will time-windowed audio chunking still matter once the model can swallow an entire podcast in one call?

The most expensive part of a multimodal RAG system was never the embedding model. It was the four-pipeline preprocessing stack you built to feed it — the OCR, the transcription, the image captioning, the glue code. Gemini Embedding 2 didn’t make embeddings cheaper. It made everything around them unnecessary.

The GA release and higher per-modality input limits will determine whether the chunking workarounds in this tutorial become permanent best practices or temporary scaffolding. Track both on the Gemini API changelog — and start deleting preprocessing code now, before the next sprint buries it under another quarter of tech debt.