> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sglang.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Embedding models

> Dense and sparse embedding models with FlashInfer acceleration and SGLang's batching infrastructure.

SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment.

> [!WARNING]
>
> Native encoder embedding architectures, `google/embeddinggemma-300m` and `nvidia/Nemotron-3-Embed-1B-BF16` are detected automatically. Decoder-style embedding models require `--is-embedding`; add `--trust-remote-code` when the model requires it.

## Quick Start

### Launch Server

```bash theme={null}
sglang serve \
  --model-path Qwen/Qwen3-Embedding-4B \
  --is-embedding
```

### EmbeddingGemma

EmbeddingGemma uses bidirectional attention and is auto-detected, so its best default command is simply:

```bash theme={null}
sglang serve --model-path google/embeddinggemma-300m
```

On CUDA, SGLang automatically uses breakable CUDA graph (BCG) for its full encoder prefill and disables incompatible radix-cache and chunked-prefill behavior. Do not add the deprecated piecewise CUDA graph knobs.

### Nemotron-3-Embed

`nvidia/Nemotron-3-Embed-1B-BF16` uses a bidirectional Ministral3 backbone with normalized mean pooling. It is auto-detected from its checkpoint config, so `--is-embedding` is not needed:

```bash theme={null}
sglang serve --model-path nvidia/Nemotron-3-Embed-1B-BF16
```

It is not supported on ROCm. SGLang disables radix cache and chunked prefill for this model, and skips the KV cache allocation when the prefill attention backend is FA3 or FA4. Prepend the task prefix to each input text as described in the model card, for example `query: ...` or `passage: ...`.

### Client Request

```python theme={null}
import requests

url = "http://127.0.0.1:30000"

payload = {
    "model": "Qwen/Qwen3-Embedding-4B",
    "input": "What is the capital of France?",
    "encoding_format": "float"  # or "base64" for compact FP32 responses
}

response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embedding:", response["data"][0]["embedding"])
```

## Multimodal Embedding Example

For multimodal models like GME that support both text and images:

```bash theme={null}
sglang serve \
  --model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
  --is-embedding \
  --chat-template gme-qwen2-vl
```

```python Example theme={null}
import requests

url = "http://127.0.0.1:30000"

text_input = "Represent this image in embedding space."
image_path = "https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild/resolve/main/images/023.jpg"

payload = {
    "model": "gme-qwen2-vl",
    "input": [
        {
            "text": text_input
        },
        {
            "image": image_path
        }
    ],
}

response = requests.post(url + "/v1/embeddings", json=payload).json()

print("Embeddings:", [x.get("embedding") for x in response.get("data", [])])
```

## Matryoshka Embedding Example

[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows user to trade off between performance and cost.

### 1. Launch a Matryoshka‑capable model

If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below:

```bash Command theme={null}
sglang serve \
    --model-path Qwen/Qwen3-Embedding-0.6B \
    --is-embedding \
    --json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}'
```

1. Setting `"is_matryoshka": true` allows truncating to any dimension. Otherwise, the server will validate that the specified dimension in the request is one of `matryoshka_dimensions`.
2. Omitting `dimensions` in a request returns the full vector.

### 2. Make requests with different output dimensions

```python theme={null}
import requests

url = "http://127.0.0.1:30000"

# Request a truncated (Matryoshka) embedding by specifying a supported dimension.
payload = {
    "model": "Qwen/Qwen3-Embedding-0.6B",
    "input": "Explain diffusion models simply.",
    "dimensions": 512  # change to 128 / 1024 / omit for full size
}

response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embedding:", response["data"][0]["embedding"])
```

## Supported Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Example Model</th>
      <th>Chat template</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>EmbeddingGemma</td>
      <td>`google/embeddinggemma-300m`</td>
      <td>N/A</td>
      <td>Bidirectional Gemma3 text encoder; auto-detected and served with BCG by default</td>
    </tr>

    <tr>
      <td>Nemotron-3-Embed</td>
      <td>`nvidia/Nemotron-3-Embed-1B-BF16`</td>
      <td>N/A</td>
      <td>Bidirectional Ministral3 encoder with normalized mean pooling; auto-detected</td>
    </tr>

    <tr>
      <td>E5 (Llama/Mistral based)</td>
      <td>`intfloat/e5-mistral-7b-instruct`</td>
      <td>N/A</td>
      <td>High-quality text embeddings based on Mistral/Llama architectures</td>
    </tr>

    <tr>
      <td>GTE-Qwen2</td>
      <td>`Alibaba-NLP/gte-Qwen2-7B-instruct`</td>
      <td>N/A</td>
      <td>Alibaba's text embedding model with multilingual support</td>
    </tr>

    <tr>
      <td>Qwen3-Embedding</td>
      <td>`Qwen/Qwen3-Embedding-4B`</td>
      <td>N/A</td>
      <td>Latest Qwen3-based text embedding model for semantic representation</td>
    </tr>

    <tr>
      <td>Qwen3 (bare backbone)</td>
      <td>`microsoft/harrier-oss-v1-0.6b`</td>
      <td>N/A</td>
      <td>Bare `Qwen3Model` backbone (no LM head); served natively on SGLang's fused Qwen3 kernels and auto-classified as an embedding model</td>
    </tr>

    <tr>
      <td>BGE</td>
      <td>`BAAI/bge-large-en-v1.5`</td>
      <td>N/A</td>
      <td>BAAI's text embeddings (requires `attention-backend` triton/torch\_native)</td>
    </tr>

    <tr>
      <td>GME (Multimodal)</td>
      <td>`Alibaba-NLP/gme-Qwen2-VL-2B-Instruct`</td>
      <td>`gme-qwen2-vl`</td>
      <td>Multimodal embedding for text and image cross-modal tasks</td>
    </tr>

    <tr>
      <td>CLIP</td>
      <td>`openai/clip-vit-large-patch14-336`</td>
      <td>N/A</td>
      <td>OpenAI's CLIP for image and text embeddings</td>
    </tr>
  </tbody>
</table>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.