OpenAI APIs - Embedding - SGLang Documentation

SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference. This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the corresponding overview page

Launch A Server

Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and google/embeddinggemma-300m are detected automatically. Decoder-style embedding models still require --is-embedding.

Example

from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

embedding_process, port = launch_server_cmd(
    """
sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
    --is-embedding --log-level warning
"""
)

wait_for_server(f"http://localhost:{port}")

Using cURL

Example

import subprocess, json

text = "Once upon a time"

curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'"""

result = subprocess.check_output(curl_text, shell=True)

print(result)

text_embedding = json.loads(result)["data"][0]["embedding"]

print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")

Using Python Requests

Example

import requests

text = "Once upon a time"

response = requests.post(
    f"http://localhost:{port}/v1/embeddings",
    json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text},
)

text_embedding = response.json()["data"][0]["embedding"]

print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")

Using OpenAI Python Client

Example

import openai

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

# Text embedding example
response = client.embeddings.create(
    model="Alibaba-NLP/gte-Qwen2-1.5B-instruct",
    input=text,
)

embedding = response.data[0].embedding[:10]
print_highlight(f"Text embedding (first 10): {embedding}")

Using Input IDs

SGLang also supports input_ids as input to get the embedding.

Example

import json
import os
from transformers import AutoTokenizer

os.environ["TOKENIZERS_PARALLELISM"] = "false"

tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct")
input_ids = tokenizer.encode(text)

curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'"""

input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
    0
]["embedding"]

print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")

Compact Base64 Responses

Set encoding_format to base64 when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients.

Example

response = requests.post(
    f"http://localhost:{port}/v1/embeddings",
    json={
        "model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct",
        "input": text,
        "encoding_format": "base64",
    },
)

base64_embedding = response.json()["data"][0]["embedding"]
print_highlight(f"Base64 embedding: {base64_embedding[:20]}...")

Example

terminate_process(embedding_process)

Please refer to Multi-Modal Embedding Model

Basic Usage

Advanced Features

Supported Models

Developer Guide

References

OpenAI APIs - Embedding

Launch A Server

Using cURL

Using Python Requests

Using OpenAI Python Client

Using Input IDs

Compact Base64 Responses

​Launch A Server

​Using cURL

​Using Python Requests

​Using OpenAI Python Client

​Using Input IDs

​Compact Base64 Responses

​Multi-Modal Embedding Model

Launch A Server

Using cURL

Using Python Requests

Using OpenAI Python Client

Using Input IDs

Compact Base64 Responses

Multi-Modal Embedding Model