FLUX - SGLang Documentation

1. Model Introduction

FLUX is Black Forest Labs’ rectified-flow image model family. FLUX.1-dev is the smaller 12B text-to-image checkpoint; FLUX.2-dev is a 32B model that adds instruction-based editing plus single- and multi-reference composition. FLUX is a strong default when prompt adherence, polished image quality, or reference consistency matters. The tradeoff is deployment weight: FLUX.2 needs substantially more memory than FLUX.1, and the dev checkpoints use the FLUX non-commercial license, so review the model license before production use.

Checkpoint	Best fit	Main limitation
`black-forest-labs/FLUX.1-dev`	High-quality text-to-image with the lighter FLUX deployment	No native multi-reference editing path
`black-forest-labs/FLUX.2-dev`	Text-to-image, editing, and reference-guided composition in one model	32B model with a larger memory footprint

2. SGLang-diffusion Installation

SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the official SGLang-diffusion installation guide for installation instructions.

3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

3.1 Basic Configuration

FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version. Interactive Command Generator: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2, A3 NPUs.

3.2 Configuration Tips

See Performance Optimization for acceleration features and their runtime requirements.

--vae-path: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
--num-gpus: Number of GPUs to use
--tp-size: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
--sp-degree: Sequence parallelism size (typically should match the number of GPUs)
--ulysses-degree: The degree of DeepSpeed-Ulysses-style SP in USP
--ring-degree: The degree of ring attention-style SP in USP

4. API Usage

For complete API documentation, please refer to the official API usage guide.

4.1 Generate an Image

Example

import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1")

response = client.images.generate(
    model="black-forest-labs/FLUX.1-dev",
    prompt="A cat holding a sign that says hello world",
    size="1024x1024",
    n=1,
    response_format="b64_json",
)

# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)

4.2 Advanced Usage

4.2.1 Cache-DiT Acceleration

SGLang integrates Cache-DiT, a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set SGLANG_CACHE_DIT_ENABLED=True to enable it. For more details, please refer to the SGLang Cache-DiT documentation. Basic Usage

Command

SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path black-forest-labs/FLUX.1-dev

Advanced Usage

DBCache Parameters: DBCache controls block-level caching behavior:

Parameter	Env Variable	Default	Description
Fn	`SGLANG_CACHE_DIT_FN`	1	Number of first blocks to always compute
Bn	`SGLANG_CACHE_DIT_BN`	0	Number of last blocks to always compute
W	`SGLANG_CACHE_DIT_WARMUP`	4	Warmup steps before caching starts
R	`SGLANG_CACHE_DIT_RDT`	0.24	Residual difference threshold
MC	`SGLANG_CACHE_DIT_MC`	3	Maximum continuous cached steps

TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

Parameter	Env Variable	Default	Description
Enable	`SGLANG_CACHE_DIT_TAYLORSEER`	false	Enable TaylorSeer calibrator
Order	`SGLANG_CACHE_DIT_TS_ORDER`	1	Taylor expansion order (1 or 2)

Combined Configuration Example:

Command

SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path black-forest-labs/FLUX.1-dev

4.2.2 CPU Offload

--dit-cpu-offload: Use CPU offload for DiT inference. Enable if run out of memory.
--text-encoder-cpu-offload: Use CPU offload for text encoder inference.
--vae-cpu-offload: Use CPU offload for VAE.
--pin-cpu-memory: Pin memory for CPU offload. Only added as a temp workaround if it throws “CUDA error: invalid argument”.

4.2.3 Known LoRA examples

Use --lora-path at startup or the LoRA management API to load an adapter. Known FLUX examples include:

5. Benchmark

5.1 Speedup Benchmark

5.1.1 Generate a image

Test Environment:

Hardware: NVIDIA B200 GPU (1x)
Model: black-forest-labs/FLUX.1-dev
sglang diffusion version: 0.5.6.post2

NVIDIA B200
Ascend A3

Server Command:

Command

sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000

Benchmark Command:

Command

python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1

Result:

Output

================= Serving Benchmark Result =================
Model:                                   black-forest-labs/FLUX.1-dev
Dataset:                                 vbench
Task:                                    text-to-image
--------------------------------------------------
Benchmark duration (s):                  50.97
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.02
Latency Mean (s):                        50.9681
Latency Median (s):                      50.9681
Latency P99 (s):                         50.9681
--------------------------------------------------
Peak Memory Max (MB):                    27905.19
Peak Memory Mean (MB):                   27905.19
Peak Memory Median (MB):                 27905.19
============================================================

Server Command:

Command

#One A3 card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2

Benchmark Command:

Command

python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1

Result:

Output

================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        black-forest-labs/FLUX.1-dev
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       16.30
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.06
Output throughput (outputs/s):                0.06
Latency Mean (s):                             16.30
Latency Median (s):                           16.30
Latency P90 (s):                              16.30
Latency P95 (s):                              16.30
Latency P99 (s):                              16.30
--------------------------------------------------
Peak Memory Max (MB):                         19972.00
Peak Memory Mean (MB):                        19972.00
Peak Memory Median (MB):                      19972.00
------------------------------------------------------------

5.1.2 Generate images with high concurrency

NVIDIA B200
Ascend A3

Server Command :

Command

sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000

Benchmark Command :

Command

python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20

Result :

Output

================= Serving Benchmark Result =================
Model:                                   black-forest-labs/FLUX.1-dev
Dataset:                                 vbench
Task:                                    text-to-image
--------------------------------------------------
Benchmark duration (s):                  111.79
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     20/20
--------------------------------------------------
Request throughput (req/s):              0.18
Latency Mean (s):                        67.0646
Latency Median (s):                      66.9691
Latency P99 (s):                         110.8949
--------------------------------------------------
Peak Memory Max (MB):                    27917.19
Peak Memory Mean (MB):                   27916.59
Peak Memory Median (MB):                 27917.19
============================================================

Server Command :

Command

#One A3 card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2

Benchmark Command :

Command

python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20

Result :

Output

================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        black-forest-labs/FLUX.1-dev
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       300.85
Request rate:                                 inf
Max request concurrency:                      20
Successful requests:                          18/20
Completed outputs:                            18
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.06
Output throughput (outputs/s):                0.06
Latency Mean (s):                             155.16
Latency Median (s):                           155.11
Latency P90 (s):                              266.30
Latency P95 (s):                              280.15
Latency P99 (s):                              291.23
--------------------------------------------------
Peak Memory Max (MB):                         19972.00
Peak Memory Mean (MB):                        19972.00
Peak Memory Median (MB):                      19972.00
------------------------------------------------------------

​1. Model Introduction

​2. SGLang-diffusion Installation

​3. Model Deployment

​3.1 Basic Configuration

​3.2 Configuration Tips

​4. API Usage

​4.1 Generate an Image

​4.2 Advanced Usage

​4.2.1 Cache-DiT Acceleration

​4.2.2 CPU Offload

​4.2.3 Known LoRA examples

​5. Benchmark

​5.1 Speedup Benchmark

​5.1.1 Generate a image

​5.1.2 Generate images with high concurrency

1. Model Introduction

2. SGLang-diffusion Installation

3. Model Deployment

3.1 Basic Configuration

3.2 Configuration Tips

4. API Usage

4.1 Generate an Image

4.2 Advanced Usage

4.2.1 Cache-DiT Acceleration

4.2.2 CPU Offload

4.2.3 Known LoRA examples

5. Benchmark

5.1 Speedup Benchmark

5.1.1 Generate a image

5.1.2 Generate images with high concurrency