Skip to main content
While the DiT denoises, the text and image encoders are idle — and while they encode, the whole DiT replica is idle. --encoder-parallel decides how to use those otherwise-unused GPUs for the encoding stage.
The two accelerated modes are mutually exclusive per encoder: folding shards the weights for the lifetime of the loaded model, so a folded encoder cannot also be data-parallel.

​
Which Mode Wins

Measured on H100 across T5 (hidden 4096), Qwen3 (2560), and CLIP-L (768) at batch 1–8 and replica sizes 2 and 4:
  • Folding pays when the encoder is wide enough that sharding its GEMMs beats the per-layer all-reduce it adds. T5 gains; Qwen3 (+35%) and CLIP-L (+50%) get slower, so folding is gated at hidden ≥ 4096. Its benefit also saturates as the replica grows, since each rank’s slice keeps shrinking.
  • Data-parallel pays only when the encode is compute-bound, which needs a wide encoder (hidden ≥ 1024 — CLIP-L is slower at every batch and replica measured) and more than one prompt in a single encode call.
  • Replication is the right answer whenever neither condition holds, which is most single-request latency work.
auto encodes exactly these rules, so prefer it unless you are pinning a configuration you measured yourself.

​
Numerics

replicate matches single-GPU encoding bit-for-bit only when the encoder TP degree is one. Both the existing DiT TP layout and fold can reorder parallel reductions; they are mathematically equivalent but are not generally bitwise identical to a single-GPU kernel. dp is also not bitwise-identical: each encoder copy sees a smaller batch, so GEMM tiling can differ from the unsplit reference. It may compose with encoder TP: all ranks in one TP group receive the same batch slice, and corresponding TP ranks gather outputs across the orthogonal encoder-DP group. The result remains mathematically equivalent and deterministic for a fixed topology and batch shape, but long video sampling can amplify small floating-point differences. Throughput serving. A single encode call must carry more than one prompt for DP to engage, so raise the batching ceiling too; an encoder flag deliberately does not change DiT batching for you:
Single-request latency with one wide text encoder:
Bit-exact reproducibility against a single-GPU reference:

​
Interaction With Other Flags

  • Tensor parallel: encoder DP composes with TP. A TP group jointly encodes one batch slice; the orthogonal ranks inside the same pipeline replica split and gather the batch. A pure-TP replica has one encoder copy, so there is no additional batch-DP degree.
  • Data parallel: encoder collectives never cross pipeline replicas. With --dp-size > 1, each replica independently uses its own TP/SP/CFG ranks.
  • Dynamic batching: dp only pays with a wide batch. Selecting it does not change --batching-max-size; configure that separately. See Inference Batching.
  • Sequence parallelism: SP splits the DiT latent sequence. During encoding, those ranks either hold encoder copies for batch DP or join a folded encoder. See Sequence Parallelism.