LingBot World

1. Model Introduction

LingBot World is a realtime camera-controlled video world model. It keeps a live causal session, applies prompt and camera events between chunks, reuses DiT/VAE state, and streams decoded frames instead of finishing a bounded clip before returning. Choose it for interactive exploration and continuous control, not one-shot cinematic generation. The causal window makes long sessions practical but gives up the global bidirectional context available to offline Wan or LTX pipelines; session state and cache policy are therefore part of the serving contract.

2. Deployment

Command

sglang serve \
  --model-path robbyant/lingbot-world-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false

Optional KV-Cache Compression

Long-running sessions can enable lossy int4 PRQ compression for completed causal KV-cache chunks by installing quant-videogen as described in the quantization guide and adding --kv-cache-quant int4 to the server command. The current and most recent completed chunks remain in BF16. See Causal KV-Cache Quantization for the algorithm, tuning options, measured memory-latency tradeoff, and support limits.

3. Realtime WebUI

The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior.

Command

python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui

Open http://127.0.0.1:18080 and use:

Example

ws://127.0.0.1:30000/v1/realtime_video/generate

4. HTTP and WebSocket API

LingBot World uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running.

Endpoints

API	Method	Purpose	Notes
`/v1/models`	`GET`	Query the served model id before opening a session.	The WebUI uses this to fill the model field when the server exposes model metadata.
`/v1/realtime_video/generate`	`WebSocket`	Create one realtime LingBot session and stream generated video chunks.	The first client message must be an `init` message encoded with MessagePack.

`init` message

Send this MessagePack map immediately after the WebSocket opens.

Parameter	Type	Required	Meaning
`type`	string	Yes	Must be `"init"`.
`model`	string	No	Model id. Leave empty to use the served model.
`prompt`	string	Yes	Text prompt for the initial scene and motion style.
`first_frame`	bytes or string	Yes	Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string.
`size`	string	Yes	Generation size as `WIDTHxHEIGHT`, for example `832x480`.
`fps`	number	Yes	Target playback FPS for the generated stream.
`num_frames`	integer	Yes	Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size.
`seed`	integer	No	Random seed for deterministic sampling.
`num_inference_steps`	integer	No	Denoising steps per chunk. LingBot defaults to `4` when omitted.
`guidance_scale`	number	No	Classifier-free guidance scale. Realtime LingBot commonly uses `1`.
`negative_prompt`	string	No	Negative prompt passed to the diffusion pipeline.
`quality`	`"lossless"`, `"extra-high"`, `"high"`	No	`lossless` keeps FP32 VAE decode. `extra-high` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency.
`max_chunks`	integer	No	Stop after this many chunks. Omit for a continuous session.
`realtime_causal_sink_size`	integer	No	Number of sink frames/tokens retained in the causal attention window.
`realtime_causal_kv_cache_num_frames`	integer	No	Number of recent frames retained in the causal KV cache window.
`realtime_output_format`	`"webp"`, `"jpeg"`, `"raw"`	No	Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` sends raw RGB; omit for lossless delta-gzip RGB.
`output_compression`	integer	No	Preview quality for `webp` or `jpeg`, from `1` to `100`.
`enable_upscaling`	boolean	No	Enable server-side super resolution after frame decode.
`upscaling_scale`	integer	No	Super-resolution scale. Current default is `4` when upscaling is enabled.
`upscaling_model_path`	string	No	Optional Real-ESRGAN model path.
`enable_frame_interpolation`	boolean	No	Enable frame interpolation. Keep this disabled when measuring true generated FPS.
`frame_interpolation_exp`	integer	No	Interpolation multiplier exponent. `1` means 2x frames.
`frame_interpolation_scale`	number	No	RIFE internal scale for interpolation.
`frame_interpolation_model_path`	string	No	Optional RIFE model path.
`condition_inputs.camera_actions`	`list[list[string]]`	No	Initial scripted camera actions, one action list per frame.

Runtime `event` messages

After init, send MessagePack event maps to update the live session.

Parameter	Type	Required	Meaning
`type`	string	Yes	Must be `"event"`.
`kind`	`"prompt"` or `"camera_actions"`	Yes	Runtime event kind.
`payload`	string or object/list	Yes	For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload.
`event_id`	integer	No	Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled.

camera_actions supports two payload modes:

Mode	Payload shape	Meaning
Script	`list[list[string]]`	A fixed sequence of per-frame actions consumed by upcoming chunks.
State	`{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }`	Live control state transitions from keyboard or UI controls.

Supported LingBot action tokens include w, a, s, d for camera movement and i, j, k, l for look controls.

Server messages

Message	Payload	Meaning
`frame_batch`	MessagePack map with `payload` bytes	One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata.
`chunk_stats`	MessagePack map	Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`.
`error`	MessagePack map	Server-side validation or generation error.

Minimal client sketch

Python

import msgspec.msgpack
import websocket

ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate")
ws.send_binary(msgspec.msgpack.encode({
    "type": "init",
    "prompt": "A quiet rainy London alley, stable camera motion.",
    "first_frame": open("reference.jpg", "rb").read(),
    "size": "832x480",
    "fps": 25,
    "num_frames": 9,
    "num_inference_steps": 4,
    "guidance_scale": 1,
    "realtime_output_format": "webp",
    "output_compression": 95,
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "camera_actions",
    "event_id": 1,
    "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]},
}))

5. Consistency

LingBot World uses raw-frame websocket GT plus per-chunk latency guards for consistency checks.

6. Notes

Use the realtime endpoint for interactive sessions: /v1/realtime_video/generate.
Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks.
Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior.

Cookbook

1. Model Introduction

2. Deployment

Optional KV-Cache Compression

3. Realtime WebUI

4. HTTP and WebSocket API

Endpoints

`init` message

Runtime `event` messages

Server messages

Minimal client sketch

5. Consistency

6. Notes

7. Run in ComfyUI

​1. Model Introduction

​2. Deployment

​Optional KV-Cache Compression

​3. Realtime WebUI

​4. HTTP and WebSocket API

​Endpoints

​init message

​Runtime event messages

​Server messages

​Minimal client sketch

​5. Consistency

​6. Notes

​7. Run in ComfyUI

1. Model Introduction

2. Deployment

Optional KV-Cache Compression

3. Realtime WebUI

4. HTTP and WebSocket API

Endpoints

`init` message

Runtime `event` messages

Server messages

Minimal client sketch

5. Consistency

6. Notes

7. Run in ComfyUI