Skip to main content
dVLAOpenPI / LeRobotflow matching actionrobot edge

​
1. Model Introduction

Pi0.5 is an OpenPI / LeRobot diffusion Vision-Language-Action (dVLA) policy. It consumes camera images, a language instruction, and robot state, then returns a continuous action chunk for robot control. SGLang serves Pi0.5 through the native multimodal_gen runtime. The implementation uses a SigLIP/PaliGemma prefix encoder and a Gemma action expert: the prefix is encoded once, then the action expert runs the flow-matching denoising loop. This is not a token decode workload, so the Pi0.5 path does not use the LLM sampler, logits processor, token streaming, paged decode KV cache, or a separate SRT serving engine. The prefix encoder covers both stages of observation encoding: SigLIP turns resized camera pixels into continuous patch embeddings, then the PaliGemma transformer jointly encodes those patches with tokenized task/state inputs and produces per-layer prefix K/V. At flow timestep t, the action expert projects the noisy continuous action chunk x_t into action embeddings. Its queries attend to both the fixed prefix K/V and the current action K/V, while the timestep follows a separate sinusoidal-MLP path and conditions every action-expert layer through AdaRMSNorm gates. Supported public checkpoints:
CheckpointCamerasState DimOutput Action DimAction HorizonDenoise Steps
lerobot/pi05_basebase_0_rgb, left_wrist_0_rgb, right_wrist_0_rgb32325010
lerobot/pi05_libero_baseimage, image2, one empty camera875010
References:

​
2. Installation

Install SGLang with the diffusion extra. Pi0.5 lives in multimodal_gen, and the extra includes the runtime dependencies used by the policy server.
Command
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[diffusion]"
For general environment setup, see the SGLang Diffusion installation guide.

​
3. Model Deployment

Serve the base Pi0.5 policy:
Command
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
Serve the LIBERO checkpoint:
Command
sglang serve lerobot/pi05_libero_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
These registered LeRobot checkpoints dispatch to the native multimodal_gen Pi0.5 pipeline automatically. The --pipeline / --pipeline-class-name flag is only an advanced override for local or private checkpoints that cannot be resolved from the model registry. Pi0.5 serving does not start a second SRT LLM serving engine.

​
3.1 Action Request Schema

FieldTypeDescription
modelstring, optionalServed model name. When omitted, the server uses the currently loaded policy.
input.task or input.promptstringLanguage instruction for the policy.
input.observation.imagesobjectMap from camera name to RGB image. Images can be nested here or sent as OpenPI observation keys such as observation.images.base_0_rgb through the websocket adapter.
input.observation.statearray or tensor objectRobot state vector. Use the same normalization convention as the OpenPI / LeRobot checkpoint.
input.observation.noisearray or tensor object, optionalInitial action noise with shape [action_horizon, action_dim]. Use this for deterministic debugging.
parameters.num_inference_stepsinteger, optionalFlow-matching denoise steps. Defaults to 10.
parameters.action_horizoninteger, optionalOutput action horizon. Defaults to the checkpoint config.
parameters.action_diminteger, optionalInternal padded action dimension. Defaults to the checkpoint config.
runtime.return_timingboolean, optionalReturn stage timing fields. Defaults to true.
runtime.prefix_cacheboolean or "auto", optionalEnable exact full-prefix lookup for this request when the server has enable_global_prefix_cache=true. Defaults to "auto".
runtime.cuda_graphboolean or "auto", optionalEnable the action denoise CUDA graph path for this request when a matching shape bucket is available. Defaults to "auto".
runtime.output_format"list" or "numpy", optionalUse "list" for JSON compatibility. Use "numpy" with msgpack or Python clients to avoid Python-list materialization. Defaults to "list".
runtime.response_format"envelope" or "raw", optionalHTTP-only response shape. "envelope" returns the generic action envelope. "raw" returns the policy payload directly. Defaults to "envelope".

​
4. API Usage

​
4.1 Generic Action HTTP API

Use /v1/actions/generations for direct policy calls and debugging. Use /v1/actions/metadata to discover the camera keys, state size, action shape, defaults, and websocket capabilities of the currently served policy.
Example
import numpy as np
import requests

image = np.zeros((224, 224, 3), dtype=np.uint8)
payload = {
    "model": "lerobot/pi05_base",
    "input": {
        "task": "pick up the block",
        "observation": {
            "images": {
                "base_0_rgb": image.tolist(),
                "left_wrist_0_rgb": image.tolist(),
                "right_wrist_0_rgb": image.tolist(),
            },
            "state": np.zeros(32, dtype=np.float32).tolist(),
        },
    },
    "runtime": {
        "return_timing": True,
        "prefix_cache": "auto",
        "cuda_graph": "auto",
    },
}

response = requests.post(
    "http://127.0.0.1:30000/v1/actions/generations",
    json=payload,
    timeout=60,
)
response.raise_for_status()
data = response.json()

actions = data["data"][0]["action"]["values"]
print(len(actions), len(actions[0]))
print(data.get("timings"))
The same /v1/actions/generations endpoint also accepts Content-Type: application/msgpack and can return msgpack when Accept: application/msgpack is set. For msgpack clients, send numpy arrays directly using the pack_numpy_payload helper from the websocket example below. Msgpack requests default to numpy action output on the server side; set runtime.output_format to "list" only when a client explicitly needs nested Python lists. For the lowest-overhead generic HTTP path, use msgpack with the raw policy response:
Example
payload["runtime"] = {
    "return_timing": True,
    "prefix_cache": False,
    "cuda_graph": "auto",
    "response_format": "raw",
}

response = requests.post(
    "http://127.0.0.1:30000/v1/actions/generations",
    data=packb(payload),
    headers={
        "Content-Type": "application/msgpack",
        "Accept": "application/msgpack",
    },
    timeout=60,
)
result = unpackb(response.content)
actions = result["actions"]
For lerobot/pi05_libero_base, use the LIBERO camera names and an 8-dimensional state vector:
Example
payload = {
    "input": {
        "task": "pick up the object",
        "observation": {
            "images": {
                "image": image.tolist(),
                "image2": image.tolist(),
            },
            "state": np.zeros(8, dtype=np.float32).tolist(),
        },
    },
}

​
4.2 Generic Realtime WebSocket

/v1/actions/realtime is the generic msgpack websocket path. It sends action.metadata on connect and returns the same action.generation envelope as the HTTP API for each request.

​
4.3 OpenPI-Compatible WebSocket

Robot clients can use the OpenPI-compatible msgpack websocket endpoint at /openpi/policy. The server sends metadata immediately after connection, then each client message should contain one observation.
Example
import asyncio

import msgpack
import numpy as np
import websockets


def pack_array(obj):
    if isinstance(obj, np.ndarray):
        return {
            b"__ndarray__": True,
            b"data": obj.tobytes(),
            b"dtype": obj.dtype.str,
            b"shape": obj.shape,
        }
    if isinstance(obj, np.generic):
        return {
            b"__npgeneric__": True,
            b"data": obj.item(),
            b"dtype": obj.dtype.str,
        }
    return obj


def unpack_array(obj):
    ndarray_marker = obj.get("__ndarray__") or obj.get(b"__ndarray__")
    npgeneric_marker = obj.get("__npgeneric__") or obj.get(b"__npgeneric__")
    data = obj.get("data", obj.get(b"data"))
    dtype = obj.get("dtype", obj.get(b"dtype"))
    shape = obj.get("shape", obj.get(b"shape"))
    if ndarray_marker:
        return np.ndarray(
            buffer=data,
            dtype=np.dtype(dtype),
            shape=shape,
        )
    if npgeneric_marker:
        return np.dtype(dtype).type(data)
    return obj


def packb(payload):
    return msgpack.packb(payload, default=pack_array, use_bin_type=True)


def unpackb(payload):
    return msgpack.unpackb(payload, object_hook=unpack_array, raw=False)


async def main():
    image = np.zeros((224, 224, 3), dtype=np.uint8)
    observation = {
        "task": "pick up the block",
        "observation.images.base_0_rgb": image,
        "observation.images.left_wrist_0_rgb": image,
        "observation.images.right_wrist_0_rgb": image,
        "observation.state": np.zeros(32, dtype=np.float32),
    }

    async with websockets.connect(
        "ws://127.0.0.1:30000/openpi/policy",
        max_size=None,
    ) as websocket:
        metadata = unpackb(await websocket.recv())
        print(metadata)

        await websocket.send(packb(observation))
        result = unpackb(await websocket.recv())
        actions = result["actions"]
        print(len(actions), len(actions[0]))
        print(result.get("server_timing"))


asyncio.run(main())

​
5. Configuration Tips

  • Request-local PrefixContext is always reused across all denoise steps in one request. The prefix K/V is not cloned per step.
  • The optional global prefix cache is a bounded exact-match LRU. It is disabled by default because changing robot frames rarely hit it and enabling it prevents unrelated misses from entering grouped prefix execution. Set enable_global_prefix_cache=true for repeated observations, retries, or multiple policy calls over the same camera/state sample; runtime.prefix_cache can then disable lookup per request.
  • Partial-prefix reuse is not supported because Pi0.5 combines image and tokenized task/state inputs under full attention. Changing any input can change every deeper-layer prefix K/V tensor. The exact key hashes resized and normalized pixels before SigLIP, plus effective token IDs, token masks, camera masks, model revision, dtype, and parallel layout. Tensor content hashing reuses SRT’s CPU/CUDA implementation; hashing the pre-SigLIP input lets an exact hit skip both the vision encoder and prefix transformer.
  • CUDA graph capture targets one action-denoise step by shape bucket, then replays it across the flow-matching loop. Shape buckets include batch size, prefix length, action horizon, action dim, dtype, and parallel layout. With action SP enabled, the graph bucket uses the local action shard length and rank-specific position offset.
  • Cache-DiT is not used in the default Pi0.5 path. The current robot policy target is numerically lossless inference, while Cache-DiT-style reuse is an image/video DiT approximation that needs separate policy-quality validation before it can be recommended for action control.
  • Do not use CFG parallelism to split the 10 Euler steps. Use it only for independent branches such as multiple candidate actions or future conditional/unconditional branches.
  • Prefix TP uses native SGLang parallel linear layers for the PaliGemma language prefix model when model parallel TP is initialized and the VLA split broadcast group is not active. The action expert does not share that TP layout. The v1 split prefix/action path instead uses the SP group: prefix root computes/broadcasts PrefixContext, while action ranks run the SP action path.
  • The split path uses the SP group as the action group: prefix root computes/broadcasts PrefixContext, all action ranks broadcast the initial action noise once, shard the action horizon, and run the action expert through Ulysses attention when the prefix is full-attention, ring degree is one, heads are divisible by SP size, and the horizon is evenly shardable. Otherwise it falls back to action-root execution.
  • lerobot/pi05_libero_base returns 7 action dimensions even though the internal padded action tensor uses 32 dimensions.

​
6. VRAM Tuning

For current public Pi0.5 checkpoints, a stable 16GB discrete GPU target is a reasonable v1 deployment bar for robot workstations. In practice, leave headroom for the driver, camera middleware, robot process, and allocator fragmentation. Jetson/Orin unified-memory devices need extra caution because system RAM and GPU memory share the same pool. OpenPI inference is mixed precision, not full fp32: most weights and compute run in bf16, selected stability-sensitive weights stay in fp32, and returned actions are float32. SGLang mirrors that policy by default. The validated pi05_aloha OpenPI PyTorch checkpoint keeps 119,720,608 parameters in fp32; SGLang reports the same fp32 stability set, plus 3,233,713,264 bf16 runtime parameters after skipping unused LM heads for continuous action inference. The fp32 set includes SigLIP patch/position embeddings, Gemma layer norms/final norms, and the action/time projection heads. Keep materialize_dtype at bf16 unless you are debugging numerical parity. Current H100 pressure validation shows that the bf16 model path fits a 16GB-free budget without layerwise offload. The Pi0.5 path batches all camera frames for a grouped request into one SigLIP forward before splitting the embeddings back by camera, which is important for multi-camera robot workloads. The latency numbers below use the Python grouped API with global prefix cache disabled and CUDA graph enabled on unconstrained H100; rerun HTTP/OpenPI websocket on your target server before using the policy in a closed-loop robot.
ModeCommand ShapeSteady VRAM SnapshotNotes
Single GPU bf16--num-gpus 1, no offloadfit with 16383 MiB free before load in H100 pressure testRecommended first path for 16GB-class discrete GPUs.
ALOHA bf16 groupedPython API, batch size 1 / 2 / 4 / 8unconstrained H10052.4 ms single; 65.1 ms / 2; 91.9 ms / 4; 147.4 ms / 8.
ALOHA 16GB-free pressurePython API, no offloadsame 16GB-free pressureFit was validated after the bf16 correction; rerun latency on the target because older pressure latency was collected before the final OpenPI precision fix.
Global prefix cache disabledenable_global_prefix_cache=false or request runtime.prefix_cache=falseavoids cache growth across changing framesRecommended for tight edge budgets unless repeated exact frames are common.
Offload fallbackconfig in 6.2target-dependentUse only if the default bf16 path OOMs on the target device.
Use these knobs first because they do not change action numerics for a fixed input/noise:

​
6.1 16GB Edge Config

Use this single-GPU config first for 16GB-class robot workstations. It keeps parameters resident on GPU, keeps CUDA graph enabled, and disables global prefix cache growth.
File
{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_action_cuda_graph": true
}
Start a single-GPU server with the override:
Command
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --pipeline-config-path pi05_edge_16gb.json \
  --num-gpus 1 \
  --warmup-mode off \
  --host 127.0.0.1 \
  --port 30000
Treat the H100 pressure result as a memory-budget validation, not a Jetson latency guarantee. Jetson/Orin devices use shared system memory and have much lower effective memory bandwidth than H100, so measure closed-loop control latency on the target device before deciding the action chunk cadence.

​
6.2 Offload Fallback

Use offload only when the default bf16 config still does not fit on the target. These modes are numerically lossless for fixed inputs/noise, but they move weights between CPU and GPU and can hurt latency substantially. For a moderate fallback, offload cache growth and selected stage-resident modules first:
File
{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_action_cuda_graph": false,
  "offload_prefix_image_encoder_after_embed": true,
  "offload_prefix_token_embedding": true,
  "offload_prefix_language_layer_count_after_prefix": 2,
  "offload_action_expert_after_denoise": true,
  "empty_cache_after_prefix": true
}
If that still does not fit, full prefix layerwise CPU offload keeps every PaliGemma language layer on CPU and moves one layer at a time to GPU during prefix compute:
File
{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_action_cuda_graph": false,
  "offload_prefix_image_encoder": true,
  "offload_prefix_token_embedding": true,
  "offload_prefix_language_layers": true,
  "offload_prefix_language_layers_empty_cache": true,
  "empty_cache_after_prefix": true
}
Offload validation should be repeated on the target hardware after any dtype or loader change. Earlier fp32-runtime offload numbers are not comparable to the current bf16 path.

​
6.3 Per-Request Controls

For HTTP calls, disable cache or CUDA graph without restarting the server:
Example
payload = {
    "input": {
        "task": "pick up the block",
        "observation": {
            "images": images,
            "state": state,
        },
    },
    "runtime": {
        "prefix_cache": False,
        "cuda_graph": False,
    },
}
For OpenPI websocket clients, include equivalent enable_prefix_cache and enable_cuda_graph fields in each raw msgpack observation if you need per-request compatibility controls.

​
6.4 Deployment Choices

  • Keep batch size to one control stream unless grouped robot streams are explicitly validated. More concurrent observations increase activation and PrefixContext residency.
  • Keep materialize_dtype at the default bf16. fp32 is useful only for debugging numerical issues and will increase memory and latency.
  • Use split prefix/action only when you have multiple GPUs and have validated the robot control latency on that topology. On two GPUs with --sp-degree 2 --ulysses-degree 2, the action horizon can be sequence-sharded while the prefix root still computes and broadcasts PrefixContext. A single 16GB-class GPU should try the default bf16 config first.
  • Reducing num_inference_steps lowers latency but is not a pure memory fix and can change policy behavior. Validate closed-loop task success before using fewer than the checkpoint default.
  • CPU offload is a compatibility fallback, not the preferred 16GB path. Quantization and deeper prefix/vision sharding are the next steps for sub-16GB devices.

​
6.5 Loader And Run:ai Model Streamer

Run:ai Model Streamer can improve cold-start and checkpoint loading by reading safetensors concurrently and streaming tensors toward GPU memory. It is useful for local SSD, object storage, and cloud deployments where startup time is dominated by model file IO. The python[diffusion] extra includes runai_model_streamer. If the package is installed, SGLang enables it by default through SGLANG_USE_RUNAI_MODEL_STREAMER=true. Set it to false to force the plain safetensors loader:
Command
SGLANG_USE_RUNAI_MODEL_STREAMER=false \
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
Pi0.5 uses the direct SSD-to-GPU Run:ai path when all load targets for the current process are GPU-resident. This check is rank-local: distributed action ranks can stream their action-expert subset directly to GPU, while any rank with CPU/offloaded target tensors stays on the CPU safetensors fallback. The single-GPU validation streamed 13.5 GiB of safetensors directly to cuda:0 in about 1.5 s, then returned [50, 32] actions over both HTTP and OpenPI websocket. For mixed CPU offload and low-VRAM 16GB-class modes, Pi0.5 still uses the header-filtered safe loader for ranks whose target tensors are not fully CUDA-resident. Direct GPU streaming can increase the exact VRAM pressure the low-memory path is trying to avoid. When debugging distributed startup, compare with SGLANG_USE_RUNAI_MODEL_STREAMER=false to separate streamer behavior from model execution behavior. Run:ai Model Streamer does not reduce steady inference VRAM after parameters, caches, activations, CUDA contexts, and graph buffers are resident. For Pi0.5 low-VRAM work, prioritize component placement, prefix cache size, CUDA graph residency, and CPU/offload first. Model Streamer is a cold-start optimization after the steady-memory budget is correct.

​
7. OpenPI Comparison Benchmark

Use bench_pi05_openpi.py when you need a side-by-side latency and action-difference report against the OpenPI policy implementation. Start the SGLang Pi0.5 server first for HTTP or websocket modes, then run the benchmark from the SGLang repository root:
Command
python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \
  --profile aloha \
  --sglang-url http://127.0.0.1:30000 \
  --openpi-checkpoint gs://openpi-assets/checkpoints/pi05_base \
  --num-inference-steps 10 \
  --batch-size 4 \
  --repeats 20 \
  --warmup 3 \
  --deterministic-noise \
  --output pi05_openpi_compare.json
The benchmark reports:
  • SGLang HTTP latency for /v1/actions/generations, including stage timings when the server returns them.
  • SGLang msgpack HTTP latency when --sglang-api http_msgpack is set. This keeps the generic HTTP endpoint but avoids JSON image-array overhead. Use --sglang-http-response-format raw to benchmark the compact raw policy response.
  • SGLang OpenPI-compatible websocket latency when --sglang-api openpi_ws is set. This uses persistent msgpack websocket connections and is closer to the robot client path than JSON-over-HTTP.
  • SGLang Python in-process latency when --sglang-api python is set. This loads the native Pi0.5 pipeline in the benchmark process and avoids HTTP, websocket, scheduler, and serialization overhead. Use --sglang-python-batch-mode grouped to exercise the conservative native grouped-batch path. The Python path also reports actual SGLang module parameter dtype counts and example parameter names.
  • OpenPI single-request latency through Policy.infer.
  • Batch latency for grouped robot streams. SGLang uses concurrent HTTP requests in HTTP mode and persistent multi-connection msgpack calls in websocket mode. The Python backend can use true grouped model execution for fresh-prefix requests. OpenPI defaults to its internal direct model batch path because the public Policy.infer API is single-observation.
  • Action difference on the common output prefix. Use --deterministic-noise for strict debugging when the SGLang and OpenPI horizons match. The aloha profile supports this directly; the LIBERO profile compares the common prefix because OpenPI’s released LIBERO config uses a shorter output horizon than the LeRobot Pi0.5 checkpoint metadata.
For one-sided 16GB-class checks, run each backend separately under the same VRAM pressure. The SGLang Python path accepts the same pipeline config override as serving:
Command
python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \
  --profile aloha \
  --sglang-api python \
  --sglang-python-batch-mode grouped \
  --skip-openpi \
  --sglang-pipeline-config-path pi05_edge_16gb.json \
  --num-inference-steps 10 \
  --batch-size 4 \
  --repeats 5 \
  --warmup 2 \
  --deterministic-noise \
  --disable-prefix-cache \
  --disable-cuda-graph
Use --skip-sglang --openpi-pytorch-compile-mode none to measure an OpenPI eager baseline in the same environment. For a PyTorch-native OpenPI baseline, point --openpi-checkpoint at a checkpoint directory containing model.safetensors and the OpenPI assets/ norm-stat tree. The keep mode preserves OpenPI’s checkpoint default compile setting. The grouped Python path currently requires compatible fresh-prefix requests without split prefix/action workers or effective prefix-cache hits; other cases fall back to per-request execution.

​
8. Validation Notes

The following checks were run on H100 GPUs with the native SGLang Pi0.5 path:
CheckResult
lerobot/pi05_base direct end-to-endPrefix length 968, output shape [1, 50, 32], peak allocated memory 12.817 GiB.
LeRobot reference parityOne-step velocity max absolute difference 1.17e-6; final 10-step action max absolute difference 1.17e-7.
Action denoise CUDA graphEager 10-step denoise 125.4 ms; steady graph replay 50.8 ms; max output difference 0.
Exact full-prefix cacheFirst prefix pass about 203 ms; exact cache hit prefix stage about 0.2 ms.
lerobot/pi05_libero_base direct end-to-endImage keys image, image2, empty_camera_0; state dim 8; output action dim 7; output tensor shape [1, 50, 32].
Python grouped executionALOHA batch=4 grouped path measured 91.9 ms / 4 on current mixed precision: prefix 18.4 ms, action denoise 61.2 ms, preprocess about 2.4 ms per request. Sequential Python loop batch=4 measured 211.9 ms / 4.
sglang serve HTTP/v1/actions/generations returned action shape [50, 32]. JSON-over-HTTP remains compatible but image-array serialization dominates. Msgpack HTTP with prefix cache disabled measured 57.2 ms single and 221.9 ms / 4 for the envelope response, and 56.4 ms single and 219.7 ms / 4 for runtime.response_format="raw".
OpenPI websocket/openpi/policy returned action shape [50, 32]; with persistent msgpack connections and prefix cache disabled, ALOHA measured about 77.0 ms single and 162.0 ms / 4.
2-GPU prefix/action splitEarlier split validation returned action shape [50, 32] and matched single-GPU HTTP with max absolute difference 0. After the true action-SP change, rerun this check with --num-gpus 2 --sp-degree 2 --ulysses-degree 2 and verify both action ranks enter denoise kernels.
OpenPI/SGLang precisionOfficial OpenPI JAX inference restores the public GCS checkpoint as bf16 with selected fp32 stability compute and returns float32 actions. The converted OpenPI PyTorch pi05_aloha checkpoint keeps 119,720,608 fp32 stability params; SGLang reports the same fp32 set and 3,233,713,264 bf16 runtime params after skipping unused LM heads.
Native attention dtypeCheckpoint source tensors may be fp32, but SGLang finalizes PiGemma and SigLIP compute dtype before native attention backend selection; backend logs showed Using fa attention backend for the PiGemma path in the prior run.
16GB-free Python pressureWith an H100 artificially constrained to 16381 MiB free before model load, single-GPU bf16 no-offload Python grouped path completed without OOM. Re-run latency after precision or loader changes before using pressure numbers for deployment sizing.
Low-VRAM switchesDisabling prefix cache prevents cache growth across changing robot frames. CUDA graph can stay enabled when the action expert remains resident; disable it only for offload fallback modes.
Offload fallbackCPU/offload modes are retained as numerically lossless compatibility fallbacks, but earlier fp32-runtime offload latency numbers are stale after the bf16 dtype correction and should be revalidated before deployment decisions.
Run:ai direct loaderSingle-GPU serve streamed 13.5 GiB safetensors to cuda:0 in about 1.5 s and returned [50, 32] actions. Distributed direct streaming is now rank-local and should be revalidated on the target split topology; offload ranks with CPU targets still use the safe loader.
OpenPI comparison statusOfficial OpenPI GCS pi05_base is a JAX checkpoint; converted PyTorch eager was validated without torch.compile. On 80GB H100, ALOHA OpenPI PyTorch eager was about 125-130 ms single and about 164 ms / 4 in the direct-model batch path. Current SGLang Python grouped measured 52.4 ms single and 91.9 ms / 4; JAX OpenPI was 53.0 ms single and 59.5 ms / 2 in a short check.
Run a full HTTP or websocket smoke test in your robot deployment environment before using the policy in a closed-loop controller.