LFM2.5

​

Deployment

pip install --upgrade pip
pip install uv
uv pip install sglang

docker pull lmsysorg/sglang:dev-cu13

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    --ipc=host \
    lmsysorg/sglang:dev-cu13 \
    sglang serve <use args below>

​

Playground

​

1. Model Introduction

• Hybrid LIV-conv + GQA architecture: the 1.2B / 350M dense models are 16 layers (10 conv + 6 GQA); the 8B-A1B MoE is 24 layers (18 conv + 6 GQA).
• Pythonic tool calling: function calls are emitted as a Python list between <|tool_call_start|> and <|tool_call_end|> tokens. The lfm2 tool-call parser surfaces these as standard message.tool_calls.
• Reasoning variants: the 8B-A1B and 1.2B-Thinking checkpoints emit an explicit <think>...</think> chain-of-thought before the answer.
• Multilingual: up to 10 languages, with dedicated Japanese chat checkpoints.
• Vision: LFM2.5-VL-1.6B pairs the 1.2B language backbone with a SigLIP2 NaFlex 400M encoder for OCR, document understanding, and multilingual vision; LFM2.5-VL-450M pairs the 350M backbone with a SigLIP2 86M encoder for captioning and object detection at edge sizes.

​

2. Configuration Tips

• Reasoning parser: LFM2.5 reasoning models wrap their chain-of-thought in <think>...</think> tags. The command generator passes --reasoning-parser qwen3 for 8B-A1B (it emits an explicit opening <think>) and --reasoning-parser qwen3-thinking for 1.2B-Thinking (always-on reasoning). This splits the thinking process into reasoning_content; without it the chain-of-thought stays inline in content.
• Tool calling: --tool-call-parser lfm2 surfaces LFM2.5’s Pythonic <|tool_call_start|>[...]<|tool_call_end|> calls as standard message.tool_calls. The original 1.2B-JP does not expose tool calling; Base has no chat template (use completions).
• Attention backend on Blackwell (B200/sm100): SGLang defaults to the trtllm_mha backend on sm100, which is fastest for the dense text models. The 8B-A1B uses a mamba-style state cache that runs on a page-size-1 backend, so the generator picks --attention-backend flashinfer for it. The VL language model also uses that state cache and offers two backends: --attention-backend flashinfer (keeps prefix/radix caching — what the generator emits), or --attention-backend trtllm_mha --disable-radix-cache to run the language model on Blackwell trtllm_mha attention (--disable-radix-cache lifts the page-size-1 requirement, at the cost of prefix caching). Pair either with --mm-attention-backend fa4 for the vision tower.
• VL vision tower (--mm-attention-backend): on sm100 the trtllm_mha default is fastest for text but applies causal attention to image tokens. For the VL model, pass --mm-attention-backend fa4 on B200/B300 (or fa3 on H100/H200) to restore bidirectional image-token attention and full vision quality.
• VL multimodal feature transport: the generator launches the VL models with SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected.
• VL-450M memory headroom (--mem-fraction-static 0.8): with the default memory fraction, the 450M’s small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps --mem-fraction-static 0.8 for VL-450M; the pool is still far larger than this model ever needs.
• Mamba scheduling: LFM2.5 runs on the default no_buffer mamba scheduler strategy — no --mamba-scheduler-strategy flag is needed. The extra_buffer strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use mamba_chunk_size=1.
• Hardware requirements: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant.

​

3. Advanced Usage

​

3.1 Basic Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

# Non-OpenAI fields (top_k / min_p / repetition_penalty) ride in extra_body.
SAMPLING = {
    "LiquidAI/LFM2.5-8B-A1B":         dict(temperature=0.2,  extra_body={"top_k": 80, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-1.2B-Instruct":  dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-1.2B-Thinking":  dict(temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-350M":           dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-1.2B-JP-202606": dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-VL-1.6B":        dict(temperature=0.1,  extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-VL-450M":        dict(temperature=0.1,  extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
}

def chat(model, messages, **overrides):
    cfg = SAMPLING[model]
    body = cfg["extra_body"] | overrides.pop("extra_body", {})
    return client.chat.completions.create(
        model=model, messages=messages,
        temperature=cfg["temperature"], extra_body=body, **overrides,
    )

resp = chat(
    "LiquidAI/LFM2.5-1.2B-Instruct",
    [{"role": "user", "content": "What is C. elegans? Answer in one sentence."}],
)
print(resp.choices[0].message.content)

​

3.2 Reasoning

resp = chat(
    "LiquidAI/LFM2.5-8B-A1B",
    [{"role": "user", "content": "If a train travels 60 km/h for 2.5 hours, how far does it go?"}],
)
msg = resp.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Answer:", msg.content)

​

3.3 Tool Calling

resp = chat(
    "LiquidAI/LFM2.5-1.2B-Instruct",
    [{"role": "user", "content": "What's the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            },
        },
    }],
)
for call in resp.choices[0].message.tool_calls or []:
    print(call.function.name, call.function.arguments)

​

3.4 Vision Input

resp = chat(
    "LiquidAI/LFM2.5-VL-1.6B",
    [{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {
                "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
            {"type": "text", "text": "What is in this image?"},
        ],
    }],
)
print(resp.choices[0].message.content)

​

3.5 Base Checkpoint

comp = client.completions.create(
    model="LiquidAI/LFM2.5-1.2B-Base",
    prompt="The capital of France is",
    temperature=0.3,
    extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(comp.choices[0].text)