Installation

1. Option 1: For single node serving (typically when the model size fits into GPUs on one node) Execute command kubectl apply -f docker/k8s-sglang-service.yaml, to create k8s deployment and service, with llama-31-8b as example.
2. Option 2: For multi-node serving (usually when a large model requires more than one GPU node, such as DeepSeek-R1) Modify the LLM model path and arguments as necessary, then execute command kubectl apply -f docker/k8s-sglang-distributed-sts.yaml, to create two nodes k8s statefulset and serving service.

This method is recommended if you plan to serve it as a service. A better approach is to use the k8s-sglang-service.yaml.

1. Copy the compose.yml to your local machine
2. Execute the command docker compose up -d in your terminal.

1. Install SkyPilot and set up Kubernetes cluster or cloud access: see SkyPilot’s documentation.
2. Deploy on your own infra with a single command and get the HTTP API endpoint:

# sglang.yaml
envs:
  HF_TOKEN: null

resources:
  image_id: docker:lmsysorg/sglang:latest
  accelerators: A100
  ports: 30000

run: |
  conda deactivate
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 30000

# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml

# Get the HTTP API endpoint
sky status --endpoint 30000 sglang

3. To further scale up your deployment with autoscaling and failure recovery, check out the SkyServe + SGLang guide.

1. Build a docker container with sagemaker.Dockerfile alongside the serve script.
2. Push your container onto AWS ECR.

#!/bin/bash
AWS_ACCOUNT="<YOUR_AWS_ACCOUNT>"
AWS_REGION="<YOUR_AWS_REGION>"
REPOSITORY_NAME="<YOUR_REPOSITORY_NAME>"
IMAGE_TAG="<YOUR_IMAGE_TAG>"

ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}"

echo "Starting build and push process..."

# Login to ECR
echo "Logging into ECR..."
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}

# Build the image
echo "Building Docker image..."
docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile .

echo "Pushing ${IMAGE_URI}"
docker push ${IMAGE_URI}

echo "Build and push completed successfully!"

3. Deploy a model for serving on AWS Sagemaker, refer to deploy_and_serve_endpoint.py. For more information, check out sagemaker-python-sdk.
   1. By default, the model server on SageMaker will run with the following command: python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080. This is optimal for hosting your own model with SageMaker.
   2. To modify your model serving parameters, the serve script allows for all available options within python3 -m sglang.launch_server --help cli by specifying environment variables with prefix SM_SGLANG_.
   3. The serve script will automatically convert all environment variables with prefix SM_SGLANG_ from SM_SGLANG_INPUT_ARGUMENT into --input-argument to be parsed into python3 -m sglang.launch_server cli.
   4. For example, to run Qwen/Qwen3-0.6B with reasoning parser, simply add additional environment variables SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B and SM_SGLANG_REASONING_PARSER=qwen3.