Production Request Tracing

​

Setup Guide

1. Install the required packages and tools
   • install Docker and Docker Compose
   • install the dependencies
   Command

   # enter the SGLang root directory
   pip install -e "python[tracing]"
   
   # or manually install the dependencies using pip
   pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-grpc
   
2. Launch OpenTelemetry collector and Jaeger
   Command

   docker compose -f examples/monitoring/tracing_compose.yaml up -d
   
3. Start your SGLang server with tracing enabled
   Command

   # set env variables
   export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500
   export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64
   # start the prefill and decode server
   python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other option>
   # start the model-gate-way
   python -m sglang_router.launch_router --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other option>
   
   Replace 0.0.0.0:4317 with the actual endpoint of the OpenTelemetry collector. If you launched the openTelemetry collector with tracing_compose.yaml, the default receiving port is 4317. To use the HTTP/protobuf span exporter, set the following environment variable and point to an HTTP endpoint, for example, http://0.0.0.0:4318/v1/traces.
   Command

   export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
   
4. Raise some requests
5. Observe whether trace data is being exported
   • Access port 16686 of Jaeger using a web browser to visualize the request traces.
   • The OpenTelemetry Collector also exports trace data in JSON format to /tmp/otel_trace.json. In a follow-up patch, we will provide a tool to convert this data into a Perfetto-compatible format, enabling visualization of requests in the Perfetto UI.
6. Dynamically adjust trace level The trace level accepts configurable values from 0 to 3. The meanings of different trace level values are as follows:
   0: disable tracing
   1: Trace important slices
   2: Trace all slices except nested ones
   3: Trace all slices
   
   The trace level can be dynamically set via HTTP API, for example:
   Command

   curl http://0.0.0.0:30000/set_trace_level?level=2
   
   Replace 0.0.0.0:30000 with your actual server address, and replace level=2 with the level you want to set. Note: You must set the parameter --enable-trace; otherwise, the trace capability will not be enabled regardless of any dynamic adjustments to the trace level.

​

How to add Tracing for slices you’re interested in?(API introduction)

1. Initialization Every process involved in tracing during the initialization phase should execute:
   Example

   process_tracing_init(otlp_traces_endpoint, server_name)
   
   The otlp_traces_endpoint is obtained from the arguments, and you can set server_name freely, but it should remain consistent across all processes. Every thread involved in tracing during the initialization phase should execute:
   Example

   trace_set_thread_info("thread label", tp_rank, dp_rank)
   
   The “thread label” can be regarded as the name of the thread, used to distinguish different threads in the visualization view.
2. Create a trace context for a request Each request needs to call TraceReqContext() to initialize a request context, which is used to generate slice spans and record request stage info. You can either store it within the request object or maintain it as a global variable.
3. Mark the beginning and end of a request
   trace_ctx.trace_req_start().
   trace_ctx.trace_req_finish()
   
   trace_req_start() and trace_req_finish() must be called within the same process, for example, in the tokenizer.
4. Add tracing for a slice
   • Add slice tracing normally:
     Example

     trace_ctx.trace_slice_start(RequestStage.TOKENIZER.stage_name)
     trace_ctx.trace_slice_end(RequestStage.TOKENIZER.stage_name)
     
     or
     trace_ctx.trace_slice(slice: TraceSliceContext)
     
   • The end of the last slice in a thread must be marked with thread_finish_flag=True, or explicitly call trace_ctx.abort(); otherwise, the thread’s span will not be properly generated.
     Example

     trace_ctx.slice_end(RequestStage.D.stage_name, thread_finish_flag = True)
     trace_ctx.abort()
     
5. When the request execution flow transfers to another thread, the thread context needs to be explicitly rebuilt.
   • receiver: Execute the following code after receiving the request via ZMQ
     Example

     trace_ctx.rebuild_thread_context()
     

​

How to Extend the Tracing Framework to Support Complex Tracing Scenarios

TraceReqContext (req_id="req-123")
├── TraceThreadContext(thread_label="scheduler", tp_rank=0)
|     └── TraceSliceContext(slice_name="prefill")
|
└── TraceThreadContext(thread_label="scheduler", tp_rank=1)
      └── TraceSliceContext(slice_name="prefill")