Skip to content

[Bug]: Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" during running performance(online, offline), sglang 0.5.15.post1, 0.5.18 #497

Open
Open
[Bug]: Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" during running performance(online, offline), sglang 0.5.15.post1, 0.5.18#497
Labels
status: needs-triageNew issue, awaiting reviewtype: bugSomething isn't working

Description

@maria-18-git
maria-18-git
opened on Sep 8, 2026

Bug Description

Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" appears when running a performance experiments for sglang 0.5.15.post1, sglang 0.5.18.

Steps to Reproduce

  1. Run a server
docker run --gpus all --shm-size 32g -p 30000:30000 --ipc=host   -v "$HF_CACHE:/root/.cache/huggingface"   -e "HF_TOKEN=$HF_TOKEN"   lmsysorg/sglang:v0.5.18   python3 -m sglang.launch_server   --model-path deepseek-ai/DeepSeek-R1   --trust-remote-code   --tp 8   --context-length 32768   --mem-fraction-static 0.65   --disable-radix-cache   --max-running-requests 2048   --cuda-graph-max-bs-decode 2048   --cuda-graph-bs 128 256 512 1024 2048   --chunked-prefill-size 65536   --max-prefill-tokens 65536   --attention-backend flashinfer   --stream-interval 10   --decode-log-interval 1   --host 0.0.0.0   --port 30000   2>&1 | tee -a "$RUN_DIR/run.log"
  1. Run a performance experiment
uv run inference-endpoint benchmark from-config --config examples/07_DeepSeekR1_Example/performance_h200_online_max_concurrency_test.yaml
...
2026-09-03 08:56:52,162 - inference_endpoint.endpoint_client.http_client - INFO - EndpointClient initialized with num_workers=4, endpoints=['http://localhost:30000/generate'], adapter=SGLangGen
erateAdapter, accumulator=SGLangSSEAccumulator, transport=zmq
2026-09-03 08:56:52,162 - inference_endpoint.load_generator.session - INFO - Starting phase: warmup (warmup)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W2/3618620] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
...

Solution:

diff --git a/src/inference_endpoint/endpoint_client/adapter_protocol.py b/src/inference_endpoint/endpoint_client/adapter_protocol.py
index a96faaa..1f58c59 100644
--- a/src/inference_endpoint/endpoint_client/adapter_protocol.py
+++ b/src/inference_endpoint/endpoint_client/adapter_protocol.py
@@ -127,12 +127,11 @@ class HttpRequestAdapter(ABC):
         """
         json_docs = cls.SSE_DATA_PATTERN.findall(buffer[:end_pos])
         parsed: list[Any] = []
-        # Note: if one frame is malformed, remaining frames are skipped
-        try:
-            for json_doc in json_docs:
+        for json_doc in json_docs:
+            try:
                 content = cls.decode_sse_message(json_doc)
                 if content is not None:
                     parsed.append(content)
-        except (msgspec.DecodeError, msgspec.ValidationError) as exc:
-            logger.warning("skipping malformed SSE batch (%s)", type(exc).__name__)
+            except (msgspec.DecodeError, msgspec.ValidationError) as exc:
+                logger.warning("skipping malformed SSE frame (%s)", type(exc).__name__)
         return parsed
diff --git a/src/inference_endpoint/sglang/adapter.py b/src/inference_endpoint/sglang/adapter.py
index a6614fa..c19e3d2 100644
--- a/src/inference_endpoint/sglang/adapter.py
+++ b/src/inference_endpoint/sglang/adapter.py
@@ -122,7 +122,7 @@ class SGLangGenerateAdapter(HttpRequestAdapter):
         # the output_tokens is the delta from the previous chunk
         token_delta = resp.output_ids
 
-        has_retractions = resp.meta_info.total_retractions > 0
+        has_retractions = resp.meta_info.num_retractions > 0
         return SGLangSSEDelta(
             text=total_text,
             token_delta=token_delta,
diff --git a/src/inference_endpoint/sglang/types.py b/src/inference_endpoint/sglang/types.py
index 9e75b3a..89f71ee 100644
--- a/src/inference_endpoint/sglang/types.py
+++ b/src/inference_endpoint/sglang/types.py
@@ -52,13 +52,13 @@ class SGLangGenerateRequest(
 
 class MetaInfo(msgspec.Struct, frozen=True, kw_only=True, omit_defaults=True, gc=False):  # type: ignore[call-arg]
     id: str
-    finish_reason: dict[str, Any]
+    finish_reason: dict[str, Any] | None
     prompt_tokens: int
     weight_version: str
-    total_retractions: int
+    num_retractions: int
     completion_tokens: int
     cached_tokens: int
-    e2e_latency: float
+    e2e_latency: float | None = None
 
 
 class SGLangGenerateResponse(
diff --git a/tests/performance/sglang/test_sglang_adapter.py b/tests/performance/sglang/test_sglang_adapter.py
index f614cc0..3e0fa90 100644
--- a/tests/performance/sglang/test_sglang_adapter.py
+++ b/tests/performance/sglang/test_sglang_adapter.py
@@ -64,7 +64,7 @@ def make_response_bytes(text: str, n_tokens: int) -> bytes:
                 "finish_reason": {"type": "stop"},
                 "prompt_tokens": 10,
                 "weight_version": "v1",
-                "total_retractions": 0,
+                "num_retractions": 0,
                 "completion_tokens": n_tokens,
                 "cached_tokens": 0,
                 "e2e_latency": 0.1,
diff --git a/tests/unit/endpoint_client/test_adapter_protocol.py b/tests/unit/endpoint_client/test_adapter_protocol.py
index 591dc42..3a177a6 100644
--- a/tests/unit/endpoint_client/test_adapter_protocol.py
+++ b/tests/unit/endpoint_client/test_adapter_protocol.py
@@ -49,7 +49,7 @@ class _SimpleAdapter(HttpRequestAdapter):
 @pytest.mark.unit
 def test_parse_sse_chunk_skips_bad_frame_and_keeps_valid():
     """A malformed SSE frame is skipped; surrounding valid frames are preserved."""
-    buffer = b'data: {"ok":1}\n\ndata: BAD\n\ndata: {"ok":1}\n\n'
+    buffer = b'data: {"ok":1}\n\ndata: {"bad":}\n\ndata: {"ok":1}\n\n'
     end_pos = len(buffer)
     result = _SimpleAdapter.parse_sse_chunk(buffer, end_pos)
     assert result == [{"x": 1}, {"x": 1}]

Environment

git log -1
commit 73981a84fbb8c2dbd761686b3d002ca7f434fe72 (HEAD -> main, origin/main, origin/HEAD)

Relevant Logs

Short run:

uv run inference-endpoint benchmark from-config \
> --config examples/07_DeepSeekR1_Example/performance_h200_online_max_concurrency_test.yaml
2026-09-03 08:56:39,955 - inference_endpoint.endpoint_client.cpu_affinity - INFO - CPU affinity: 128 online CPUs available to process
2026-09-03 08:56:39,960 - inference_endpoint.endpoint_client.cpu_affinity - INFO - CPU affinity: 64 physical cores across 2 NUMA nodes, requesting 5 for loadgen, 4 workers
2026-09-03 08:56:39,988 - inference_endpoint.endpoint_client.cpu_affinity - INFO - LoadGen pinned to 10 CPUs (5 physical cores)
2026-09-03 08:56:40,169 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/models/deepseek-ai/DeepSeek-R1 "HTTP/1.1 200 OK"
2026-09-03 08:56:40,170 - inference_endpoint.commands.benchmark.execute - INFO - Tokenizer available for model: deepseek-ai/DeepSeek-R1
2026-09-03 08:56:40,170 - inference_endpoint.commands.benchmark.execute - INFO - Streaming: enabled (on)
2026-09-03 08:56:40,170 - inference_endpoint.commands.benchmark.execute - INFO - No separate accuracy datasets provided
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - INFO - Loaded 4388 samples
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - WARNING - Warmup is enabled without salt (--warmup-salt): warmup prompts are issued verbatim, so the server may serve t
he measured phase from a warm KV/prefix cache and understate latency. Enable --warmup-salt on text-'prompt' datasets to bust the cache.
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - INFO - Mode: TestMode.PERF, Target QPS: None, Responses: False
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - INFO - Expected samples: 12
deepseek-ai/DeepSeek-R1 (Streaming: True):   0%|                                                                                                                          | 0/12 [00:00<?, ?it/s]
2026-09-03 08:56:40,910 - inference_endpoint.async_utils.transport.zmq.pubsub - INFO - Publisher bound to ipc:///dev/shm/zmq_cjpkdbk5/ev_pub_abfd9bee
2026-09-03 08:56:40,911 - inference_endpoint.async_utils.transport.zmq.pubsub - INFO - Subscriber connected to ipc:///dev/shm/zmq_cjpkdbk5/metrics_pub_b8200cce
2026-09-03 08:56:40,911 - inference_endpoint.async_utils.services.launcher - INFO - Launching service: inference_endpoint.async_utils.services.metrics_aggregator (id=0)
2026-09-03 08:56:40,911 - inference_endpoint.async_utils.services.launcher - INFO - Launching service: inference_endpoint.async_utils.services.event_logger (id=1)
2026-09-03 08:56:41,342 - inference_endpoint.async_utils.transport.zmq.pubsub - INFO - Subscriber connected to ipc:///dev/shm/zmq_cjpkdbk5/ev_pub_abfd9bee
2026-09-03 08:56:42,767 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/deepseek-ai/DeepSeek-R1/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
...
2026-09-03 08:56:52,162 - inference_endpoint.endpoint_client.http_client - INFO - EndpointClient initialized with num_workers=4, endpoints=['http://localhost:30000/generate'], adapter=SGLangGen
erateAdapter, accumulator=SGLangSSEAccumulator, transport=zmq
2026-09-03 08:56:52,162 - inference_endpoint.load_generator.session - INFO - Starting phase: warmup (warmup)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W2/3618620] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W0/3618618] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W1/3618619] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
...
2026-09-03 09:02:25,578 - inference_endpoint.endpoint_client.adapter_protocol[W3/3618621] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
2026-09-03 09:02:25,669 - inference_endpoint.endpoint_client.adapter_protocol[W3/3618621] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
deepseek-ai/DeepSeek-R1 (Streaming: True): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [05:44<00:00, 28.73s/it]2026-09-03 09:02:25,670 - inference_endpoint.load_generator.session - INFO - Phase performance complete: 12 samples issued
2026-09-03 09:02:25,670 - inference_endpoint.commands.benchmark.pipeline - INFO - Closing publisher (buffer=0, pending=0)...
2026-09-03 09:02:25,670 - inference_endpoint.commands.benchmark.pipeline - INFO - Waiting for services to finish processing...
2026-09-03 09:02:25,671 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - ENDED event received, shutting down aggregator
2026-09-03 09:02:25,671 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - Draining 0 pending tokenizations...
2026-09-03 09:02:25,671 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - Tokenizations fully drained (n_pending_tasks=0)
2026-09-03 09:02:25,675 - inference_endpoint.async_utils.services.metrics_aggregator.registry - INFO - sample_latency_ns early-stopping detail (confidence 0.99): p99.9: estimate=None empirical=96469560463 n=12 min_queries=6636 discarded=0; p99.0: estimate=None empirical=96469560463 n=12 min_queries=662 discarded=0; p97.0: estimate=None empirical=96469560463 n=12 min_queries=219 discarded=0; p95.0: estimate=None empirical=96469560463 n=12 min_queries=130 discarded=0; p90.0: estimate=None empirical=82004097015 n=12 min_queries=64 discarded=0; p80.0: estimate=None empirical=72922931653 n=12 min_queries=31 discarded=0; p75.0: estimate=None empirical=72922931653 n=12 min_queries=24 discarded=0; p50.0: estimate=164572395605 empirical=48464537381 n=12 min_queries=11 discarded=0
2026-09-03 09:02:25,676 - inference_endpoint.async_utils.services.metrics_aggregator.registry - INFO - ttft_ns early-stopping detail (confidence 0.99): p99.9: estimate=None empirical=None n=0 min_queries=6636 discarded=0; p99.0: estimate=None empirical=None n=0 min_queries=662 discarded=0; p97.0: estimate=None empirical=None n=0 min_queries=219 discarded=0; p95.0: estimate=None empirical=None n=0 min_queries=130 discarded=0; p90.0: estimate=None empirical=None n=0 min_queries=64 discarded=0; p80.0: estimate=None empirical=None n=0 min_queries=31 discarded=0; p75.0: estimate=None empirical=None n=0 min_queries=24 discarded=0; p50.0: estimate=None empirical=None n=0 min_queries=11 discarded=0
2026-09-03 09:02:25,676 - inference_endpoint.async_utils.services.metrics_aggregator.registry - INFO - tpot_ns early-stopping detail (confidence 0.99): p99.9: estimate=None empirical=None n=0 min_queries=6636 discarded=0; p99.0: estimate=None empirical=None n=0 min_queries=662 discarded=0; p97.0: estimate=None empirical=None n=0 min_queries=219 discarded=0; p95.0: estimate=None empirical=None n=0 min_queries=130 discarded=0; p90.0: estimate=None empirical=None n=0 min_queries=64 discarded=0; p80.0: estimate=None empirical=None n=0 min_queries=31 discarded=0; p75.0: estimate=None empirical=None n=0 min_queries=24 discarded=0; p50.0: estimate=None empirical=None n=0 min_queries=11 discarded=0
2026-09-03 09:02:25,679 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - Aggregator finalized: 36 total records processed
2026-09-03 09:02:25,804 - inference_endpoint.commands.benchmark.pipeline - INFO - Built report from final_snapshot.json
deepseek-ai/DeepSeek-R1 (Streaming: True): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [05:44<00:00, 28.74s/it]
2026-09-03 09:02:25,815 - inference_endpoint.endpoint_client.http_client - INFO - [e54f55c3] Shutting down...
2026-09-03 09:02:26,316 - inference_endpoint.endpoint_client.http_client - INFO - [e54f55c3] Shutdown complete.
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - ----------------- Summary -----------------
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Version: 0.1.0
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Git SHA: 73981a8
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Run config:
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   load_pattern: type=concurrency, target_qps=None, target_concurrency=4
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   warmup: enabled=True, n_requests=8, salt=False, drain=True, warmup_random_seed=42
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   scheduler_random_seed: 42
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   dataloader_random_seed: 42
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples issued: 12
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples succeeded: 12
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples failed: 0
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples dropped: 0
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Duration: 261.04 seconds
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - QPS: 0.05
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - TPS: N/A
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - ----------------- End of Summary -----------------
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Report written to results/deepseek_r1_h200/online_max_concurrency_4_20260903_sglang_0_5_18/report.txt
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Completed in 261.0s
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Results: 12/12 successful
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Estimated QPS: 0.0
2026-09-03 09:02:26,324 - inference_endpoint.commands.benchmark.execute - INFO - Partial results saved to results/deepseek_r1_h200/online_max_concurrency_4_20260903_sglang_0_5_18

Before submitting

  • I searched existing issues and found no duplicates

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: needs-triageNew issue, awaiting reviewtype: bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions