Sampling is deterministic (does not re-randomize per request) on fallback-loaded models

[lmargit]

Models loaded through OpenArc's fallback path for architectures the installed transformers/optimum-intel version doesn't fully recognize (logged as You are using a model of type to instantiate a model of type . This is not supported for all configurations of models and can yield errors.) show three related generation-quality problems that do not appear on a natively-supported architecture. Tested across three models on the same host/image/config to isolate the pattern.

Environment

• Image: ghcr.io/searchsavior/openarc-battlemage:latest (digest sha256:3036d10c7bb27705eb5f801c3967bba2969a9c126e3b2620b499c58a679aa414, built 2026-09-05)
• OpenARC Version: v2.0.5-dirty, Git ref fb39ea7942459580fa0af47cdaa624993e307bd4
• openvino 2026.5.0.dev20260903, openvino-genai 2026.5.0.0.dev20260903, openvino-tokenizers 2026.5.0.0.dev20260903
• optimum 2.3.0, optimum-intel 2.2.0.dev0+4632054
• Hardware: Intel Arc Pro B70 (Battlemage), engine ovgenai, device GPU
• Host: TrueNAS Scale, docker compose deployment

Models tested

Model | Provenance | Architecture warning at load? | Loaded as -- | -- | -- | -- gemma4-31b | Locally converted via optimum-cli export openvino --model google/gemma-4-31b-it --task image-text-to-text --weight-format int4 --group-size 128 --ratio 1.0 --trust-remote-code | Yes — type gemma4 | vlm qwen3.8-27b-int4 | Pre-converted, downloaded verbatim from OpenVINO/Qwen3.8-27B-int4-ov on HuggingFace (no local conversion) | Yes — type qwen3_5 | vlm qwen3-vl-8b-int4 | OpenVINO-provided, natively supported | No | vlm

Note the two affected models have different provenance — one a local optimum-cli export, the other an official pre-converted release downloaded as-is — yet both trigger the identical fallback warning and show the identical determinism bug. This makes a local-conversion mistake an unlikely explanation for at least the qwen3.8-27b-int4 case, pointing more toward a genuine OpenArc/optimum-intel architecture-support gap for both gemma4 and qwen3_5 types.

---

Bug 1 — Sampling is deterministic (does not re-randomize per request) on fallback-loaded models

Repro:

bash

for i in 1 2 3 4 5; do
  curl -s http://<host>:<port>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"<model>","messages":[{"role":"user","content":"Please, provide the history of the James Webb Telescope, the new functionalities and advances it represents versus the previous telescopes."}],"stream":false,"temperature":0.7}' \
    | jq -r '.choices[0].message.content' > /tmp/run_$i.txt
done
md5sum /tmp/run_*.txt

Expected: with do_sample: true, temperature: 0.7, top_p: 0.95 (from generation_config.json), and no seed supplied, five independent requests should produce five different completions.

Actual:

• gemma4-31b: (natively supported), reproduced across a full model unload/reload cycle.
• qwen3.8-27b-int4: 5/5 identical (same md5), independently confirmed.
• qwen3-vl-8b-int4 (natively supported, no fallback warning): 5/5 different md5s — normal sampling variation, as expected.

Passing an explicit seed value restores variation on gemma4-31b, but the same seed value does not reproduce identical output across repeated calls either — i.e. seed is accepted but does not appear to be wired into deterministic reproduction. This suggests the RNG state used for sampling is fixed/not re-initialized per request specifically on the fallback-loaded code path, rather than a temperature/top_p handling bug.

Attempted native-only isolation: confirmed via direct requests to OpenArc's port (bypassing a custom reverse proxy sitting in front of it), ruling out anything proxy-side as the cause.

---

Bug 2 — Reasoning/content field separation breaks down on fallback-loaded reasoning model, even with a large token budget

Model: qwen3.8-27b-int4 (fallback-loaded, type qwen3_5)

Repro:

bash

curl -s http://<host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.8-27b-int4","messages":[{"role":"user","content":"Please, provide the history of the James Webb Telescope, the new functionalities and advances it represents versus the previous telescopes."}],"stream":false,"temperature":0.7,"max_tokens":4000}' \
  | jq '{reasoning: .choices[0].message.reasoning_content, content: .choices[0].message.content}'

Expected: internal reasoning/thinking content routed to reasoning_content; final answer routed to content (this works correctly for this same model on a trivial prompt, e.g. "Hello" — reasoning_content and content are populated separately and correctly).

Actual: for this longer/more complex prompt, reasoning_content is null and the entire internal reasoning/planning trace (visibly in a "we need to..." planning voice, second-guessing facts, drafting an outline) is dumped into content instead — and even at max_tokens: 4000, generation is still deep in unstructured planning with no final answer ever produced, i.e. it isn't merely a token-budget issue at moderate limits (already confirmed not to be one at 1200 tokens, then re-confirmed by more than tripling the budget).

Control: the natively-supported qwen3-vl-8b-int4 produces a clean, complete, well-terminated answer with reasoning_content: null and a fully-formed content on the identical prompt at the same max_tokens: 4000 — though this model may not implement a reasoning/thinking mode at all, so this is not a fully like-for-like comparison for this specific bug; noted for completeness rather than as definitive negative evidence.

---

Bug 3 — Intermittent stray token substitution in markdown table divider rows

Model: gemma4-31b (fallback-loaded, type gemma4)

Repro: ask for a response containing a 3-column markdown comparison table; inspect the header divider row.

Actual: across 3 independent generations (2 at temperature: 0.7 with real sampling variation, one from an unrelated frozen-RNG run), the third column's divider cell renders as own instead of --- in 2 of the 3:

| Feature | Hubble Space Telescope | James Webb Space Telescope |
| :--- | :--- | own |

| Feature | Hubble Space Telescope | James Webb Space Telescope |
| :--- | :--- | : own |

• Verified own is an ordinary, unremarkable vocabulary token (id 931) via direct inspection of tokenizer.json — not a corrupted or control-token-adjacent entry, so this doesn't look like a tokenizer/vocab mismatch.
• The model's chat_template.jinja (Gemma4's channel/turn/tool-call structured template) was inspected and appears well-formed; no evidence the leaked token is a mishandled control-token fragment.
• Control: the natively-supported qwen3-vl-8b-int4, asked to produce comparable tables (once via explicit table request, once inside a long free-form answer), produced clean, correct --- divider rows both times, no anomaly.

Given Bug 1, we can't fully rule out that this is a downstream symptom of whatever mechanism causes the frozen sampling (e.g., a bad early logits pass being replayed) rather than an independent defect — flagging both possibilities for the maintainers to assess given visibility into the relevant code paths.

---

Suggested framing

All three symptoms appear only on models routed through the "unsupported model type" fallback path (gemma4, qwen3_5), and none appear on a natively-supported architecture (qwen3-vl) tested identically on the same host/build. This suggests the fallback instantiation path is missing some initialization step — plausibly around per-request generation/RNG state setup — that the natively-supported path performs correctly. Happy to provide full raw request/response logs, the exact docker-compose.yaml, or run further isolation tests on request.

[SearchSavior]

@lmargit Hey thanks for the issue.

What exactly is a fallback model? Also, can you condense this down for me? The agents troubleshooting steps are making it hard to reason about what the issue is, especially since there are errors. gemma4 and qwen35 are supported in openvino genai

[lmargit]

Hey @SearchSavior,
Thanks for the reply. Apologies for the lack of clarity. I named the fallback models the 2 models that returned the warning below when loading using openarc (from the docker compose logs of openarc):

"You are using a model of type gemma4 to instantiate a model of type . This is not supported for all configurations of models and can yield errors."
This same warning appears in Qwen 27B-INT4.

Qwen3-vl-8b-int4 did not produce this warning.

Since the small Qwen3 model did not produce the warning and works fine no matter the context size my initial hypothesis was that it may be related to an issue with my conversion of Gemma4. So, I downloaded the OpenVino Qwen3.8 27B-int4 from HuggingFace to rule out the conversion. This model returned the same fallback warning in the openarc logs. Also, same issue with the deterministic responses. I tested with different temperatures (from 0 to 0.95):

With temperature equal to zero the docker crashed completely and needed to rebuilt it.
The deterministic responses occurs no matter the temperature level.

So, probably it is an issue with the model conversions of (mine and coincidentally with the published in HF).

Here the steps I used to convert Gemma4:

optimum-cli export openvino
--model google/gemma-4-31b-it
--task image-text-to-text
--weight-format int4
--group-size 128
--ratio 1.0
--trust-remote-code
~/Downloads/gemma4-31b-int4-ov

Using these dependencies:
pip install transformers==5.0.0
pip install "optimum-intel @ git+https://github.com/huggingface/optimum-intel.git" (2.2.0.dev0+4632054)
pip install openvino==2026.5.0.dev20260903
pip install openvino-tokenizers
pip install openvino-genai
pip install huggingface_hub
pip install accelerate

I have not tested different conversion/optimization alternatives:
For example, later I noticed Transformers in OpenVino 2026.3 was 5.5 and for 2026.5 is 5.0.

I hope this helps. I will continue testing, but slow during the weekdays.
Regards

[SearchSavior]

I am pretty sure this warning is more like a disclaimer about the limited implementation of gemma4- i don't know if 12b, e4b, e2b cover audio, or if 31b, 26ba4b are covering video and audio in all cases. So I don't think it's an export issue.

However I have found some other gaps in the gemma4 implementation in the paged attention scheduler, so maybe there are gaps elsewhere? Also be sure to test the openarc side behavior of do_sample parameter. Might be worth having a look into the ovgenai source, specially if around sampler implementation in python vs c++ apis. C++ is more mature and causes me tons of pain in some narrow cases.

Yeah I have seen the temp 0 issue and don't understand, haven't investigated yet. Try adding -vvvv to the openarc serve command to see verbose logs.

Let me know how you make out @lmargit

[lmargit]

Hi Emerson @SearchSavior, I was able to get some time.
I started the testing from scratch, using 2 Qwen models (the small qwen3-vl-8b-int4 and the medium size qwen3.6-35b-a3b-int4).
The idea was to prevent noise from Gemma4 because I converted it myself.

I explored further the issues about the temperature set to zero (Temp=0) and the apparent deterministic results:

1. I found that when openvino_genai correctly rejects an invalid GenerationConfig (in this case do_sample=True with temperature=0):

• OpenArc's failure handler (worker_registry.py, _commit_completed_packet) does not handle this gracefully.
• It treats it as a fatal device/runtime failure, triggering an unconditional model unload.
• That unload clears the model registry but does not release the underlying VLMPipeline's GPU allocation, leaving a "ghost" model resident in VRAM that no longer appears in /openarc/status and cannot be reclaimed via /openarc/unload (returns 404: Model not found).
• I think OpenArc should catch openvino_genai's specific validation-class exceptions before triggering the failure path, so that a client sending an invalid but harmless parameter (like temperature=0) receives a clean error without any model-lifecycle side effect.

2. The apparent deterministic sampling (identical-output symptom) is not an actual issue:

• I checked https://github.com/openvinotoolkit/openvino.genai/blob/master/src/cpp/include/openvino/genai/generation_config.hpp and found "size_t rng_seed = 0;"
• So, I think the deterministic output I was getting was just the result of the openvino_genai's non-auto-randomized rng_seed convention.
• I assume OpenArc has a guard in vlm.py/llm.py to prevent identical seeds. So, that's why I was getting identical outputs when bypassing OpenArc during my tests.
• In case OpenArc does not have a guard mechanism then it would be great to assign a random rng_seed per request when the client (app or user) doesn't supply one, in both vlm.py and llm.py to prevent the behavior I saw.

I will continue testing OpenArc in my Truenas Server.
Best,
Luis

1. Confirm baseline state before triggering anything
   curl -s http://127.0.0.1:18000/status | jq '.vram'
   curl -s http://192.168.1.217:11435/openarc/status | jq

2. Load the model fresh
   curl -X POST http://127.0.0.1:18000/proxy/load
   -H "Content-Type: application/json"
   -d '{"model_name":"qwen3.6-35b-a3b-int4"}'

3. Confirm it's actually loaded, note the VRAM figure
   curl -s http://192.168.1.217:11435/openarc/status | jq
   curl -s http://127.0.0.1:18000/status | jq '.vram'

4. Send the request that triggers the temperature=0 validation error
   curl -s -X POST http://127.0.0.1:18000/v1/chat/completions
   -H "Content-Type: application/json"
   -d '{"model":"qwen3.6-35b-a3b-int4","messages":[{"role":"user","content":"Say hello."}],"temperature":0,"stream":false}' | jq

5. Check registry vs. VRAM immediately after — this is the demonstration
   curl -s http://192.168.1.217:11435/openarc/status | jq
   curl -s http://127.0.0.1:18000/status | jq '.vram'

6. Confirm there's no way to reclaim it via the normal API
   curl -s -X POST http://192.168.1.217:11435/openarc/unload
   -H "Content-Type: application/json"
   -d '{"model_name":"qwen3.6-35b-a3b-int4"}'

The OpenArc code that sends the unload:
File: /app/src/server/worker_registry.py, function _commit_completed_packet (lines 93–110):

def _commit_completed_packet(
packet: WorkerPacket,
completed: WorkerPacket,
model_name: str,
registry: ModelRegistry,
) -> bool:
"""Complete the request future. Return True if the worker should exit."""
if completed.error is not None:
logger.error(
f"[{model_name}] Inference failed, triggering model unload..."
)
if packet.result_future is not None and not packet.result_future.done():
packet.result_future.set_exception(completed.error)
asyncio.create_task(registry.register_unload(model_name))
return True
...