Hardening gemma4

README.md

@@ -62,6 +62,12 @@ train_dataset = prepare_data(
 )
 ```
 
+With a live Gemma 4 tokenizer, omit `chat_template_kwargs` to let Teich choose
+thinking or non-thinking independently for each row. See [Training](docs/training.md#live-gemma-4-models)
+for auto-mode rules and the E4B, 26B-A4B, and 31B template contract.
+Qwen 3.8 keeps its own template-native defaults, including historical reasoning
+preservation and `reasoning_effort`; see [Live Qwen 3.8 Models](docs/training.md#live-qwen-38-models).
+
 Then create your trainer and call `mask_data()`:
 
 ```python

docs/cli.md

@@ -98,7 +98,7 @@ teich anonymize output --output output_anonymized
 teich anonymize data --in-place
 ```
 
-Anonymization replaces known credential formats, high-confidence secret assignments, personal email addresses, and home-directory usernames with deterministic dummy values while preserving embedded base64 media payloads. Reserved example-domain addresses, known public bot addresses, provider thinking signatures, placeholders, and common public IDs are preserved to avoid corrupting training data. Assistant-authored prose, reasoning, and code examples are also preserved from credential/email heuristics; local usernames in paths are still anonymized, and structured tool inputs continue through the full scanner. Reported totals are replacement occurrences rather than estimates of unique secrets. It is a best-effort pass; review data before publishing.
+Anonymization replaces known credential formats, high-confidence secret assignments, personal email addresses, contextual PII, home-directory usernames, and embedded base64 media with deterministic dummy values. Media replacements are tiny decoder-valid payloads; unsupported subtypes are relabeled to the actual image, audio, or video placeholder format so multimodal loaders do not receive false MIME declarations. Reserved example-domain addresses, known public bot addresses, provider thinking signatures, placeholders, and common public IDs are preserved to avoid corrupting training data. User, assistant, reasoning, and structured tool content all pass through the same high-confidence privacy scanner. Reported totals are replacement occurrences rather than estimates of unique secrets. It is a best-effort pass; review data before publishing.
 
 ## Studio
 

docs/generation.md

@@ -81,7 +81,7 @@ teich extract codex --model gpt-5-codex --out codex-data
 
 `--model` filters by provider model metadata, not by arbitrary prompt text. This keeps traces that actually ran with matching model identifiers such as `claude-fable-5` and excludes traces that only mention the model name in conversation text.
 
-After extraction, Teich automatically scrubs API keys, emails, and home-directory usernames while preserving embedded media payloads for conversation context. It then prints the replacement counts and asks whether to upload to Hugging Face. If you need a raw, unchanged local export, pass `--no-anon` or `--no-anonymize`:
+After extraction, Teich automatically scrubs API keys, emails, contextual PII, home-directory usernames, and embedded media payloads. Redacted media remains decoder-valid, with the MIME type normalized when a fallback image, audio, or video format is required. Teich then prints the replacement counts and asks whether to upload to Hugging Face. If you need a raw, unchanged local export, pass `--no-anon` or `--no-anonymize`:
 
 ```bash
 teich extract codex --sessions-dir /path/to/.codex --out raw-codex-data --no-anon

docs/prepare-data.md

@@ -48,7 +48,17 @@ print(prep_report.max_token_length)
 print(prep_report.oversized_rows[:3])
 ```
 
-`PrepareReport` includes dropped rows, oversized rows, trimmed rows, token lengths, max token length, kept-row ids, and returned row count.
+`PrepareReport` includes dropped rows, oversized rows, trimmed rows, token lengths,
+max token length, kept-row ids, and returned row count. With a live Gemma 4
+template it also reports per-row mode counts in `gemma4_modes` and migrated
+leading `<|think|>` markers in `gemma4_legacy_triggers_normalized`. When
+`reasoning_policy="strip"` is used, it reports affected rows and messages in
+`reasoning_stripped_rows` and `reasoning_stripped_messages`.
+
+Gemma 4 defaults to per-row auto mode when `enable_thinking` is omitted:
+reasoning-bearing rows enable thinking and history preservation, while direct
+rows use the loaded model's non-thinking protocol. See [Live Gemma 4 Models](training.md#live-gemma-4-models)
+for explicit overrides and validation rules.
 
 Original columns are removed after formatting unless `preserve_columns=True` or an explicit list is passed. The default provenance set is `source`, `metadata`, `raw_index`, and `source_key`.
 
@@ -81,6 +91,7 @@ train_dataset = prepare_data(
         "instruct-chat": {
             "source": "TeichAI/polaris-alpha-1000x",
             "percentage": 20,
+            "reasoning_policy": "strip",
             "chat_template_kwargs": {"enable_thinking": False, "preserve_thinking": False},
         },
     },
@@ -96,7 +107,8 @@ train_dataset = prepare_data(
 
 If one source cannot fill its share after filtering or context-window drops, Teich scales the total row count down instead of silently changing the realized mix.
 
-Global `chat_template_kwargs` are the default for every source. A source-level `chat_template_kwargs` mapping overrides those keys for that dataset only.
+Global `chat_template_kwargs` and `reasoning_policy` values are the defaults for
+every source. Source-level values override them for that dataset only.
 
 You can also pass a simple list of sources:
 

docs/python-api.md

@@ -65,6 +65,7 @@ Useful options:
 - `teich_masking`
 - `tokenize`
 - `chat_template_kwargs`
+- `reasoning_policy` (`"keep"` or `"strip"`)
 
 See [Preparing Data](prepare-data.md).
 

docs/studio.md

@@ -119,7 +119,7 @@ https://huggingface.co/datasets/<owner>/<dataset>/embed/viewer
 
 That official embed works for datasets already available on the Hub. For unpublished local output, Studio uses Teich's local converter to approximate the parts of the viewer that matter before upload. The full Hugging Face viewer backend is hosted by Hugging Face and adds Parquet-backed row serving, search, filtering, SQL, and statistics after the dataset is uploaded and processed.
 
-The Studio upload button regenerates the dataset card before publishing. The card stays intentionally short and points readers to the maintained training docs; large dataset-level tool snapshots are written to `tools.json` and uploaded alongside the JSONL files.
+The Studio upload button regenerates the dataset card before publishing. The card stays intentionally short and points readers to the maintained training docs; dataset JSONL, generated metadata JSON, `README.md`, and `tools.json` are allowlisted for upload. Unrelated files in the output directory are not published.
 
 ## Requirements
 

docs/training.md

@@ -57,23 +57,137 @@ are:
 - `google/gemma-4-26B-A4B-it`
 - `google/gemma-4-31B-it`
 
-Pass `chat_template_kwargs={"enable_thinking": True, "preserve_thinking": True}`
-to `prepare_data()`. Do not replace `tokenizer.chat_template` unless you are
-intentionally testing a maintained fork. Teich supervises the closing
-`<turn|>` token for completed Gemma responses while keeping system, user, and
-tool-response context masked.
+Gemma 4's thinking mode is a prompt protocol, so Teich resolves it for every
+row before rendering. Leave `enable_thinking` unset for the recommended auto
+mode:
+
+- a row containing assistant `reasoning_content`, `thinking`, or `reasoning`
+  becomes a thinking row;
+- a row without assistant reasoning becomes a non-thinking row;
+- thinking rows automatically receive `enable_thinking=True` and
+  `preserve_thinking=True`, preserving historical reasoning in multi-turn data;
+- a leading legacy `<|think|>` in system/developer content is removed and used
+  as a thinking-mode hint. `PrepareReport` records the migration, while triggers
+  in any other position are rejected.
+
+The live template, not the source data, inserts `<|think|>` into the system
+turn. The 26B-A4B and 31B generation prompts also insert an empty thought
+channel in non-thinking mode while E4B does not. Teich discovers that behavior
+from the loaded template and mirrors the prefix in completed SFT turns while
+keeping the synthetic prefix masked.
+
+Explicit `{"enable_thinking": True}` and `{"enable_thinking": False}` remain
+available when an entire source must be forced to one mode. Teich automatically
+enables history preservation for the explicit thinking case. Contradictions
+fail closed: forcing non-thinking on a reasoning-bearing row, or explicitly
+disabling `preserve_thinking` when historical reasoning would disappear,
+raises an error instead of producing inconsistent training text.
+
+Do not replace `tokenizer.chat_template` unless you are intentionally testing a
+maintained fork. Teich supervises the closing `<turn|>` token for completed
+Gemma responses while keeping system, user, tool-response, and generated prompt
+prefix context masked.
+
+Thinking and non-thinking examples can therefore be mixed in the same source
+without any template configuration:
+
+```python
+train_dataset, prep_report = prepare_data(
+    "username/mixed-gemma-traces",
+    tokenizer,
+    tokenize=True,
+    strict=True,
+    return_report=True,
+)
+```
+
+With `return_report=True`, inspect `prep_report.gemma4_modes` for the resolved
+thinking/non-thinking counts and
+`prep_report.gemma4_legacy_triggers_normalized` for migrated legacy rows.
+Source-level `chat_template_kwargs` are still useful as explicit policy
+overrides, but are no longer required merely to mix the two modes.
 
 `gemma4_example.py` uses the live remote template by default. Set
-`CHAT_TEMPLATE_PATH` only to opt into a local custom template, and set
-`MODEL_REVISION` when a reproducible non-`main` revision is required. Set
+`CHAT_TEMPLATE_PATH` only to opt into a local custom template. Its
+`GEMMA4_THINKING_MODE` defaults to `auto`; use `thinking` or `nonthinking` only
+to force a homogeneous source. The older `GEMMA4_ENABLE_THINKING` variable is
+still accepted for compatibility. Set `MODEL_REVISION` when a reproducible
+non-`main` revision is required. Set
 `HF_TOKEN` to an account that has accepted the Gemma repository terms when the
 checkpoint is not already available through the local Hugging Face login.
 
+The example keeps reasoning in its agent source and strips reasoning from its
+direct-chat source. Override those source policies with
+`AGENT_REASONING_POLICY` or `CHAT_REASONING_POLICY` when using datasets with a
+different contract. It prints the resolved Gemma mode counts, stripped-row
+count, and maximum token length before training. Do not add `<turn|>` to source
+messages: Teich derives and supervises the live template's completed-turn
+terminator automatically.
+
 Run the example in a dedicated, internally consistent Unsloth training
 environment and check it with `python -m pip check` before a long run. Teich's
 core environment does not pin the CUDA, PyTorch, Unsloth, and TRL stack because
 those versions depend on the host GPU and CUDA runtime.
 
+## Live Qwen 3.8 Models
+
+Qwen 3.8 has a different native contract and Teich does not apply Gemma's auto
+mode rules to it. The live `Qwen/Qwen3.8-27B` template defaults to thinking,
+defaults `reasoning_effort` to `xhigh`, and preserves historical
+`reasoning_content` unless `preserve_thinking=False` is supplied. To train its
+native reasoning behavior, retain those defaults or set only the desired
+effort:
+
+```python
+train_dataset = prepare_data(
+    "username/qwen38-reasoning-traces",
+    tokenizer,
+    chat_template_kwargs={"reasoning_effort": "medium"},
+    tokenize=True,
+    strict=True,
+)
+```
+
+Supported live efforts are `low`, `medium`, and `xhigh`. Continue to use
+`train_on_reasoning=True` in `mask_data()` when those reasoning tokens should
+receive loss.
+
+For direct instruction tuning from a dataset that still contains reasoning,
+remove the reasoning before rendering and explicitly select Qwen's
+non-thinking template mode:
+
+```python
+train_dataset, prep_report = prepare_data(
+    "username/mixed-source-traces",
+    tokenizer,
+    reasoning_policy="strip",
+    chat_template_kwargs={
+        "enable_thinking": False,
+        "preserve_thinking": False,
+    },
+    return_report=True,
+    tokenize=True,
+    strict=True,
+)
+```
+
+Qwen 3.8's non-thinking prompt contains an empty `<think>...</think>` primer.
+Teich keeps that inference-alignment prefix in the rendered text but masks it
+from loss. The final answer and closing `<|im_end|>` remain supervised.
+
+## Source Reasoning Policy
+
+`reasoning_policy="keep"` is the default and leaves structured assistant
+reasoning for the loaded chat template to handle according to its own model
+contract. `reasoning_policy="strip"` removes normalized `reasoning_content`
+before rendering. This is deliberately independent of
+`mask_data(train_on_reasoning=False)`: masking keeps gold reasoning in the
+causal context, whereas stripping creates a true instruction-only example.
+
+The policy can be set per source in a mixed dataset. `PrepareReport` records
+the affected row and message counts in `reasoning_stripped_rows` and
+`reasoning_stripped_messages`.
+
 ## What `mask_data()` Does
 
 Before `mask_data()`, the trainer dataset usually contains:
@@ -108,6 +222,12 @@ each dataset-map batch together instead of issuing one tokenizer call per row.
 
 For Qwen-style templates, the initial `<think>` tag is intentionally included in supervision.
 
+For Gemma 4, Teich supervises exactly one closing `<turn|>` for a completed
+model turn whenever reasoning, final-answer, or tool-call supervision is
+enabled for that turn. It does not add a second terminator inside a continuing
+tool-call chain. The terminator remains a target even when the final answer is
+masked, so reasoning-only and tool-only fine-tunes still learn to stop.
+
 ## Masking Policy
 
 `mask_data()` trains on these by default:

gemma4_example.py

@@ -17,6 +17,29 @@
 HUB_REPO_ID = os.environ.get("HUB_REPO_ID") or ""
 HF_TOKEN = os.environ.get("HF_TOKEN", "")
 CHAT_TEMPLATE_PATH = os.environ.get("CHAT_TEMPLATE_PATH")
+AGENT_REASONING_POLICY = os.environ.get("AGENT_REASONING_POLICY", "keep").strip().lower()
+CHAT_REASONING_POLICY = os.environ.get("CHAT_REASONING_POLICY", "strip").strip().lower()
+for policy_name, policy in {
+    "AGENT_REASONING_POLICY": AGENT_REASONING_POLICY,
+    "CHAT_REASONING_POLICY": CHAT_REASONING_POLICY,
+}.items():
+    if policy not in {"keep", "strip"}:
+        raise ValueError(f"{policy_name} must be keep or strip")
+_thinking_mode = os.environ.get("GEMMA4_THINKING_MODE")
+_legacy_thinking = os.environ.get("GEMMA4_ENABLE_THINKING")
+if _thinking_mode is None and _legacy_thinking is not None:
+    _thinking_mode = (
+        "nonthinking"
+        if _legacy_thinking.strip().lower() in {"0", "false", "no"}
+        else "thinking"
+    )
+GEMMA4_THINKING_MODE = (_thinking_mode or "auto").strip().lower().replace("-", "")
+if GEMMA4_THINKING_MODE not in {"auto", "thinking", "nonthinking"}:
+    raise ValueError("GEMMA4_THINKING_MODE must be auto, thinking, or nonthinking")
+CHAT_TEMPLATE_KWARGS = {
+    "thinking": {"enable_thinking": True},
+    "nonthinking": {"enable_thinking": False},
+}.get(GEMMA4_THINKING_MODE)
 
 model, tokenizer = FastModel.from_pretrained(
     model_name=MODEL_NAME,
@@ -51,26 +74,46 @@
     random_state = 3407,
 )
 
-train_dataset = prepare_data(
+train_dataset, prep_report = prepare_data(
     {
         "max_examples": 30,
         "agent": {
             "source": "armand0e/ag-datagen-v2-test",
             "percentage": 80,
+            # Keep structured reasoning. In auto mode, Teich renders these as
+            # thinking rows and preserves reasoning across multi-turn history.
+            "reasoning_policy": AGENT_REASONING_POLICY,
         },
         "chat": {
             "source": "armand0e/DeepSeek-v4-Flash-Chat",
             "percentage": 20,
+            # Make this a true direct-instruction source even if an upstream
+            # row happens to contain reasoning fields. This differs from only
+            # masking reasoning loss, which would leave it in causal context.
+            "reasoning_policy": CHAT_REASONING_POLICY,
         },
     },
     tokenizer,
     split="train",
     hf_token=HF_TOKEN,
-    chat_template_kwargs={"enable_thinking": True, "preserve_thinking": True},
+    # Auto mode classifies every Gemma 4 row independently. Reasoning-bearing
+    # rows enable thinking and preserve history; direct rows use the exact
+    # non-thinking inference prefix of the loaded live template.
+    chat_template_kwargs=CHAT_TEMPLATE_KWARGS,
     max_length=MAX_SEQ_LEN,
     oversized_policy="trim_followups",
     tokenize=True,
     strict=True,
+    return_report=True,
+)
+
+print(
+    "Prepared Gemma 4 modes:",
+    prep_report.gemma4_modes,
+    "| stripped reasoning rows:",
+    prep_report.reasoning_stripped_rows,
+    "| max tokens:",
+    prep_report.max_token_length,
 )
 
 trainer = SFTTrainer(
@@ -109,6 +152,10 @@
     train_on_tools=True,
 )
 
+# Teich keeps exactly one <turn|> target for each completed Gemma model turn
+# that has an enabled reasoning, answer, or tool-call target. Do not append a
+# terminator to dataset content manually.
+
 print(trainer.train_dataset.preview())
 
 trainer_stats = trainer.train(resume_from_checkpoint=False)

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "teich"
-version = "0.3.4"
+version = "0.3.5"
 description = "Turn coding agent traces into auditable supervised fine-tuning data"
 readme = "README.md"
 license = {text = "Apache-2.0"}

src/teich/__init__.py

@@ -1,6 +1,6 @@
 from __future__ import annotations
 
-__version__ = "0.3.4"
+__version__ = "0.3.5"
 
 from .audit import SFTAuditReport, audit_sft_dataset
 from .config import Config, load_config