Show context window size and warn on exceedance in summarization workflow

[spencer-crook]

Motivation

When running the summarization workflow we may hit model context-window limits without clear visibility into how much of the input was used or which parts were dropped. We need a way to surface the effective context window size, current token usage, and provide clear warnings/visual indicators when the input will exceed the model's context window so engineers and users can make informed decisions.

User stories

• As an engineer running batch summarization, I want to see the token count of the original document and the model context limit so I can trim or split inputs proactively.
• As a user of the summarization UI, I want an inline warning when my input will exceed the model's context window and an option to auto-split or truncate.
• As a developer debugging summarization quality, I want logs and metrics showing if any part of the input was omitted due to context-window exceedance.

Proposed solution

1. Surface token counts and context limit
   • Display model context window (e.g., 8k, 32k) used for each run.
   • Show token count for: original input, prompt + instruction overhead, and estimated summary output. Have pop out to show what prompt was actually made of
2. Warnings & UX
   • In UI: show a progress bar / meter with green/yellow/red states; show explicit warning if input > (context limit - safety margin).
   • Provide buttons?: "Proceed anyway" (with confirmation).
3. Automatic splitting/truncation?
   • Add an option to split inputs into chunks with overlap and run summarization iteratively (and combine results), with configurable chunksize and overlap.?
4. Logging & telemetry
   • Emit structured logs/metrics: tokens_in, tokens_prompt_overhead, tokens_used_by_model, chunks_created, truncated=true/false.
   • Record warnings as events to make it easier to find runs that hit limits.
5. Tests & docs
   • Unit tests for token counting and splitting logic.
   • End-to-end test that simulates an input exceeding the context window and verifies warning and split/truncate behavior.
   • Update README/docs with explanation and screenshots.

Acceptance criteria

• The UI/CLI shows token counts and the context window limit for every summarization run.
• A clear warning is shown when inputs will exceed the context window, with options to auto-split or truncate.
• Logs/metrics record whether inputs were split or truncated and the token counts.
• Tests and documentation updated.

Implementation notes / suggestions

• Reuse existing tokenization utilities (specify tokenizer used) to ensure counts match model usage.
• Default safety margin: 90% context window (configurable by environment variable).
• Splitting approach: greedy chunking with configurable overlap?

Labels

• enhancement

Deliverables

• Feature implemented behind feature flag
• Unit + e2e tests
• Docs with screenshots and configuration options