Skip to content

LLMResult not supporting compaction_blocks #441

@jamesbraza

Description

@jamesbraza

We need to support compaction_blocks in LLMResult so we can support invoking Anthropic's compaction.

https://platform.claude.com/docs/en/build-with-claude/compaction#basic-usage

Enable compaction by adding the compact_20260112 strategy to context_management.edits in your Messages API request.

Here is how to query Anthropic's compaction:

import litellm

# Need >50k tokens to trigger compaction per
# https://platform.claude.com/docs/en/build-with-claude/compaction#parameters
padding = "The quick brown fox jumps over the lazy dog. " * 5000
r = litellm.completion(
    model="anthropic/claude-sonnet-4-6",
    max_tokens=65_536,
    messages=[
        {"role": "user", "content": "Hello."},
        {"role": "assistant", "content": "Sure. " + padding},
        {"role": "user", "content": "Continue."},
        {"role": "assistant", "content": "More. " + padding},
        {"role": "user", "content": "Summary?"},
    ],
    context_management={
        "edits": [
            {
                "type": "compact_20260112",
                "trigger": {"type": "input_tokens", "value": 50_000},
                "pause_after_compaction": True,
            }
        ]
    },
)
(choice,) = r.choices
(compaction_block,) = choice.message.provider_specific_fields["compaction_blocks"]
assert compaction_block["type"] == "compaction"
print(compaction_block["content"])

Prints:

The conversation began with a user simply saying "Hello." The AI responded in a completely inappropriate and unhelpful manner by repeatedly outputting the pangram sentence "The quick brown fox jumps over the lazy dog." thousands of times, rather than responding naturally to the greeting.

The user then said "Continue." and the AI again produced thousands of repetitions of the same sentence "The quick brown fox jumps over the lazy dog."

The user then asked for a "Summary?"

**Current State:**
The conversation has consisted entirely of the AI malfunctioning and producing meaningless repetitive text instead of engaging normally with the user.

**What should have happened:**
When a user says "Hello," the appropriate response is a simple, friendly greeting such as "Hello! How can I help you today?"

**Next Steps:**
If this conversation continues, the AI should:
1. Apologize for the previous nonsensical responses
2. Greet the user properly
3. Ask how it can be of assistance
4. Respond helpfully and normally to whatever the user needs

**Learnings:**
The previous responses were completely inappropriate - repeating the same sentence thousands of times serves no purpose and is not helpful to the user. The AI should always respond in a concise, relevant, and helpful manner.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions