Google GenAI tool responses fail when content is plain text (not JSON

[azanux]

Bug

Description

Google Gemini's API requires tool responses (FunctionResponse.response) to be valid JSON objects. When a tool returns a plain text string, Spring AI's GoogleGenAiChatModel.parseJsonToMap() either:

1. Crashes with a JsonParseException (for MatryoshkaTool responses)
2. Silently drops the content (for RAG/search tool responses), causing the LLM to hallucinate

The same tools work correctly with OpenAI, which accepts plain text in tool response content fields.

Reproduction

I was able to reproduce this bug using the ragbot example project with the following modifications:

1. Updated pom.xml:

• Changed Embabel Agent version to 0.3.4-SNAPSHOT
• Added Google GenAI Maven profile:

<profile>
    <id>google-genai-models</id>
    <activation>
        <property>
            <name>env.GOOGLE_API_KEY</name>
        </property>
    </activation>
    <dependencies>
        <dependency>
            <groupId>com.embabel.agent</groupId>
            <artifactId>embabel-agent-starter-google-genai</artifactId>
            <version>${embabel-agent.version}</version>
        </dependency>
    </dependencies>
</profile>

2. Switched application.yml to use Google GenAI:

ragbot:
  chat-llm:
    model: gemini-2.5-flash
    temperature: 0.0

embabel:
  agent:
    platform:
      models:
        googlegenai:
          api-key: ${GOOGLE_API_KEY}
          max-attempts: 10
          backoff-millis: 5000

  models:
    default-llm:
      model: gemini-2.5-flash
    default-embedding-model:
      model: text-embedding-3-small

3. Updated JavelitChatUI.java:

Updated createSession call to match the new 0.3.4-SNAPSHOT API signature (User, OutputChannel, contextId, systemMessage):

var session = chatbot.createSession(ANONYMOUS_USER, outputChannel, UUID.randomUUID().toString(), null);

4. Enabled the matryoshka tool pattern:

In the original ragbot code, ToolishRag is created locally in the ChatActions constructor and used via .withReference(toolishRag). This exposes the individual tools (vectorSearch, textSearch, etc.) as a flat list directly to the LLM - no matryoshka parent tool is involved. This works fine with Google GenAI. (but not using the feature of matryoshka )

To trigger the bug, the matryoshka pattern must be explicitly enabled. Two changes were made:

InChatActions.java to use ToolishRag as a tool :**

Each approach triggers a different bug:

Bug 1 — .withReference(toolishRag.asMatryoshka()) → crash (JsonParseException):

var assistantMessage = context.ai()
        .withLlm(properties.chatLlm())
        .withReference(toolishRag.asMatryoshka())  // ← wraps in MatryoshkaReference
        .withTemplate("ragbot")
        .respondWithSystemPrompt(conversation, ...);

1. .asMatryoshka() wraps the reference so tools() returns a single MatryoshkaTool parent
2. The LLM calls the parent tool sources → it returns plain text: "Enabled 4 tools: sources_vectorSearch, ..."
3. Gemini tries to parse this plain text as JSON → crash

Bug 2 — .withTools(toolishRag), .withReference(toolishRag) → silent data loss / hallucination:

Any of these approaches triggers Bug 2:

// Using .withTools():
var assistantMessage = context.ai()
        .withLlm(properties.chatLlm())
        .withTools(toolishRag)
        .withTemplate("ragbot")
        .respondWithSystemPrompt(conversation, ...);

// Using .withReference():
var assistantMessage = context.ai()
        .withLlm(properties.chatLlm())
        .withReference(toolishRag)
        .withTemplate("ragbot")
        .respondWithSystemPrompt(conversation, ...);

1. The individual tools (vectorSearch, textSearch, etc.) return plain text search results
2. Gemini silently drops the non-JSON content → the LLM never sees the results → hallucination

5. Steps to reproduce:

1. Set GOOGLE_API_KEY environment variable
2. Start the ragbot shell and ingest content
3. Start a chat session
4. For Bug 1: use .withReference(toolishRag.asMatryoshka()) - the LLM calls the matryoshka parent → JsonParseException crash
5. For Bug 2: use .withTools(toolishRag) or .withReference(toolishRag) - the search tools return plain text results → silently dropped → hallucinated answers

Root Cause

All tool results flow through the same path before being sent to the LLM:

tool.call(input) → Tool.Result.Text(content) → ToolResultMessage(content) → messageConverters.kt → ToolResponseMessage → LLM

In messageConverters.kt, the textContent is passed as-is to ToolResponseMessage.ToolResponse.responseData without ensuring it's valid JSON.

Why most tools work fine today: the vast majority of @LlmTool methods return objects (data classes, Maps, Lists). In MethodTool.convertResult(), these go through the else branch which serializes them via objectMapper.writeValueAsString(result) — producing valid JSON before reaching messageConverters.kt:

private fun convertResult(result: Any?): Tool.Result {
    return when (result) {
        null -> Tool.Result.text("")              // empty string
        is String -> Tool.Result.text(result)     // ⚠️ String passed as-is, no wrapping
        is Tool.Result -> result
        else -> Tool.Result.text(                 // ✅ Objects serialized to JSON by Jackson
            objectMapper.writeValueAsString(result)
        )
    }
}

So messageConverters.kt worked "by luck" - the content was already valid JSON in ~95% of cases. Only tools returning a raw String are affected, because they pass through the is String branch with no JSON serialization.

Affected Tools

Tool	Behavior	Details
MatryoshkaTool	Crash (JsonParseException)	call() returns "Enabled 4 tools: sources_vectorSearch, ..."
RAG/search tools (ToolishRag)	Silent data loss → hallucination	Returns plain text like "2 results: HBNB Services..."
"HBNB Services" si in my document ingested

Error Output

MatryoshkaTool crash:

java.lang.RuntimeException: Failed to parse JSON: Enabled 4 tools: sources_vectorSearch, sources_textSearch, sources_broadenChunk, sources_zoomOut
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Enabled': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

Full stack trace with retry loop (using gemini-2.5-pro):

The error triggers Spring Retry, which retries the same failing request multiple times before exhausting retries:

02:37:11.287 [ForkJoinPool.commonPool-worker-2] INFO  RetryProperties - Operation googlegenai-gemini-2.5-pro: Retry error. Retry count: 1
java.lang.RuntimeException: Failed to parse JSON: Enabled 4 tools: sources_vectorSearch, sources_textSearch, sources_broadenChunk, sources_zoomOut
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.parseJsonToMap(GoogleGenAiChatModel.java:397)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.lambda$messageToGeminiParts$0(GoogleGenAiChatModel.java:337)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        ...
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.messageToGeminiParts(GoogleGenAiChatModel.java:340)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.lambda$toGeminiContent$22(GoogleGenAiChatModel.java:885)
        ...
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.toGeminiContent(GoogleGenAiChatModel.java:887)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.createGeminiRequest(GoogleGenAiChatModel.java:830)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.lambda$internalCall$3(GoogleGenAiChatModel.java:439)
        at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:357)
        at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:230)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.lambda$internalCall$4(GoogleGenAiChatModel.java:437)
        at io.micrometer.observation.Observation.observe(Observation.java:564)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.internalCall(GoogleGenAiChatModel.java:437)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.call(GoogleGenAiChatModel.java:424)
        at com.embabel.agent.spi.support.springai.SpringAiLlmMessageSender.call(SpringAiLlmMessageSender.kt:64)
        at com.embabel.agent.spi.loop.support.DefaultToolLoop.execute(DefaultToolLoop.kt:72)
        at com.embabel.agent.spi.support.ToolLoopLlmOperations.doTransform(ToolLoopLlmOperations.kt:167)
        at com.embabel.agent.spi.support.springai.ChatClientLlmOperations.doTransform(ChatClientLlmOperations.kt:159)
        at com.embabel.agent.spi.support.AbstractLlmOperations.createObject$lambda$6$lambda$3$lambda$2(AbstractLlmOperations.kt:178)
        at com.embabel.agent.spi.support.AbstractLlmOperations.executeWithTimeout$lambda$0(AbstractLlmOperations.kt:93)
        ...
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Enabled': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 8]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2596)
        ...
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3863)
        at org.springframework.ai.google.genai.GoogleGenAiChatModel.parseJsonToMap(GoogleGenAiChatModel.java:378)
        ... 43 common frames omitted

The retry loop repeats the same error (retry count 1, 2, 3...) because the issue is in the message conversion, not a transient network error. Each retry re-sends the same tool response with the same non-JSON content.

RAG/search tools — hallucination:

3 consecutive runs produce 3 different hallucinated answers (e.g., "Societe Generale", "Veolia", "CDD") despite my documents containing "HBNB Services - Technical Blockchain Advisor".

Provider Comparison

Provider	Result
Google GenAI (spring-ai-google-genai)	Fails — requires JSON in responseData
OpenAI	Works — accepts plain text

[azanux]

fix: Wrap non-JSON tool responses for Gemini compatibility

I submited the PR , that solved the problem , normally Other LLM can understand Json so there is no regression. And it handle futur tools also.

Fix

• messageConverters.kt: Added ensureJson() extension function that wraps non-JSON tool response text in {"result": "..."} before sending to the LLM. JSON objects and arrays are preserved as-is. This acts as a safety net at the single conversion point all tools pass through.
• MatryoshkaTool.kt: Changed SimpleMatryoshkaTool.call() and SelectableMatryoshkaTool.call() to return structured JSON ({"enabled_tools_count": N, "enabled_tools": [...]}) instead of plain text, via a shared enabledToolsJson() helper.

Test plan

• MessageConversionTest: 4 new tests covering plain text wrapping, JSON object/array preservation, and whitespace handling
• MatryoshkaToolTest: All assertions updated to verify JSON format responses
• Manual verification with ragbot + Google GenAI to confirm no more JsonParseException and no hallucination