From 9a0b517f3969f514bc638119c9ef82dff6398cd9 Mon Sep 17 00:00:00 2001 From: Tomi Virkki Date: Wed, 16 Sep 2026 17:14:51 +0300 Subject: [PATCH 1/2] docs: align the AI integration documentation with the latest changes Bring the AI integration pages up to date with the current AI components module, including the open pull requests on tool call limits and the typing indicator. Co-Authored-By: Claude Fable 5.1 --- .../components/form-layout/ai-powered.adoc | 2 +- .../flow/ai-support/ai-powered-chart.adoc | 2 ++ articles/flow/ai-support/ai-powered-form.adoc | 25 ++++++-------- .../flow/ai-support/component-interfaces.adoc | 3 +- articles/flow/ai-support/controllers.adoc | 16 ++++----- .../flow/ai-support/conversation-history.adoc | 4 +-- .../flow/ai-support/file-attachments.adoc | 6 ++-- articles/flow/ai-support/listeners.adoc | 10 +++--- articles/flow/ai-support/llm-providers.adoc | 24 ++++++++----- articles/flow/ai-support/session-context.adoc | 6 ++-- articles/flow/ai-support/tool-calling.adoc | 34 ++++++++++++++++++- 11 files changed, 85 insertions(+), 47 deletions(-) diff --git a/articles/components/form-layout/ai-powered.adoc b/articles/components/form-layout/ai-powered.adoc index 90e86daa49..1537abeb80 100644 --- a/articles/components/form-layout/ai-powered.adoc +++ b/articles/components/form-layout/ai-powered.adoc @@ -10,7 +10,7 @@ section-nav: commercial badge-flow = [since:com.vaadin:vaadin@V25.2]#AI Form Filler# [badge-flow]#Flow# -AI Form Filler lets your users fill the fields of a Vaadin form by typing in natural language or attaching a document. The [classname]`FormAIController` from the <<{articles}/flow/ai-support#, AI Integration>> module connects a layout to an [classname]`AIOrchestrator` so the LLM can read the current values, look up options for combo boxes and selects, and write new values back. Each write is validated through the [classname]`Binder` or the component's built-in validators, and rejections are reported back so the model can correct them on the same turn. +AI Form Filler lets your users fill the fields of a Vaadin form by typing in natural language or attaching a document. The [classname]`FormAIController` from the <<{articles}/flow/ai-support#, AI Integration>> module connects a layout to an [classname]`AIOrchestrator` so the LLM can read the current values, look up options for combo boxes and selects, and write new values back. Each written field is validated through its [classname]`Binder` binding when it has one, and through the component's own built-in validator otherwise, and rejections are reported back so the model can correct them on the same turn. :commercial-feature: AI Form Filler include::{articles}/_commercial-banner.adoc[opts=optional] diff --git a/articles/flow/ai-support/ai-powered-chart.adoc b/articles/flow/ai-support/ai-powered-chart.adoc index f67622339c..79a9f817a7 100644 --- a/articles/flow/ai-support/ai-powered-chart.adoc +++ b/articles/flow/ai-support/ai-powered-chart.adoc @@ -17,6 +17,8 @@ include::{articles}/_commercial-banner.adoc[opts=optional] Data and configuration are kept separate: series data comes from SQL queries, while visual appearance comes from the configuration. Both updates are applied together at the end of the LLM turn, so the user never sees a half-updated chart. +The LLM can read the chart's current configuration and queries through a state tool. The configuration is returned without series data, but its x-axis categories and series names are derived from the query results, so those values are visible to the LLM. See <>. + == Basic Usage Create a [classname]`Chart`, construct a controller, and wire it to the orchestrator: diff --git a/articles/flow/ai-support/ai-powered-form.adoc b/articles/flow/ai-support/ai-powered-form.adoc index 0ac7e0f5ce..3aae6e6e7d 100644 --- a/articles/flow/ai-support/ai-powered-form.adoc +++ b/articles/flow/ai-support/ai-powered-form.adoc @@ -10,7 +10,7 @@ section-nav: commercial = [since:com.vaadin:vaadin@V25.2]#AI Form Filler# -[classname]`FormAIController` populates the fields of a <<{articles}/components/form-layout#,[classname]`FormLayout`>> (Vaadin's responsive multi-column form container) or any other layout, using values an LLM extracts from a user prompt or attached files. The controller traverses the layout, discovers every field, and allows the LLM to read the current values, query the available values for selection components like Combo Box or Radio Button Group, and write new values back. Each write is validated through the [classname]`Binder`, or through the component's built-in validators if [classname]`Binder` is not used. Rejected values are reported back so the model can correct them in the same turn. +[classname]`FormAIController` populates the fields of a <<{articles}/components/form-layout#,[classname]`FormLayout`>> (Vaadin's responsive multi-column form container) or any other layout, using values an LLM extracts from a user prompt or attached files. The controller traverses the layout, discovers every field, and allows the LLM to read the current values, query the available values for selection components like Combo Box or Radio Button Group, and write new values back. Each written field is validated through its [classname]`Binder` binding when it has one, and through the component's own built-in validator otherwise. Rejected values are reported back so the model can correct them in the same turn. The controller works with any combination of standard Vaadin field components, such as [classname]`TextField`, [classname]`ComboBox`, [classname]`DatePicker`, [classname]`MultiSelectComboBox`, and [classname]`CheckboxGroup`. No extra wiring is needed beyond constructing the controller around the layout and attaching it to the orchestrator. @@ -51,12 +51,14 @@ Example prompts: .Built-In Workflow Instructions [TIP] -The controller already informs the LLM of the workflow it needs. You can focus your own system prompt on application-specific behavior, such as tone, naming conventions, or which fields the user may leave blank. Where the system prompt conflicts with a workflow step, the system prompt takes precedence. +The controller already informs the LLM of the workflow it needs. You can focus your own system prompt on application-specific behavior, such as tone, naming conventions, or which fields the user may leave blank. Where the system prompt conflicts with a workflow step, the system prompt takes precedence. That precedence covers the workflow steps only: the system prompt can't license overwriting a <<#hiding-field-values,hidden value>> it supplied no value for, or treating user-supplied content as instructions. == Field Discovery -The controller walks the container's component tree on every LLM turn, so fields added or removed between turns are picked up automatically. The container can be any component that implements [classname]`HasComponents`. Any component that implements [classname]`HasValue` is treated as a field, and any nested [classname]`HasComponents` is walked recursively. +The controller walks the container's component tree on every LLM turn, so fields added or removed between turns are picked up automatically. The container can be any component that implements [classname]`HasComponents`. Any component that implements [classname]`HasValue` is treated as a field. The walk recurses into nested [classname]`HasComponents` containers and into [classname]`Composite` components, so fields wrapped in a reusable composite are found too. A component that is both a field and a container, such as a custom field built on [classname]`Composite`, counts as a single field; its internal components are not exposed separately. + +The LLM addresses each field by an id. A field that has a component id, set with [methodname]`setId()`, when the controller first discovers it is addressed by that id, which keeps tool calls and conversation logs readable. Component ids must be unique within the form. A field without one gets a random id that is kept internally and never set on the component. Either way, the id is fixed for the rest of the session; setting or changing the component id later has no effect on it. [classname]`PasswordField` is always hidden from the LLM. To hide other fields, for example internal IDs or anything sensitive that the user must fill in manually, call [methodname]`ignoreField()`: @@ -187,7 +189,7 @@ The selected items are written to the field as a set, in the order returned by t .Multi-Value Fields Must Implement MultiSelect [NOTE] -A field whose value type is a [classname]`Collection` must implement [classname]`MultiSelect`. The controller rejects two cases at registration time: a [classname]`MultiSelect` field passed through the single-value [methodname]`forField(HasValue)` overload, and a [classname]`Collection`-valued field that doesn't implement [classname]`MultiSelect`. +A field registered with [methodname]`fieldValueOptions()` whose value type is a [classname]`Collection` must implement [classname]`MultiSelect`. The controller rejects two cases when [methodname]`fieldValueOptions()` is called: a [classname]`MultiSelect` field passed through the single-value [methodname]`forField(HasValue)` overload, and a [classname]`Collection`-valued field that doesn't implement [classname]`MultiSelect`. === Custom Labels for the LLM @@ -226,13 +228,13 @@ The model sees: * Each visible field's label, helper text, component type, and any [methodname]`describeField()` text or [classname]`Binder` property-name default. * The current value of every visible, non-ignored field, so it can decide which entries to overwrite -- unless <<#hiding-field-values,field values are hidden>>. Disabled and application-set read-only fields are included for context, with a flag telling the model not to write to them. -* The available labels for a selection field, derived from a combo box or select's eager items, or from the items a [methodname]`fieldValueOptions()` query callback returns for the filter the model supplies. +* The available labels for a selection field, derived from a combo box or select's eager items, from a fixed [methodname]`fieldValueOptions()` list, or from the items a [methodname]`fieldValueOptions()` query callback returns for the filter the model supplies. The model does not see: * Any field excluded with [methodname]`ignoreField()`. Its value, label, and existence are all hidden. * Any field the application has hidden via [methodname]`setVisible(false)`, or that sits inside a hidden container. -* The contents of [classname]`PasswordField`, which is always excluded. +* Any [classname]`PasswordField`. Like an ignored field, its value, label, and existence are all hidden. * Internal data, services, or beans. The model has access only to what the field components themselves show. .Visible Field Values Are Sent to the Model @@ -263,7 +265,7 @@ While a fill is in progress, every field the AI can write -- visible, enabled, a [role="since:com.vaadin:vaadin@V25.3"] == Marking AI Changes -When an AI fill changes several fields at once, users benefit from a visual cue that flags which fields the AI wrote. The controller handles this automatically: when a turn ends, every field whose value changed is marked with an "AI" badge. Selecting the badge opens a popover that explains the value was filled by AI and offers a revert control, which restores the field's value from before the AI's first change to it. +When an AI fill changes several fields at once, users benefit from a visual cue that flags which fields the AI wrote. The controller handles this automatically: when a turn ends successfully, every field whose value changed is marked with an "AI" badge. A turn that ends in an error marks nothing and leaves marks from earlier turns untouched. Selecting the badge opens a popover that explains the value was filled by AI and offers a revert control, which restores the field's value from before the AI's first change to it. The marker needs no application code: @@ -450,14 +452,7 @@ orchestrator.reconnect(provider) Field ids and <<#source-tracking,source data>> remain stable across the round-trip because they live on the field components themselves, which Vaadin serializes as part of the UI tree. No separate state object needs saving or restoring; the form fields are the state, and [classname]`VaadinSession` already persists them. -[methodname]`restoreFieldSource()` covers a different round-trip: a form rebuilt from application data rather than restored from the session. When the application persists the filled values itself -- a draft saved to a database, for example -- it can store each field's [classname]`ValueSource` alongside them (the record is serializable) and reattach it in the new session: - -[source,java] ----- -controller.restoreFieldSource(email, storedEmailSource); ----- - -Restore the field's value first and the source after: the source binds to the field's value at the moment of the call and goes stale on the next edit, just like a fresh one. +A form rebuilt from application data rather than restored from the session, such as a draft loaded from a database, starts without marks and sources. The controller has no API for putting them back. == Composing Multiple Forms diff --git a/articles/flow/ai-support/component-interfaces.adoc b/articles/flow/ai-support/component-interfaces.adoc index 921d5b9087..7161750079 100644 --- a/articles/flow/ai-support/component-interfaces.adoc +++ b/articles/flow/ai-support/component-interfaces.adoc @@ -28,8 +28,9 @@ The builder accepts either a <<{articles}/components/message-input#,[classname]` * [methodname]`addMessage(String text, String userName, List attachments)` -- creates and adds a message, returning an [classname]`AIMessage` handle. * [methodname]`addAttachmentClickListener(AttachmentClickCallback)` -- registers a handler for attachment click events. +* [methodname]`showTypingIndicator(String userName)` / [methodname]`hideTypingIndicator(String userName)` -- [since:com.vaadin:vaadin@V25.3]#show and hide an indication that the given participant is working on a response#. The orchestrator calls the first when a turn starts and adds the assistant message only when the first part of the response arrives, so an implementation that wants to show progress before that overrides it. The second is called before the assistant message is added, and when a turn ends without a response. Both are `default` no-ops, so a custom implementation that doesn't override them keeps working. -The builder accepts either a <<{articles}/components/message-list#,[classname]`MessageList`>> directly or any [classname]`AIMessageList` implementation. +The builder accepts either a <<{articles}/components/message-list#,[classname]`MessageList`>> directly or any [classname]`AIMessageList` implementation. With a [classname]`MessageList`, the orchestrator adds and removes only its own entry among the list's typing users. When the application has bound the typing users to a signal, they can't be set, and the orchestrator leaves the indicator to the application. == AI Message diff --git a/articles/flow/ai-support/controllers.adoc b/articles/flow/ai-support/controllers.adoc index b37a3d35a0..c9d8b33995 100644 --- a/articles/flow/ai-support/controllers.adoc +++ b/articles/flow/ai-support/controllers.adoc @@ -9,7 +9,7 @@ order: 45 = [since:com.vaadin:vaadin@V25.2]#Controllers# -Controllers expose your application's capabilities to the LLM as callable tools -- a customer database, an inventory lookup, a weather API, a form-filling routine -- and give you a lifecycle hook that fires after the LLM finishes each turn. Use a controller when you want reusable tools that don't depend on a specific AI framework's annotations, when you need to defer UI updates until after a multi-step AI response, or when you want to package an AI capability once and share it across applications. +Controllers expose your application's capabilities to the LLM as callable tools -- a customer database, an inventory lookup, a weather API, a form-filling routine -- and give you lifecycle hooks that fire before and after each LLM turn. Use a controller when you want reusable tools that don't depend on a specific AI framework's annotations, when you need to defer UI updates until after a multi-step AI response, or when you want to package an AI capability once and share it across applications. If you've registered tool objects via [methodname]`withTools()` elsewhere, controllers are the framework-agnostic equivalent: the same tool calling, but defined through the [classname]`AIController` interface instead of LangChain4j's or Spring AI's [annotationname]`@Tool` annotations. @@ -40,18 +40,18 @@ var orchestrator = AIOrchestrator .build(); ---- -Only one controller can be attached per orchestrator. If you need tools from several sources, compose them into one controller that delegates, or combine a controller with tool objects registered via [methodname]`withTools()` -- the orchestrator merges their tool lists before each request. +Only one controller can be attached per orchestrator. If you need tools from several sources, compose them into one controller that delegates, or combine a controller with tool objects registered via [methodname]`withTools()` -- the orchestrator collects the controller's tools before each request and passes both to the provider, which offers them to the LLM as one tool list. [[database-provider]] == Database Provider -[classname]`DatabaseProvider` is the bridge between the LLM and an application database. The built-in grid and chart controllers use it to let the LLM discover the schema and run SQL queries on demand. Critically, the data flow is asymmetric: the LLM sees the schema so it can write valid queries, but the query results are rendered in the [classname]`Grid` or [classname]`Chart` component only -- they are never sent back to the LLM. +[classname]`DatabaseProvider` is the bridge between the LLM and an application database. The built-in grid and chart controllers use it to let the LLM discover the schema and run SQL queries on demand. Critically, the data flow is asymmetric: the LLM sees the schema so it can write valid queries, but the query results are rendered in the [classname]`Grid` or [classname]`Chart` component -- the rows themselves are never sent back to the LLM. The interface defines two methods: * [methodname]`getSchema()` -- returns a plain-text description of the tables, columns, and SQL dialect. The LLM uses this to write valid queries. -* [methodname]`executeQuery(String sql)` -- executes a SQL query and returns the rows as a list of column-name-to-value maps. These rows are handed to the grid or chart for rendering; they do not appear in any prompt. +* [methodname]`executeQuery(String sql)` -- executes a SQL query and returns the rows as a list of column-name-to-value maps. These rows are handed to the grid or chart for rendering; they do not appear in any prompt as such. The following example is a straightforward JDBC implementation: @@ -110,7 +110,7 @@ The LLM writes the SQL that gets executed. Always back a [classname]`DatabasePro .Query Results Stay in the Application [IMPORTANT] -The LLM receives only the schema, never the query results. Every row returned by [methodname]`executeQuery()` is rendered in the grid or chart component and discarded from the request cycle. This boundary is unconditional: sensitive row values cannot leak into a follow-up prompt, into the conversation history, or to the LLM provider. +The LLM receives the schema, never the query results as rows. Every row returned by [methodname]`executeQuery()` is rendered in the grid or chart component and discarded from the request cycle, so row values don't enter a follow-up prompt or the conversation history. The grid's state tool returns only the current SQL query to the LLM. The chart's state tool returns the chart configuration without series data, but the x-axis categories and the series names in that configuration are derived from the query results: the category column's values and the values of the `_series` grouping column. Keep sensitive values out of the columns the chart uses for categories and series grouping. .Schema Scope [TIP] @@ -126,7 +126,7 @@ The built-in controllers cover grid and chart data exploration. To expose your o * [methodname]`getTools()` -- returns the list of [classname]`LLMProvider.ToolSpec` instances the controller contributes to each LLM request. Tools are collected before every request, so a controller can vary its tool set based on current state. * [methodname]`onRequest(RequestListener.RequestEvent)` -- runs on the UI thread right before the LLM stream opens. Use it to lock UI surfaces, snapshot state the tool definitions depend on, or otherwise prepare for the turn. Since tools may execute on a background thread, this is also the place to capture anything that depends on Vaadin thread locals, such as [methodname]`UI.getCurrent()`. The [since:com.vaadin:vaadin@V25.3]#event# is the same one the <> receives: the user message, the id assigned to it, and its attachments. Use it, for example, to key per-turn state by message id so [methodname]`onResponse()` can find it, or to pick which tools apply based on the attachments. -* [methodname]`onResponse(ResponseListener.ResponseEvent)` -- called through `ui.access()` once the LLM stream has completed, either successfully ([methodname]`event.getError()` is empty) or with an error, so it can safely update components. Controllers use this hook to commit deferred state changes on success and release any per-turn state captured in [methodname]`onRequest()` on failure. The event also carries the provider's [since:com.vaadin:vaadin@V25.3]#response metadata#, which tells a completed turn from one truncated at the model's output limit -- a truncated turn ends without an error, so a controller that commits staged state should check it. See <>. +* [methodname]`onResponse(ResponseListener.ResponseEvent)` -- called through `ui.access()` when the turn ends: once the LLM stream has completed, either successfully ([methodname]`event.getError()` is empty) or with an error, but also when the turn fails before the stream opens, possibly without a preceding [methodname]`onRequest()` call. It doesn't fire for a prompt the <> rejected. Since it runs through `ui.access()`, it can safely update components. Controllers use this hook to commit deferred state changes on success and release any per-turn state captured in [methodname]`onRequest()` on failure. The event also carries the provider's [since:com.vaadin:vaadin@V25.3]#response metadata#, which tells a completed turn from one truncated at the model's output limit -- a truncated turn ends without an error, so a controller that commits staged state should check it. See <>. Each tool is an implementation of [classname]`LLMProvider.ToolSpec`, which has four methods: @@ -184,7 +184,7 @@ public class WeatherController implements AIController { } ---- -Tool names must match the pattern `^[a-zA-Z0-9_-]{1,64}$`, as required by popular LLM APIs. Names are validated at build time; invalid names cause an [classname]`IllegalArgumentException`. Use a prefixed name such as `"MyController_getWeather"` to avoid collisions with tools from other controllers. +Tool names must match the pattern `^[a-zA-Z0-9_-]{1,64}$`, as required by popular LLM APIs. Names and parameter schemas are validated when the controller is registered with [methodname]`withController()`, on the builder and on the reconnector alike. An invalid name, or a schema that isn't a JSON object, causes an [classname]`IllegalArgumentException` that names the tool. Use a prefixed name such as `"MyController_getWeather"` to avoid collisions with tools from other controllers. .Generating JSON Schemas [TIP] @@ -201,7 +201,7 @@ Controllers are not serialized with the orchestrator. After session restore, pas When a tool call fails, the message the LLM receives determines whether it can recover. Given the actual reason, it can correct its next attempt; given a generic error, it tends to retry the same call unchanged. -By default, any exception thrown from tool code is caught, logged, and replaced with a generic error message before reaching the LLM, so internal details such as SQL fragments, schema names, or file paths don't leak into the conversation. To let the LLM see why a call failed, throw a [classname]`ToolException` (from the `com.vaadin.flow.component.ai.provider` package) -- its message is forwarded to the LLM verbatim as the tool's error output: +By default, any exception thrown from tool code is caught, logged, and replaced with a generic error message before reaching the LLM, so internal details such as SQL fragments, schema names, or file paths don't leak into the conversation. To let the LLM see why a call failed, throw a [classname]`ToolException` (from the `com.vaadin.flow.component.ai.provider` package) -- its message is forwarded to the LLM unchanged, behind a short error prefix, as the tool's error output: [source,java] ---- diff --git a/articles/flow/ai-support/conversation-history.adoc b/articles/flow/ai-support/conversation-history.adoc index 81ca5afe5a..27e2a4d914 100644 --- a/articles/flow/ai-support/conversation-history.adoc +++ b/articles/flow/ai-support/conversation-history.adoc @@ -61,7 +61,7 @@ include::{root}/src/main/java/com/vaadin/demo/flow/aicomponents/AIOrchestratorHi endif::[] -- -This replays messages into the LLM provider's memory, populates the Message List UI, and rebuilds internal mappings for attachment click handling. +This replays messages into the LLM provider's memory, populates the Message List UI, and rebuilds internal mappings for attachment click handling. A provider whose memory lives outside the orchestrator, such as a [classname]`SpringAILLMProvider` created from a [classname]`ChatClient`, ignores the provider part of the restore; see <>. [NOTE] [methodname]`getHistory()` returns a point-in-time snapshot. If called while a streaming response is in progress, the snapshot may include the user message without its corresponding assistant response. Use [classname]`ResponseListener` to capture history at the right time. @@ -81,7 +81,7 @@ File attachments are represented as [classname]`AIAttachment` records with [meth == Session Persistence -[classname]`AIOrchestrator` is serializable. Conversation history, UI component bindings, listener registrations, and display names are preserved across serialization. However, the LLM provider and tool objects are transient -- they are not serialized and must be restored after deserialization. +[classname]`AIOrchestrator` is serializable. Conversation history, UI component bindings, listener registrations, and display names are preserved across serialization. However, the LLM provider, tool objects, and the controller are transient -- they are not serialized and must be restored after deserialization. === Reconnecting after Deserialization diff --git a/articles/flow/ai-support/file-attachments.adoc b/articles/flow/ai-support/file-attachments.adoc index 8b8b8a19b2..16654f7d9e 100644 --- a/articles/flow/ai-support/file-attachments.adoc +++ b/articles/flow/ai-support/file-attachments.adoc @@ -27,15 +27,15 @@ var orchestrator = AIOrchestrator .Upload Handler Conflict [NOTE] -When using [classname]`UploadManager` or [classname]`Upload`, the component must not already have an upload handler or receiver set. The orchestrator installs its own in-memory handler. If one is already set, an [classname]`IllegalArgumentException` is thrown at build time. This restriction does not apply to custom [classname]`AIFileReceiver` implementations. +When using [classname]`UploadManager` or [classname]`Upload`, the component must not already have an upload handler set, and an [classname]`Upload` must not have a receiver either. The orchestrator installs its own in-memory handler. If one is already set, [methodname]`withFileReceiver()` throws an [classname]`IllegalArgumentException`. This restriction does not apply to custom [classname]`AIFileReceiver` implementations. .Supported Attachment Types [NOTE] -The following MIME type categories are supported: images (`image/*`), text (`text/*`), PDF (`application/pdf`), audio (`audio/*`), and video (`video/*`). Attachments with other MIME types are silently dropped and not sent to the LLM. To send such a file anyway, convert it to a supported format with a <>. +The following MIME type categories are supported: images (`image/*`), text (`text/*`), PDF (`application/pdf` or `application/x-pdf`), audio (`audio/*`), and video (`video/*`). An attachment with any other MIME type still appears in the Message List, but the built-in providers leave its content out of the request, so the LLM never sees it. To send such a file anyway, convert it to a supported format with a <>. .Duplicate File Names [NOTE] -Uploading multiple files with the same name in a single batch throws an [classname]`IllegalArgumentException`. Ensure each file has a unique name. +Uploading a file whose name matches one that is still pending, that is, uploaded but not yet sent with a prompt, throws an [classname]`IllegalArgumentException`. Removing the earlier file from the upload list frees the name, and once a prompt has consumed the pending files, the name can be used again. == Persisting Attachments diff --git a/articles/flow/ai-support/listeners.adoc b/articles/flow/ai-support/listeners.adoc index a0ceed8807..046478456f 100644 --- a/articles/flow/ai-support/listeners.adoc +++ b/articles/flow/ai-support/listeners.adoc @@ -44,7 +44,7 @@ The listener runs on the UI thread under the session lock: components can be upd == Response Listener -The response listener is called when the turn ends -- normally when the assistant's response has completed, successfully or with an error, but also when the turn fails before a response ever starts. It fires at most once per prompt, and not at all for a prompt rejected by the interceptor or for history restored via [methodname]`withHistory()`. +The response listener is called when the turn ends -- normally when the assistant's response has completed, successfully or with an error, but also when the turn fails before a response ever starts. It fires at most once per prompt, and not at all for a prompt rejected by the interceptor, for a postponed prompt abandoned because its UI was detached, or for history restored via [methodname]`withHistory()`. The event carries: @@ -78,7 +78,7 @@ When a turn contains <> round trips, the finish reason [[truncated-responses]] === Truncated Responses -A response cut off at the model's output limit is not an error: the turn ends normally, the partial text stays in the Message List and the conversation history, and [methodname]`event.getError()` is empty. The finish reason is what tells a completed turn from a truncated one: a truncated turn reports the underlying framework's value for the output limit instead of its value for a natural stop. With the built-in providers, that value is `LENGTH` for OpenAI models through Spring AI and for every model through LangChain4j, and `max_tokens` for Anthropic models through Spring AI -- see <<#finish-reason-vocabulary,Finish Reason Vocabulary>>: +A response cut off at the model's output limit is not an error: the turn ends normally, the partial text stays in the Message List and the conversation history, and [methodname]`event.getError()` is empty. The finish reason is what tells a completed turn from a truncated one: a truncated turn reports the underlying framework's value for the output limit instead of its value for a natural stop. With the built-in providers, that value is `LENGTH` for OpenAI models through Spring AI and for OpenAI and Anthropic models through LangChain4j, and `max_tokens` for Anthropic models through Spring AI -- see <<#finish-reason-vocabulary,Finish Reason Vocabulary>>: [source,java] ---- @@ -86,7 +86,7 @@ A response cut off at the model's output limit is not an error: the turn ends no boolean truncated = event.getMetadata() .map(ResponseMetadata::finishReason) // Output-limit reason for OpenAI models through Spring AI, - // and for every model through LangChain4j + // and for OpenAI and Anthropic models through LangChain4j .filter("LENGTH"::equals) .isPresent(); if (truncated) { @@ -114,6 +114,8 @@ The two built-in providers relay the following: The built-in providers publish metadata as the turn progresses, and each publication replaces the previous one. A turn that fails or times out midway therefore still reports what was observed before the failure: [methodname]`event.getError()` carries the cause and the metadata describes the turn as far as it got. How far that is depends on which side runs the <> loop. [classname]`LangChain4JLLMProvider` runs the loop itself and publishes after every round trip, so a turn that fails after a tool call still reports the token usage of the round trips that did complete. Spring AI runs the loop inside the framework and hands [classname]`SpringAILLMProvider` only the final response, so the round trips before it are never published: a turn that fails before the final response reports nothing, in synchronous and streaming mode alike. In streaming mode, the chunks of the final response that arrived before the failure are still reported. -The built-in providers also log a warning when a turn ends in a state a completed turn can't end in: without a finish reason, or with tool calls still pending. +Both built-in providers log a warning when a turn ends without a finish reason. [classname]`SpringAILLMProvider` also warns when a turn ends with tool calls still pending, which happens when Spring AI's tool-calling loop stops before the model answers; [classname]`LangChain4JLLMProvider` runs that loop itself, so it can't end a turn that way. + +A turn that [classname]`LangChain4JLLMProvider` ends because the model requested more tool calls than allowed fails with a [classname]`ToolCallLimitExceededException` as its error; see <>. Spring AI stops its loop at the same limits without failing the turn and reports the finish reason `toolCallLimitExceeded` instead. A custom [classname]`LLMProvider` decides itself what metadata to publish -- see <>. diff --git a/articles/flow/ai-support/llm-providers.adoc b/articles/flow/ai-support/llm-providers.adoc index 42af70d07e..61ff348a69 100644 --- a/articles/flow/ai-support/llm-providers.adoc +++ b/articles/flow/ai-support/llm-providers.adoc @@ -14,7 +14,7 @@ An AI framework -- such as Spring AI or LangChain4j -- is a Java library that ha .Memory Window Limit [IMPORTANT] -Both built-in providers maintain a 30-message memory window. Older messages are evicted from the provider's working memory. The orchestrator's [methodname]`getHistory()` retains the full conversation, but the LLM only sees the most recent 30 messages. +Both built-in providers maintain a 30-message memory window of their own, except a [classname]`SpringAILLMProvider` created from a [classname]`ChatClient`, where the application owns the memory. Older messages are evicted from the provider's working memory. The orchestrator's [methodname]`getHistory()` retains the full conversation, but the LLM only sees the most recent 30 messages. == Spring AI @@ -71,6 +71,8 @@ The provider manages its own conversation memory using a 30-message window. Synchronous mode blocks the UI for the duration of each exchange; see <<#background-execution,Background Execution>>. +The provider runs the tool-calling loop itself and bounds the number of tool calls per turn; see <>. + [[framework-features]] == Framework Features @@ -157,11 +159,11 @@ Background execution moves the exchange to a background thread instead. Enable i provider.setBackgroundExecution(true); ---- -The user's message and an empty assistant message then appear immediately, the UI stays responsive, and the response is filled in when the model finishes. The setting is off by default. It's read for each prompt, so it can be changed at any time; the next prompt uses the new mode. It has no effect with a streaming model, whose response already arrives on the LLM client's own threads. +The user's message then appears immediately and the Message List shows a typing indicator while the model works, the UI stays responsive, and the response replaces the indicator when the model finishes. The setting is off by default. It's read for each prompt, so it can be changed at any time; the next prompt uses the new mode. It has no effect with a streaming model, whose response already arrives on the LLM client's own threads. -The response is now produced outside any request, so it reaches the browser through server push or polling. Enable push by annotating the application shell with [annotationname]`@Push` (see <>), or enable polling with [methodname]`UI.setPollInterval()`. Without either, the response only shows up with the next request the browser happens to make -- the page looks stuck even though the turn completed on the server. The provider logs a warning, once per provider instance, when neither is active. Manual push mode is not enough on its own, because nothing in the framework calls `ui.push()` for the application. +The response is now produced outside any request, so it reaches the browser through server push or polling. Enable push by annotating the application shell with [annotationname]`@Push` (see <>), or enable polling with [methodname]`UI.setPollInterval()`. Without either, the response only shows up with the next request the browser happens to make -- the page looks stuck even though the turn completed on the server. The provider logs a warning, once per provider instance, when neither is active; the same warning applies in streaming mode, whose response also arrives outside a request. Manual push mode is not enough on its own, because nothing in the framework calls `ui.push()` for the application. -Everything that happens before the model is called still runs in the request thread: the <>, adding the user's message and the empty assistant message to the Message List, [methodname]`AIController.onRequest()`, the request listener, and the session context supplier. The model calls, every tool execution, and the [classname]`ResponseListener` run on the background thread, where [methodname]`UI.getCurrent()` and other Vaadin thread locals return `null` and components must not be touched directly. Thread-bound framework state, such as Spring Security's [classname]`SecurityContext`, is absent there for the same reason. +Everything that happens before the model is called still runs in the request thread: the <>, adding the user's message to the Message List and showing the typing indicator, [methodname]`AIController.onRequest()`, the request listener, and the session context supplier. The model calls, every tool execution, and the [classname]`ResponseListener` run on the background thread, where [methodname]`UI.getCurrent()` and other Vaadin thread locals return `null` and components must not be touched directly. Thread-bound framework state, such as Spring Security's [classname]`SecurityContext`, is absent there for the same reason. Wrap component access in `ui.access()`, or capture what a tool needs in [methodname]`AIController.onRequest()` while the request thread is still current -- see <> and <>. The built-in controllers already handle this. [methodname]`AIController.onResponse()` is the exception: the orchestrator calls it through `ui.access()`, so it can update components directly. @@ -184,11 +186,13 @@ public class MyLLMProvider implements LLMProvider { @Override public Flux stream(LLMRequest request) { // Return a reactive stream of response tokens - // request.userMessage() -- the user's prompt - // request.attachments() -- any file attachments - // request.systemPrompt() -- the system prompt - // request.tools() -- registered tool objects - // request.metadataSink() -- consumer for response metadata + // request.userMessage() -- the user's prompt + // request.attachments() -- any file attachments + // request.systemPrompt() -- the system prompt + // request.tools() -- tool objects registered via withTools() + // request.explicitTools() -- ToolSpec tools contributed by the controller + // request.sessionContext() -- per-turn session context, or null + // request.metadataSink() -- consumer for response metadata } @Override @@ -199,6 +203,8 @@ public class MyLLMProvider implements LLMProvider { } ---- +A provider is responsible for its own conversation memory, for the tool-calling loop over both the vendor-annotated tool objects and the framework-agnostic [classname]`ToolSpec` tools (see <>), and for appending the <> to the user message when one is given. + The response stream carries text only. The provider can also [since:com.vaadin:vaadin@V25.3]#publish the finish reason and token usage# of the turn through the [methodname]`metadataSink()` consumer on the request. Each call carries everything observed so far and replaces the value of any earlier call, so publish whenever the provider learns more -- a turn that fails midway has then still reported what was observed. Pass `null` for any value the framework doesn't report; a provider that observes no metadata never calls the consumer. See <> for how applications read it. The orchestrator calls [methodname]`stream()` on the thread that triggers the prompt and subscribes to the returned stream on that same thread -- whether a turn runs in the background is decided entirely by the implementation. An implementation whose LLM call blocks should schedule that call itself; otherwise it occupies the request thread and holds the session lock for the whole turn. See <<#background-execution,Background Execution>> for how the built-in providers expose this as a setting. diff --git a/articles/flow/ai-support/session-context.adoc b/articles/flow/ai-support/session-context.adoc index dba88d3125..1e40a1004d 100644 --- a/articles/flow/ai-support/session-context.adoc +++ b/articles/flow/ai-support/session-context.adoc @@ -39,7 +39,7 @@ AIOrchestrator.builder(provider, systemPrompt) .build(); ---- -The supplier is invoked once per turn, right before the request goes out. The returned string is passed to the model without further interpretation; compose multiple pieces of context with plain string concatenation. Setting a supplier replaces the default current-date-and-time line; include it yourself if you still want it: +The supplier is invoked once per turn, right before the request goes out. The orchestrator doesn't interpret the returned string; compose multiple pieces of context with plain string concatenation. The built-in providers append it to the end of the user message in a delimited `` block, with a short note telling the model that the block comes from the application. Only the user's own text appears in the Message List and in the conversation history. Setting a supplier replaces the default current-date-and-time line; include it yourself if you still want it: [source,java] ---- @@ -76,9 +76,9 @@ AIOrchestrator.builder(provider, systemPrompt) The supplier runs on the UI thread under the session lock, so it can read [methodname]`UI.getCurrent()` and any session-scoped state. Keep it fast -- anything that touches a database, a remote service, or the file system should be done outside the supplier and cached in session-scoped state. -A supplier that throws an exception aborts the turn through the normal error path: the assistant placeholder is replaced with a generic error message, the [classname]`ResponseListener` (if registered) fires with the thrown exception, and the exception propagates to the caller of the prompt entry point. +A supplier that throws an exception aborts the turn through the normal error path: the Message List shows a generic error message under the assistant name, the [classname]`ResponseListener` (if registered) fires with the thrown exception, and the exception propagates to the caller of the prompt entry point. == Why Not the System Prompt -The system prompt is sent verbatim on every turn. Many LLM providers use it as the cache prefix to keep latency and cost down between calls; putting per-turn values such as the current time or the active tenant into the system prompt invalidates that cache on every request. Session context is delivered through a separate per-turn channel that lives outside the system prompt, so the system prompt stays cacheable. +The system prompt is sent unchanged on every turn. Many LLM providers use it as the cache prefix to keep latency and cost down between calls; putting per-turn values such as the current time or the active tenant into the system prompt invalidates that cache on every request. Session context travels with the user message instead, after the user's text, so the system prompt and the tool definitions stay identical from turn to turn and the prefix cache keeps hitting. diff --git a/articles/flow/ai-support/tool-calling.adoc b/articles/flow/ai-support/tool-calling.adoc index 44c2b3ead8..f818c993cb 100644 --- a/articles/flow/ai-support/tool-calling.adoc +++ b/articles/flow/ai-support/tool-calling.adoc @@ -52,7 +52,7 @@ With a streaming provider or <> instead. [classname]`GridAIController` and [classname]`ChartAIController` are built-in examples. Controllers and tool objects can be combined on the same orchestrator. +For a reusable set of tools that does not depend on a specific LLM framework's annotations, or when a lifecycle hook is needed after each LLM request cycle, implement <> instead. [classname]`GridAIController`, [classname]`ChartAIController`, and [classname]`FormAIController` are built-in examples. Controllers and tool objects can be combined on the same orchestrator. .Tool Errors [NOTE] @@ -63,6 +63,38 @@ Tool objects registered via [methodname]`withTools()` are executed by the vendor The tool-calling loop of a turn is bounded: by default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Exceeding a limit fails the turn with a [classname]`ToolCallLimitExceededException`, with either built-in provider. See <>. +[[tool-call-limits]] +[role="since:com.vaadin:vaadin@V25.3"] +== Tool Call Limits + +A model that keeps requesting tool calls instead of answering, for example to look for something the application doesn't have, would never end its turn, and each round costs another model call. With both built-in providers, the tool-calling loop is bounded at the same defaults: 40 calls to any one tool, and 150 tool calls in total, within a turn. Whether a limit ends the turn with an error or not depends on which side runs the loop. + +=== LangChain4j + +[classname]`LangChain4JLLMProvider` runs the tool-calling loop itself and counts the tool calls the model requests during a turn. Once a call would exceed a limit, the turn fails with a [classname]`ToolCallLimitExceededException`: none of the tool calls of that round are executed, the model is not called again, and the exception arrives as the error of the turn in [methodname]`ResponseListener.ResponseEvent.getError()` and in [methodname]`AIController.onResponse()`. The Message List shows the generic error message in place of a response. The exception's [methodname]`getToolName()` names the tool whose per-tool limit was exceeded, or is `null` when the limit on all tool calls of the turn was exceeded instead, and [methodname]`getLimit()` returns the limit in question. + +The refused round never enters the provider's chat memory, so the conversation stays usable, and the next prompt continues from the last completed round. Metadata observed before the failure is still published, so the turn reports the token usage of the round trips that did complete. + +Adjust the limits on the provider with [methodname]`setMaxCallsPerTool()` and [methodname]`setMaxTotalToolCalls()`. Passing `0` removes a limit, and a negative value throws an [classname]`IllegalArgumentException`. The values are read when a turn starts, so a change applies from the next prompt on: + +[source,java] +---- +LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel); +provider.setMaxCallsPerTool(10); +provider.setMaxTotalToolCalls(30); +---- + +=== Spring AI + +[classname]`SpringAILLMProvider` adds no limits of its own and never throws [classname]`ToolCallLimitExceededException`. Spring AI runs the tool-calling loop itself and, from version 2.0.1, stops it at the same limits without failing the turn: the turn completes with Spring AI's own message about the exceeded limit as the assistant's reply, which also enters chat memory, and with the finish reason `toolCallLimitExceeded` in the <>. Check that finish reason to tell such a turn from a completed one. Spring AI 2.0.0 applies no limit, so a turn can loop on tool calls until the model stops on its own. + +The limits belong to the `ToolCallingAdvisor` of the [classname]`ChatClient`. A provider created from a [classname]`ChatModel` builds its own client and keeps Spring AI's defaults. To change them, build the client yourself, passing `ChatClient.builder` a `ToolCallingAdvisor.Builder` that carries a `DefaultToolCallingManager` with your limits, and create the provider from that client with `new SpringAILLMProvider(chatClient)`. Its `maxCallsPerTool` and `maxTotalToolCalls` set a limit, and `unlimitedCallsPerTool()` and `unlimitedTotalToolCalls()` remove one. In a Spring Boot application, the `spring.ai.tools.limits` properties configure the same limits on the auto-configured `ChatClient.Builder`, so passing that client to the provider needs no builder code. + +.Custom Providers +[NOTE] +A <> that runs its own tool-calling loop decides itself whether and how to bound it. The [classname]`ToolCallLimitExceededException` constructor is public, so such a provider can end a turn with the same exception, and a [classname]`ResponseListener` or [classname]`AIController` can be unit-tested against it. + + == Programmatic Prompts Send prompts without a Message Input component using [methodname]`prompt()`. This is useful for triggering AI interaction from button clicks or other events: From 268a419502b963eb2db34ae85b7578687fa61815 Mon Sep 17 00:00:00 2001 From: Tomi Virkki Date: Mon, 21 Sep 2026 12:29:01 +0300 Subject: [PATCH 2/2] docs: align the AI integration documentation with the tool call limits and chart state behavior Co-Authored-By: Claude Fable 5.1 --- .../flow/ai-support/ai-powered-chart.adoc | 2 +- articles/flow/ai-support/controllers.adoc | 2 +- articles/flow/ai-support/listeners.adoc | 4 +- articles/flow/ai-support/llm-providers.adoc | 43 ++----------------- articles/flow/ai-support/tool-calling.adoc | 30 +++++++------ 5 files changed, 26 insertions(+), 55 deletions(-) diff --git a/articles/flow/ai-support/ai-powered-chart.adoc b/articles/flow/ai-support/ai-powered-chart.adoc index 79a9f817a7..f806455983 100644 --- a/articles/flow/ai-support/ai-powered-chart.adoc +++ b/articles/flow/ai-support/ai-powered-chart.adoc @@ -17,7 +17,7 @@ include::{articles}/_commercial-banner.adoc[opts=optional] Data and configuration are kept separate: series data comes from SQL queries, while visual appearance comes from the configuration. Both updates are applied together at the end of the LLM turn, so the user never sees a half-updated chart. -The LLM can read the chart's current configuration and queries through a state tool. The configuration is returned without series data, but its x-axis categories and series names are derived from the query results, so those values are visible to the LLM. See <>. +The LLM can read the chart's current configuration and queries through a state tool. The configuration is returned without series data, but its x-axis categories, series names, and the nodes of an organization chart are derived from the query results, so those values are visible to the LLM. See <>. == Basic Usage diff --git a/articles/flow/ai-support/controllers.adoc b/articles/flow/ai-support/controllers.adoc index c9d8b33995..bc38506b36 100644 --- a/articles/flow/ai-support/controllers.adoc +++ b/articles/flow/ai-support/controllers.adoc @@ -110,7 +110,7 @@ The LLM writes the SQL that gets executed. Always back a [classname]`DatabasePro .Query Results Stay in the Application [IMPORTANT] -The LLM receives the schema, never the query results as rows. Every row returned by [methodname]`executeQuery()` is rendered in the grid or chart component and discarded from the request cycle, so row values don't enter a follow-up prompt or the conversation history. The grid's state tool returns only the current SQL query to the LLM. The chart's state tool returns the chart configuration without series data, but the x-axis categories and the series names in that configuration are derived from the query results: the category column's values and the values of the `_series` grouping column. Keep sensitive values out of the columns the chart uses for categories and series grouping. +The LLM receives the schema, never the query results as rows. Every row returned by [methodname]`executeQuery()` is rendered in the grid or chart component and discarded from the request cycle, so row values don't enter a follow-up prompt or the conversation history. The grid's state tool returns only the current SQL query to the LLM. The chart's state tool returns the chart configuration without series data, but some of that configuration is derived from the query results: the x-axis categories come from the category column's values, the series names from the values of the `_series` grouping column, and the nodes of an organization chart carry the `_id`, `_name`, `_title`, `_description`, and `_image` values of every row. Keep sensitive values out of those columns. .Schema Scope [TIP] diff --git a/articles/flow/ai-support/listeners.adoc b/articles/flow/ai-support/listeners.adoc index 046478456f..f2c8ac0ed1 100644 --- a/articles/flow/ai-support/listeners.adoc +++ b/articles/flow/ai-support/listeners.adoc @@ -49,7 +49,7 @@ The response listener is called when the turn ends -- normally when the assistan The event carries: * [methodname]`getResponse()` -- the assistant's response text. On success, it may be empty when the model emitted only tool calls; on failure, it's always empty -- partial text received before the error isn't passed on, and the Message List shows a generic error message in its place. Empty responses are not appended to the conversation history. -* [methodname]`getError()` -- the failure cause, or an empty optional on success. A turn stopped at a <> fails with a [classname]`ToolCallLimitExceededException`. +* [methodname]`getError()` -- the failure cause, or an empty optional on success. A turn stopped at a <> fails with a [classname]`ToolCallLimitExceededException`. * [methodname]`getMetadata()` -- the provider's metadata for the turn; see <<#response-metadata,Response Metadata>>. A typical use is persisting the conversation after each exchange -- see <>. @@ -116,6 +116,6 @@ The built-in providers publish metadata as the turn progresses, and each publica Both built-in providers log a warning when a turn ends without a finish reason. [classname]`SpringAILLMProvider` also warns when a turn ends with tool calls still pending, which happens when Spring AI's tool-calling loop stops before the model answers; [classname]`LangChain4JLLMProvider` runs that loop itself, so it can't end a turn that way. -A turn that [classname]`LangChain4JLLMProvider` ends because the model requested more tool calls than allowed fails with a [classname]`ToolCallLimitExceededException` as its error; see <>. Spring AI stops its loop at the same limits without failing the turn and reports the finish reason `toolCallLimitExceeded` instead. +A turn that ends because the model requested more tool calls than allowed fails with a [classname]`ToolCallLimitExceededException` as its error, with either built-in provider; see <>. With [classname]`SpringAILLMProvider`, the metadata of such a turn also carries the finish reason `toolCallLimitExceeded` that Spring AI puts on the reply it stops the loop with. A custom [classname]`LLMProvider` decides itself what metadata to publish -- see <>. diff --git a/articles/flow/ai-support/llm-providers.adoc b/articles/flow/ai-support/llm-providers.adoc index 61ff348a69..cb023e6edb 100644 --- a/articles/flow/ai-support/llm-providers.adoc +++ b/articles/flow/ai-support/llm-providers.adoc @@ -90,7 +90,7 @@ ChatClient chatClient = ChatClient.builder(chatModel) SpringAILLMProvider provider = new SpringAILLMProvider(chatClient); ---- -The MCP tools then reach the model on every turn, next to any tools registered on the orchestrator through <> or a <>. The model calls all of them the same way, so tool names have to be unique across every source and match `^[a-zA-Z0-9_-]{1,64}$` -- validated at build time for controller tools, but not for names that come from an MCP server. Give the client a chat memory advisor and a default conversation id as well, since a provider created from a [classname]`ChatClient` leaves conversation memory to the application. +The MCP tools then reach the model on every turn, next to any tools registered on the orchestrator through <> or a <>. The model calls all of them the same way, so tool names have to be unique across every source and match `^[a-zA-Z0-9_-]{1,64}$` -- validated when a controller is registered, but not for names that come from an MCP server. Give the client a chat memory advisor and a default conversation id as well, since a provider created from a [classname]`ChatClient` leaves conversation memory to the application. Retrieval-augmented generation against a vector store, guardrails and moderation, observability, model options such as temperature and token limits, and structured output with schema validation all work the same way: configure them on the framework's client, then wrap it in a provider. The framework's own reference documentation is where to look for the details: @@ -106,44 +106,9 @@ The orchestrator sends its own system prompt on every turn, which replaces any [ [classname]`LangChain4JLLMProvider` is built from a [classname]`ChatModel` or [classname]`StreamingChatModel` and runs the tool-calling loop itself, so LangChain4j's [classname]`McpToolProvider` -- which attaches to an AI service built with [classname]`AiServices` -- has nowhere to plug in. With LangChain4j, MCP tools currently require a <<#custom-llm-providers,custom LLM provider>> built around an AI service, which then gets the tool provider, memory, and tool loop from LangChain4j. With Spring AI, the [classname]`ChatClient` setup above is all that's needed. -[[tool-call-limits]] -[role="since:com.vaadin:vaadin@V25.3"] -== Tool Call Limits - -A turn that uses <> is a loop: the model asks for tool calls, the provider runs them and calls the model again with the results, until the model answers. Both built-in providers bound that loop per turn. By default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Without a bound, a request the model can't satisfy keeps calling the model and the tools until the application is stopped. - -When a limit is exceeded, the turn fails with a [classname]`ToolCallLimitExceededException`, whichever provider runs it. The Message List shows its generic error message, and the exception reaches the <> and [methodname]`AIController.onResponse()` as the error of the turn. Its message names the limit that was exceeded and, for a per-tool limit, the tool: - -[source,java] ----- -.withResponseListener(event -> { - event.getError() - .filter(ToolCallLimitExceededException.class::isInstance) - .ifPresent(error -> log.warn( - "Turn stopped: {}", error.getMessage())); -}) ----- - -=== LangChain4j - -[classname]`LangChain4JLLMProvider` runs the loop itself and enforces the limits. Change them with [methodname]`setMaxCallsPerTool()` and [methodname]`setMaxTotalToolCalls()`. A value of `0` removes that limit. The values are read when a turn starts, so a change applies from the next prompt. - -[source,java] ----- -LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel); -provider.setMaxCallsPerTool(10); -provider.setMaxTotalToolCalls(0); // No limit across all tools ----- - -The provider checks the limits before running the tool calls of a response. When a call would take a count past its limit, none of the tool calls in that response run, and the model isn't called again. The failed turn leaves only its prompt in the provider's memory: tool calls and their results are sent to the model only within the turn they belong to and are never kept between turns, so the next prompt continues from where the last completed turn left the conversation. - -=== Spring AI - -[classname]`SpringAILLMProvider` adds no limit of its own. Spring AI bounds the loop with the same defaults, 40 calls per tool and 150 in total per turn, with either constructor of the provider. When Spring AI stops the loop, the provider fails the turn with the [classname]`ToolCallLimitExceededException` described above, carrying Spring AI's own message about the limit. That message may remain in the chat memory, with either constructor: the provider doesn't rewrite what Spring AI's advisors stored. - -The limits belong to the [classname]`ChatClient`, so tune or remove them there and pass the client to the [classname]`ChatClient` constructor of the provider; the [classname]`ChatModel` constructor builds a client with Spring AI's defaults. How to configure them is described under https://docs.spring.io/spring-ai/reference/api/tools.html#tool-call-limits[Tool Call Limits] in the Spring AI documentation. - -A <<#custom-llm-providers,custom provider>> decides itself whether, and how, it bounds its loop. +.Tool Call Limits +[NOTE] +Both built-in providers bound the tool-calling loop of a turn: by default, at most 40 calls per tool and 150 in total. [classname]`LangChain4JLLMProvider` enforces the limits itself and has setters for them, while [classname]`SpringAILLMProvider` leaves them to the [classname]`ChatClient`. See <>. [[background-execution]] diff --git a/articles/flow/ai-support/tool-calling.adoc b/articles/flow/ai-support/tool-calling.adoc index f818c993cb..bbb2ad182f 100644 --- a/articles/flow/ai-support/tool-calling.adoc +++ b/articles/flow/ai-support/tool-calling.adoc @@ -58,37 +58,43 @@ For a reusable set of tools that does not depend on a specific LLM framework's a [NOTE] Tool objects registered via [methodname]`withTools()` are executed by the vendor framework, whose own error handling decides what the LLM sees when a tool throws -- by default, both LangChain4j and Spring AI relay the raw message of any exception. To control what the LLM learns about failures, define the tool through a controller instead and throw a [classname]`ToolException` for messages the LLM is meant to see; see <>. -.Tool Call Limits -[NOTE] -The tool-calling loop of a turn is bounded: by default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Exceeding a limit fails the turn with a [classname]`ToolCallLimitExceededException`, with either built-in provider. See <>. - [[tool-call-limits]] [role="since:com.vaadin:vaadin@V25.3"] == Tool Call Limits -A model that keeps requesting tool calls instead of answering, for example to look for something the application doesn't have, would never end its turn, and each round costs another model call. With both built-in providers, the tool-calling loop is bounded at the same defaults: 40 calls to any one tool, and 150 tool calls in total, within a turn. Whether a limit ends the turn with an error or not depends on which side runs the loop. +A turn that uses tools is a loop: the model asks for tool calls, the provider runs them and calls the model again with the results, until the model answers. Both built-in providers bound that loop per turn. By default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Without a bound, a request the model can't satisfy keeps calling the model and the tools until the application is stopped. The limits count every tool the model can call, whether it comes from [methodname]`withTools()`, a <>, or an <>. -=== LangChain4j +When a limit is exceeded, the turn fails with a [classname]`ToolCallLimitExceededException`, whichever provider runs it. The Message List shows its generic error message, and the exception reaches the <> and [methodname]`AIController.onResponse()` as the error of the turn. Its message names the limit that was exceeded and, for a per-tool limit, the tool: -[classname]`LangChain4JLLMProvider` runs the tool-calling loop itself and counts the tool calls the model requests during a turn. Once a call would exceed a limit, the turn fails with a [classname]`ToolCallLimitExceededException`: none of the tool calls of that round are executed, the model is not called again, and the exception arrives as the error of the turn in [methodname]`ResponseListener.ResponseEvent.getError()` and in [methodname]`AIController.onResponse()`. The Message List shows the generic error message in place of a response. The exception's [methodname]`getToolName()` names the tool whose per-tool limit was exceeded, or is `null` when the limit on all tool calls of the turn was exceeded instead, and [methodname]`getLimit()` returns the limit in question. +[source,java] +---- +.withResponseListener(event -> { + event.getError() + .filter(ToolCallLimitExceededException.class::isInstance) + .ifPresent(error -> log.warn( + "Turn stopped: {}", error.getMessage())); +}) +---- -The refused round never enters the provider's chat memory, so the conversation stays usable, and the next prompt continues from the last completed round. Metadata observed before the failure is still published, so the turn reports the token usage of the round trips that did complete. +=== LangChain4j -Adjust the limits on the provider with [methodname]`setMaxCallsPerTool()` and [methodname]`setMaxTotalToolCalls()`. Passing `0` removes a limit, and a negative value throws an [classname]`IllegalArgumentException`. The values are read when a turn starts, so a change applies from the next prompt on: +[classname]`LangChain4JLLMProvider` runs the loop itself and enforces the limits. Change them with [methodname]`setMaxCallsPerTool()` and [methodname]`setMaxTotalToolCalls()`. A value of `0` removes that limit, and a negative value throws an [classname]`IllegalArgumentException`. The values are read when a turn starts, so a change applies from the next prompt. [source,java] ---- LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel); provider.setMaxCallsPerTool(10); -provider.setMaxTotalToolCalls(30); +provider.setMaxTotalToolCalls(0); // No limit across all tools ---- +The provider checks the limits before running the tool calls of a response. When a call would take a count past its limit, none of the tool calls in that response run, and the model isn't called again. The failed turn leaves only its prompt in the provider's memory: tool calls and their results are sent to the model only within the turn they belong to and are never kept between turns, so the next prompt continues from where the last completed turn left the conversation. Metadata observed before the failure is still published, so the turn reports the token usage of the round trips that did complete. + === Spring AI -[classname]`SpringAILLMProvider` adds no limits of its own and never throws [classname]`ToolCallLimitExceededException`. Spring AI runs the tool-calling loop itself and, from version 2.0.1, stops it at the same limits without failing the turn: the turn completes with Spring AI's own message about the exceeded limit as the assistant's reply, which also enters chat memory, and with the finish reason `toolCallLimitExceeded` in the <>. Check that finish reason to tell such a turn from a completed one. Spring AI 2.0.0 applies no limit, so a turn can loop on tool calls until the model stops on its own. +[classname]`SpringAILLMProvider` adds no limit of its own. Spring AI bounds the loop with the same defaults, 40 calls per tool and 150 in total per turn, with either constructor of the provider. When Spring AI stops the loop, the provider fails the turn with the [classname]`ToolCallLimitExceededException` described above, carrying Spring AI's own message about the limit. The finish reason `toolCallLimitExceeded` that Spring AI puts on that reply is still published in the <>, and the reply may remain in the chat memory, with either constructor: the provider doesn't rewrite what Spring AI's advisors stored. -The limits belong to the `ToolCallingAdvisor` of the [classname]`ChatClient`. A provider created from a [classname]`ChatModel` builds its own client and keeps Spring AI's defaults. To change them, build the client yourself, passing `ChatClient.builder` a `ToolCallingAdvisor.Builder` that carries a `DefaultToolCallingManager` with your limits, and create the provider from that client with `new SpringAILLMProvider(chatClient)`. Its `maxCallsPerTool` and `maxTotalToolCalls` set a limit, and `unlimitedCallsPerTool()` and `unlimitedTotalToolCalls()` remove one. In a Spring Boot application, the `spring.ai.tools.limits` properties configure the same limits on the auto-configured `ChatClient.Builder`, so passing that client to the provider needs no builder code. +The limits belong to the [classname]`ChatClient`, so tune or remove them there and pass the client to the [classname]`ChatClient` constructor of the provider; the [classname]`ChatModel` constructor builds a client with Spring AI's defaults. In a Spring Boot application, the `spring.ai.tools.limits` properties configure them on the auto-configured [classname]`ChatClient.Builder`. How to configure them is described under https://docs.spring.io/spring-ai/reference/api/tools.html#tool-call-limits[Tool Call Limits] in the Spring AI documentation. .Custom Providers [NOTE]