-
Notifications
You must be signed in to change notification settings - Fork 0
Description
One common pattern for LLM apps is to keep calling generateWithOptionalTools until the LLM has decided it's used enough tools to finally generate a natural language output.
This pattern is subject to a halting problem (the LLM may never decide it is satisfied, and may not be making progress toward a discernable goal - it may even lose track of the goal over time).
However the pattern is common enough (especially in quick prototypes) that it's worth supporting a simple way of doing it. Our current code sample for this is kinda ugly and verbose, so we'd like to provide a high-level looping function in Kurt that can accept:
- the standard options for generating (prompt, extraMessages, etc)
- a map of named tool parameter schemas
- a map of named tool functions that take those data types as inputs
- some additional settings, including:
- a maximum number of iterations to allow before calling a final time with no tools (to force natural language output)
- perhaps a way to remove tools that have already been called? (or been called a certain number of times?)
- perhaps some control over if/how/what messages get added into the extraMessages history
- whatever other options make sense
The output should get streamed just like we do now with KurtStream. However we need some way to have multiple consecutive streams, because we'll get multiple finished messages for each stream. Maybe an AsyncIterable<KurtStream> is the right return type.