Support json schemas to force the model to generate valid JSON

[k33g]

From the Llamacpp documentation:

GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp. For example, you can use it to force the model to generate valid JSON, or speak only in emojis.

See: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md#json-schemas--gbnf

Usage examples

Extract data

schema:

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "capital": {
      "type": "string"
    },
    "languages": {
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  },
  "required": ["name", "capital", "languages"]
}

prompt:

Tell me about Canada

response:

{
  "name": "Canada",
  "capital": "Ottawa",
  "languages": ["English", "French"]
}

Intent / Routing

schema:

{
  "type": "object",
  "properties": {
    "action": {
      "type": "string",
      "description": "The action to take with the selected agent",
      "enum": ["talk to", "ignore", "call"]
    },
    "agent_name": {
      "type": "string",
      "description": "The name of the agent to route the request to",
      "enum": ["Bob", "Sam", "John"]
    }
  },
  "required": ["action", "agent_name"]
}

system prompt:

You are an AI assistant helping route user requests to the appropriate agent.

Available agents:
- Bob: Handles sales inquiries
- Sam: Handles technical support
- John: Handles billing questions

Available actions:
- "talk to": Direct conversation with the agent
- "ignore": Skip this request
- "call": Schedule a phone call with the agent

Based on the user's message, determine which action to take and which agent to contact.
Respond with the appropriate action and agent name.

User message:

I need help with my invoice"

response:

{
  "action": "talk to",
  "agent_name": "John"
}

[ardan-bkennedy]

@k33g Kronk supports Grammars and there is an example for this

https://github.com/ardanlabs/kronk/blob/main/examples/grammar/main.go