dactyl-generation is a Python library developed to support batch inference text generation from different LLM APIs.
Batch inference enables sending multiple prompts at once, often at a 50% discount compared to standard streaming or live APIs.
pip install dactyl-generationCheck the examples/ directories to see how to use different batch inference providers seamlessly.
We support the following LLM API providers.
- OpenAI
- Anthropic
- Mistral
- Google (Gemini)
- AWS Bedrock
- Fireworks AI
- Together AI
Icons by Icons8.