> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anyapi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LiteLLM Integration

> Use AnyAPI with LiteLLM for unified access to multiple language models

# LiteLLM Integration

LiteLLM provides a unified interface to call 100+ language models using the OpenAI format. AnyAPI integrates seamlessly with LiteLLM, allowing you to access all AnyAPI models through LiteLLM's standardized interface.

## Overview

LiteLLM simplifies working with multiple LLM providers by:

* **Unified API format** - Use OpenAI's format for all models
* **Automatic retries** - Built-in retry logic and error handling
* **Cost tracking** - Monitor usage and costs across providers
* **Fallback support** - Automatically failover between models
* **Streaming support** - Real-time response streaming

<CardGroup cols={2}>
  <Card title="Easy Setup" icon="rocket">
    Single installation, unified interface
  </Card>

  <Card title="Cost Tracking" icon="dollar-sign">
    Built-in usage and cost monitoring
  </Card>

  <Card title="Error Handling" icon="shield">
    Automatic retries and fallbacks
  </Card>

  <Card title="Streaming" icon="bolt">
    Real-time response streaming
  </Card>
</CardGroup>

## Installation

Install LiteLLM via pip:

```bash theme={"system"}
pip install litellm
```

For additional features like cost tracking and logging:

```bash theme={"system"}
pip install 'litellm[proxy]'
```

## Quick Start

### Basic Usage

```python theme={"system"}
from litellm import completion
import os

# Set your AnyAPI key
os.environ["ANYAPI_API_KEY"] = "your-anyapi-key"

# Call any AnyAPI model using LiteLLM
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    api_base="https://api.anyapi.ai/v1"
)

print(response.choices[0].message.content)
```

### Environment Configuration

Set up environment variables for automatic configuration:

```bash theme={"system"}
# .env file
ANYAPI_API_KEY=your-anyapi-key
ANYAPI_API_BASE=https://api.anyapi.ai/v1
```

```python theme={"system"}
from litellm import completion
import os
from dotenv import load_dotenv

load_dotenv()

response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "What is machine learning?"}]
)
```

## Model Configuration

### Available Models

Access all AnyAPI models through LiteLLM by prefixing with `anyapi/`:

```python theme={"system"}
# OpenAI models
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

# Anthropic models  
response = completion(
    model="anyapi/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "Hello"}]
)

# Google models
response = completion(
    model="anyapi/gemini-pro",
    messages=[{"role": "user", "content": "Hello"}]
)

# Open source models
response = completion(
    model="anyapi/llama-2-70b",
    messages=[{"role": "user", "content": "Hello"}]
)
```

### Custom Configuration

Configure model-specific parameters:

```python theme={"system"}
from litellm import completion

# Configure specific model parameters
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Write a story"}],
    temperature=0.8,
    max_tokens=1000,
    top_p=0.9,
    frequency_penalty=0.1,
    presence_penalty=0.1
)
```

## Advanced Features

### Streaming Responses

Stream responses in real-time:

```python theme={"system"}
from litellm import completion

def stream_response():
    response = completion(
        model="anyapi/gpt-4o",
        messages=[{"role": "user", "content": "Write a long story about AI"}],
        stream=True
    )
    
    for chunk in response:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)

stream_response()
```

### Async Support

Use LiteLLM with async/await:

```python theme={"system"}
import asyncio
from litellm import acompletion

async def async_completion():
    response = await acompletion(
        model="anyapi/gpt-4o",
        messages=[{"role": "user", "content": "What is quantum computing?"}]
    )
    return response.choices[0].message.content

# Run async function
result = asyncio.run(async_completion())
print(result)
```

### Batch Processing

Process multiple requests efficiently:

```python theme={"system"}
import asyncio
from litellm import acompletion

async def batch_process(questions):
    tasks = []
    
    for question in questions:
        task = acompletion(
            model="anyapi/gpt-4o",
            messages=[{"role": "user", "content": question}]
        )
        tasks.append(task)
    
    responses = await asyncio.gather(*tasks)
    return [r.choices[0].message.content for r in responses]

# Example usage
questions = [
    "What is machine learning?",
    "Explain quantum computing",
    "What is blockchain technology?"
]

answers = asyncio.run(batch_process(questions))
for q, a in zip(questions, answers):
    print(f"Q: {q}")
    print(f"A: {a}\n")
```

## Error Handling and Retries

### Automatic Retries

LiteLLM includes built-in retry logic:

```python theme={"system"}
from litellm import completion

# Automatic retries on failure
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    num_retries=3,  # Retry up to 3 times
    timeout=30      # 30 second timeout
)
```

### Custom Error Handling

Implement custom error handling:

```python theme={"system"}
from litellm import completion
from litellm.exceptions import APIError, RateLimitError, Timeout
import time

def robust_completion(model, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = completion(
                model=model,
                messages=messages,
                timeout=30
            )
            return response
            
        except RateLimitError:
            print(f"Rate limit hit, waiting {2**attempt} seconds...")
            time.sleep(2**attempt)
            
        except Timeout:
            print(f"Request timeout on attempt {attempt + 1}")
            
        except APIError as e:
            print(f"API error: {e}")
            if attempt == max_retries - 1:
                raise
                
        except Exception as e:
            print(f"Unexpected error: {e}")
            if attempt == max_retries - 1:
                raise
    
    raise Exception("Max retries exceeded")

# Usage
response = robust_completion(
    "anyapi/gpt-4o",
    [{"role": "user", "content": "Hello"}]
)
```

## Fallback Configuration

### Model Fallbacks

Configure automatic fallbacks between models:

```python theme={"system"}
from litellm import completion

def completion_with_fallback(messages, models=None):
    if models is None:
        models = [
            "anyapi/gpt-4o",
            "anyapi/claude-3-5-sonnet", 
            "anyapi/gemini-pro"
        ]
    
    for model in models:
        try:
            response = completion(
                model=model,
                messages=messages,
                timeout=30
            )
            print(f"✅ Success with {model}")
            return response
            
        except Exception as e:
            print(f"❌ Failed with {model}: {e}")
            continue
    
    raise Exception("All models failed")

# Usage
response = completion_with_fallback([
    {"role": "user", "content": "Explain artificial intelligence"}
])
```

### Load Balancing

Distribute requests across multiple models:

```python theme={"system"}
import random
from litellm import completion

class LoadBalancer:
    def __init__(self, models):
        self.models = models
        self.model_weights = {model: 1.0 for model in models}
    
    def get_model(self):
        # Weighted random selection
        models = list(self.model_weights.keys())
        weights = list(self.model_weights.values())
        return random.choices(models, weights=weights)[0]
    
    def update_weight(self, model, success):
        # Increase weight on success, decrease on failure
        if success:
            self.model_weights[model] = min(2.0, self.model_weights[model] * 1.1)
        else:
            self.model_weights[model] = max(0.1, self.model_weights[model] * 0.9)
    
    def completion(self, messages, **kwargs):
        model = self.get_model()
        
        try:
            response = completion(
                model=model,
                messages=messages,
                **kwargs
            )
            self.update_weight(model, True)
            return response
            
        except Exception as e:
            self.update_weight(model, False)
            raise e

# Usage
balancer = LoadBalancer([
    "anyapi/gpt-4o",
    "anyapi/claude-3-5-sonnet",
    "anyapi/gemini-pro"
])

response = balancer.completion([
    {"role": "user", "content": "What is the future of AI?"}
])
```

## Cost Tracking

### Usage Monitoring

Track costs and usage across models:

```python theme={"system"}
from litellm import completion, cost_per_token
import json

class CostTracker:
    def __init__(self):
        self.usage_log = []
    
    def tracked_completion(self, model, messages, **kwargs):
        response = completion(
            model=model,
            messages=messages,
            **kwargs
        )
        
        # Calculate cost
        usage = response.usage
        cost = cost_per_token(
            model=model,
            prompt_tokens=usage.prompt_tokens,
            completion_tokens=usage.completion_tokens
        )
        
        # Log usage
        self.usage_log.append({
            "model": model,
            "prompt_tokens": usage.prompt_tokens,
            "completion_tokens": usage.completion_tokens,
            "total_tokens": usage.total_tokens,
            "cost": cost,
            "timestamp": time.time()
        })
        
        return response
    
    def get_total_cost(self):
        return sum(entry["cost"] for entry in self.usage_log)
    
    def get_usage_summary(self):
        summary = {}
        for entry in self.usage_log:
            model = entry["model"]
            if model not in summary:
                summary[model] = {
                    "requests": 0,
                    "tokens": 0,
                    "cost": 0
                }
            
            summary[model]["requests"] += 1
            summary[model]["tokens"] += entry["total_tokens"]
            summary[model]["cost"] += entry["cost"]
        
        return summary

# Usage
tracker = CostTracker()

response = tracker.tracked_completion(
    "anyapi/gpt-4o",
    [{"role": "user", "content": "Explain machine learning"}]
)

print(f"Total cost: ${tracker.get_total_cost():.4f}")
print("Usage summary:", json.dumps(tracker.get_usage_summary(), indent=2))
```

## LiteLLM Proxy Server

### Setup Proxy Server

Run LiteLLM as a proxy server for team usage:

```yaml theme={"system"}
# config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: anyapi/gpt-4o
      api_base: https://api.anyapi.ai/v1
      api_key: os.environ/ANYAPI_API_KEY
      
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anyapi/claude-3-5-sonnet
      api_base: https://api.anyapi.ai/v1
      api_key: os.environ/ANYAPI_API_KEY

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  set_verbose: True
```

Start the proxy server:

```bash theme={"system"}
export ANYAPI_API_KEY="your-api-key"
litellm --config config.yaml --port 8000
```

### Using the Proxy

Connect to the proxy server:

```python theme={"system"}
from openai import OpenAI

# Connect to LiteLLM proxy
client = OpenAI(
    api_key="anything",  # Proxy handles auth
    base_url="http://localhost:8000"
)

response = client.chat.completions.create(
    model="gpt-4o",  # Model name from config
    messages=[{"role": "user", "content": "Hello proxy!"}]
)

print(response.choices[0].message.content)
```

## Function Calling

### Using Functions with AnyAPI Models

```python theme={"system"}
from litellm import completion

# Define functions
functions = [
    {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "City name"
                }
            },
            "required": ["location"]
        }
    }
]

# Call with function
response = completion(
    model="anyapi/gpt-4o",
    messages=[
        {"role": "user", "content": "What's the weather like in New York?"}
    ],
    functions=functions,
    function_call="auto"
)

# Check if function was called
if response.choices[0].message.function_call:
    function_call = response.choices[0].message.function_call
    print(f"Function called: {function_call.name}")
    print(f"Arguments: {function_call.arguments}")
```

## Best Practices

### Configuration Management

Use configuration files for complex setups:

```python theme={"system"}
# config.py
ANYAPI_CONFIG = {
    "api_key": "your-api-key",
    "api_base": "https://api.anyapi.ai/v1",
    "default_model": "anyapi/gpt-4o",
    "timeout": 30,
    "max_retries": 3,
    "temperature": 0.7
}

# main.py
from litellm import completion
from config import ANYAPI_CONFIG

def smart_completion(messages, **kwargs):
    # Merge config with kwargs
    params = {**ANYAPI_CONFIG, **kwargs}
    
    return completion(
        model=params.pop("default_model"),
        messages=messages,
        **params
    )
```

### Performance Optimization

Optimize for speed and efficiency:

```python theme={"system"}
import asyncio
from litellm import acompletion

class OptimizedClient:
    def __init__(self, max_concurrent=10):
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.session_cache = {}
    
    async def optimized_completion(self, model, messages, **kwargs):
        async with self.semaphore:
            return await acompletion(
                model=model,
                messages=messages,
                **kwargs
            )
    
    async def batch_with_limit(self, requests):
        tasks = [
            self.optimized_completion(**req) 
            for req in requests
        ]
        return await asyncio.gather(*tasks)

# Usage
client = OptimizedClient(max_concurrent=5)

requests = [
    {
        "model": "anyapi/gpt-4o",
        "messages": [{"role": "user", "content": f"Question {i}"}]
    }
    for i in range(20)
]

responses = asyncio.run(client.batch_with_limit(requests))
```

## Troubleshooting

### Common Issues

#### Authentication Errors

```
Error: Invalid API key
```

**Solution**: Verify your AnyAPI key is set correctly:

```bash theme={"system"}
echo $ANYAPI_API_KEY
```

#### Model Not Found

```
Error: Model 'gpt-4o' not found
```

**Solution**: Use the full model path:

```python theme={"system"}
# Correct
model="anyapi/gpt-4o"

# Incorrect  
model="gpt-4o"
```

#### Connection Issues

```
Error: Connection timeout
```

**Solution**: Check your network connection and increase timeout:

```python theme={"system"}
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    timeout=60  # Increase timeout
)
```

### Debug Mode

Enable debug logging:

```python theme={"system"}
import litellm
import logging

# Enable debug mode
litellm.set_verbose = True
logging.basicConfig(level=logging.DEBUG)

response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Debug test"}]
)
```

### Health Check

Test your configuration:

```python theme={"system"}
def test_anyapi_connection():
    try:
        response = completion(
            model="anyapi/gpt-4o-mini",
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=1
        )
        print("✅ Connection successful")
        return True
    except Exception as e:
        print(f"❌ Connection failed: {e}")
        return False

test_anyapi_connection()
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Langflow Integration" icon="workflow" href="/integrations/langflow">
    Visual AI workflow builder
  </Card>

  <Card title="Continue.dev" icon="code" href="/integrations/continue-dev">
    VS Code AI coding assistant
  </Card>

  <Card title="API Reference" icon="book" href="/api-reference/text-models/overview">
    Complete API documentation
  </Card>

  <Card title="Quick Start" icon="rocket" href="/quickstart/setting-up">
    Get started with AnyAPI
  </Card>
</CardGroup>

For more information about LiteLLM, visit the [official documentation](https://docs.litellm.ai/).