LiteLLM Integration

LiteLLM provides a unified interface to call 100+ language models using the OpenAI format. AnyAPI integrates seamlessly with LiteLLM, allowing you to access all AnyAPI models through LiteLLM’s standardized interface.

Overview

LiteLLM simplifies working with multiple LLM providers by:

Unified API format - Use OpenAI’s format for all models
Automatic retries - Built-in retry logic and error handling
Cost tracking - Monitor usage and costs across providers
Fallback support - Automatically failover between models
Streaming support - Real-time response streaming

Easy Setup

Single installation, unified interface

Cost Tracking

Built-in usage and cost monitoring

Error Handling

Automatic retries and fallbacks

Streaming

Real-time response streaming

Installation

Install LiteLLM via pip:

pip install litellm

For additional features like cost tracking and logging:

pip install 'litellm[proxy]'

Quick Start

Basic Usage

from litellm import completion
import os

# Set your AnyAPI key
os.environ["ANYAPI_API_KEY"] = "your-anyapi-key"

# Call any AnyAPI model using LiteLLM
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
    api_base="https://api.anyapi.ai/v1"
)

print(response.choices[0].message.content)

Environment Configuration

Set up environment variables for automatic configuration:

# .env file
ANYAPI_API_KEY=your-anyapi-key
ANYAPI_API_BASE=https://api.anyapi.ai/v1

from litellm import completion
import os
from dotenv import load_dotenv

load_dotenv()

response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "What is machine learning?"}]
)

Model Configuration

Available Models

Access all AnyAPI models through LiteLLM by prefixing with anyapi/:

# OpenAI models
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

# Anthropic models  
response = completion(
    model="anyapi/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "Hello"}]
)

# Google models
response = completion(
    model="anyapi/gemini-pro",
    messages=[{"role": "user", "content": "Hello"}]
)

# Open source models
response = completion(
    model="anyapi/llama-2-70b",
    messages=[{"role": "user", "content": "Hello"}]
)

Custom Configuration

Configure model-specific parameters:

from litellm import completion

# Configure specific model parameters
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Write a story"}],
    temperature=0.8,
    max_tokens=1000,
    top_p=0.9,
    frequency_penalty=0.1,
    presence_penalty=0.1
)

Advanced Features

Streaming Responses

Stream responses in real-time:

from litellm import completion

def stream_response():
    response = completion(
        model="anyapi/gpt-4o",
        messages=[{"role": "user", "content": "Write a long story about AI"}],
        stream=True
    )
    
    for chunk in response:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)

stream_response()

Async Support

Use LiteLLM with async/await:

import asyncio
from litellm import acompletion

async def async_completion():
    response = await acompletion(
        model="anyapi/gpt-4o",
        messages=[{"role": "user", "content": "What is quantum computing?"}]
    )
    return response.choices[0].message.content

# Run async function
result = asyncio.run(async_completion())
print(result)

Batch Processing

Process multiple requests efficiently:

import asyncio
from litellm import acompletion

async def batch_process(questions):
    tasks = []
    
    for question in questions:
        task = acompletion(
            model="anyapi/gpt-4o",
            messages=[{"role": "user", "content": question}]
        )
        tasks.append(task)
    
    responses = await asyncio.gather(*tasks)
    return [r.choices[0].message.content for r in responses]

# Example usage
questions = [
    "What is machine learning?",
    "Explain quantum computing",
    "What is blockchain technology?"
]

answers = asyncio.run(batch_process(questions))
for q, a in zip(questions, answers):
    print(f"Q: {q}")
    print(f"A: {a}\n")

Error Handling and Retries

Automatic Retries

LiteLLM includes built-in retry logic:

from litellm import completion

# Automatic retries on failure
response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    num_retries=3,  # Retry up to 3 times
    timeout=30      # 30 second timeout
)

Custom Error Handling

Implement custom error handling:

from litellm import completion
from litellm.exceptions import APIError, RateLimitError, Timeout
import time

def robust_completion(model, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = completion(
                model=model,
                messages=messages,
                timeout=30
            )
            return response
            
        except RateLimitError:
            print(f"Rate limit hit, waiting {2**attempt} seconds...")
            time.sleep(2**attempt)
            
        except Timeout:
            print(f"Request timeout on attempt {attempt + 1}")
            
        except APIError as e:
            print(f"API error: {e}")
            if attempt == max_retries - 1:
                raise
                
        except Exception as e:
            print(f"Unexpected error: {e}")
            if attempt == max_retries - 1:
                raise
    
    raise Exception("Max retries exceeded")

# Usage
response = robust_completion(
    "anyapi/gpt-4o",
    [{"role": "user", "content": "Hello"}]
)

Fallback Configuration

Model Fallbacks

Configure automatic fallbacks between models:

from litellm import completion

def completion_with_fallback(messages, models=None):
    if models is None:
        models = [
            "anyapi/gpt-4o",
            "anyapi/claude-3-5-sonnet", 
            "anyapi/gemini-pro"
        ]
    
    for model in models:
        try:
            response = completion(
                model=model,
                messages=messages,
                timeout=30
            )
            print(f"✅ Success with {model}")
            return response
            
        except Exception as e:
            print(f"❌ Failed with {model}: {e}")
            continue
    
    raise Exception("All models failed")

# Usage
response = completion_with_fallback([
    {"role": "user", "content": "Explain artificial intelligence"}
])

Load Balancing

Distribute requests across multiple models:

import random
from litellm import completion

class LoadBalancer:
    def __init__(self, models):
        self.models = models
        self.model_weights = {model: 1.0 for model in models}
    
    def get_model(self):
        # Weighted random selection
        models = list(self.model_weights.keys())
        weights = list(self.model_weights.values())
        return random.choices(models, weights=weights)[0]
    
    def update_weight(self, model, success):
        # Increase weight on success, decrease on failure
        if success:
            self.model_weights[model] = min(2.0, self.model_weights[model] * 1.1)
        else:
            self.model_weights[model] = max(0.1, self.model_weights[model] * 0.9)
    
    def completion(self, messages, **kwargs):
        model = self.get_model()
        
        try:
            response = completion(
                model=model,
                messages=messages,
                **kwargs
            )
            self.update_weight(model, True)
            return response
            
        except Exception as e:
            self.update_weight(model, False)
            raise e

# Usage
balancer = LoadBalancer([
    "anyapi/gpt-4o",
    "anyapi/claude-3-5-sonnet",
    "anyapi/gemini-pro"
])

response = balancer.completion([
    {"role": "user", "content": "What is the future of AI?"}
])

Cost Tracking

Usage Monitoring

Track costs and usage across models:

from litellm import completion, cost_per_token
import json

class CostTracker:
    def __init__(self):
        self.usage_log = []
    
    def tracked_completion(self, model, messages, **kwargs):
        response = completion(
            model=model,
            messages=messages,
            **kwargs
        )
        
        # Calculate cost
        usage = response.usage
        cost = cost_per_token(
            model=model,
            prompt_tokens=usage.prompt_tokens,
            completion_tokens=usage.completion_tokens
        )
        
        # Log usage
        self.usage_log.append({
            "model": model,
            "prompt_tokens": usage.prompt_tokens,
            "completion_tokens": usage.completion_tokens,
            "total_tokens": usage.total_tokens,
            "cost": cost,
            "timestamp": time.time()
        })
        
        return response
    
    def get_total_cost(self):
        return sum(entry["cost"] for entry in self.usage_log)
    
    def get_usage_summary(self):
        summary = {}
        for entry in self.usage_log:
            model = entry["model"]
            if model not in summary:
                summary[model] = {
                    "requests": 0,
                    "tokens": 0,
                    "cost": 0
                }
            
            summary[model]["requests"] += 1
            summary[model]["tokens"] += entry["total_tokens"]
            summary[model]["cost"] += entry["cost"]
        
        return summary

# Usage
tracker = CostTracker()

response = tracker.tracked_completion(
    "anyapi/gpt-4o",
    [{"role": "user", "content": "Explain machine learning"}]
)

print(f"Total cost: ${tracker.get_total_cost():.4f}")
print("Usage summary:", json.dumps(tracker.get_usage_summary(), indent=2))

LiteLLM Proxy Server

Setup Proxy Server

Run LiteLLM as a proxy server for team usage:

# config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: anyapi/gpt-4o
      api_base: https://api.anyapi.ai/v1
      api_key: os.environ/ANYAPI_API_KEY
      
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anyapi/claude-3-5-sonnet
      api_base: https://api.anyapi.ai/v1
      api_key: os.environ/ANYAPI_API_KEY

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  set_verbose: True

Start the proxy server:

export ANYAPI_API_KEY="your-api-key"
litellm --config config.yaml --port 8000

Using the Proxy

Connect to the proxy server:

from openai import OpenAI

# Connect to LiteLLM proxy
client = OpenAI(
    api_key="anything",  # Proxy handles auth
    base_url="http://localhost:8000"
)

response = client.chat.completions.create(
    model="gpt-4o",  # Model name from config
    messages=[{"role": "user", "content": "Hello proxy!"}]
)

print(response.choices[0].message.content)

Function Calling

Using Functions with AnyAPI Models

from litellm import completion

# Define functions
functions = [
    {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "City name"
                }
            },
            "required": ["location"]
        }
    }
]

# Call with function
response = completion(
    model="anyapi/gpt-4o",
    messages=[
        {"role": "user", "content": "What's the weather like in New York?"}
    ],
    functions=functions,
    function_call="auto"
)

# Check if function was called
if response.choices[0].message.function_call:
    function_call = response.choices[0].message.function_call
    print(f"Function called: {function_call.name}")
    print(f"Arguments: {function_call.arguments}")

Best Practices

Configuration Management

Use configuration files for complex setups:

# config.py
ANYAPI_CONFIG = {
    "api_key": "your-api-key",
    "api_base": "https://api.anyapi.ai/v1",
    "default_model": "anyapi/gpt-4o",
    "timeout": 30,
    "max_retries": 3,
    "temperature": 0.7
}

# main.py
from litellm import completion
from config import ANYAPI_CONFIG

def smart_completion(messages, **kwargs):
    # Merge config with kwargs
    params = {**ANYAPI_CONFIG, **kwargs}
    
    return completion(
        model=params.pop("default_model"),
        messages=messages,
        **params
    )

Performance Optimization

Optimize for speed and efficiency:

import asyncio
from litellm import acompletion

class OptimizedClient:
    def __init__(self, max_concurrent=10):
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.session_cache = {}
    
    async def optimized_completion(self, model, messages, **kwargs):
        async with self.semaphore:
            return await acompletion(
                model=model,
                messages=messages,
                **kwargs
            )
    
    async def batch_with_limit(self, requests):
        tasks = [
            self.optimized_completion(**req) 
            for req in requests
        ]
        return await asyncio.gather(*tasks)

# Usage
client = OptimizedClient(max_concurrent=5)

requests = [
    {
        "model": "anyapi/gpt-4o",
        "messages": [{"role": "user", "content": f"Question {i}"}]
    }
    for i in range(20)
]

responses = asyncio.run(client.batch_with_limit(requests))

Troubleshooting

Common Issues

Authentication Errors

Error: Invalid API key

Solution: Verify your AnyAPI key is set correctly:

echo $ANYAPI_API_KEY

Model Not Found

Error: Model 'gpt-4o' not found

Solution: Use the full model path:

# Correct
model="anyapi/gpt-4o"

# Incorrect  
model="gpt-4o"

Connection Issues

Error: Connection timeout

Solution: Check your network connection and increase timeout:

response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    timeout=60  # Increase timeout
)

Debug Mode

Enable debug logging:

import litellm
import logging

# Enable debug mode
litellm.set_verbose = True
logging.basicConfig(level=logging.DEBUG)

response = completion(
    model="anyapi/gpt-4o",
    messages=[{"role": "user", "content": "Debug test"}]
)

Health Check

Test your configuration:

def test_anyapi_connection():
    try:
        response = completion(
            model="anyapi/gpt-4o-mini",
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=1
        )
        print("✅ Connection successful")
        return True
    except Exception as e:
        print(f"❌ Connection failed: {e}")
        return False

test_anyapi_connection()

Next Steps

Langflow Integration

Visual AI workflow builder

Continue.dev

VS Code AI coding assistant

API Reference

Complete API documentation

Quick Start

Get started with AnyAPI

For more information about LiteLLM, visit the official documentation.

​LiteLLM Integration

​Overview

Easy Setup

Cost Tracking

Error Handling

Streaming

​Installation

​Quick Start

​Basic Usage

​Environment Configuration

​Model Configuration

​Available Models

​Custom Configuration

​Advanced Features

​Streaming Responses

​Async Support

​Batch Processing

​Error Handling and Retries

​Automatic Retries

​Custom Error Handling

​Fallback Configuration

​Model Fallbacks

​Load Balancing

​Cost Tracking

​Usage Monitoring

​LiteLLM Proxy Server

​Setup Proxy Server

​Using the Proxy

​Function Calling

​Using Functions with AnyAPI Models

​Best Practices

​Configuration Management

​Performance Optimization

​Troubleshooting

​Common Issues

​Authentication Errors

​Model Not Found

​Connection Issues

​Debug Mode

​Health Check

​Next Steps

Langflow Integration

Continue.dev

API Reference

Quick Start

LiteLLM Integration

Overview

Installation

Quick Start

Basic Usage

Environment Configuration

Model Configuration

Available Models

Custom Configuration

Advanced Features

Streaming Responses

Async Support

Batch Processing

Error Handling and Retries

Automatic Retries

Custom Error Handling

Fallback Configuration

Model Fallbacks

Load Balancing

Cost Tracking

Usage Monitoring

LiteLLM Proxy Server

Setup Proxy Server

Using the Proxy

Function Calling

Using Functions with AnyAPI Models

Best Practices

Configuration Management

Performance Optimization

Troubleshooting

Common Issues

Authentication Errors

Model Not Found

Connection Issues

Debug Mode

Health Check

Next Steps