Skip to main content

Provider Configuration

Configure connections to LLM providers.

Supported Providers​

OpenAI​

export TOKLIGENCE_OPENAI_API_KEY=sk-...

Available Models:

  • gpt-4, gpt-4-turbo, gpt-4o
  • gpt-3.5-turbo
  • text-embedding-ada-002

Anthropic​

export TOKLIGENCE_ANTHROPIC_API_KEY=sk-ant-...

Available Models:

  • claude-3-opus, claude-3-sonnet, claude-3-haiku
  • claude-3-5-sonnet, claude-3-5-haiku
  • claude-2.1, claude-2.0

Google Gemini​

export TOKLIGENCE_GOOGLE_API_KEY=AIza...

Available Models:

  • gemini-pro
  • gemini-pro-vision
  • gemini-ultra

Local LLMs​

Ollama​

# Start Ollama
ollama serve
ollama run llama3

# Configure gateway (auto-detected)
# No API key needed

vLLM​

export TOKLIGENCE_LLM_ENDPOINT=http://localhost:8000/v1

LM Studio​

export TOKLIGENCE_LLM_ENDPOINT=http://localhost:1234/v1

Custom Endpoint​

export TOKLIGENCE_LLM_ENDPOINT=http://your-llm.com/v1
export TOKLIGENCE_LLM_API_KEY=optional-key

Model Routing​

Configure model-to-provider routing:

export TOKLIGENCE_ROUTES="claude*=>anthropic,gpt-*=>openai,gemini-*=>google"

Pattern Syntax​

  • * matches any characters
  • => separates pattern from provider

Examples​

# Default routing
TOKLIGENCE_ROUTES="claude*=>anthropic,gpt-*=>openai,o1*=>openai,gemini-*=>google"

# Custom vendor prefixes
TOKLIGENCE_ROUTES="qwen*=>ali,llama*=>local"

Sidecar Translation​

For Claude Code → OpenAI translation:

# Model mapping
export TOKLIGENCE_SIDECAR_MODEL_MAP="claude-3-5-sonnet-20241022=gpt-4o
claude-3-5-haiku-20241022=gpt-4o-mini"

# Default model for unmapped requests
export TOKLIGENCE_SIDECAR_DEFAULT_OPENAI_MODEL=gpt-4o

# Max tokens (Anthropic allows larger than OpenAI)
export TOKLIGENCE_OPENAI_COMPLETION_MAX_TOKENS=16384

Provider-Specific Settings​

Rate Limits​

# config/gateway.ini
[openai]
rate_limit_rpm = 100

[anthropic]
rate_limit_rpm = 50

Timeouts​

export TOKLIGENCE_REQUEST_TIMEOUT=60000  # milliseconds

Prompt Caching​

Enable Anthropic's prompt caching for cost optimization:

{
  "model": "claude-3-5-sonnet-20241022",
  "messages": [...],
  "cache_control": {
    "type": "ephemeral"
  }
}

Health Checks​

Verify provider connectivity:

curl http://localhost:8081/health

Response:

{
  "status": "ok",
  "providers": {
    "openai": "connected",
    "anthropic": "connected",
    "google": "connected"
  }
}