Skip to main content

Routing & Work Modes

Configure intelligent request routing across multiple LLM providers.

Model-Based Routing​

The gateway routes requests based on model name patterns:

# Configuration via environment variable
export TOKLIGENCE_ROUTES="claude*=>anthropic,gpt-*=>openai,gemini-*=>google"

Examples:

  • model: "claude-3-haiku" → Anthropic API
  • model: "claude-3.5-sonnet" → Anthropic API
  • model: "gpt-4" → OpenAI API
  • model: "gpt-3.5-turbo" → OpenAI API
  • model: "gemini-pro" → Google API

Work Modes​

The gateway supports three work modes for flexible request handling:

ModeBehaviorUse Case
auto (default)Smart routing - automatically chooses passthrough or translation based on endpoint+model matchBest for mixed workloads
passthroughDelegation-only - direct passthrough to upstream providers, rejects translation requestsForce all requests to native providers
translationTranslation-only - only allows translation between API formats, rejects passthrough requestsForce all requests through translation layer

Configuration​

# Via environment variable
export TOKLIGENCE_WORK_MODE=auto

# Or in config/dev/gateway.ini
work_mode=auto

Auto Mode Behavior​

In work_mode=auto, the gateway first infers the provider from the requested model (via model_provider_routes). The endpoint (/v1/messages, /v1/chat/completions, /v1/responses) decides whether to translate or passthrough once the provider is known.

Examples:

  • /v1/responses with gpt-4 → delegates to OpenAI
  • /v1/responses with claude-3.5-sonnet → translates to Anthropic
  • /anthropic/v1/messages with gpt-4o → translates to OpenAI

Multi-Port Architecture​

Default Mode (Single-Port)​

The gateway runs on port :8081 by default, exposing all endpoints on a single port.

Multi-Port Mode (Optional)​

Enable multiport_mode=true for strict endpoint isolation:

multiport_mode = true
facade_port = :8081      # Main aggregator (all endpoints)
admin_port = :8079       # Admin-only endpoints
openai_port = :8082      # OpenAI-only endpoints
anthropic_port = :8083   # Anthropic-only endpoints
gemini_port = :8084      # Gemini-only endpoints

Endpoint Keys​

KeyRoutes
openai_core/v1/chat/completions, /v1/embeddings, /v1/models
openai_responses/v1/responses
anthropic/anthropic/v1/messages, /v1/messages
gemini_native/v1beta/models/*
admin/api/v1/admin/...
health/health

Example Multi-Port Configuration​

openai_endpoints = openai_core,openai_responses,health
anthropic_endpoints = anthropic,health
gemini_endpoints = gemini_native,health
admin_endpoints = admin,health

Protocol Translation​

The gateway supports bidirectional protocol translation:

OpenAI → Anthropic​

When Codex CLI (OpenAI Responses API) talks to Claude models:

Codex CLI → /v1/responses → Gateway → Translation → Anthropic API

Anthropic → OpenAI​

When Claude Code talks to GPT models via gateway:

Claude Code → /anthropic/v1/messages → Gateway → Translation → OpenAI API

Enabling Chat-to-Anthropic Translation​

# Enable OpenAI Chat endpoint translation to Anthropic
export TOKLIGENCE_CHAT_TO_ANTHROPIC=on

With this enabled, /v1/chat/completions with a claude* model is translated to Anthropic /v1/messages.

Sidecar Model Mapping​

For Claude Code → OpenAI translation, configure model mapping:

export TOKLIGENCE_SIDECAR_MODEL_MAP="claude-3-5-sonnet-20241022=gpt-4o
claude-3-5-haiku-20241022=gpt-4o-mini"
export TOKLIGENCE_SIDECAR_DEFAULT_OPENAI_MODEL=gpt-4o

Loopback Model​

The built-in loopback model echoes input without calling external LLMs:

curl -X POST http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"loopback","messages":[{"role":"user","content":"Hello"}]}'

Ideal for testing connectivity and SSE streaming.