Documentation

Overview

SwitchBoard is a self-hosted gateway that sits between your application and upstream LLM providers. It exposes a single OpenAI-compatible endpoint, POST /v1/chat/completions, and does three jobs behind it:

• Semantic caching. Prompts are embedded and matched against previous requests by cosine similarity, so a rephrased question is answered from Redis instead of the provider.
• Multi-key routing. Any number of API keys can be registered per provider. The router picks the key with the most remaining quota and fails over automatically on rate limits and auth errors.
• Observability. Cache hit rate, provider latency quantiles, key rotations and token throughput are exported to Prometheus and rendered by a Grafana dashboard that provisions itself.

Supported providers are groq, google and anthropic. There is no hosted version — you run the whole stack yourself from one Compose file.

Prerequisites

• Docker and Docker Compose v2 — the only requirement for the quickstart below.
• Python 3.11+ — only if you intend to run the gateway outside Docker.

Provider keys

All provider keys are optional at install time; you can add them later from the control panel or the admin API without restarting anything.

Quickstart

Three commands, and the script handles the rest.

git clone https://github.com/sankalp-happy/switchboard.git
cd switchboard
./setup.sh

setup.sh will:

1. Verify your Docker and Compose v2 installation.
2. Generate the Fernet ENCRYPTION_KEY for you.
3. Prompt for any provider keys you want to seed.
4. Write .env, remapping any host port that is already occupied.
5. Bring the stack up and wait until every service reports healthy.

It is safe to re-run. Existing .env values are reused and never overwritten, and the file is backed up only when something actually changes.

Flags

# lighter stack, no metrics UI
./setup.sh --minimal

# unattended, seeding a key from the environment
GROQ_API_KEY=gsk_... ./setup.sh --yes

Port conflicts

If a port the stack needs is already taken, setup remaps the service rather than fighting for the port. It never kills another process — including Docker itself, which owns every published port on macOS.

▲ Gateway port 8000 is in use by Python (pid 68964).
▲ Gateway moved to port 8001 (saved in .env).

The chosen ports are written to .env as GATEWAY_PORT, ADMIN_UI_PORT, REDIS_PORT, PROMETHEUS_PORT and GRAFANA_PORT, and the summary prints the real URLs. They persist across runs, so your URLs stay stable — edit .env to move a service back.

Containers left behind by an interrupted run belong to SwitchBoard, so setup clears them with docker compose down --remove-orphans before starting. Your database volume is never touched.

If setup fails

Containers started by the run are rolled back automatically, so a failed install never leaves ports occupied. Full logs are written to setup-failure-<timestamp>.log first, so the cleanup doesn't cost you the evidence. Use --keep-on-failure to debug live instead.

Verify

curl http://localhost:8000/health
# {"status":"ok","version":"0.2.0"}

Manual setup

If you'd rather not run the script, the same result takes three steps.

cp .env.example .env

# generate a Fernet key and paste it as ENCRYPTION_KEY in .env
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

docker compose up --build

Local development

Running the gateway directly on the host, with Redis in a container.

1 · Virtual environment

python3.11 -m venv .venv
source .venv/bin/activate

2 · Dependencies

pip install -r requirements.txt

3 · Redis

# using Docker
docker run -d --name switchboard-redis -p 6379:6379 redis:7-alpine

# or a locally installed Redis
redis-server

4 · Environment

export GROQ_API_KEY="gsk_..."
export GOOGLE_API_KEY="AIza..."
export ANTHROPIC_API_KEY="sk-ant-..."
export ENCRYPTION_KEY="$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"
export REDIS_URL="redis://localhost:6379/0"

5 · Run

uvicorn gateway.main:app --host 0.0.0.0 --port 8000 --reload

Settings are read by Pydantic from the process environment and from a .env file in the working directory, so exporting is optional if .env is already populated.

Usage

Chat completions

The endpoint mirrors the OpenAI request shape. provider is the one addition — a SwitchBoard extension that pins this single call to a vendor.

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instant",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7,
    "provider": "groq"
  }'

Request fields

Python client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",  # SwitchBoard holds the provider keys
)

response = client.chat.completions.create(
    model="llama-3.1-8b-instant",
    messages=[{"role": "user", "content": "Explain quantum computing in one sentence."}],
)
print(response.choices[0].message.content)

The api_key value is ignored — the gateway does not authenticate callers. See Auth & exposure.

Response headers

Every response carries routing and cache metadata.

Cache behaviour in practice

The cache matches on meaning, not on exact text. Asking the same thing a different way still hits:

"Say hi in five words."      → X-Cache: MISS   X-Provider: groq   X-Latency-Ms: 424.0
"Say hi in five words."      → X-Cache: HIT    X-Semantic-Similarity: 1.0000
"Greet me using five words." → X-Cache: HIT    X-Semantic-Similarity: 0.9144

A hit costs one embedding call and no completion tokens. Entries expire one hour after they are written.

Configuration

Environment variables

Read by core/config.py from the environment or a .env file. Unknown keys are permitted, so extra entries in .env won't break startup.

Host port mappings

These control the ports published on your machine by Compose. setup.sh sets them automatically, remapping any that are already in use; set them yourself to pin a service.

Semantic cache

Cache behaviour is currently defined in code, in cache/redis_client.py, not through environment variables. To change any of it, edit the constructor:

The stored key is a SHA-256 of model + temperature + messages, so identical prompts at different temperatures occupy different entries. Retrieval, however, scans all keys under the prefix and compares embeddings in Python — see Limitations before pointing production traffic at it.

Turning the cache off

Leave GOOGLE_API_KEY unset. The embedding client is never constructed, _get_embedding returns None, and both the read and write paths skip the cache while the gateway keeps serving normally.

Clearing the cache

docker compose exec redis redis-cli --scan --pattern 'nexus:cache:*' \
  | xargs -r docker compose exec -T redis redis-cli DEL

Routing & key rotation

Provider selection resolves in this order:

1. provider in the request body, if present.
2. SWITCHBOARD_PROVIDER from the environment.
3. groq as the final fallback.

An unrecognised name raises immediately rather than silently falling back.

Key selection

KeyManager.get_available_key orders enabled keys for the provider like this:

1. Keys never used before — rate_limit_remaining_tokens IS NULL — are treated as unlimited and preferred.
2. Then keys with remaining tokens above MIN_TOKENS_THRESHOLD (100), highest remaining first.
3. If every key is below the threshold, the one with the soonest reset is used anyway, with a warning logged.

Failover

Background maintenance

Both are asyncio tasks started in the FastAPI lifespan and cancelled on shutdown. The intervals live in gateway/main.py.

Auth & exposure

What the gateway does protect:

• Keys at rest. Provider credentials are Fernet-encrypted (AES-128-CBC with HMAC) before insertion into SQLite and decrypted only at the moment a request is sent.
• Keys in responses. /admin/keys and /admin/stats return masked values — first four and last four characters only.
• Prompt residency. Prompts and responses stay on your host except for the call to the provider you selected, plus the embedding call to Google when the semantic cache is enabled.

Two further deployment notes from docker-compose.yml:

• The gateway service bind-mounts the project directory as .:/app. That is convenient for development but means container code follows your working tree — pin an image build for anything long-lived.
• Redis is published on the host at REDIS_PORT with no password. Remove that port mapping if the host is not private.

Admin API

Mounted under /admin. Interactive documentation for every endpoint is generated at /docs.

Examples

# add a key
curl -X POST http://localhost:8000/admin/keys \
  -H "Content-Type: application/json" \
  -d '{"provider": "groq", "api_key": "gsk_...", "label": "personal-key"}'

# take a key out of rotation without deleting it
curl -X PATCH http://localhost:8000/admin/keys/3 \
  -H "Content-Type: application/json" \
  -d '{"is_enabled": false}'

# who has quota left?
curl http://localhost:8000/admin/providers

Control panel

A static dashboard ships as the admin-ui service on port 3000 — a single vis/index.html served by nginx, which reverse-proxies /v1/, /admin/, /health and /metrics to the gateway over the Compose network. Because everything is same-origin there, the panel can read the X-Cache and X-Provider headers directly.

Observability

Metrics

The gateway exposes /metrics in Prometheus text format. Seven SwitchBoard metric families are defined in core/metrics.py, alongside standard HTTP metrics contributed automatically by prometheus-fastapi-instrumentator.

Prometheus

prometheus/prometheus.yml defines a single scrape job, switchboard-gateway, hitting gateway:8000/metrics every 15 seconds and attaching a service="switchboard" label.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "switchboard-gateway"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["gateway:8000"]
        labels:
          service: "switchboard"

Grafana

Both the Prometheus datasource and the dashboard are provisioned from grafana/provisioning/ at container start — there is nothing to import by hand. The Switchboard Gateway dashboard carries nine panels:

Skip both services entirely with ./setup.sh --minimal if you only want the gateway, Redis and the control panel.

Architecture

The gateway is a single FastAPI process. Redis, Prometheus, Grafana and the nginx control panel are sibling containers on the Compose network. Configuration state lives in a SQLite file on a named volume, switchboard-data.

A full request walkthrough with a diagram is on the home page. The lifecycle in detail:

Request lifecycle

1. Validation. The body is parsed into ChatCompletionRequest. Unknown providers are rejected before any network call.
2. Cache read. Message contents are joined into one string and embedded. Every key under nexus:cache:* is fetched and scored by cosine similarity; the best score is kept.
3. Hit. At ≥ 0.90 the stored ChatCompletionResponse is returned with X-Cache: HIT and the similarity score. No provider is contacted. A failure anywhere in this path is logged as a warning and treated as a miss — the cache can never take the gateway down.
4. Miss. CACHE_MISSES increments and the router resolves the provider, loads every enabled key for it, and starts with the best-quota key.
5. Provider call. The adapter posts over HTTPX with a 30-second timeout and maps the vendor's reply onto the unified schema. Anthropic requests are translated — the first system message becomes the top-level system field and max_tokens is set to 1024.
6. Bookkeeping. x-ratelimit-* headers update the key's row, an absolute reset timestamp is computed from Groq-style durations such as 1m6s, per-minute usage buckets increment, and the four provider metrics record the outcome.
7. Cache write. The response is stored under a SHA-256 of model, temperature and messages, with its embedding, for one hour. A write failure is logged and the response is still returned.

Storage

The connection is a singleton in WAL mode with a 5-second busy timeout and foreign keys on, so concurrent async tasks don't collide. The schema is created idempotently at startup and a lightweight migration adds rate_limit_resets_at to databases created before it existed.

Testing

# full suite
pytest

# one file
pytest tests/test_routing.py

# one test
pytest tests/test_routing.py::test_router_picks_key_and_calls_provider

# verbose, or with print output
pytest -v
pytest -s

Project structure

switchboard/
├── gateway/
│   ├── main.py                 # FastAPI app, lifespan, /v1/chat/completions
│   └── admin.py                # Admin API router (/admin/*)
├── core/
│   ├── config.py               # Pydantic settings (env vars)
│   ├── database.py             # SQLite schema, usage buckets, cleanup
│   ├── key_manager.py          # Key CRUD, Fernet, rate-limit ledger
│   ├── metrics.py              # Prometheus metric definitions
│   └── schemas.py              # Pydantic request/response models
├── routing/
│   └── router.py               # Key-availability routing + failover
├── providers/
│   ├── base.py                 # Abstract LLMProvider interface
│   ├── groq_provider.py
│   ├── google_provider.py
│   └── anthropic_provider.py
├── cache/
│   └── redis_client.py         # Semantic cache (embeddings + Redis)
├── vis/
│   ├── index.html              # Static control panel
│   ├── default.conf.template   # nginx reverse proxy to the gateway
│   └── Dockerfile
├── prometheus/prometheus.yml
├── grafana/
│   ├── dashboards/             # Pre-built dashboard JSON
│   └── provisioning/           # Datasource + dashboard provisioning
├── site/                       # This documentation site
├── tests/
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
└── setup.sh

Limitations

Known constraints in v0.2.0, stated plainly so you can decide where this fits.

Streaming is not implemented

ChatCompletionRequest accepts a stream field, but both the Groq and Google adapters overwrite it with False before sending ("For MVP, disable streaming as it requires SSE handling"), and the Anthropic adapter never forwards it. Every response is returned whole. Setting stream: true will not error — it simply has no effect.

Cache lookup is a linear scan

Retrieval calls KEYS nexus:cache:* and then fetches and scores every entry in Python. Cost grows linearly with cache size, and KEYS blocks Redis while it runs. The source calls this out as an MVP choice: a vector index — Redis Stack, or a dedicated vector database — is the intended replacement.

No authentication

Neither the completions endpoint nor the admin API checks a caller identity, and CORS allows every origin. See Auth & exposure for what to do about it.

Smaller things

• switchboard_active_keys is set once at startup and does not track keys added later.
• Anthropic requests are capped at max_tokens: 1024, hardcoded in the adapter.
• get_cost_per_token exists on the provider interface but returns rough estimates for Groq and zeros elsewhere; nothing consumes it yet.
• The cache key prefix is still nexus:cache:, a leftover from an earlier name.
• Provider requests use a fixed 30-second HTTPX timeout that is not configurable.

License & contributing