SwitchBoard sits between your application and Groq, Google and Anthropic. It answers repeat questions from a semantic cache instead of the provider, rotates across your API keys the moment one hits a rate limit, and reports every hop to Prometheus. The endpoint is OpenAI-compatible — change the base URL, keep your client.
git clone https://github.com/sankalp-happy/switchboard.git && cd switchboard && ./setup.sh
None of them are exotic. They show up the first week you put a prototype in front of real traffic, and every one of them is a gateway's job — not your application's.
Provider keys carry hard per-minute token and request ceilings. One 429 and the call fails — while a second key sits idle in your .env. SwitchBoard marks the exhausted key, retries the next one, and only surfaces an error once every key is genuinely spent.
People ask the same thing in different words all day long. Hash-based caches miss every rephrasing. SwitchBoard embeds the prompt and compares it by cosine similarity, so "greet me using five words" hits the entry stored for "say hi in five words."
A provider SDK hands back a response, not a hit rate, a p95, or a count of key rotations. SwitchBoard exposes all of it on /metrics and ships a Grafana dashboard that is provisioned for you at first boot.
Roughly two thousand lines of Python you can read in an afternoon. Every part of it is listed here, and every claim on this page maps to a file in the repository.
POST /v1/chat/completions accepts and returns the shapes your client already speaks. Point the SDK at SwitchBoard and delete nothing else.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-needed", # SwitchBoard holds the real keys
)Prompts are embedded with gemini-embedding-001 and compared by cosine similarity. Anything at or above 0.90 is served from Redis with a one-hour TTL — no provider call, no tokens spent.
Register as many keys per provider as you like. The router picks the one with the most remaining quota, then walks the rest on 429, 401 or 5xx. A sweeper revives expired keys every five seconds.
Provider credentials are sealed with Fernet before they touch SQLite and are only ever returned masked — gsk_…4f2a. The ENCRYPTION_KEY stays in your environment.
Set "provider" in the request body to pin a single call to groq, google or anthropic. Omit it and SWITCHBOARD_PROVIDER decides.
There is no SwitchBoard service to sign up for. The whole stack — gateway, Redis, Prometheus, Grafana, admin UI — runs from one Compose file on hardware you control. Prompts leave your host only when they reach the provider you chose.
Seven purpose-built metric families plus automatic FastAPI instrumentation, scraped every 15 seconds and rendered by a Grafana dashboard that provisions itself.
Add, disable and delete keys without a restart, and read live quota state per key. A static control panel ships in the stack on port 3000, proxied to the gateway by nginx.
Everything inside the dashed boundary is a single FastAPI process. Redis, Prometheus and Grafana are sibling containers; the providers are the only hop that leaves your network.
grafana/dashboards/switchboard.json is provisioned at container start against a Prometheus datasource that is also provisioned. Nothing to click through on first boot — these nine panels are simply there.
| Panel | Type | Query |
|---|---|---|
| Cache Hit Rate | timeseries | rate(switchboard_cache_hits_total[5m]) / (rate(hits[5m]) + rate(misses[5m])) |
| Total Request Throughput | timeseries | rate(http_requests_total{handler="/v1/chat/completions"}[5m]) |
| Total Requests | stat | http_requests_total{handler="/v1/chat/completions"} |
| Cache Hits | stat | switchboard_cache_hits_total |
| Provider Latency | timeseries | histogram_quantile(0.50 / 0.95 / 0.99, rate(switchboard_provider_latency_seconds_bucket[5m])) |
| Key Switches | stat | increase(switchboard_key_switches_total[1h]) |
| Active Keys | stat | switchboard_active_keys |
| Tokens Processed | timeseries | rate(switchboard_tokens_processed_total[5m]) |
| Cache Hits vs Misses | timeseries | rate(switchboard_cache_hits_total[5m]) · rate(switchboard_cache_misses_total[5m]) |
Grafana ships with anonymous viewer access enabled and an admin login of admin / switchboard — change both before exposing it.
setup.sh checks your Docker install, generates the Fernet encryption key, prompts for whichever provider keys you want to seed, writes .env, brings the stack up, and waits until every service reports healthy. Re-running it is safe — existing values are reused, never overwritten.
git clone https://github.com/sankalp-happy/switchboard.git
cd switchboard
./setup.shDocker and Docker Compose v2 are the only prerequisites. Provider keys are optional at install time — add them later from the control panel. Without GOOGLE_API_KEY the gateway still serves traffic; it just runs with the semantic cache disabled.
If a port is already taken, setup remaps the service and records the new port in .env rather than killing whatever holds it.