Local inference

Local inference servers share one BitRouter integration: expose an OpenAI-compatible endpoint, add it under providers, and declare the model ids that endpoint serves.

Common servers

Ports and authentication can be changed by the server. Treat the table as a starting point and confirm the endpoint locally.

Configure BitRouter

providers:
  local:
    api_base: http://127.0.0.1:11434/v1
    api_protocol:
      - "*": chat_completions
    models:
      - id: llama3.1

For an authenticated server, add an environment-backed key:

providers:
  local:
    api_base: http://127.0.0.1:8000/v1
    api_key: ${LOCAL_INFERENCE_API_KEY}
    api_protocol:
      - "*": chat_completions
    models:
      - id: my-served-model

The model id must match what the server accepts. For vLLM, that may be the full model repository id or the value supplied with --served-model-name. For Ollama, it is the pulled model tag. For Unsloth, use the id returned by its models endpoint.

Verify and route

bro config validate
bro models --provider local
bro route local:llama3.1
bro start

The provider-qualified id pins the local server. A bare model id lets BitRouter use another configured source serving the same model, which is useful for local-first fallback.

Server-specific references

• Ollama OpenAI compatibility
• vLLM OpenAI-compatible server
• Unsloth API endpoint