# OpenRouter fallback proxy for local Ollama on resource-constrained GPUs

Use this when a user's GPU is too old / small for the latest Ollama builds
(`CUDA error: the provided PTX was compiled with an unsupported toolchain`,
or OOM on any model the user actually wants to run), and they've concluded
they want a fallback path that:

1. Tries free cloud models first (OpenRouter `:free` tier)
2. Tracks rate-limit cooldowns so the same model isn't hammered
3. Falls back to a local Ollama model when all free models are exhausted

OpenRouter is the cheapest free tier we know of — one key gives you access
to ~17 free models from Google, NVIDIA, Cohere, Mistral, Meta, etc. Each
free model has its own rate limit; the proxy rotates between them.

## Components

1. **API key** — saved to `~/.hermes/openrouter.env` with `icacls` ACL
   restricting to owner only. Plaintext is fine for single-user local boxes.
2. **Config** — `~/.hermes/openrouter_proxy.json` lists the model order
   (primary then secondary) and the Ollama fallback tag.
3. **Proxy script** — `~/.hermes/scripts/openrouter_proxy.py` — listens on
   `127.0.0.1:11435` and exposes an OpenAI-compatible `/v1/chat/completions`
   endpoint.
4. **State file** — `~/.hermes/openrouter_proxy_state.json` — tracks
   per-model cooldowns (default 60s).

## Setup

```bash
# 1. Get a free key from https://openrouter.ai/keys
# 2. Store it (Windows, restrict to owner)
echo "OPENROUTER_API_KEY=sk-or-v1-..." > ~/.hermes/openrouter.env
icacls ~/.hermes/openrouter.env /inheritance:r /grant:r "%USERNAME%:F"

# 3. Write config (~/.hermes/openrouter_proxy.json)
#    - 3-5 free models in primary tier, 3+ in secondary
#    - 1 local Ollama model as the final fallback

# 4. Run the proxy in the background
~/.hermes/scripts/openrouter_proxy.py &

# 5. Point Hermes at it
# Update ~/.hermes/config.yaml:
#   model:
#     provider: custom:ollama
#     base_url: http://127.0.0.1:11435/v1
#     default: google/gemma-4-31b-it:free
```

## Discovery endpoints (often forgotten)

Hermes probes `/v1/models`, `/api/v1/models`, `/api/tags`, `/version`,
and `/props` BEFORE issuing a chat request. If any of these return 404,
Hermes rejects the model — even if the actual chat endpoint works.

The proxy must serve **all five** endpoints:

| Path              | Format                              |
|-------------------|-------------------------------------|
| `/v1/models`      | OpenAI-style `{"object":"list","data":[{"id":"m","object":"model",...}]}` |
| `/api/v1/models`  | Same as above (older clients)       |
| `/api/tags`       | Ollama-style `{"models":[{"name":"tag","model":"tag",...}]}` |
| `/version`        | Ollama-style `{"version":"0.0.1"}`  |
| `/props`          | Accept any JSON                     |
| `/health`         | `{"status":"ok"}`                   |

## Known model failures (avoid triage time)

- `google/gemma-4-31b-it:free` — frequently 429 from Google AI Studio
  upstream. Don't put it alone in the primary tier.
- `google/gemma-4-26b-a4b-it:free` — same upstream pool, same rate limits.
- `nvidia/nemotron-3-ultra-550b-a55b:free` — 18 sec first-token latency,
  often empty content with `finish_reason: length` and `reasoning_tokens:
  22`. Useful only for short direct answers.
- Reasoning-heavy models (`inclusionai/ling-3.0-flash:free`,
  `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free`) burn `max_tokens`
  on reasoning before any visible output. Set `max_tokens=100+` or
  `max_tokens=20` and prepare for `content: null` in the response.

## Hermes integration gotcha

Hermes validates the model against the `/v1/models` list **before** sending
the request. If the listed model has a `:` in its name (e.g.
`google/gemma-4-31b-it:free`), Hermes will sometimes reject it with
`HTTP 404: model not found` even though the proxy returns a 200 response
to the actual request. Workarounds:

- Cache warmer: send a Redis-style preflight via `curl` to populate
  whatever validation cache Hermes keeps
- Rename alias: in the proxy, expose alias names without the `:` and
  rewrite them on the way out
- Or: don't use the proxy as the Hermes default — instead use it for
  cron jobs and ad-hoc scripts only

The proxy still works fine for: any tool that hits
`http://127.0.0.1:11435/v1/chat/completions` directly (`curl`, Python
`openai` SDK, custom scripts, scheduled jobs). It's the Hermes connector
that's finicky.

## Operational notes

- **Health check:** `curl http://127.0.0.1:11435/health` — returns
  `{"status":"ok","models_available":N,"cooldowns":{...}}`
- **Cooldown decay:** the proxy clears expired cooldowns on every request
  by checking `time.time() < until`.
- **Don't bounce the proxy to "clear cooldowns"** — the state file
  persists; killing the process and restarting will hit the same
  rate-limited models again immediately.
- **Rotate the API key** if you suspect leakage — OpenRouter keys are
  tied to a free tier and can be regenerated from the dashboard.