---
name: cloud-llm-fallback-proxy
description: "Stand up a local OpenAI-compatible HTTP proxy that fans requests out to multiple free-tier cloud LLMs (OpenRouter, etc.) and falls back to a local Ollama model when every cloud option is rate-limited. Use when the user wants to maximize free inference, route around per-model 429s, or reduce spend on a local-only Ollama box. Covers the proxy script, the discovery endpoints picky clients need (`/v1/models`, `/api/tags`, `/version`), model-name sanitization, persistent cooldown state, and Windows service registration."
version: 1.0.0
author: hermes
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [LLM, proxy, openrouter, ollama, fallback, rate-limit, free-tier]
---

# Cloud LLM Fallback Proxy

A standalone Python HTTP proxy that presents an OpenAI-compatible
`/v1/chat/completions` endpoint to clients, then tries multiple free-tier
cloud LLMs in order. When every cloud model is rate-limited (HTTP 429), it
falls back to a local Ollama model. Designed for **budget-constrained, VRAM-
limited Windows machines** (4 GB VRAM class) where local-only is too slow for
chat but paid API is undesirable.

## Architecture

```
┌──────────────┐    POST /v1/chat/completions     ┌────────────────────┐
│  client      │ ───────────────────────────────► │  Proxy on          │
│  (Hermes,    │                                  │  127.0.0.1:11435   │
│   curl, etc) │ ◄─────────────────────────────── │  (Python stdlib)   │
└──────────────┘    { choices: [...] }             └──────┬─────────────┘
                                                          │
                       ┌──────────────────────────────────┴──────────────┐
                       ▼                                  ▼              ▼
              ┌─────────────────┐               ┌────────────────┐  ┌────────────┐
              │ OpenRouter free │               │ OpenRouter free │  │ Local      │
              │ model #1        │  ...─►        │ model #N        │  │ Ollama     │
              │ (Google Gemma)  │               │ (Mistral, etc) │  │ (qwen3:4b) │
              └─────────────────┘               └────────────────┘  └────────────┘
              (429 → cooldown 60s + next)        (exhausted)        (last resort)
```

## When to use this

- User wants to use **free cloud models** as primary inference with **local
  fallback** instead of paid providers.
- User runs a **4 GB VRAM** local Ollama which is too slow for chat but
  usable as emergency fallback.
- User already has an **OpenRouter free-tier key** (or wants Cloudflare,
  Gemini free tier, etc.) and wants automatic failover when one model 429s.
- User wants Hermes / any OpenAI-compatible client to point at a single
  stable URL while the rotation happens behind it.

## When NOT to use this

- User has a paid API key with sufficient quota — just point Hermes directly.
- User wants to add a brand-new provider — extend the `call_<provider>`
  function, don't reuse this skill.
- User insists on a single fixed model — use Ollama directly.

## Prerequisites

- Python 3.11+ (uses `urllib.request` + `http.server` from stdlib — no deps)
- `OPENROUTER_API_KEY` (or equivalent) — get one at https://openrouter.ai/keys
- For Ollama fallback: Ollama installed and a model pulled (`ollama pull qwen3:4b`)
- For Windows-as-service: `nssm` (https://nssm.cc) or `pythonw.exe` + Task Scheduler

## File layout

```
~/.hermes/
├── openrouter.env                     # OPENROUTER_API_KEY=sk-or-...  (chmod 600)
├── openrouter_proxy.json              # models, URL, cooldown config
├── openrouter_proxy_state.json        # cooldown timestamps (auto-managed)
└── scripts/
    └── openrouter_proxy.py            # the proxy itself
```

## Setup (start to finish)

```bash
# 1. Save the API key with restrictive permissions
mkdir -p ~/.hermes
cat > ~/.hermes/openrouter.env << 'EOF'
OPENROUTER_API_KEY=sk-or-v1-...
EOF
# Windows: use icacls to lock it down (see "Windows file ACL" below)

# 2. Drop the config and the proxy script (see templates/ folder)

# 3. Start the proxy (background)
PATH="$HOME/.local/bin:$PATH" uv run --no-project python ~/.hermes/scripts/openrouter_proxy.py &

# 4. Verify
curl -s http://127.0.0.1:11435/health
# -> {"status": "ok", "models_available": 7, "cooldowns": {}}
```

## Verification

```bash
curl -sS -X POST http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"sag: ok"}],"max_tokens":15}'
```

A successful response includes `"choices": [{"message": {"content": "..."}}]`.
If all OpenRouter models are rate-limited, you'll see the response augmented
with `"content": "[ollama-fallback] <actual response>"`.

## Pitfalls (READ THESE — they cost an hour each)

### 1. Picky clients probe `/api/v1/models` BEFORE any chat call

Hermes (and other strict OpenAI clients) calls `GET /api/v1/models` first
and **rejects the model if it isn't in the list**. Your proxy MUST return a
JSON list with an `id` field for every model. If you skip this, the client
refuses to make any `chat/completions` call and you see "404: model not
found" without any chat traffic ever hitting your proxy.

The `scripts/openrouter_proxy.py` template implements these endpoints:

| Endpoint                              | Purpose                                 |
|---------------------------------------|-----------------------------------------|
| `GET /v1/models`                      | OpenAI-style list (REQUIRED)            |
| `GET /api/v1/models`                  | OpenAI-style list, alt path             |
| `GET /api/tags`                       | Ollama-style list (some clients probe)  |
| `GET /version`, `GET /props`          | Ollama-style version (some clients probe)|
| `GET /health`                         | for manual debug                        |

### 2. Model names with `:` get rejected by Hermes

`google/gemma-4-31b-it:free` looks like a valid OpenRouter slug, but Hermes
treats `:` as an invalid character in model IDs and rejects it. The proxy
MUST:
- Expose **sanitized** IDs in `/v1/models` (replace `:` with `_`, `/` with `-`)
- Map them back to the real OpenRouter slug in the POST handler
- Keep a single source-of-truth map (e.g. `SANITIZED_TO_REAL = {...}`)

### 3. Cooldowns must persist across restarts

If your proxy restarts, in-memory 429 cooldowns are lost. Persist them to
`~/.hermes/openrouter_proxy_state.json` after every 429. The template does
this with a `state = {"cooldowns": {model: timestamp}}` dict.

### 4. Free models claim different things than they deliver

A few of the OpenRouter `:free` models will answer with `finish_reason: "length"`
and `content: null` because they spend all output tokens on internal reasoning.
If your "test" model does this, don't conclude the proxy is broken — try another
model from the list. The template lists 6 working alternatives.

### 5. The `bash` shell on Windows MSYS corrupts some arguments

When passing Windows paths to `terminal()` that contain `$_` or `\\`, MSYS
expands them. Use `execute_code` with `subprocess.run([...])` for any call
that touches paths, JSON parsing, or process spawn. The proxy script itself
is plain Python — independent of shell quirks.

### 6. The `api_keys.openrouter` config entry does NOT register OpenRouter as a provider

Hermes' `config set api_keys.openrouter <key>` saves the key but does NOT
make `openrouter` a recognized provider. To actually use OpenRouter from
Hermes, you must either:
- Run `hermes model` interactively and pick OpenRouter from the wizard, OR
- Add `OPENROUTER_API_KEY` to `~/.hermes/.env` (the env file, not config.yaml), OR
- Use the proxy as in this skill and point `model.base_url` at it.

### 7. Ollama's failure mode on legacy GPUs (GTX 970, etc.)

Ollama 0.32+ ships CUDA kernels that require a newer nvidia driver than the
GTX 970 supports. Symptom: `Error: 500 Internal Server Error: ... CUDA
error: the provided PTX was compiled with an unsupported toolchain.`
**Fix:** install a pinned older Ollama release (e.g. v0.5.7 via
`OllamaSetup.exe` from GitHub releases) that matches the legacy driver.
Verify with `nvidia-smi` (Driver Version should be 5xx+ for CUDA 12.x).

## Windows file ACL for the API key

```powershell
# Only the current user can read the key file
icacls "C:\Users\<user>\.hermes\openrouter.env" /inheritance:r `
  /grant:r "$env:USERNAME:(R,W)"
```

## Running as a background service (Windows)

Use NSSM to install the proxy as a Windows service that auto-starts on boot:

```powershell
# Download nssm from https://nssm.cc, then:
nssm install OpenRouterProxy "C:\Users\server\AppData\Local\Programs\Python\Python311\python.exe" `
  "C:\Users\server\.hermes\scripts\openrouter_proxy.py"
nssm set OpenRouterProxy AppDirectory "C:\Users\server\.hermes"
nssm set OpenRouterProxy DisplayName "OpenRouter -> Ollama fallback proxy"
nssm set OpenRouterProxy Start SERVICE_AUTO_START
nssm start OpenRouterProxy
```

## Running on Linux / macOS

```bash
# systemd unit
cat > /etc/systemd/system/openrouter-proxy.service << 'EOF'
[Unit]
Description=OpenRouter -> Ollama fallback proxy
After=network.target ollama.service

[Service]
User=YOUR_USER
WorkingDirectory=/home/YOUR_USER/.hermes
ExecStart=/usr/bin/python3 /home/YOUR_USER/.hermes/scripts/openrouter_proxy.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now openrouter-proxy
```

## Verifying it all works

```bash
# 1. Health
curl -s http://127.0.0.1:11435/health

# 2. Models list
curl -s http://127.0.0.1:11435/v1/models | python -m json.tool

# 3. Chat
curl -s -X POST http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":20}'

# 4. Direct Ollama fallback path (force all cloud models to cooldown)
# Skip — instead, just observe that the proxy already routes to Ollama when
# all upstream models return 429.
```

## References

- `scripts/openrouter_proxy.py` — full proxy implementation, ready to copy
- `templates/openrouter_proxy.json` — config with 6 working OpenRouter free models
- `references/openrouter-free-models.md` — list of currently-free models, refreshed weekly
- `references/windows-ollama-legacy-gpu.md` — fix for CUDA toolchain errors on GTX 970 etc.