provider-drift

Did your LLM inference provider quietly change the deal? provider-drift snapshots every model + provider combination listed on OpenRouter every 6 hours and surfaces the diffs: silent price hikes, context-window downgrades, quantization swaps, uptime regressions, and model removals.

Live at: https://holyai.me/provider-drift/

Why this exists

In March 2026, researchers at CISPA published "Real Money, Fake Models: Deceptive Claims in Shadow APIs," showing that production LLM gateways routinely substitute models, swap quantization, and version-arbitrage their backends without telling the user. Open-source LLM fingerprinting tooling such as Julius (Feb 2026) is now spreading. OpenRouter is the de-facto neutral marketplace, and they expose all this metadata publicly — but nobody is logging it longitudinally and showing diffs.

provider-drift is the public, append-only audit log of that marketplace. Subscribe to alerts on the models you depend on; use it to negotiate with providers; use it to call out silent rerouting.

Data sources (real, no mocks)

| Source | URL | Auth | Frequency |
|---|---|---|---|
| Model catalog | https://openrouter.ai/api/v1/models | none | every 6h |
| Per-model endpoints | https://openrouter.ai/api/v1/models/{slug}/endpoints | none | every 6h, top-N models (N=200 by default), 250 ms between requests, exp backoff on 429/5xx |
| Manual refresh | POST /provider-drift/api/refresh | none | on demand, 5-minute debounce |

Every number on this site is fetched live from OpenRouter. Nothing here is mocked, seeded, or randomly generated. If a snapshot fails, the failure is recorded in the runs table and skipped — it is never replaced with synthetic data.

Stack

• Node.js 20+ (uses global fetch)
• Express 4
• better-sqlite3 (WAL mode)
• node-cron
• helmet + compression
• Vanilla JS SPA, dark theme, Chart.js loaded from cdnjs (no build step)

Mounted at BASE_PATH = /provider-drift. Default port 4763.

Quick start

cp .env.example .env
npm install
npm start
# open http://localhost:4763/provider-drift/
# trigger first snapshot manually:
curl -X POST http://localhost:4763/provider-drift/api/refresh

The first snapshot fetches the catalog (~360 models, ~430 KB JSON) and then per-provider endpoints for the top 200 by created desc — roughly 200 sequential requests at 250 ms apart, so plan on ~1 minute. After the first run, the cron job at 15 /6 keeps it fresh.

API reference (all endpoints public, no auth)

| Method | Path | Description |
|---|---|---|
| GET | /provider-drift/health | Health, db size, last run, counts |
| GET | /provider-drift/api/stats | Top-line counters + last_run summary |
| GET | /provider-drift/api/drift?days=7&type=price_up&severity=2&model=…&provider=…&limit=100&offset=0 | Drift events feed |
| GET | /provider-drift/api/models?limit=500&q=… | Models with provider count + 24h drift |
| GET | /provider-drift/api/models/:org/:name/timeline?days=30&provider=… | Per-model snapshot timeline |
| GET | /provider-drift/api/providers | All providers with our derived honesty score |
| GET | /provider-drift/api/providers/:name/drift?days=30 | Drift events per provider |
| POST | /provider-drift/api/refresh | Trigger a manual snapshot (5-min debounce) |
| GET | /provider-drift/rss | RSS 2.0 of last 100 events with severity ≥ 2 |
| GET | /provider-drift/feed.json | JSON Feed 1.1 equivalent |
| GET | /provider-drift/api/export.csv?days=7 | CSV export of recent drift |
| GET | /provider-drift/ | Single-page app |

Drift events

Each row in drift_events is one detected change between consecutive snapshots for the same (model_id, provider_name, endpoint_tag) triple.

| Event | Severity | Trigger |
|---|---|---|
| price_up | 3 if > 20%, else 2 | prompt/completion price increased ≥ 1% |
| price_down | 1 | prompt/completion price decreased ≥ 1% |
| context_down | 3 | context length dropped |
| context_up | 1 | context length grew |
| max_completion_change | 2 | max completion tokens changed |
| quant_change | 2 | quantization string changed |
| uptime_drop | 2 | uptime_1d dropped by ≥ 5 points |
| provider_added | 1 | first time we saw this triple |
| provider_removed | 2 | the triple disappeared from the latest run |
| model_added | 1 | first time we saw the model |
| model_removed | 2 | every provider for the model is gone |

Honesty score

A provider's honesty score is 100 − (3·price_up + 5·context_down + 2·quant_change + uptime_drop) over the last 30 days, clamped to [0, 100]. It is intentionally simple — context downgrades hurt the most because they actually break user code.

Score colour bands: ≥ 85 green, ≥ 60 amber, otherwise red.

Roadmap

• v2: deterministic prompt fingerprinting via OpenRouter (temperature: 0, single-token prompts) to detect actual model substitution, not just claim drift.
• v2: cross-check OpenRouter pricing against the model creator's official pricing page (Anthropic, OpenAI, Google) and flag suspicious spreads.
• v2: webhook + email alerts for individual (model, provider) watchlists.
• v2: CDN snapshot of every drift event for forensic citation.

Honest about limits

• OpenRouter is the only data source today. We trust their reported uptime and latency numbers; we don't measure them ourselves yet.
• We don't probe model outputs yet — claim drift is detected, output substitution is not (planned for v2).
• "Provider removed" needs two consecutive snapshots to fire — a 6h window can hide a brief outage.

License

MIT — Cowork (Claude Opus 4.7), built for holyai.me.