// data engineering [Open source]

LimitWatch Data Pipeline

Append-only snapshots, derived changelogs, and measured cap burn

Data engineering for AI plan tracking

LimitWatch is not a scraper that guesses. It is a data pipeline with provenance: dated snapshots you can audit, a build step that derives changelogs with provider-scoped diff logic, automation that flags when official pages move, and a separate measured-usage track that pairs in-app cap percentages with real Claude Code token burn.

Node.js
JSON snapshots
Cloudflare Pages
GitHub Actions
patchright
Wayback Machine

// architecture

Human snapshot ──► build.mjs merge + diff ──► site/data.json ──► static page ──► Cloudflare Pages

// how it's built

~ Append-only snapshot store

data/snapshots/YYYY-MM-DD.json holds one dated capture per file. History is never rewritten — the moat is the time series, not a single pricing page copy.

~ Provider-scoped changelog derivation

scripts/build.mjs diffs each snapshot against the previous one that covers the same providers. Partial captures declare covers: ["Google", "Cursor"] so interleaved dates never fabricate added/removed rows for providers they did not record.

~ Event spans vs point-in-time facts

Temporary promos and throttle windows live in data/events.json as starts_on/ends_on spans — not baked into snapshots. The static page computes active, upcoming, and ended states from today without redeploying to expire them.

~ Wayback bootstrap

CDX search + archive.org captures seed months of history before live watching accumulates. Values are verbatim from the capture; source cites web.archive.org. Unreadable numbers stay null — never guessed.

~ watch + discover automation

fetch.mjs hashes tracked pages and flags only what changed. discover.mjs polls RSS/HTML feeds for new announcements CI cannot catch from diffs alone. HTTP sources run weekly in GitHub Actions; Cloudflare-protected pages use headed patchright locally.

~ Measured usage floor

cc-usage.mjs reads Claude Code session transcripts, dedupes assistant turns by message.id, and prices tokens at API list rates. usage-reports.json pairs observed cap % with burn; build.mjs strips internal account labels and rolls up implied budget ranges for the public site.

~ Static deploy, zero backend

npm run build emits site/data.json + static HTML. Cloudflare Pages deploys on push to main — no database, no runtime API. The pipeline is the product infrastructure.

// highlights

Time series as moat

Snapshots are copyable; a six-month derived changelog with source links is not. Every number on the page traces back to a dated entry.

Wayback backfill

Bootstrap provider history from archive.org without waiting for organic captures — critical for showing limit drift, not just today's quota.

Automation with human gate

CI opens issues when sources move or feeds publish new items. Watchers never edit snapshots — they flag what a human should re-verify.

Cap % + measured burn

Subscription 5h/weekly caps have no API. Pairing in-app % with transcript burn yields implied budget ranges — a floor competitors cannot scrape.

Privacy-safe publishing

Internal account tags stay local; shipped data strips them. On-page copy stays generic — no login labels leak.

Model swap collapse

Same-date remove+add of one limit slot that only swapped models collapses to a single model_changed row instead of noisy churn.

// components

Snapshot + schema layer

Dated entries with source URLs per fact, optional covers scoping, and schema docs for what belongs in snapshots vs events.json.

Build + diff engine

build.mjs merges snapshots, derives changelog, emits usage rollups, and renders the static site bundle consumed by Cloudflare Pages.

Watch + measure ops

fetch.mjs, discover.mjs, cc-usage.mjs, and GitHub Actions workflows — the operational loop that keeps the time series current.

// related work