Ops log

Permanent record of operational incidents and data gaps — things a future reader must be able to discover without re-deriving them from the database. Newest first. (Registry classification changes have their own log: registry-changelog.md.)

2026-07-05 — Neon project storage limit hit; ingest FAILED (DiskFull)

• The daily cron (fired late, 14:58 UTC) downloaded all three CSVs cleanly — no 429; the new retry ladder was not even needed — but the load failed: DiskFull: could not extend file because project size limit (512 MB) has been exceeded (ingest_runs id 8, status failed). Zero 2026-07-05 snapshot rows landed.
• Cause is capacity, not bloat: neondb = 463 MB with 0 dead tuples (price_snapshots 286 MB, sales_volume_snapshots 85 MB, products 83 MB); the test DB is 8 MB. Three snapshot days ≈ 460 MB → ~130 MB/day of growth. The 512 MB free tier cannot hold this pipeline; a storage upgrade (or migration) is required — a billing decision, not an engineering one.
• Recovery window for 07-05: the 17:30 UTC catch-up pass (or a manual python -m ingest snapshot --catch-up) re-downloads and loads the still-current CSVs at any point until PriceCharting’s next refresh (~12:00 UTC 2026-07-06). After that the day is permanently lost like 2026-07-04’s Pokemon.
• Resolution: Mason upgraded the Neon plan the same day (~21:00 UTC). A manual python -m ingest snapshot --catch-up at 21:30 UTC recovered the full 2026-07-05 snapshot (ingest_runs id 10, success: 703,959 price rows + 216,002 volumes, all three categories, no 429). engine daily recomputed the day’s levels — stale 0/25 across all four indexes, partial flags cleared. No day was lost. (Run id 9 was the 17:30 UTC catch-up cron doing exactly its job but hitting DiskFull pre-upgrade.) Watch storage growth (~130 MB/day) against the new plan’s ceiling.

2026-07-04 — Pokemon category snapshot permanently missed (429)

• The 13:30 UTC cron’s Pokemon CSV download got HTTP 429 (second consecutive day, after 2026-07-03); unlike 07-03, no same-day manual re-run happened, so 2026-07-04 has no pokemon-category price or sales-volume snapshots. MTG and One Piece landed normally (ingest_runs id 7, status partial).
• Consequence: the pokemon-25-psa10 (flagship) and pokemon-japan-25-psa10 levels for 2026-07-04 (both 1000.00) are 100% C5 carry-forward of 2026-07-03 prices — index_levels status partial, stale 25/25.
• The gap is permanent: PriceCharting serves only current prices, so the day cannot be backfilled. price_snapshots is append-only history; nothing is interpolated or rewritten.
• Discoverability: index_events rows (event_type='data_gap', event_date 2026-07-04) were inserted for both Pokemon indexes on 2026-07-05 as a documented one-off (SQL: insert into index_events with details naming this entry; recorded here rather than as engine code).
• Follow-up shipped 2026-07-05: in-run 429/5xx retry ladder (~2/8/20 min backoff, RETRY_WAITS_SECONDS in ingest/pricecharting.py) and the 17:30 UTC catch-up trigger (--catch-up mode) in .github/workflows/daily.yml.
• Raw-file note (added 2026-07-05 with the CSV archive): 2026-07-04 also has no raw CSVs for any category — mtg/one_piece were downloaded on the ephemeral Actions runner, pokemon never downloaded. Raw archives exist for 07-02, 07-03 (rescued from local temp), and 07-05 onward.