Files
goods/migrations/0004_ingest_state.up.sql
John Doe a35bcd6647
CI / Go (api) (pull_request) Has been cancelled
CI / Python (ingestion) (pull_request) Has been cancelled
CI / Migrations (postgres) (pull_request) Has been cancelled
M4: ingestion management (incremental, GS1 supplement, dedup/conflict, quality, scheduler)
- OFF incremental fetch via search API + persistent watermark (ingest_state, migration 0004)
- GS1 barcode supplement adapter (offline mapping + GS1-style API) filling only gaps with field-level provenance
- Non-GTIN dedup with canonical selection + merge_log; field-level conflict resolution (source trust > recency)
- Quality scoring (0.4 completeness + 0.3 source trust + 0.2 multi-source + 0.1 freshness) wired into load/merge
- Jobs: update_off, dedup, schedule; docs/ingestion-management.md
- 19 new tests (pure + DB-integration), ruff clean

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-08 09:27:42 +00:00

11 lines
447 B
SQL

-- M4 ingestion management: persistent per-source incremental watermark.
-- The updater reads/writes one row per source to resume incremental imports
-- (e.g. Open Food Facts `last_modified_t`) and to record run statistics.
CREATE TABLE ingest_state (
source TEXT PRIMARY KEY,
last_modified_t BIGINT NOT NULL DEFAULT 0,
last_run_at TIMESTAMPTZ,
cursor TEXT,
stats JSONB NOT NULL DEFAULT '{}'
);