M4: ingestion management (incremental, GS1 supplement, dedup/conflict, quality, scheduler)
- OFF incremental fetch via search API + persistent watermark (ingest_state, migration 0004) - GS1 barcode supplement adapter (offline mapping + GS1-style API) filling only gaps with field-level provenance - Non-GTIN dedup with canonical selection + merge_log; field-level conflict resolution (source trust > recency) - Quality scoring (0.4 completeness + 0.3 source trust + 0.2 multi-source + 0.1 freshness) wired into load/merge - Jobs: update_off, dedup, schedule; docs/ingestion-management.md - 19 new tests (pure + DB-integration), ruff clean Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1,16 @@
|
||||
from opengoods.etl.state import get_watermark, set_watermark
|
||||
|
||||
|
||||
def test_watermark_roundtrip_and_monotonic(db_conn):
|
||||
src = "test-source"
|
||||
assert get_watermark(db_conn, src) == 0
|
||||
|
||||
set_watermark(db_conn, src, 100, stats={"loaded": 1})
|
||||
assert get_watermark(db_conn, src) == 100
|
||||
|
||||
# A lower watermark must not rewind progress.
|
||||
set_watermark(db_conn, src, 50)
|
||||
assert get_watermark(db_conn, src) == 100
|
||||
|
||||
set_watermark(db_conn, src, 150)
|
||||
assert get_watermark(db_conn, src) == 150
|
||||
Reference in New Issue
Block a user