M4: ingestion management (incremental, GS1 supplement, dedup/conflict, quality, scheduler)
- OFF incremental fetch via search API + persistent watermark (ingest_state, migration 0004) - GS1 barcode supplement adapter (offline mapping + GS1-style API) filling only gaps with field-level provenance - Non-GTIN dedup with canonical selection + merge_log; field-level conflict resolution (source trust > recency) - Quality scoring (0.4 completeness + 0.3 source trust + 0.2 multi-source + 0.1 freshness) wired into load/merge - Jobs: update_off, dedup, schedule; docs/ingestion-management.md - 19 new tests (pure + DB-integration), ruff clean Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
"""Shared test fixtures.
|
||||
|
||||
`db_conn` yields a psycopg connection inside a transaction that is rolled back
|
||||
after each test, so DB tests stay isolated and leave no residue. Tests are
|
||||
skipped automatically when no database is reachable or M4 migrations are not
|
||||
applied (e.g. local runs without docker).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import psycopg
|
||||
import pytest
|
||||
|
||||
from opengoods.etl.load import default_dsn
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db_conn():
|
||||
try:
|
||||
conn = psycopg.connect(default_dsn(), connect_timeout=3)
|
||||
except psycopg.OperationalError as exc: # pragma: no cover - env dependent
|
||||
pytest.skip(f"no database available: {exc}")
|
||||
has_state = conn.execute("SELECT to_regclass('public.ingest_state') IS NOT NULL").fetchone()[0]
|
||||
if not has_state:
|
||||
conn.close()
|
||||
pytest.skip("M4 migrations not applied")
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
conn.rollback()
|
||||
conn.close()
|
||||
Reference in New Issue
Block a user