feat(ingestion): harden OFF ingestion for bulk seeding
- Add retry/backoff (429 + 5xx, Retry-After aware) to the OFF adapter so transient API errors no longer abort a run. - Clamp bounded text fields (serving_size, net_content_unit, country_of_origin) to their column widths in transform; long OFF values previously raised StringDataRightTruncation and rolled back the batch. - Load each record inside a savepoint (load_record_safe) so one malformed source record is skipped instead of aborting the whole import; jobs now report an errored count. - Tests for retry behaviour, serving_size clamping, and per-record isolation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
@@ -65,3 +65,14 @@ def test_transform_full_record():
|
||||
|
||||
def test_transform_drops_unnamed():
|
||||
assert transform({"code": "0000000000000"}) is None
|
||||
|
||||
|
||||
def test_transform_clamps_long_serving_size():
|
||||
raw = {
|
||||
"code": "3017624010701",
|
||||
"product_name": "X",
|
||||
"serving_size": "1 portion (30 g) / 1 portion (30 g) / extra long descriptive text here",
|
||||
}
|
||||
rec = transform(raw)
|
||||
assert rec is not None
|
||||
assert len(rec["food"]["serving_size"]) == 32
|
||||
|
||||
Reference in New Issue
Block a user