Makes the Open Food Facts ingestion resilient enough to bulk-seed the production database. Before this, a single transient API error or one malformed source record would abort an entire import (the loader runs one big transaction and commits at the end). Verified by loading ~460 real products into prod — https://goods.tangshasha.com now returns real search results and product detail pages (previously an empty repository).
Three independent failure modes are addressed:
Transient OFF API errors (429/5xx) — OpenFoodFactsAdapter now routes both the per-barcode and search calls through a single _get(url, params) helper that retries on connection errors and retryable statuses with exponential backoff, honouring Retry-After when present. OFF's search endpoint intermittently 503s under load, which previously killed seed runs.
Over-long source values — OFF fields can exceed our bounded columns (e.g. food_detail.serving_size VARCHAR(32)), which raised StringDataRightTruncation and rolled back the whole batch. transform() now clamps serving_size, net_content_unit, and country_of_origin to their column widths via a small _clamp(value, max_len) helper.
Per-record isolation — new load_record_safe() wraps load_record in a savepoint so one bad record is skipped instead of aborting the import. Both seed_off and update_off use it and now report an errored count alongside loaded/skipped.
Tests
test_off_retry.py: retries transient 5xx then succeeds, exhausts retries, does not retry 4xx, retries connection errors.
test_transform.py: long serving_size is clamped to 32 chars.
test_load_integration.py: a NOT-NULL-violating record is skipped while a preceding good record survives (savepoint isolation).
All ruff lint/format checks and the full pytest suite (39 passed) pass locally.
## Summary
Makes the Open Food Facts ingestion resilient enough to bulk-seed the production database. Before this, a single transient API error or one malformed source record would abort an entire import (the loader runs one big transaction and commits at the end). Verified by loading ~460 real products into prod — https://goods.tangshasha.com now returns real search results and product detail pages (previously an empty repository).
Three independent failure modes are addressed:
1. **Transient OFF API errors (429/5xx)** — `OpenFoodFactsAdapter` now routes both the per-barcode and search calls through a single `_get(url, params)` helper that retries on connection errors and retryable statuses with exponential backoff, honouring `Retry-After` when present. OFF's search endpoint intermittently 503s under load, which previously killed seed runs.
2. **Over-long source values** — OFF fields can exceed our bounded columns (e.g. `food_detail.serving_size VARCHAR(32)`), which raised `StringDataRightTruncation` and rolled back the whole batch. `transform()` now clamps `serving_size`, `net_content_unit`, and `country_of_origin` to their column widths via a small `_clamp(value, max_len)` helper.
3. **Per-record isolation** — new `load_record_safe()` wraps `load_record` in a savepoint so one bad record is skipped instead of aborting the import. Both `seed_off` and `update_off` use it and now report an `errored` count alongside `loaded`/`skipped`.
## Tests
- `test_off_retry.py`: retries transient 5xx then succeeds, exhausts retries, does not retry 4xx, retries connection errors.
- `test_transform.py`: long `serving_size` is clamped to 32 chars.
- `test_load_integration.py`: a NOT-NULL-violating record is skipped while a preceding good record survives (savepoint isolation).
All ruff lint/format checks and the full pytest suite (39 passed) pass locally.
---
## 更新：以「国内商品为主」的市场化采集（本次提交）
之前线上首批 ~460 条以欧美英文商品为主（仅 4 条 69 开头国产码），按需求改为以国内商品为主：
- `adapter.fetch_by_country(country)`：按 OFF `countries_tags_en` 过滤（如 `china`），按 `unique_scans_n`（最常被扫描）排序，跨页去重（OFF 热度排序在分页间不稳定）。
- `is_cn_gs1(code)`：判断 GS1 中国前缀（690–699），即真正的国产商品，排除「仅在中国销售」的进口品。
- `seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]`：例如 `seed_off --country china --domestic-only` 只采集 69x 国产码商品（农夫山泉/康师傅/怡宝/娃哈哈/海天/蒙牛/伊利…）。
已在生产库执行该任务导入国产商品；现有英文为主的旧数据的处理方式待与维护者确认。
- Add retry/backoff (429 + 5xx, Retry-After aware) to the OFF adapter so
transient API errors no longer abort a run.
- Clamp bounded text fields (serving_size, net_content_unit,
country_of_origin) to their column widths in transform; long OFF values
previously raised StringDataRightTruncation and rolled back the batch.
- Load each record inside a savepoint (load_record_safe) so one malformed
source record is skipped instead of aborting the whole import; jobs now
report an errored count.
- Tests for retry behaviour, serving_size clamping, and per-record isolation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add a market-focused seeding path so the catalogue can be built from
domestic products rather than the English-heavy global default:
- adapter.fetch_by_country(country): OFF search filtered by
countries_tags_en, sorted by unique_scans_n (most-scanned first),
de-duplicated across pages since OFF popularity ordering is unstable.
- is_cn_gs1(code): True for GS1-China company prefixes (690-699),
i.e. genuinely domestic items vs. imports merely sold in China.
- seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]:
e.g. 'seed_off --country china --domestic-only' loads only 69x
barcodes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Makes the Open Food Facts ingestion resilient enough to bulk-seed the production database. Before this, a single transient API error or one malformed source record would abort an entire import (the loader runs one big transaction and commits at the end). Verified by loading ~460 real products into prod — https://goods.tangshasha.com now returns real search results and product detail pages (previously an empty repository).
Three independent failure modes are addressed:
Transient OFF API errors (429/5xx) —
OpenFoodFactsAdapternow routes both the per-barcode and search calls through a single_get(url, params)helper that retries on connection errors and retryable statuses with exponential backoff, honouringRetry-Afterwhen present. OFF's search endpoint intermittently 503s under load, which previously killed seed runs.Over-long source values — OFF fields can exceed our bounded columns (e.g.
food_detail.serving_size VARCHAR(32)), which raisedStringDataRightTruncationand rolled back the whole batch.transform()now clampsserving_size,net_content_unit, andcountry_of_originto their column widths via a small_clamp(value, max_len)helper.Per-record isolation — new
load_record_safe()wrapsload_recordin a savepoint so one bad record is skipped instead of aborting the import. Bothseed_offandupdate_offuse it and now report anerroredcount alongsideloaded/skipped.Tests
test_off_retry.py: retries transient 5xx then succeeds, exhausts retries, does not retry 4xx, retries connection errors.test_transform.py: longserving_sizeis clamped to 32 chars.test_load_integration.py: a NOT-NULL-violating record is skipped while a preceding good record survives (savepoint isolation).All ruff lint/format checks and the full pytest suite (39 passed) pass locally.
更新：以「国内商品为主」的市场化采集（本次提交）
之前线上首批 ~460 条以欧美英文商品为主（仅 4 条 69 开头国产码），按需求改为以国内商品为主：
adapter.fetch_by_country(country)：按 OFFcountries_tags_en过滤（如china），按unique_scans_n（最常被扫描）排序，跨页去重（OFF 热度排序在分页间不稳定）。is_cn_gs1(code)：判断 GS1 中国前缀（690–699），即真正的国产商品，排除「仅在中国销售」的进口品。seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]：例如seed_off --country china --domestic-only只采集 69x 国产码商品（农夫山泉/康师傅/怡宝/娃哈哈/海天/蒙牛/伊利…）。已在生产库执行该任务导入国产商品；现有英文为主的旧数据的处理方式待与维护者确认。