feat(ingestion): harden OFF ingestion for bulk seeding #6

Merged

lixu merged 3 commits from devin/1781938549-ingestion-resilience into main 2026-06-20 15:54:07 +08:00

Conversation 0 Commits 3 Files Changed 9

+333 -25

lixu commented 2026-06-20 14:57:12 +08:00

Owner

Copy Link

Copy Source

Summary

Makes the Open Food Facts ingestion resilient enough to bulk-seed the production database. Before this, a single transient API error or one malformed source record would abort an entire import (the loader runs one big transaction and commits at the end). Verified by loading ~460 real products into prod — https://goods.tangshasha.com now returns real search results and product detail pages (previously an empty repository).

Three independent failure modes are addressed:

1. Transient OFF API errors (429/5xx) — OpenFoodFactsAdapter now routes both the per-barcode and search calls through a single _get(url, params) helper that retries on connection errors and retryable statuses with exponential backoff, honouring Retry-After when present. OFF's search endpoint intermittently 503s under load, which previously killed seed runs.

2. Over-long source values — OFF fields can exceed our bounded columns (e.g. food_detail.serving_size VARCHAR(32)), which raised StringDataRightTruncation and rolled back the whole batch. transform() now clamps serving_size, net_content_unit, and country_of_origin to their column widths via a small _clamp(value, max_len) helper.

3. Per-record isolation — new load_record_safe() wraps load_record in a savepoint so one bad record is skipped instead of aborting the import. Both seed_off and update_off use it and now report an errored count alongside loaded/skipped.

Tests

• test_off_retry.py: retries transient 5xx then succeeds, exhausts retries, does not retry 4xx, retries connection errors.
• test_transform.py: long serving_size is clamped to 32 chars.
• test_load_integration.py: a NOT-NULL-violating record is skipped while a preceding good record survives (savepoint isolation).

All ruff lint/format checks and the full pytest suite (39 passed) pass locally.

---

更新：以「国内商品为主」的市场化采集（本次提交）

之前线上首批 ~460 条以欧美英文商品为主（仅 4 条 69 开头国产码），按需求改为以国内商品为主：

• adapter.fetch_by_country(country)：按 OFF countries_tags_en 过滤（如 china），按 unique_scans_n（最常被扫描）排序，跨页去重（OFF 热度排序在分页间不稳定）。
• is_cn_gs1(code)：判断 GS1 中国前缀（690–699），即真正的国产商品，排除「仅在中国销售」的进口品。
• seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]：例如 seed_off --country china --domestic-only 只采集 69x 国产码商品（农夫山泉/康师傅/怡宝/娃哈哈/海天/蒙牛/伊利…）。

已在生产库执行该任务导入国产商品；现有英文为主的旧数据的处理方式待与维护者确认。

## Summary Makes the Open Food Facts ingestion resilient enough to bulk-seed the production database. Before this, a single transient API error or one malformed source record would abort an entire import (the loader runs one big transaction and commits at the end). Verified by loading ~460 real products into prod — https://goods.tangshasha.com now returns real search results and product detail pages (previously an empty repository). Three independent failure modes are addressed: 1. **Transient OFF API errors (429/5xx)** — `OpenFoodFactsAdapter` now routes both the per-barcode and search calls through a single `_get(url, params)` helper that retries on connection errors and retryable statuses with exponential backoff, honouring `Retry-After` when present. OFF's search endpoint intermittently 503s under load, which previously killed seed runs. 2. **Over-long source values** — OFF fields can exceed our bounded columns (e.g. `food_detail.serving_size VARCHAR(32)`), which raised `StringDataRightTruncation` and rolled back the whole batch. `transform()` now clamps `serving_size`, `net_content_unit`, and `country_of_origin` to their column widths via a small `_clamp(value, max_len)` helper. 3. **Per-record isolation** — new `load_record_safe()` wraps `load_record` in a savepoint so one bad record is skipped instead of aborting the import. Both `seed_off` and `update_off` use it and now report an `errored` count alongside `loaded`/`skipped`. ## Tests - `test_off_retry.py`: retries transient 5xx then succeeds, exhausts retries, does not retry 4xx, retries connection errors. - `test_transform.py`: long `serving_size` is clamped to 32 chars. - `test_load_integration.py`: a NOT-NULL-violating record is skipped while a preceding good record survives (savepoint isolation). All ruff lint/format checks and the full pytest suite (39 passed) pass locally. --- ## 更新：以「国内商品为主」的市场化采集（本次提交） 之前线上首批 ~460 条以欧美英文商品为主（仅 4 条 69 开头国产码），按需求改为以国内商品为主： - `adapter.fetch_by_country(country)`：按 OFF `countries_tags_en` 过滤（如 `china`），按 `unique_scans_n`（最常被扫描）排序，跨页去重（OFF 热度排序在分页间不稳定）。 - `is_cn_gs1(code)`：判断 GS1 中国前缀（690–699），即真正的国产商品，排除「仅在中国销售」的进口品。 - `seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]`：例如 `seed_off --country china --domestic-only` 只采集 69x 国产码商品（农夫山泉/康师傅/怡宝/娃哈哈/海天/蒙牛/伊利…）。 已在生产库执行该任务导入国产商品；现有英文为主的旧数据的处理方式待与维护者确认。

lixu added 1 commit 2026-06-20 14:57:13 +08:00

feat(ingestion): harden OFF ingestion for bulk seeding ... 044c870df7

- Add retry/backoff (429 + 5xx, Retry-After aware) to the OFF adapter so
  transient API errors no longer abort a run.
- Clamp bounded text fields (serving_size, net_content_unit,
  country_of_origin) to their column widths in transform; long OFF values
  previously raised StringDataRightTruncation and rolled back the batch.
- Load each record inside a savepoint (load_record_safe) so one malformed
  source record is skipped instead of aborting the whole import; jobs now
  report an errored count.
- Tests for retry behaviour, serving_size clamping, and per-record isolation.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

lixu added 1 commit 2026-06-20 15:14:53 +08:00

feat(ingest): country-focused seeding (collect domestic CN products) ... 2255243081

Add a market-focused seeding path so the catalogue can be built from
domestic products rather than the English-heavy global default:

- adapter.fetch_by_country(country): OFF search filtered by
  countries_tags_en, sorted by unique_scans_n (most-scanned first),
  de-duplicated across pages since OFF popularity ordering is unstable.
- is_cn_gs1(code): True for GS1-China company prefixes (690-699),
  i.e. genuinely domestic items vs. imports merely sold in China.
- seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]:
  e.g. 'seed_off --country china --domestic-only' loads only 69x
  barcodes.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

lixu added 1 commit 2026-06-20 15:32:34 +08:00

style(ingest): ruff format country-seed tests ... c090bcdc9b

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

lixu merged commit 7d7a8f1baf into main 2026-06-20 15:54:07 +08:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lixu

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lixu/goods#6