lixu/goods
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(ingestion): harden OFF ingestion for bulk seeding #6

Merged
lixu merged 3 commits from devin/1781938549-ingestion-resilience into main 2026-06-20 15:54:07 +08:00
Conversation 0 Commits 3 Files Changed 9
+333 -25
3 Commits
Author SHA1 Message Date
novaalphastrikeomegaz663andDevin AI c090bcdc9b style(ingest): ruff format country-seed tests
CI / Python (ingestion) (pull_request) Successful in 9s
Details
CI / Migrations (postgres) (pull_request) Successful in 13s
Details
CI / Go (api) (pull_request) Successful in 29s
Details 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-06-20 07:32:30 +00:00
novaalphastrikeomegaz663andDevin AI 2255243081 feat(ingest): country-focused seeding (collect domestic CN products)
CI / Python (ingestion) (pull_request) Failing after 6s
Details
CI / Migrations (postgres) (pull_request) Successful in 16s
Details
CI / Go (api) (pull_request) Failing after 11m35s
Details 
Add a market-focused seeding path so the catalogue can be built from
domestic products rather than the English-heavy global default:

- adapter.fetch_by_country(country): OFF search filtered by
  countries_tags_en, sorted by unique_scans_n (most-scanned first),
  de-duplicated across pages since OFF popularity ordering is unstable.
- is_cn_gs1(code): True for GS1-China company prefixes (690-699),
  i.e. genuinely domestic items vs. imports merely sold in China.
- seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]:
  e.g. 'seed_off --country china --domestic-only' loads only 69x
  barcodes.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-06-20 07:14:45 +00:00
novaalphastrikeomegaz663andDevin AI 044c870df7 feat(ingestion): harden OFF ingestion for bulk seeding
CI / Go (api) (pull_request) Failing after 22s
Details
CI / Python (ingestion) (pull_request) Successful in 14s
Details
CI / Migrations (postgres) (pull_request) Failing after 18s
Details 
- Add retry/backoff (429 + 5xx, Retry-After aware) to the OFF adapter so
  transient API errors no longer abort a run.
- Clamp bounded text fields (serving_size, net_content_unit,
  country_of_origin) to their column widths in transform; long OFF values
  previously raised StringDataRightTruncation and rolled back the batch.
- Load each record inside a savepoint (load_record_safe) so one malformed
  source record is skipped instead of aborting the whole import; jobs now
  report an errored count.
- Tests for retry behaviour, serving_size clamping, and per-record isolation.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-06-20 06:55:56 +00:00