Stage-0 caching from docs/scalability.md. The Go API now caches hot
product-detail and search-result reads in Redis with a fail-open,
epoch-versioned scheme; Python ingestion bumps the epoch after a write
run to invalidate the cache globally in O(1).
- api/internal/cache: fail-open Cache (GetJSON/SetJSON) namespaced by an
epoch counter (og:cache:epoch); disabled when Redis is unconfigured.
- store: ProductByID/ProductByGTIN (24h TTL) and SearchProducts (1h TTL)
read-through the cache via WithCache.
- ingestion: bump_cache_epoch() called after update_off/seed_off/
import_bypos/dedup commits when rows changed; best-effort, never fails
a run. Adds redis dependency.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add a market-focused seeding path so the catalogue can be built from
domestic products rather than the English-heavy global default:
- adapter.fetch_by_country(country): OFF search filtered by
countries_tags_en, sorted by unique_scans_n (most-scanned first),
de-duplicated across pages since OFF popularity ordering is unstable.
- is_cn_gs1(code): True for GS1-China company prefixes (690-699),
i.e. genuinely domestic items vs. imports merely sold in China.
- seed_off --country <slug> [--domestic-only] [--page-size/--max-pages]:
e.g. 'seed_off --country china --domestic-only' loads only 69x
barcodes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Add retry/backoff (429 + 5xx, Retry-After aware) to the OFF adapter so
transient API errors no longer abort a run.
- Clamp bounded text fields (serving_size, net_content_unit,
country_of_origin) to their column widths in transform; long OFF values
previously raised StringDataRightTruncation and rolled back the batch.
- Load each record inside a savepoint (load_record_safe) so one malformed
source record is skipped instead of aborting the whole import; jobs now
report an errored count.
- Tests for retry behaviour, serving_size clamping, and per-record isolation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
rosemariejebbjtxbfp
lixu