lixu/goods
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
73 Commits 41 Branches 0 Tags
a75318b811e8fd42a266101fa9d840d4a4c59394
Commit Graph

3 Commits

Author SHA1 Message Date
rosemariejebbjtxbfp a75318b811 feat(api): Redis read cache for product details and search
CI / Go (api) (pull_request) Successful in 54s
Details
CI / Python (ingestion) (pull_request) Successful in 19s
Details
CI / Migrations (postgres) (pull_request) Successful in 27s
Details 
Stage-0 caching from docs/scalability.md. The Go API now caches hot
product-detail and search-result reads in Redis with a fail-open,
epoch-versioned scheme; Python ingestion bumps the epoch after a write
run to invalidate the cache globally in O(1).

- api/internal/cache: fail-open Cache (GetJSON/SetJSON) namespaced by an
  epoch counter (og:cache:epoch); disabled when Redis is unconfigured.
- store: ProductByID/ProductByGTIN (24h TTL) and SearchProducts (1h TTL)
  read-through the cache via WithCache.
- ingestion: bump_cache_epoch() called after update_off/seed_off/
  import_bypos/dedup commits when rows changed; best-effort, never fails
  a run. Adds redis dependency.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-06-24 06:52:56 +00:00
novaalphastrikeomegaz663 044c870df7 feat(ingestion): harden OFF ingestion for bulk seeding
CI / Go (api) (pull_request) Failing after 22s
Details
CI / Python (ingestion) (pull_request) Successful in 14s
Details
CI / Migrations (postgres) (pull_request) Failing after 18s
Details 
- Add retry/backoff (429 + 5xx, Retry-After aware) to the OFF adapter so
  transient API errors no longer abort a run.
- Clamp bounded text fields (serving_size, net_content_unit,
  country_of_origin) to their column widths in transform; long OFF values
  previously raised StringDataRightTruncation and rolled back the batch.
- Load each record inside a savepoint (load_record_safe) so one malformed
  source record is skipped instead of aborting the whole import; jobs now
  report an errored count.
- Tests for retry behaviour, serving_size clamping, and per-record isolation.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-06-20 06:55:56 +00:00
John Doe a35bcd6647 M4: ingestion management (incremental, GS1 supplement, dedup/conflict, quality, scheduler)
CI / Go (api) (pull_request) Has been cancelled
Details
CI / Python (ingestion) (pull_request) Has been cancelled
Details
CI / Migrations (postgres) (pull_request) Has been cancelled
Details 
- OFF incremental fetch via search API + persistent watermark (ingest_state, migration 0004)
- GS1 barcode supplement adapter (offline mapping + GS1-style API) filling only gaps with field-level provenance
- Non-GTIN dedup with canonical selection + merge_log; field-level conflict resolution (source trust > recency)
- Quality scoring (0.4 completeness + 0.3 source trust + 0.2 multi-source + 0.1 freshness) wired into load/merge
- Jobs: update_off, dedup, schedule; docs/ingestion-management.md
- 19 new tests (pure + DB-integration), ruff clean

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2026-06-08 09:27:42 +00:00