Compare commits

..

16 Commits

Author SHA1 Message Date
oyaegeli98668 5bfb2d3a5f chore: 站名改为「天工商品档案公共仓」(公开站 + 后台标题/页头/页脚/API 文案)
CI / Go (api) (pull_request) Failing after 15s
CI / Python (ingestion) (pull_request) Successful in 7s
CI / Migrations (postgres) (pull_request) Failing after 19s
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 05:37:23 +00:00
oyaegeli98668 8909ddcb69 chore(public): 页脚添加 ICP 备案号 鲁ICP备2025185218号-6
CI / Go (api) (pull_request) Failing after 20s
CI / Python (ingestion) (pull_request) Successful in 8s
CI / Migrations (postgres) (pull_request) Failing after 16s
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 05:33:58 +00:00
oyaegeli98668 2d95052bd9 feat(public): 首页新增 API 调用说明页(只读端点/参数/示例/免责声明)
CI / Python (ingestion) (pull_request) Successful in 6s
CI / Go (api) (pull_request) Failing after 17s
CI / Migrations (postgres) (pull_request) Failing after 20s
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 05:26:53 +00:00
oyaegeli98668 c9a4404052 feat: 公开首页(搜索+商品详情) + 好心人投稿 + 后台审核收纳
CI / Go (api) (pull_request) Failing after 20s
CI / Python (ingestion) (pull_request) Successful in 8s
CI / Migrations (postgres) (pull_request) Failing after 17s
- 公开前端 SPA(根路径 /):首页大搜索框、检索结果、只读商品详情、贡献档案表单
- 公开写入端点 POST /api/public/submissions(无需登录,基础频率限流),投稿进入 submission 待审核队列,不直接写 product
- 迁移 0006:submission 投稿表 + community 来源(trust=0.50)
- 后台审核队列:列表(待审核/已通过/已驳回) → 查看投稿 → 通过(创建/补全商品 + 记 source=community + 字段级溯源 + 审计 + 重算质量分) / 驳回(记原因)
- 公开只读 api 服务内嵌公开 SPA;Dockerfile.prod 增加 node 构建阶段 + 内嵌 dist

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 05:11:07 +00:00
oyaegeli98668 bad771e172 build(admin): npm 走 npmmirror 镜像以适配部署环境网络
CI / Go (api) (pull_request) Failing after 19s
CI / Python (ingestion) (pull_request) Successful in 7s
CI / Migrations (postgres) (pull_request) Failing after 21s
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 02:54:32 +00:00
oyaegeli98668 d90a539e6b feat(admin): 运营后台(登录/查看/审核编辑/补全)+ 写入API + 审计留痕
CI / Go (api) (pull_request) Failing after 18s
CI / Python (ingestion) (pull_request) Successful in 7s
CI / Migrations (postgres) (pull_request) Failing after 18s
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 02:53:30 +00:00
lixu c272b2c5d7 Merge pull request 'chore(deploy): 生产 docker compose 部署配置' (#3) from devin/1781922421-prod-deploy into main
CI / Go (api) (push) Failing after 20s
CI / Python (ingestion) (push) Successful in 6s
CI / Migrations (postgres) (push) Failing after 19s
2026-06-20 10:32:22 +08:00
oyaegeli98668 6777461268 chore(deploy): 生产 docker compose + scratch Dockerfile + 部署文档
CI / Go (api) (pull_request) Failing after 41s
CI / Python (ingestion) (pull_request) Successful in 43s
CI / Migrations (postgres) (pull_request) Failing after 17s
- docker-compose.prod.yml:仅 api 绑 127.0.0.1:8120,DB/redis/minio 不暴露,restart=unless-stopped,凭据走 .env
- api/Dockerfile.prod:runtime 改用 scratch + 拷贝 ca-certs(gcr.io/distroless 不可达环境),Go 模块走 goproxy.cn
- .env.example / docs/deploy.md

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-20 02:27:09 +00:00
lixu e3e5c7979c ci: 修复 Gitea Actions 数据库连接(用服务名 postgres) (#2)
CI / Go (api) (push) Successful in 6s
CI / Python (ingestion) (push) Successful in 6s
CI / Migrations (postgres) (push) Successful in 16s
2026-06-19 17:47:41 +08:00
rosemariejebbjtxbfp 19b7c43f37 ci: connect to postgres via service hostname for Gitea Actions
CI / Go (api) (pull_request) Successful in 30s
CI / Python (ingestion) (pull_request) Successful in 7s
CI / Migrations (postgres) (pull_request) Successful in 16s
Gitea Actions runs jobs inside a container, so a service container is reachable by its service name (postgres), not localhost; localhost:5432 yields connection refused. Also drop the unnecessary 5432:5432 host port mapping that caused 'port already allocated' collisions.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-19 08:54:18 +00:00
lixu 57537c880d M4: ingestion management (#1)
CI / Go (api) (push) Failing after 14s
CI / Migrations (postgres) (push) Failing after 33s
CI / Python (ingestion) (push) Failing after 13m10s
Incremental OFF + GS1 supplement + dedup/conflict + quality scoring + scheduler
2026-06-08 17:40:26 +08:00
John Doe a35bcd6647 M4: ingestion management (incremental, GS1 supplement, dedup/conflict, quality, scheduler)
CI / Go (api) (pull_request) Has been cancelled
CI / Python (ingestion) (pull_request) Has been cancelled
CI / Migrations (postgres) (pull_request) Has been cancelled
- OFF incremental fetch via search API + persistent watermark (ingest_state, migration 0004)
- GS1 barcode supplement adapter (offline mapping + GS1-style API) filling only gaps with field-level provenance
- Non-GTIN dedup with canonical selection + merge_log; field-level conflict resolution (source trust > recency)
- Quality scoring (0.4 completeness + 0.3 source trust + 0.2 multi-source + 0.1 freshness) wired into load/merge
- Jobs: update_off, dedup, schedule; docs/ingestion-management.md
- 19 new tests (pure + DB-integration), ruff clean

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-08 09:27:42 +00:00
lixu e750501b44 feat(M3): 只读 API 端点实现
CI / Go (api) (push) Has been cancelled
CI / Python (ingestion) (push) Has been cancelled
CI / Migrations (postgres) (push) Has been cancelled
- store: pgx 只读数据访问层(productByGTIN/ByID/search/nutriments/msrp/brands/categories/source)
- handler: 真实查询替换 501 占位, 统一分页 + 错误信封, MSRP 带免责声明无购买入口
- main: pgxpool 连接池接线
- search: 名称模糊 + 分类子树过滤(ltree <@)
- 测试: healthz/pageParams 单测 + DB-backed handler 集成测试(无库自动跳过)
- CI: Go job 增加 postgres service + migrate up, 实跑 DB 测试
- 依赖: pgx v5.7.2 (固定到兼容 go1.23 的版本)
- 本地实跑: 8 个端点对真实 OFF 数据返回正确(barcode/search/nutriments/msrp/brands/categories/source/404)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-08 07:14:18 +00:00
lixu 766a573989 feat(M2): Open Food Facts 采集导入 ETL
- adapters/openfoodfacts.py: OFF API 适配器(限速+User-Agent) + JSONL/.gz dump 读取
- etl/transform.py: 纯函数转换(GTIN 校验/净含量解析+归一/营养 per_100g 双能量/过敏原添加剂清洗/关键词分类映射)
- etl/load.py: psycopg upsert(product/food_detail/product_image) + product_source 字段级溯源
- jobs/seed_off.py: CLI(--barcodes API / --dump 文件 / --limit)
- 测试: 13 个离线 transform 单测(fixture) + 可跳过的 DB 集成测试
- 依赖: 增加 psycopg[binary]
- 实跑: 从 OFF API 拉 5 个真实条码入本地 postgres 验证通过
- docs/etl-openfoodfacts.md: 流程/运行/字段映射

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-08 07:01:04 +00:00
lixu b0b816b0ee feat(M1): 数据模型迁移 + GS1 GPC 食品分类 + 单位字典
- migrations/0001_init: 全部核心表(product/food_detail/product_msrp/product_source/brand/manufacturer/category/category_schema/unit/attribute_definition/product_image/merge_log) + 索引(gtin唯一/name trigram/JSONB GIN/category ltree/tsvector) + tsvector/updated_at 触发器
- 0002_seed_units: 单位字典(与 units.py 一致, 含中文别名) + 常用营养参数定义
- 0003_seed_categories: 食品品类骨架(GS1 GPC 映射 + 自建中文树, ltree) + 品类参数模板(营养基准 per_100g/ml)
- CI 增加 migrations job: 用 postgres service 跑 migrate up + down 验证可逆
- 本地实跑 up/down/re-up 全部通过

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-08 06:32:16 +00:00
lixu 786b7d3721 feat(M0): 工程地基 - Go API + Python 采集骨架 + CI + 数据契约
- api/: Go(chi) 只读 API 骨架, /healthz + 版本化路由(占位), Dockerfile, 单测
- ingestion/: Python 采集/ETL 包骨架, units 单位归一化(纯函数+测试), adapter 协议
- docker-compose.yml: postgres + redis + minio + api
- .github/workflows/ci.yml: Go build/vet/test + Python ruff/pytest
- docs/data-contract.md(两端共享契约) + docs/disclaimer.md(不提供购买声明)
- migrations/ 占位(M1 起填充)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-06-08 06:18:31 +00:00
127 changed files with 14165 additions and 1486 deletions
+14
View File
@@ -0,0 +1,14 @@
.git
**/node_modules
admin-frontend/dist
public-frontend/dist
api/server
*.test
*.out
__pycache__
.venv
.pytest_cache
.ruff_cache
.env
.env.*
!.env.example
+12
View File
@@ -0,0 +1,12 @@
# Copy to .env and fill in real values before running docker-compose.prod.yml.
# Used by docker-compose.prod.yml for production deployment.
POSTGRES_USER=opengoods
POSTGRES_PASSWORD=change-me
POSTGRES_DB=opengoods
MINIO_ROOT_USER=opengoods
MINIO_ROOT_PASSWORD=change-me
# Admin console (served at /ping). Set a strong password and a random JWT secret.
GOODS_ADMIN_USER=admin
GOODS_ADMIN_PASSWORD=change-me
GOODS_ADMIN_JWT_SECRET=change-me-to-a-long-random-string
+89
View File
@@ -0,0 +1,89 @@
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
go:
name: Go (api)
runs-on: ubuntu-latest
defaults:
run:
working-directory: api
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: opengoods
POSTGRES_PASSWORD: opengoods
POSTGRES_DB: opengoods
options: >-
--health-cmd "pg_isready -U opengoods"
--health-interval 5s --health-timeout 5s --health-retries 10
env:
OPENGOODS_DATABASE_URL: postgres://opengoods:opengoods@postgres:5432/opengoods?sslmode=disable
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.23"
cache-dependency-path: api/go.sum
- name: Apply migrations
working-directory: .
run: |
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@v4.18.1
migrate -path migrations -database "$OPENGOODS_DATABASE_URL" up
- name: Verify gofmt
run: test -z "$(gofmt -l .)"
- run: go vet ./...
- run: go build ./...
- run: go test ./...
python:
name: Python (ingestion)
runs-on: ubuntu-latest
defaults:
run:
working-directory: ingestion
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install
run: pip install -e ".[dev]"
- name: Ruff lint
run: ruff check .
- name: Ruff format check
run: ruff format --check .
- name: Pytest
run: pytest -q
migrations:
name: Migrations (postgres)
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: opengoods
POSTGRES_PASSWORD: opengoods
POSTGRES_DB: opengoods
options: >-
--health-cmd "pg_isready -U opengoods"
--health-interval 5s --health-timeout 5s --health-retries 10
env:
DBURL: postgres://opengoods:opengoods@postgres:5432/opengoods?sslmode=disable
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.23"
- name: Install golang-migrate
run: go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@v4.18.1
- name: Migrate up
run: migrate -path migrations -database "$DBURL" up
- name: Migrate down (reversibility)
run: migrate -path migrations -database "$DBURL" down -all
+34
View File
@@ -0,0 +1,34 @@
# Go
/api/server
*.test
*.out
# Python
__pycache__/
*.py[cod]
.venv/
.pytest_cache/
.ruff_cache/
*.egg-info/
build/
dist/
# Env / local
.env
.env.*
!.env.example
# Node / admin frontend
node_modules/
# Keep the embedded SPA placeholders (real builds are injected during Docker build)
!api/internal/adminweb/dist/
!api/internal/adminweb/dist/index.html
!api/internal/publicweb/dist/
!api/internal/publicweb/dist/index.html
# OS / editors
.DS_Store
*.swp
.idea/
.vscode/
+2 -24
View File
@@ -1,24 +1,2 @@
# 天工·商品标签 (OpenGoods)
商品档案公开 API —— 公益网站/服务:**采集全网商品信息,对外提供商品参数查询 API**。
> 核心原则:**只采集 + 只提供信息,绝不涉及任何购买/下单/比价导购。**
## 这是什么
- 开放、中立、可溯源的「商品参数百科 + 开放 API」
- 首批聚焦 **食品快消**,对外提供按条码/名称查询商品参数(成分、营养、规格、官方建议零售价等)
- 技术栈:**Go**(对外只读 API) + **Python**(采集/ETL),经 PostgreSQL + Redis/队列解耦
## 项目状态
规划阶段。完整方案见 [`docs/planning/`](./docs/planning/README.md)
- [最终规划](./docs/planning/00-final-plan.md)(决策 + 架构 + 任务清单)
- [详细设计 v2.0](./docs/planning/01-detailed-design-v2.0.md)(分类/单位/数据库/API/治理/采集/部署)
- [进阶专题 v3.0](./docs/planning/02-advanced-topics-v3.0.md)OpenAPI/DDL/合规/测试/安全/SLA/竞品)
## 许可
- 代码:拟用 Apache-2.0 / MIT(待定)
- 数据:拟用 **ODbL + 署名**(因采用 Open Food Facts 等开放数据源)
# goods
商品档案公开API
+12
View File
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="zh">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>天工商品档案公共仓 · 管理后台</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>
+2690
View File
File diff suppressed because it is too large Load Diff
+26
View File
@@ -0,0 +1,26 @@
{
"name": "opengoods-admin-frontend",
"private": true,
"version": "1.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"preview": "vite preview"
},
"dependencies": {
"react": "^18.2.0",
"react-dom": "^18.2.0",
"lucide-react": "^0.344.0"
},
"devDependencies": {
"@types/react": "^18.2.55",
"@types/react-dom": "^18.2.19",
"@vitejs/plugin-react": "^4.2.1",
"autoprefixer": "^10.4.17",
"postcss": "^8.4.35",
"tailwindcss": "^3.4.1",
"typescript": "^5.3.3",
"vite": "^5.1.0"
}
}
+6
View File
@@ -0,0 +1,6 @@
export default {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
};
+125
View File
@@ -0,0 +1,125 @@
import { useEffect, useState } from "react";
import { api, clearToken, getToken } from "./api";
import Login from "./components/Login";
import ProductList from "./components/ProductList";
import ProductDetail from "./components/ProductDetail";
import SubmissionsPage from "./components/SubmissionsPage";
import { Inbox, LogOut, Package } from "lucide-react";
type Tab = "products" | "submissions";
type View = { name: "list" } | { name: "detail"; id: string };
export default function App() {
const [authed, setAuthed] = useState(false);
const [checking, setChecking] = useState(true);
const [username, setUsername] = useState("");
const [tab, setTab] = useState<Tab>("products");
const [pending, setPending] = useState<number | null>(null);
const [view, setView] = useState<View>({ name: "list" });
useEffect(() => {
if (!authed) return;
api
.listSubmissions("pending", 1, 1)
.then((r) => setPending(r.pending))
.catch(() => undefined);
}, [authed]);
useEffect(() => {
if (!getToken()) {
setChecking(false);
return;
}
api
.me()
.then((r) => {
setUsername(r.username);
setAuthed(true);
})
.catch(() => clearToken())
.finally(() => setChecking(false));
}, []);
function onLoggedIn(name: string) {
setUsername(name);
setAuthed(true);
setView({ name: "list" });
}
function logout() {
clearToken();
setAuthed(false);
setUsername("");
}
if (checking) {
return (
<div className="flex h-full items-center justify-center text-gray-500">
</div>
);
}
if (!authed) return <Login onLoggedIn={onLoggedIn} />;
return (
<div className="flex h-full flex-col">
<header className="flex items-center justify-between bg-white px-6 py-3 shadow-sm">
<div className="flex items-center gap-6">
<div className="flex items-center gap-2 text-lg font-semibold text-gray-800">
<Package className="h-5 w-5 text-emerald-600" />
·
</div>
<nav className="flex items-center gap-1 text-sm">
<button
onClick={() => {
setTab("products");
setView({ name: "list" });
}}
className={`px-3 py-1.5 rounded-md flex items-center gap-1.5 ${
tab === "products"
? "bg-emerald-50 text-emerald-700"
: "text-gray-600 hover:bg-gray-100"
}`}
>
<Package className="h-4 w-4" />
</button>
<button
onClick={() => setTab("submissions")}
className={`px-3 py-1.5 rounded-md flex items-center gap-1.5 ${
tab === "submissions"
? "bg-emerald-50 text-emerald-700"
: "text-gray-600 hover:bg-gray-100"
}`}
>
<Inbox className="h-4 w-4" /> 稿
{pending != null && pending > 0 && (
<span className="ml-1 text-xs bg-amber-500 text-white rounded-full px-1.5">
{pending}
</span>
)}
</button>
</nav>
</div>
<div className="flex items-center gap-4 text-sm text-gray-600">
<span>{username}</span>
<button
onClick={logout}
className="flex items-center gap-1 rounded px-2 py-1 text-gray-500 hover:bg-gray-100 hover:text-gray-800"
>
<LogOut className="h-4 w-4" /> 退
</button>
</div>
</header>
<main className="flex-1 overflow-auto p-6">
{tab === "submissions" ? (
<SubmissionsPage onPending={setPending} />
) : view.name === "list" ? (
<ProductList onOpen={(id) => setView({ name: "detail", id })} />
) : (
<ProductDetail id={view.id} onBack={() => setView({ name: "list" })} />
)}
</main>
</div>
);
}
+112
View File
@@ -0,0 +1,112 @@
// API base derives from Vite's BASE_URL (/ping/) so it matches the nginx prefix.
const API_BASE = `${import.meta.env.BASE_URL}api`;
const TOKEN_KEY = "opengoods_admin_token";
export function getToken(): string | null {
return localStorage.getItem(TOKEN_KEY);
}
export function setToken(token: string) {
localStorage.setItem(TOKEN_KEY, token);
}
export function clearToken() {
localStorage.removeItem(TOKEN_KEY);
}
export class ApiError extends Error {
status: number;
constructor(status: number, message: string) {
super(message);
this.status = status;
}
}
async function request<T>(path: string, options: RequestInit = {}): Promise<T> {
const headers: Record<string, string> = {
"Content-Type": "application/json",
...(options.headers as Record<string, string>),
};
const token = getToken();
if (token) headers.Authorization = `Bearer ${token}`;
const res = await fetch(`${API_BASE}${path}`, { ...options, headers });
if (res.status === 401) {
clearToken();
throw new ApiError(401, "登录已过期,请重新登录");
}
const text = await res.text();
const data = text ? JSON.parse(text) : null;
if (!res.ok) {
const msg = data?.error?.message || `请求失败 (${res.status})`;
throw new ApiError(res.status, msg);
}
return data as T;
}
export const api = {
login: (username: string, password: string) =>
request<{ token: string; username: string }>("/login", {
method: "POST",
body: JSON.stringify({ username, password }),
}),
me: () => request<{ username: string }>("/me"),
listProducts: (q: string, page: number, size: number) =>
request<{
items: import("./types").ProductRow[];
page: number;
size: number;
total: number;
completeness_fields: string[];
}>(`/products?q=${encodeURIComponent(q)}&page=${page}&size=${size}`),
getProduct: (id: string) =>
request<import("./types").ProductDetail>(`/products/${id}`),
updateProduct: (id: string, body: unknown) =>
request<import("./types").ProductDetail>(`/products/${id}`, {
method: "PUT",
body: JSON.stringify(body),
}),
listAudit: (id: string) =>
request<{ items: import("./types").AuditEntry[] }>(`/products/${id}/audit`),
addImage: (id: string, url: string, kind: string) =>
request<import("./types").ProductImage>(`/products/${id}/images`, {
method: "POST",
body: JSON.stringify({ url, kind }),
}),
deleteImage: (id: string, imageId: string) =>
request<{ status: string }>(`/products/${id}/images/${imageId}`, {
method: "DELETE",
}),
addMsrp: (id: string, body: unknown) =>
request<import("./types").MSRP>(`/products/${id}/msrp`, {
method: "POST",
body: JSON.stringify(body),
}),
deleteMsrp: (id: string, msrpId: string) =>
request<{ status: string }>(`/products/${id}/msrp/${msrpId}`, {
method: "DELETE",
}),
listBrands: () =>
request<{ items: import("./types").Brand[] }>("/brands"),
listCategories: () =>
request<{ items: import("./types").Category[] }>("/categories"),
listSubmissions: (status: string, page: number, size: number) =>
request<{
items: import("./types").SubmissionRow[];
page: number;
size: number;
total: number;
pending: number;
}>(`/submissions?status=${encodeURIComponent(status)}&page=${page}&size=${size}`),
getSubmission: (id: string) =>
request<import("./types").SubmissionDetail>(`/submissions/${id}`),
approveSubmission: (id: string) =>
request<import("./types").ProductDetail>(`/submissions/${id}/approve`, {
method: "POST",
}),
rejectSubmission: (id: string, note: string) =>
request<{ status: string }>(`/submissions/${id}/reject`, {
method: "POST",
body: JSON.stringify({ note }),
}),
};
+75
View File
@@ -0,0 +1,75 @@
import { useState } from "react";
import { api, setToken } from "../api";
import { Package } from "lucide-react";
export default function Login({
onLoggedIn,
}: {
onLoggedIn: (username: string) => void;
}) {
const [username, setUsername] = useState("");
const [password, setPassword] = useState("");
const [error, setError] = useState("");
const [loading, setLoading] = useState(false);
async function submit(e: React.FormEvent) {
e.preventDefault();
setError("");
setLoading(true);
try {
const r = await api.login(username, password);
setToken(r.token);
onLoggedIn(r.username);
} catch (err) {
setError(err instanceof Error ? err.message : "登录失败");
} finally {
setLoading(false);
}
}
return (
<div className="flex h-full items-center justify-center">
<form
onSubmit={submit}
className="w-80 rounded-xl bg-white p-8 shadow-md"
>
<div className="mb-6 flex flex-col items-center gap-2">
<Package className="h-8 w-8 text-emerald-600" />
<h1 className="text-lg font-semibold text-gray-800">
·
</h1>
</div>
{error && (
<div className="mb-4 rounded bg-red-50 px-3 py-2 text-sm text-red-600">
{error}
</div>
)}
<label className="mb-3 block">
<span className="mb-1 block text-sm text-gray-600"></span>
<input
value={username}
onChange={(e) => setUsername(e.target.value)}
className="w-full rounded border border-gray-300 px-3 py-2 text-sm focus:border-emerald-500 focus:outline-none"
autoFocus
/>
</label>
<label className="mb-5 block">
<span className="mb-1 block text-sm text-gray-600"></span>
<input
type="password"
value={password}
onChange={(e) => setPassword(e.target.value)}
className="w-full rounded border border-gray-300 px-3 py-2 text-sm focus:border-emerald-500 focus:outline-none"
/>
</label>
<button
type="submit"
disabled={loading}
className="w-full rounded bg-emerald-600 py-2 text-sm font-medium text-white hover:bg-emerald-700 disabled:opacity-60"
>
{loading ? "登录中…" : "登录"}
</button>
</form>
</div>
);
}
@@ -0,0 +1,642 @@
import { useEffect, useMemo, useState } from "react";
import { api } from "../api";
import {
AuditEntry,
Brand,
Category,
FIELD_LABELS,
ProductDetail as Detail,
} from "../types";
import {
ArrowLeft,
Plus,
Save,
Trash2,
AlertCircle,
History,
} from "lucide-react";
const NUTRIMENT_KEYS: { key: string; label: string }[] = [
{ key: "energy_kcal", label: "能量 (kcal)" },
{ key: "energy_kj", label: "能量 (kJ)" },
{ key: "fat", label: "脂肪 (g)" },
{ key: "saturated_fat", label: "饱和脂肪 (g)" },
{ key: "carbohydrates", label: "碳水 (g)" },
{ key: "sugars", label: "糖 (g)" },
{ key: "proteins", label: "蛋白质 (g)" },
{ key: "salt", label: "盐 (g)" },
];
const STATUS_OPTIONS = [
{ value: "active", label: "在用" },
{ value: "merged", label: "已合并" },
{ value: "deprecated", label: "已停用" },
];
const ACTION_LABEL: Record<string, string> = {
update: "编辑",
add_image: "新增图片",
delete_image: "删除图片",
add_msrp: "新增建议零售价",
delete_msrp: "删除建议零售价",
};
function Card({
title,
children,
}: {
title: string;
children: React.ReactNode;
}) {
return (
<div className="rounded-lg border border-gray-200 bg-white p-5">
<h3 className="mb-4 text-sm font-semibold text-gray-700">{title}</h3>
{children}
</div>
);
}
function Field({
label,
children,
}: {
label: string;
children: React.ReactNode;
}) {
return (
<label className="block">
<span className="mb-1 block text-xs text-gray-500">{label}</span>
{children}
</label>
);
}
const inputCls =
"w-full rounded border border-gray-300 px-3 py-2 text-sm focus:border-emerald-500 focus:outline-none";
export default function ProductDetail({
id,
onBack,
}: {
id: string;
onBack: () => void;
}) {
const [d, setD] = useState<Detail | null>(null);
const [brands, setBrands] = useState<Brand[]>([]);
const [categories, setCategories] = useState<Category[]>([]);
const [audit, setAudit] = useState<AuditEntry[]>([]);
const [loading, setLoading] = useState(true);
const [saving, setSaving] = useState(false);
const [msg, setMsg] = useState("");
const [error, setError] = useState("");
// editable form state
const [name, setName] = useState("");
const [gtin, setGtin] = useState("");
const [brandName, setBrandName] = useState("");
const [categoryId, setCategoryId] = useState("");
const [netValue, setNetValue] = useState("");
const [netUnit, setNetUnit] = useState("");
const [country, setCountry] = useState("");
const [status, setStatus] = useState("active");
const [ingredients, setIngredients] = useState("");
const [allergens, setAllergens] = useState("");
const [additives, setAdditives] = useState("");
const [nutriments, setNutriments] = useState<Record<string, string>>({});
const [basis, setBasis] = useState("");
const [serving, setServing] = useState("");
const [nutriScore, setNutriScore] = useState("");
function hydrate(detail: Detail) {
setD(detail);
setName(detail.name);
setGtin(detail.gtin || "");
setBrandName(detail.brand || "");
setCategoryId(detail.category_id || "");
setNetValue(detail.net_content_value?.toString() || "");
setNetUnit(detail.net_content_unit || "");
setCountry(detail.country_of_origin || "");
setStatus(detail.status);
setIngredients(detail.ingredients_text || "");
setAllergens(detail.allergens.join(", "));
setAdditives(detail.additives.join(", "));
const nm: Record<string, string> = {};
if (detail.nutriments) {
for (const [k, v] of Object.entries(detail.nutriments)) nm[k] = String(v);
}
setNutriments(nm);
setBasis(detail.nutrition_basis || "");
setServing(detail.serving_size || "");
setNutriScore(detail.nutri_score || "");
}
function reload() {
setLoading(true);
Promise.all([api.getProduct(id), api.listAudit(id)])
.then(([detail, a]) => {
hydrate(detail);
setAudit(a.items);
})
.catch((e) => setError(e.message))
.finally(() => setLoading(false));
}
useEffect(() => {
reload();
api.listBrands().then((r) => setBrands(r.items)).catch(() => {});
api.listCategories().then((r) => setCategories(r.items)).catch(() => {});
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [id]);
const missing = useMemo(() => d?.missing ?? [], [d]);
function parseList(s: string): string[] {
return s
.split(",")
.map((x) => x.trim())
.filter(Boolean);
}
async function save() {
setSaving(true);
setMsg("");
setError("");
const nm: Record<string, number> = {};
for (const [k, v] of Object.entries(nutriments)) {
const n = parseFloat(v);
if (!Number.isNaN(n)) nm[k] = n;
}
const body = {
gtin: gtin.trim() || null,
name: name.trim(),
brand_name: brandName.trim() || null,
brand_id: brandName.trim() ? undefined : null,
category_id: categoryId || null,
net_content_value: netValue.trim() ? parseFloat(netValue) : null,
net_content_unit: netUnit.trim() || null,
country_of_origin: country.trim() || null,
status,
ingredients_text: ingredients.trim() || null,
allergens: parseList(allergens),
additives: parseList(additives),
nutriments: nm,
nutrition_basis: basis || null,
serving_size: serving.trim() || null,
nutri_score: nutriScore || null,
};
try {
const updated = await api.updateProduct(id, body);
hydrate(updated);
const a = await api.listAudit(id);
setAudit(a.items);
setMsg("已保存");
setTimeout(() => setMsg(""), 2500);
} catch (e) {
setError(e instanceof Error ? e.message : "保存失败");
} finally {
setSaving(false);
}
}
if (loading) {
return <div className="text-gray-400"></div>;
}
if (!d) {
return (
<div>
<button onClick={onBack} className="text-emerald-600">
</button>
<p className="mt-4 text-red-600">{error || "未找到商品"}</p>
</div>
);
}
return (
<div className="mx-auto max-w-5xl space-y-5">
<div className="flex items-center justify-between">
<button
onClick={onBack}
className="flex items-center gap-1 text-sm text-gray-600 hover:text-gray-900"
>
<ArrowLeft className="h-4 w-4" />
</button>
<div className="flex items-center gap-3">
{msg && <span className="text-sm text-emerald-600">{msg}</span>}
{error && <span className="text-sm text-red-600">{error}</span>}
<span className="text-xs text-gray-400">
{Math.round(d.quality_score * 100)}
</span>
<button
onClick={save}
disabled={saving}
className="flex items-center gap-1 rounded bg-emerald-600 px-4 py-2 text-sm text-white hover:bg-emerald-700 disabled:opacity-60"
>
<Save className="h-4 w-4" /> {saving ? "保存中…" : "保存"}
</button>
</div>
</div>
{missing.length > 0 && (
<div className="flex items-center gap-2 rounded-lg border border-amber-200 bg-amber-50 px-4 py-3 text-sm text-amber-700">
<AlertCircle className="h-4 w-4" />
{missing.map((f) => FIELD_LABELS[f] || f).join("、")}
</div>
)}
<Card title="基础信息">
<div className="grid grid-cols-2 gap-4">
<Field label="名称 *">
<input
className={inputCls}
value={name}
onChange={(e) => setName(e.target.value)}
/>
</Field>
<Field label="条码 (GTIN)">
<input
className={inputCls}
value={gtin}
onChange={(e) => setGtin(e.target.value)}
/>
</Field>
<Field label="品牌(不存在将自动创建)">
<input
className={inputCls}
list="brand-list"
value={brandName}
onChange={(e) => setBrandName(e.target.value)}
/>
<datalist id="brand-list">
{brands.map((b) => (
<option key={b.id} value={b.name} />
))}
</datalist>
</Field>
<Field label="品类">
<select
className={inputCls}
value={categoryId}
onChange={(e) => setCategoryId(e.target.value)}
>
<option value=""></option>
{categories.map((c) => (
<option key={c.id} value={c.id}>
{"\u00A0".repeat(c.level * 2)}
{c.name_zh} ({c.path})
</option>
))}
</select>
</Field>
<Field label="净含量">
<input
className={inputCls}
type="number"
step="any"
value={netValue}
onChange={(e) => setNetValue(e.target.value)}
/>
</Field>
<Field label="净含量单位 (g/ml/cl…)">
<input
className={inputCls}
value={netUnit}
onChange={(e) => setNetUnit(e.target.value)}
/>
</Field>
<Field label="产地">
<input
className={inputCls}
value={country}
onChange={(e) => setCountry(e.target.value)}
/>
</Field>
<Field label="状态">
<select
className={inputCls}
value={status}
onChange={(e) => setStatus(e.target.value)}
>
{STATUS_OPTIONS.map((o) => (
<option key={o.value} value={o.value}>
{o.label}
</option>
))}
</select>
</Field>
</div>
</Card>
<Card title="配料与营养">
<div className="mb-4 grid grid-cols-2 gap-4">
<Field label="配料表">
<textarea
className={inputCls}
rows={3}
value={ingredients}
onChange={(e) => setIngredients(e.target.value)}
/>
</Field>
<div className="grid grid-cols-2 gap-4">
<Field label="过敏原(逗号分隔)">
<input
className={inputCls}
value={allergens}
onChange={(e) => setAllergens(e.target.value)}
/>
</Field>
<Field label="添加剂(逗号分隔)">
<input
className={inputCls}
value={additives}
onChange={(e) => setAdditives(e.target.value)}
/>
</Field>
<Field label="营养基准">
<select
className={inputCls}
value={basis}
onChange={(e) => setBasis(e.target.value)}
>
<option value=""></option>
<option value="per_100g"> 100g</option>
<option value="per_100ml"> 100ml</option>
<option value="per_serving"></option>
</select>
</Field>
<Field label="份量">
<input
className={inputCls}
value={serving}
onChange={(e) => setServing(e.target.value)}
/>
</Field>
<Field label="Nutri-Score (A-E)">
<input
className={inputCls}
maxLength={1}
value={nutriScore}
onChange={(e) =>
setNutriScore(e.target.value.toUpperCase())
}
/>
</Field>
</div>
</div>
<div className="grid grid-cols-4 gap-3">
{NUTRIMENT_KEYS.map((n) => (
<Field key={n.key} label={n.label}>
<input
className={inputCls}
type="number"
step="any"
value={nutriments[n.key] ?? ""}
onChange={(e) =>
setNutriments((prev) => ({ ...prev, [n.key]: e.target.value }))
}
/>
</Field>
))}
</div>
</Card>
<ImagesCard
product={d}
onChange={reload}
onError={setError}
/>
<MsrpCard product={d} onChange={reload} onError={setError} />
<Card title="操作记录">
{audit.length === 0 ? (
<p className="text-sm text-gray-400"></p>
) : (
<ul className="space-y-2 text-sm">
{audit.map((a) => (
<li
key={a.id}
className="flex items-center gap-3 text-gray-600"
>
<History className="h-3.5 w-3.5 text-gray-400" />
<span className="text-gray-400">{a.created_at}</span>
<span className="font-medium text-gray-700">{a.actor}</span>
<span>{ACTION_LABEL[a.action] || a.action}</span>
{a.fields.length > 0 && (
<span className="text-gray-400">
[{a.fields.map((f) => FIELD_LABELS[f] || f).join("、")}]
</span>
)}
</li>
))}
</ul>
)}
</Card>
</div>
);
}
function ImagesCard({
product,
onChange,
onError,
}: {
product: Detail;
onChange: () => void;
onError: (m: string) => void;
}) {
const [url, setUrl] = useState("");
const [kind, setKind] = useState("front");
async function add() {
if (!url.trim()) return;
try {
await api.addImage(product.id, url.trim(), kind);
setUrl("");
onChange();
} catch (e) {
onError(e instanceof Error ? e.message : "添加失败");
}
}
async function remove(imageId: string) {
try {
await api.deleteImage(product.id, imageId);
onChange();
} catch (e) {
onError(e instanceof Error ? e.message : "删除失败");
}
}
return (
<Card title="图片(仅存 URL">
<div className="mb-3 flex flex-wrap gap-3">
{product.images.length === 0 && (
<span className="text-sm text-gray-400"></span>
)}
{product.images.map((im) => (
<div
key={im.id}
className="relative h-24 w-24 overflow-hidden rounded border border-gray-200"
>
<img
src={im.url}
alt={im.kind}
className="h-full w-full object-cover"
/>
<button
onClick={() => remove(im.id)}
className="absolute right-1 top-1 rounded bg-black/50 p-1 text-white hover:bg-black/70"
>
<Trash2 className="h-3 w-3" />
</button>
<span className="absolute bottom-0 left-0 bg-black/50 px-1 text-[10px] text-white">
{im.kind}
</span>
</div>
))}
</div>
<div className="flex items-center gap-2">
<input
className={inputCls}
placeholder="图片 URL"
value={url}
onChange={(e) => setUrl(e.target.value)}
/>
<select
className="rounded border border-gray-300 px-2 py-2 text-sm"
value={kind}
onChange={(e) => setKind(e.target.value)}
>
<option value="front"></option>
<option value="ingredients"></option>
<option value="nutrition"></option>
<option value="other"></option>
</select>
<button
onClick={add}
className="flex items-center gap-1 whitespace-nowrap rounded bg-gray-700 px-3 py-2 text-sm text-white hover:bg-gray-800"
>
<Plus className="h-4 w-4" />
</button>
</div>
</Card>
);
}
function MsrpCard({
product,
onChange,
onError,
}: {
product: Detail;
onChange: () => void;
onError: (m: string) => void;
}) {
const [amount, setAmount] = useState("");
const [currency, setCurrency] = useState("CNY");
const [region, setRegion] = useState("CN");
const [date, setDate] = useState("");
const [note, setNote] = useState("");
async function add() {
const a = parseFloat(amount);
if (Number.isNaN(a)) return;
try {
await api.addMsrp(product.id, {
amount: a,
currency,
region,
effective_date: date || null,
note: note.trim() || null,
});
setAmount("");
setNote("");
setDate("");
onChange();
} catch (e) {
onError(e instanceof Error ? e.message : "添加失败");
}
}
async function remove(msrpId: string) {
try {
await api.deleteMsrp(product.id, msrpId);
onChange();
} catch (e) {
onError(e instanceof Error ? e.message : "删除失败");
}
}
return (
<Card title="官方建议零售价(MSRP 快照,非售卖)">
<div className="mb-3 space-y-2">
{product.msrp.length === 0 && (
<span className="text-sm text-gray-400"></span>
)}
{product.msrp.map((m) => (
<div
key={m.id}
className="flex items-center gap-3 rounded border border-gray-100 bg-gray-50 px-3 py-2 text-sm"
>
<span className="font-medium text-gray-800">
{m.amount} {m.currency}
</span>
<span className="text-gray-500">{m.region}</span>
<span className="text-gray-400">{m.effective_date || ""}</span>
<span className="flex-1 text-gray-400">{m.note || ""}</span>
<button
onClick={() => remove(m.id)}
className="text-gray-400 hover:text-red-600"
>
<Trash2 className="h-4 w-4" />
</button>
</div>
))}
</div>
<div className="flex flex-wrap items-end gap-2">
<Field label="金额">
<input
className="w-28 rounded border border-gray-300 px-3 py-2 text-sm"
type="number"
step="any"
value={amount}
onChange={(e) => setAmount(e.target.value)}
/>
</Field>
<Field label="币种">
<input
className="w-20 rounded border border-gray-300 px-3 py-2 text-sm"
value={currency}
onChange={(e) => setCurrency(e.target.value.toUpperCase())}
/>
</Field>
<Field label="地区">
<input
className="w-20 rounded border border-gray-300 px-3 py-2 text-sm"
value={region}
onChange={(e) => setRegion(e.target.value.toUpperCase())}
/>
</Field>
<Field label="生效日期">
<input
className="rounded border border-gray-300 px-3 py-2 text-sm"
type="date"
value={date}
onChange={(e) => setDate(e.target.value)}
/>
</Field>
<Field label="备注">
<input
className="w-40 rounded border border-gray-300 px-3 py-2 text-sm"
value={note}
onChange={(e) => setNote(e.target.value)}
/>
</Field>
<button
onClick={add}
className="flex items-center gap-1 rounded bg-gray-700 px-3 py-2 text-sm text-white hover:bg-gray-800"
>
<Plus className="h-4 w-4" />
</button>
</div>
</Card>
);
}
@@ -0,0 +1,178 @@
import { useEffect, useState } from "react";
import { api } from "../api";
import { FIELD_LABELS, ProductRow } from "../types";
import { Search, AlertCircle } from "lucide-react";
const STATUS_LABEL: Record<string, string> = {
active: "在用",
merged: "已合并",
deprecated: "已停用",
};
function QualityBadge({ score }: { score: number }) {
const pct = Math.round(score * 100);
const color =
score >= 0.8
? "bg-emerald-100 text-emerald-700"
: score >= 0.5
? "bg-amber-100 text-amber-700"
: "bg-red-100 text-red-700";
return (
<span className={`rounded px-2 py-0.5 text-xs font-medium ${color}`}>
{pct}
</span>
);
}
export default function ProductList({
onOpen,
}: {
onOpen: (id: string) => void;
}) {
const [q, setQ] = useState("");
const [input, setInput] = useState("");
const [page, setPage] = useState(1);
const [size] = useState(20);
const [rows, setRows] = useState<ProductRow[]>([]);
const [total, setTotal] = useState(0);
const [loading, setLoading] = useState(false);
const [error, setError] = useState("");
useEffect(() => {
setLoading(true);
setError("");
api
.listProducts(q, page, size)
.then((r) => {
setRows(r.items);
setTotal(r.total);
})
.catch((e) => setError(e.message))
.finally(() => setLoading(false));
}, [q, page, size]);
const pages = Math.max(1, Math.ceil(total / size));
return (
<div className="mx-auto max-w-6xl">
<div className="mb-4 flex items-center justify-between">
<h2 className="text-xl font-semibold text-gray-800">
<span className="text-sm font-normal text-gray-400"> {total} </span>
</h2>
<form
onSubmit={(e) => {
e.preventDefault();
setPage(1);
setQ(input.trim());
}}
className="flex items-center gap-2"
>
<div className="relative">
<Search className="absolute left-2 top-2.5 h-4 w-4 text-gray-400" />
<input
value={input}
onChange={(e) => setInput(e.target.value)}
placeholder="按名称 / 条码搜索"
className="w-64 rounded border border-gray-300 py-2 pl-8 pr-3 text-sm focus:border-emerald-500 focus:outline-none"
/>
</div>
<button className="rounded bg-emerald-600 px-3 py-2 text-sm text-white hover:bg-emerald-700">
</button>
</form>
</div>
{error && (
<div className="mb-3 rounded bg-red-50 px-3 py-2 text-sm text-red-600">
{error}
</div>
)}
<div className="overflow-hidden rounded-lg border border-gray-200 bg-white">
<table className="w-full text-sm">
<thead className="bg-gray-50 text-left text-xs uppercase text-gray-500">
<tr>
<th className="px-4 py-3"></th>
<th className="px-4 py-3"></th>
<th className="px-4 py-3"></th>
<th className="px-4 py-3"></th>
<th className="px-4 py-3"></th>
<th className="px-4 py-3"></th>
<th className="px-4 py-3"></th>
</tr>
</thead>
<tbody className="divide-y divide-gray-100">
{loading ? (
<tr>
<td colSpan={7} className="px-4 py-8 text-center text-gray-400">
</td>
</tr>
) : rows.length === 0 ? (
<tr>
<td colSpan={7} className="px-4 py-8 text-center text-gray-400">
</td>
</tr>
) : (
rows.map((r) => (
<tr
key={r.id}
onClick={() => onOpen(r.id)}
className="cursor-pointer hover:bg-emerald-50/50"
>
<td className="px-4 py-3 font-medium text-gray-800">{r.name}</td>
<td className="px-4 py-3 text-gray-600">{r.brand || "—"}</td>
<td className="px-4 py-3 font-mono text-xs text-gray-500">
{r.gtin || "—"}
</td>
<td className="px-4 py-3 text-xs text-gray-500">
{r.category_path || "—"}
</td>
<td className="px-4 py-3 text-gray-600">
{STATUS_LABEL[r.status] || r.status}
</td>
<td className="px-4 py-3">
<QualityBadge score={r.quality_score} />
</td>
<td className="px-4 py-3">
{r.missing.length === 0 ? (
<span className="text-xs text-emerald-600"></span>
) : (
<span className="flex items-center gap-1 text-xs text-amber-600">
<AlertCircle className="h-3.5 w-3.5" />
{r.missing
.map((f) => FIELD_LABELS[f] || f)
.join("、")}
</span>
)}
</td>
</tr>
))
)}
</tbody>
</table>
</div>
<div className="mt-4 flex items-center justify-end gap-2 text-sm text-gray-600">
<button
disabled={page <= 1}
onClick={() => setPage((p) => p - 1)}
className="rounded border border-gray-300 px-3 py-1 disabled:opacity-50"
>
</button>
<span>
{page} / {pages}
</span>
<button
disabled={page >= pages}
onClick={() => setPage((p) => p + 1)}
className="rounded border border-gray-300 px-3 py-1 disabled:opacity-50"
>
</button>
</div>
</div>
);
}
@@ -0,0 +1,314 @@
import { useEffect, useState } from "react";
import { api, ApiError } from "../api";
import type { SubmissionDetail, SubmissionRow } from "../types";
import { FIELD_LABELS } from "../types";
import { ArrowLeft, Check, X } from "lucide-react";
const STATUS_TABS = [
{ key: "pending", label: "待审核" },
{ key: "approved", label: "已通过" },
{ key: "rejected", label: "已驳回" },
];
const STATUS_BADGE: Record<string, string> = {
pending: "bg-amber-50 text-amber-700",
approved: "bg-emerald-50 text-emerald-700",
rejected: "bg-red-50 text-red-700",
};
const STATUS_TEXT: Record<string, string> = {
pending: "待审核",
approved: "已通过",
rejected: "已驳回",
};
export default function SubmissionsPage({ onPending }: { onPending?: (n: number) => void }) {
const [tab, setTab] = useState("pending");
const [rows, setRows] = useState<SubmissionRow[]>([]);
const [openId, setOpenId] = useState<string | null>(null);
const [error, setError] = useState("");
async function load() {
setError("");
try {
const res = await api.listSubmissions(tab, 1, 50);
setRows(res.items);
onPending?.(res.pending);
} catch (e) {
setError(e instanceof ApiError ? e.message : "加载失败");
}
}
useEffect(() => {
load();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [tab]);
if (openId) {
return (
<SubmissionView
id={openId}
onBack={() => {
setOpenId(null);
load();
}}
/>
);
}
return (
<div>
<div className="flex items-center gap-2 mb-4">
{STATUS_TABS.map((t) => (
<button
key={t.key}
onClick={() => setTab(t.key)}
className={`px-3 py-1.5 rounded-md text-sm ${
tab === t.key
? "bg-emerald-600 text-white"
: "bg-white border text-gray-600 hover:bg-gray-50"
}`}
>
{t.label}
</button>
))}
</div>
{error && (
<div className="mb-3 bg-red-50 text-red-700 text-sm rounded px-4 py-2">{error}</div>
)}
<div className="bg-white border rounded-lg overflow-hidden">
<table className="w-full text-sm">
<thead className="bg-gray-50 text-gray-500 text-left">
<tr>
<th className="px-4 py-2 font-medium"></th>
<th className="px-4 py-2 font-medium"></th>
<th className="px-4 py-2 font-medium">稿</th>
<th className="px-4 py-2 font-medium"></th>
<th className="px-4 py-2 font-medium"></th>
<th className="px-4 py-2 font-medium"></th>
</tr>
</thead>
<tbody className="divide-y">
{rows.length === 0 ? (
<tr>
<td colSpan={6} className="px-4 py-8 text-center text-gray-400">
稿
</td>
</tr>
) : (
rows.map((r) => (
<tr
key={r.id}
onClick={() => setOpenId(r.id)}
className="cursor-pointer hover:bg-gray-50"
>
<td className="px-4 py-2 text-gray-800">{r.name}</td>
<td className="px-4 py-2 text-gray-500">{r.gtin || "—"}</td>
<td className="px-4 py-2 text-gray-500">{r.submitter_name || "匿名"}</td>
<td className="px-4 py-2">
<span className="text-xs text-gray-500">
{r.matched ? "补全已有商品" : "新建商品"}
</span>
</td>
<td className="px-4 py-2 text-gray-500">
{new Date(r.created_at).toLocaleString()}
</td>
<td className="px-4 py-2">
<span className={`text-xs rounded px-2 py-0.5 ${STATUS_BADGE[r.status]}`}>
{STATUS_TEXT[r.status]}
</span>
</td>
</tr>
))
)}
</tbody>
</table>
</div>
</div>
);
}
function Field({ label, value }: { label: string; value: React.ReactNode }) {
if (value === null || value === undefined || value === "") return null;
return (
<div className="flex py-1.5 text-sm">
<div className="w-28 shrink-0 text-gray-400">{label}</div>
<div className="text-gray-800 break-all">{value}</div>
</div>
);
}
function SubmissionView({ id, onBack }: { id: string; onBack: () => void }) {
const [d, setD] = useState<SubmissionDetail | null>(null);
const [error, setError] = useState("");
const [busy, setBusy] = useState(false);
const [rejecting, setRejecting] = useState(false);
const [reason, setReason] = useState("");
useEffect(() => {
api.getSubmission(id).then(setD).catch((e) => setError(e.message));
}, [id]);
async function approve() {
if (!confirm("确认通过该投稿?将写入正式商品库并记录来源 community。")) return;
setBusy(true);
setError("");
try {
await api.approveSubmission(id);
onBack();
} catch (e) {
setError(e instanceof ApiError ? e.message : "操作失败");
} finally {
setBusy(false);
}
}
async function reject() {
setBusy(true);
setError("");
try {
await api.rejectSubmission(id, reason.trim());
onBack();
} catch (e) {
setError(e instanceof ApiError ? e.message : "操作失败");
} finally {
setBusy(false);
}
}
if (error && !d) {
return (
<div>
<button onClick={onBack} className="text-sm text-gray-500 flex items-center gap-1 mb-4">
<ArrowLeft className="h-4 w-4" />
</button>
<div className="bg-red-50 text-red-700 text-sm rounded px-4 py-3">{error}</div>
</div>
);
}
if (!d) return <div className="text-gray-400"></div>;
const p = d.payload;
const nutri = Object.entries(p.nutriments || {});
return (
<div className="max-w-3xl">
<button onClick={onBack} className="text-sm text-gray-500 flex items-center gap-1 mb-4">
<ArrowLeft className="h-4 w-4" /> 稿
</button>
<div className="flex items-center justify-between">
<h2 className="text-lg font-semibold text-gray-800">{p.name}</h2>
<span className={`text-xs rounded px-2 py-0.5 ${STATUS_BADGE[d.status]}`}>
{STATUS_TEXT[d.status]}
</span>
</div>
{error && <div className="mt-3 bg-red-50 text-red-700 text-sm rounded px-4 py-2">{error}</div>}
{d.target_product_id && (
<div className="mt-3 text-sm bg-blue-50 text-blue-700 rounded px-4 py-2">
<b></b>
</div>
)}
<div className="bg-white border rounded-lg p-5 mt-4">
<h3 className="font-medium text-gray-700 mb-2">稿</h3>
<Field label="商品名称" value={p.name} />
<Field label="条码" value={p.gtin} />
<Field label="品牌" value={p.brand_name} />
<Field
label="净含量"
value={p.net_content_value != null ? `${p.net_content_value} ${p.net_content_unit || ""}` : null}
/>
<Field label="产地" value={p.country_of_origin} />
<Field label="配料" value={p.ingredients_text} />
{nutri.length > 0 && (
<Field
label="营养成分"
value={
<span>
{p.nutrition_basis ? `(${p.nutrition_basis}) ` : ""}
{nutri.map(([k, v]) => `${FIELD_LABELS[k] || k}:${v}`).join("")}
</span>
}
/>
)}
{p.images && p.images.length > 0 && (
<Field
label="图片"
value={
<div className="flex flex-wrap gap-2">
{p.images.map((im, i) => (
<a key={i} href={im.url} target="_blank" rel="noreferrer">
<img src={im.url} alt="" className="h-20 w-20 object-cover rounded border" />
</a>
))}
</div>
}
/>
)}
</div>
<div className="bg-white border rounded-lg p-5 mt-4">
<h3 className="font-medium text-gray-700 mb-2">稿 / </h3>
<Field label="称呼" value={p.submitter_name || "匿名"} />
<Field label="联系方式" value={p.submitter_contact} />
<Field label="备注" value={p.note} />
<Field label="提交时间" value={new Date(d.created_at).toLocaleString()} />
{d.reviewed_by && <Field label="审核人" value={d.reviewed_by} />}
{d.review_note && <Field label="驳回原因" value={d.review_note} />}
{d.result_product_id && <Field label="收录商品ID" value={d.result_product_id} />}
</div>
{d.status === "pending" && (
<div className="mt-5">
{rejecting ? (
<div className="bg-white border rounded-lg p-4">
<label className="block text-xs text-gray-500 mb-1"></label>
<input
className="w-full border rounded-md px-3 py-2 text-sm"
value={reason}
onChange={(e) => setReason(e.target.value)}
placeholder="例如:资料无法核实 / 重复投稿"
/>
<div className="mt-3 flex gap-2">
<button
onClick={reject}
disabled={busy}
className="px-4 py-2 rounded bg-red-600 text-white text-sm hover:bg-red-700 disabled:opacity-60"
>
</button>
<button
onClick={() => setRejecting(false)}
className="px-4 py-2 rounded border text-sm text-gray-600"
>
</button>
</div>
</div>
) : (
<div className="flex gap-3">
<button
onClick={approve}
disabled={busy}
className="px-5 py-2.5 rounded-lg bg-emerald-600 text-white font-medium hover:bg-emerald-700 disabled:opacity-60 flex items-center gap-1.5"
>
<Check className="h-4 w-4" />
</button>
<button
onClick={() => setRejecting(true)}
disabled={busy}
className="px-5 py-2.5 rounded-lg border text-gray-700 font-medium hover:bg-gray-50 flex items-center gap-1.5"
>
<X className="h-4 w-4" />
</button>
</div>
)}
</div>
)}
</div>
);
}
+16
View File
@@ -0,0 +1,16 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
html,
body,
#root {
height: 100%;
}
body {
margin: 0;
background: #f3f4f6;
font-family: system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue",
Arial, "PingFang SC", "Microsoft YaHei", sans-serif;
}
+10
View File
@@ -0,0 +1,10 @@
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
import "./index.css";
ReactDOM.createRoot(document.getElementById("root")!).render(
<React.StrictMode>
<App />
</React.StrictMode>,
);
+140
View File
@@ -0,0 +1,140 @@
export interface ProductRow {
id: string;
gtin: string | null;
name: string;
brand: string | null;
category_path: string | null;
status: string;
quality_score: number;
missing: string[];
updated_at: string;
}
export interface ProductImage {
id: string;
url: string;
kind: string;
license: string | null;
}
export interface MSRP {
id: string;
amount: number;
currency: string;
region: string;
effective_date: string | null;
source_url: string | null;
note: string | null;
}
export interface ProductDetail {
id: string;
gtin: string | null;
name: string;
brand_id: string | null;
brand: string | null;
category_id: string | null;
category_path: string | null;
net_content_value: number | null;
net_content_unit: string | null;
country_of_origin: string | null;
status: string;
quality_score: number;
ingredients_text: string | null;
allergens: string[];
additives: string[];
nutriments: Record<string, number> | null;
nutrition_basis: string | null;
serving_size: string | null;
nutri_score: string | null;
images: ProductImage[];
msrp: MSRP[];
missing: string[];
updated_at: string;
}
export interface Brand {
id: string;
name: string;
}
export interface Category {
id: string;
name_zh: string;
name_en: string | null;
path: string;
level: number;
}
export interface AuditEntry {
id: string;
actor: string;
action: string;
fields: string[];
created_at: string;
}
export interface SubmissionRow {
id: string;
gtin: string | null;
name: string;
status: string;
submitter_name: string | null;
matched: boolean;
created_at: string;
reviewed_at: string | null;
}
export interface SubmissionImage {
url: string;
kind: string;
}
export interface SubmissionPayload {
gtin: string | null;
name: string;
brand_name: string | null;
category_id: string | null;
net_content_value: number | null;
net_content_unit: string | null;
country_of_origin: string | null;
ingredients_text: string | null;
nutriments: Record<string, number> | null;
nutrition_basis: string | null;
serving_size: string | null;
nutri_score: string | null;
images: SubmissionImage[] | null;
submitter_name: string | null;
submitter_contact: string | null;
note: string | null;
}
export interface SubmissionDetail {
id: string;
status: string;
gtin: string | null;
name: string;
submitter_name: string | null;
submitter_contact: string | null;
note: string | null;
review_note: string | null;
reviewed_by: string | null;
reviewed_at: string | null;
created_at: string;
target_product_id: string | null;
result_product_id: string | null;
payload: SubmissionPayload;
existing_product?: ProductDetail;
}
export const FIELD_LABELS: Record<string, string> = {
name: "名称",
gtin: "条码",
brand: "品牌",
category: "品类",
net_content: "净含量",
country_of_origin: "产地",
nutriments: "营养成分",
ingredients: "配料",
image: "图片",
};
+1
View File
@@ -0,0 +1 @@
/// <reference types="vite/client" />
+6
View File
@@ -0,0 +1,6 @@
/** @type {import('tailwindcss').Config} */
export default {
content: ["./index.html", "./src/**/*.{ts,tsx}"],
theme: { extend: {} },
plugins: [],
};
+21
View File
@@ -0,0 +1,21 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"module": "ESNext",
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "react-jsx",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true
},
"include": ["src"],
"references": [{ "path": "./tsconfig.node.json" }]
}
+11
View File
@@ -0,0 +1,11 @@
{
"compilerOptions": {
"composite": true,
"skipLibCheck": true,
"module": "ESNext",
"moduleResolution": "bundler",
"allowSyntheticDefaultImports": true,
"strict": true
},
"include": ["vite.config.ts"]
}
+9
View File
@@ -0,0 +1,9 @@
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
// Served under /ping by the admin Go binary; base must match the nginx prefix.
export default defineConfig({
base: "/ping/",
plugins: [react()],
build: { outDir: "dist", emptyOutDir: true },
});
+13
View File
@@ -0,0 +1,13 @@
# Build stage
FROM golang:1.23-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server
# Runtime stage
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/server /server
EXPOSE 8080
ENTRYPOINT ["/server"]
+29
View File
@@ -0,0 +1,29 @@
# Admin console image: builds the SPA (node), embeds it into the Go admin
# binary, and ships a static scratch runtime. Build context is the repo root.
# Stage 1: build the admin SPA.
FROM node:22-alpine AS web
WORKDIR /web
ENV npm_config_registry=https://registry.npmmirror.com
COPY admin-frontend/package.json admin-frontend/package-lock.json* ./
RUN npm ci || npm install
COPY admin-frontend/ ./
RUN npm run build
# Stage 2: build the Go admin binary with the SPA embedded.
FROM golang:1.23-alpine AS build
ENV GOPROXY=https://goproxy.cn,direct
WORKDIR /src
COPY api/go.mod api/go.sum ./
RUN go mod download
COPY api/ ./
RUN rm -rf internal/adminweb/dist && mkdir -p internal/adminweb/dist
COPY --from=web /web/dist/ internal/adminweb/dist/
RUN CGO_ENABLED=0 go build -o /out/admin ./cmd/admin
# Stage 3: minimal runtime.
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/admin /admin
EXPOSE 8080
ENTRYPOINT ["/admin"]
+31
View File
@@ -0,0 +1,31 @@
# Public API image: builds the public SPA (homepage + search + contribute),
# embeds it into the Go read-only server binary, and ships a static scratch
# runtime (used where gcr.io/distroless is not reachable). Build context is the
# repo root.
# Stage 1: build the public SPA.
FROM node:22-alpine AS web
WORKDIR /web
ENV npm_config_registry=https://registry.npmmirror.com
COPY public-frontend/package.json public-frontend/package-lock.json* ./
RUN npm ci || npm install
COPY public-frontend/ ./
RUN npm run build
# Stage 2: build the Go server binary with the SPA embedded.
FROM golang:1.23-alpine AS build
ENV GOPROXY=https://goproxy.cn,direct
WORKDIR /src
COPY api/go.mod api/go.sum ./
RUN go mod download
COPY api/ ./
RUN rm -rf internal/publicweb/dist && mkdir -p internal/publicweb/dist
COPY --from=web /web/dist/ internal/publicweb/dist/
RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server
# Stage 3: minimal runtime.
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/server /server
EXPOSE 8080
ENTRYPOINT ["/server"]
+83
View File
@@ -0,0 +1,83 @@
// Command admin starts the OpenGoods admin console (authenticated write API +
// embedded SPA), served under a base path (default /ping).
package main
import (
"context"
"crypto/rand"
"log"
"net/http"
"os"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"golang.org/x/crypto/bcrypt"
"github.com/baicai2026-baicai/goods/api/internal/adminhandler"
"github.com/baicai2026-baicai/goods/api/internal/adminstore"
"github.com/baicai2026-baicai/goods/api/internal/adminweb"
"github.com/baicai2026-baicai/goods/api/internal/auth"
)
func getenv(key, fallback string) string {
if v, ok := os.LookupEnv(key); ok && v != "" {
return v
}
return fallback
}
func main() {
addr := getenv("GOODS_ADMIN_ADDR", ":8080")
dbURL := getenv("OPENGOODS_DATABASE_URL", "postgres://opengoods:opengoods@localhost:5432/opengoods?sslmode=disable")
basePath := getenv("GOODS_ADMIN_BASE_PATH", "/ping")
username := getenv("GOODS_ADMIN_USER", "admin")
// Password: prefer a bcrypt hash; otherwise hash a plaintext password.
var passwordHash []byte
if h := os.Getenv("GOODS_ADMIN_PASSWORD_HASH"); h != "" {
passwordHash = []byte(h)
} else if p := os.Getenv("GOODS_ADMIN_PASSWORD"); p != "" {
hashed, err := bcrypt.GenerateFromPassword([]byte(p), bcrypt.DefaultCost)
if err != nil {
log.Fatalf("failed to hash admin password: %v", err)
}
passwordHash = hashed
} else {
log.Fatal("set GOODS_ADMIN_PASSWORD or GOODS_ADMIN_PASSWORD_HASH")
}
secret := []byte(os.Getenv("GOODS_ADMIN_JWT_SECRET"))
if len(secret) == 0 {
secret = make([]byte, 32)
if _, err := rand.Read(secret); err != nil {
log.Fatalf("failed to generate jwt secret: %v", err)
}
log.Print("warning: GOODS_ADMIN_JWT_SECRET not set; using a random secret (tokens invalidate on restart)")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, dbURL)
if err != nil {
log.Fatalf("failed to create db pool: %v", err)
}
defer pool.Close()
pingCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if err := pool.Ping(pingCtx); err != nil {
log.Printf("warning: database not reachable at startup: %v", err)
}
authn := auth.New(username, passwordHash, secret, 12*time.Hour)
h := adminhandler.New(adminstore.New(pool), authn, basePath, adminweb.Dist())
srv := &http.Server{
Addr: addr,
Handler: h.Router(),
ReadHeaderTimeout: 10 * time.Second,
}
log.Printf("OpenGoods admin console listening on %s (base path %s)", addr, basePath)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("server error: %v", err)
}
}
+46
View File
@@ -0,0 +1,46 @@
// Command server starts the OpenGoods public read-only API.
package main
import (
"context"
"log"
"net/http"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/baicai2026-baicai/goods/api/internal/config"
"github.com/baicai2026-baicai/goods/api/internal/handler"
"github.com/baicai2026-baicai/goods/api/internal/publicweb"
"github.com/baicai2026-baicai/goods/api/internal/store"
)
func main() {
cfg := config.Load()
ctx := context.Background()
pool, err := pgxpool.New(ctx, cfg.DatabaseURL)
if err != nil {
log.Fatalf("failed to create db pool: %v", err)
}
defer pool.Close()
pingCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if err := pool.Ping(pingCtx); err != nil {
log.Printf("warning: database not reachable at startup: %v", err)
}
h := handler.New(store.New(pool), publicweb.Dist())
srv := &http.Server{
Addr: cfg.Addr,
Handler: h.Router(),
ReadHeaderTimeout: 10 * time.Second,
}
log.Printf("OpenGoods API listening on %s", cfg.Addr)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("server error: %v", err)
}
}
+17
View File
@@ -0,0 +1,17 @@
module github.com/baicai2026-baicai/goods/api
go 1.23.4
require (
github.com/go-chi/chi/v5 v5.1.0
github.com/jackc/pgx/v5 v5.7.2
golang.org/x/crypto v0.31.0
)
require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
golang.org/x/sync v0.10.0 // indirect
golang.org/x/text v0.21.0 // indirect
)
+30
View File
@@ -0,0 +1,30 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-chi/chi/v5 v5.1.0 h1:acVI1TYaD+hhedDJ3r54HyA6sExp3HfXq7QWEEY/xMw=
github.com/go-chi/chi/v5 v5.1.0/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.7.2 h1:mLoDLV6sonKlvjIEsV56SkWNCnuNv531l94GaIzO+XI=
github.com/jackc/pgx/v5 v5.7.2/go.mod h1:ncY89UGWxg82EykZUwSpUKEfccBGGYq1xjrOpsbsfGQ=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+390
View File
@@ -0,0 +1,390 @@
// Package adminhandler wires up the authenticated admin console: a JSON write
// API mounted under a base path (default /ping) plus the embedded SPA.
package adminhandler
import (
"encoding/json"
"errors"
"io/fs"
"net/http"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/baicai2026-baicai/goods/api/internal/adminstore"
"github.com/baicai2026-baicai/goods/api/internal/auth"
)
// Handler holds the admin dependencies.
type Handler struct {
store *adminstore.Store
authn *auth.Authenticator
basePath string
spa fs.FS
submitLimit *rateLimiter
}
// New constructs an admin Handler. basePath is e.g. "/ping" (no trailing slash).
func New(store *adminstore.Store, authn *auth.Authenticator, basePath string, spa fs.FS) *Handler {
basePath = "/" + strings.Trim(basePath, "/")
return &Handler{
store: store,
authn: authn,
basePath: basePath,
spa: spa,
submitLimit: newRateLimiter(5, 10*time.Minute),
}
}
// Router builds the HTTP handler.
func (h *Handler) Router() http.Handler {
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.Recoverer)
r.Route(h.basePath, func(r chi.Router) {
r.Get("/healthz", func(w http.ResponseWriter, _ *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
})
r.Post("/api/login", h.Login)
r.Group(func(r chi.Router) {
r.Use(h.authn.Middleware)
r.Get("/api/me", h.Me)
r.Get("/api/products", h.ListProducts)
r.Get("/api/products/{id}", h.GetProduct)
r.Put("/api/products/{id}", h.UpdateProduct)
r.Get("/api/products/{id}/audit", h.ListAudit)
r.Post("/api/products/{id}/images", h.AddImage)
r.Delete("/api/products/{id}/images/{imageID}", h.DeleteImage)
r.Post("/api/products/{id}/msrp", h.AddMSRP)
r.Delete("/api/products/{id}/msrp/{msrpID}", h.DeleteMSRP)
r.Get("/api/brands", h.ListBrands)
r.Get("/api/categories", h.ListCategories)
r.Get("/api/submissions", h.ListSubmissions)
r.Get("/api/submissions/{id}", h.GetSubmission)
r.Post("/api/submissions/{id}/approve", h.ApproveSubmission)
r.Post("/api/submissions/{id}/reject", h.RejectSubmission)
})
r.Handle("/*", http.HandlerFunc(h.serveSPA))
})
// Public, unauthenticated contribution endpoint (proxied at /api/public/*).
// Submissions enter a moderation queue and never touch products until an
// admin approves them.
r.Post("/api/public/submissions", h.CreateSubmission)
return r
}
func (h *Handler) serveSPA(w http.ResponseWriter, r *http.Request) {
rel := strings.TrimPrefix(r.URL.Path, h.basePath)
rel = strings.TrimPrefix(rel, "/")
if rel == "" {
rel = "index.html"
}
if f, err := h.spa.Open(rel); err == nil {
f.Close()
http.StripPrefix(h.basePath+"/", http.FileServer(http.FS(h.spa))).ServeHTTP(w, r)
return
}
// SPA fallback: serve index.html for client-side routes.
index, err := h.spa.Open("index.html")
if err != nil {
http.NotFound(w, r)
return
}
defer index.Close()
data, _ := fs.ReadFile(h.spa, "index.html")
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write(data)
}
// ---------- auth ----------
// Login authenticates and returns a bearer token.
func (h *Handler) Login(w http.ResponseWriter, r *http.Request) {
var body struct {
Username string `json:"username"`
Password string `json:"password"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "invalid body")
return
}
token, err := h.authn.Login(body.Username, body.Password)
if err != nil {
writeError(w, http.StatusUnauthorized, "unauthorized", "用户名或密码错误")
return
}
writeJSON(w, http.StatusOK, map[string]string{"token": token, "username": body.Username})
}
// Me returns the current authenticated user.
func (h *Handler) Me(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"username": auth.UserFrom(r.Context())})
}
// ---------- products ----------
// ListProducts returns a paginated product list.
func (h *Handler) ListProducts(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query().Get("q")
page, size := pageParams(r)
items, total, err := h.store.ListProducts(r.Context(), q, size, (page-1)*size)
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items, "page": page, "size": size, "total": total,
"completeness_fields": adminstore.CompletenessFields,
})
}
// GetProduct returns full editable detail.
func (h *Handler) GetProduct(w http.ResponseWriter, r *http.Request) {
d, err := h.store.GetProduct(r.Context(), chi.URLParam(r, "id"))
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, d)
}
// UpdateProduct applies an edit.
func (h *Handler) UpdateProduct(w http.ResponseWriter, r *http.Request) {
var in adminstore.ProductInput
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "invalid body")
return
}
if strings.TrimSpace(in.Name) == "" {
writeError(w, http.StatusBadRequest, "bad_request", "名称不能为空")
return
}
d, err := h.store.UpdateProduct(r.Context(), chi.URLParam(r, "id"), auth.UserFrom(r.Context()), in)
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, d)
}
// ListAudit returns audit history for a product.
func (h *Handler) ListAudit(w http.ResponseWriter, r *http.Request) {
items, err := h.store.ListAudit(r.Context(), chi.URLParam(r, "id"), 100)
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": items})
}
// AddImage adds an image URL.
func (h *Handler) AddImage(w http.ResponseWriter, r *http.Request) {
var body struct {
URL string `json:"url"`
Kind string `json:"kind"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || strings.TrimSpace(body.URL) == "" {
writeError(w, http.StatusBadRequest, "bad_request", "图片 URL 不能为空")
return
}
im, err := h.store.AddImage(r.Context(), chi.URLParam(r, "id"), auth.UserFrom(r.Context()), body.URL, body.Kind)
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusCreated, im)
}
// DeleteImage removes an image.
func (h *Handler) DeleteImage(w http.ResponseWriter, r *http.Request) {
err := h.store.DeleteImage(r.Context(), chi.URLParam(r, "id"), chi.URLParam(r, "imageID"), auth.UserFrom(r.Context()))
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// AddMSRP adds a suggested-retail-price snapshot.
func (h *Handler) AddMSRP(w http.ResponseWriter, r *http.Request) {
var in adminstore.MSRPInput
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "invalid body")
return
}
m, err := h.store.AddMSRP(r.Context(), chi.URLParam(r, "id"), auth.UserFrom(r.Context()), in)
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusCreated, m)
}
// DeleteMSRP removes an MSRP snapshot.
func (h *Handler) DeleteMSRP(w http.ResponseWriter, r *http.Request) {
err := h.store.DeleteMSRP(r.Context(), chi.URLParam(r, "id"), chi.URLParam(r, "msrpID"), auth.UserFrom(r.Context()))
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// ListBrands returns brand options.
func (h *Handler) ListBrands(w http.ResponseWriter, r *http.Request) {
items, err := h.store.ListBrands(r.Context())
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": items})
}
// ListCategories returns category options.
func (h *Handler) ListCategories(w http.ResponseWriter, r *http.Request) {
items, err := h.store.ListCategories(r.Context())
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": items})
}
// ---------- submissions ----------
// CreateSubmission accepts an anonymous public contribution into the queue.
func (h *Handler) CreateSubmission(w http.ResponseWriter, r *http.Request) {
if !h.submitLimit.allow(realIP(r)) {
writeError(w, http.StatusTooManyRequests, "rate_limited", "提交过于频繁,请稍后再试")
return
}
var in adminstore.SubmissionInput
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&in); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "invalid body")
return
}
if strings.TrimSpace(in.Name) == "" {
writeError(w, http.StatusBadRequest, "bad_request", "商品名称不能为空")
return
}
id, err := h.store.CreateSubmission(r.Context(), in, realIP(r))
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
return
}
writeJSON(w, http.StatusCreated, map[string]string{"id": id, "status": "pending"})
}
// ListSubmissions returns the moderation queue (admin).
func (h *Handler) ListSubmissions(w http.ResponseWriter, r *http.Request) {
status := r.URL.Query().Get("status")
page, size := pageParams(r)
items, total, err := h.store.ListSubmissions(r.Context(), status, size, (page-1)*size)
if h.handleErr(w, err) {
return
}
pending, _ := h.store.PendingSubmissionCount(r.Context())
writeJSON(w, http.StatusOK, map[string]any{
"items": items, "page": page, "size": size, "total": total, "pending": pending,
})
}
// GetSubmission returns full submission detail (admin).
func (h *Handler) GetSubmission(w http.ResponseWriter, r *http.Request) {
d, err := h.store.GetSubmission(r.Context(), chi.URLParam(r, "id"))
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, d)
}
// ApproveSubmission applies a contribution to the product store (admin).
func (h *Handler) ApproveSubmission(w http.ResponseWriter, r *http.Request) {
d, err := h.store.ApproveSubmission(r.Context(), chi.URLParam(r, "id"), auth.UserFrom(r.Context()))
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, d)
}
// RejectSubmission rejects a contribution with a reviewer note (admin).
func (h *Handler) RejectSubmission(w http.ResponseWriter, r *http.Request) {
var body struct {
Note string `json:"note"`
}
_ = json.NewDecoder(r.Body).Decode(&body)
err := h.store.RejectSubmission(r.Context(), chi.URLParam(r, "id"), auth.UserFrom(r.Context()), body.Note)
if h.handleErr(w, err) {
return
}
writeJSON(w, http.StatusOK, map[string]string{"status": "rejected"})
}
// ---------- helpers ----------
func realIP(r *http.Request) string {
if ip := r.Header.Get("X-Forwarded-For"); ip != "" {
if i := strings.IndexByte(ip, ','); i >= 0 {
return strings.TrimSpace(ip[:i])
}
return strings.TrimSpace(ip)
}
if ip := r.Header.Get("X-Real-IP"); ip != "" {
return ip
}
return r.RemoteAddr
}
func (h *Handler) handleErr(w http.ResponseWriter, err error) bool {
if err == nil {
return false
}
if errors.Is(err, adminstore.ErrNotFound) {
writeError(w, http.StatusNotFound, "not_found", "资源不存在")
return true
}
if errors.Is(err, adminstore.ErrConflict) {
writeError(w, http.StatusConflict, "conflict", "该投稿已被处理")
return true
}
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
return true
}
func pageParams(r *http.Request) (page, size int) {
page = atoiDefault(r.URL.Query().Get("page"), 1)
if page < 1 {
page = 1
}
size = atoiDefault(r.URL.Query().Get("size"), 20)
if size < 1 {
size = 20
}
if size > 100 {
size = 100
}
return page, size
}
func atoiDefault(s string, fallback int) int {
if s == "" {
return fallback
}
n := 0
for _, c := range s {
if c < '0' || c > '9' {
return fallback
}
n = n*10 + int(c-'0')
}
return n
}
func writeJSON(w http.ResponseWriter, status int, body any) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(body)
}
func writeError(w http.ResponseWriter, status int, code, message string) {
writeJSON(w, status, map[string]any{"error": map[string]string{"code": code, "message": message}})
}
+41
View File
@@ -0,0 +1,41 @@
package adminhandler
import (
"sync"
"time"
)
// rateLimiter is a simple fixed-window per-key limiter used to throttle
// anonymous public submissions (basic anti-spam; captcha can be added later).
type rateLimiter struct {
mu sync.Mutex
hits map[string][]time.Time
limit int
window time.Duration
}
func newRateLimiter(limit int, window time.Duration) *rateLimiter {
return &rateLimiter{hits: map[string][]time.Time{}, limit: limit, window: window}
}
// allow reports whether the key may proceed, recording the hit if so.
func (r *rateLimiter) allow(key string) bool {
now := time.Now()
cutoff := now.Add(-r.window)
r.mu.Lock()
defer r.mu.Unlock()
kept := r.hits[key][:0]
for _, t := range r.hits[key] {
if t.After(cutoff) {
kept = append(kept, t)
}
}
if len(kept) >= r.limit {
r.hits[key] = kept
return false
}
r.hits[key] = append(kept, now)
return true
}
+282
View File
@@ -0,0 +1,282 @@
// Package adminstore is the read/write data-access layer for the admin console.
// Unlike the public store (read-only), it performs INSERT/UPDATE/DELETE and
// records field-level provenance (source = "manual") plus an audit_log entry
// for every write, then recomputes product.quality_score.
package adminstore
import (
"context"
"encoding/json"
"errors"
"strconv"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// ErrNotFound is returned when a requested row does not exist.
var ErrNotFound = errors.New("not found")
// Store wraps a pgx pool for admin operations.
type Store struct {
pool *pgxpool.Pool
}
// New constructs an admin Store.
func New(pool *pgxpool.Pool) *Store { return &Store{pool: pool} }
// Ping verifies DB connectivity.
func (s *Store) Ping(ctx context.Context) error { return s.pool.Ping(ctx) }
// CompletenessFields mirrors ingestion/opengoods/etl/quality.py COMPLETENESS_FIELDS.
var CompletenessFields = []string{
"name", "gtin", "brand", "category", "net_content",
"country_of_origin", "nutriments", "ingredients", "image",
}
// ---------- list ----------
// ProductRow is a list-view row for the admin product table.
type ProductRow struct {
ID string `json:"id"`
GTIN *string `json:"gtin"`
Name string `json:"name"`
Brand *string `json:"brand"`
CategoryPath *string `json:"category_path"`
Status string `json:"status"`
QualityScore float64 `json:"quality_score"`
Missing []string `json:"missing"`
UpdatedAt string `json:"updated_at"`
}
// ListProducts returns a paginated, optionally name/gtin-filtered list.
func (s *Store) ListProducts(ctx context.Context, q string, limit, offset int) ([]ProductRow, int, error) {
args := []any{}
where := "WHERE 1=1"
if q != "" {
args = append(args, q)
where += " AND (p.name ILIKE '%' || $1 || '%' OR p.gtin ILIKE '%' || $1 || '%')"
}
var total int
if err := s.pool.QueryRow(ctx, "SELECT count(*) FROM product p "+where, args...).Scan(&total); err != nil {
return nil, 0, err
}
args = append(args, limit, offset)
sql := `
SELECT p.id, p.gtin, p.name, b.name, c.path::text, p.status, p.quality_score,
p.updated_at,
(p.brand_id IS NOT NULL) AS has_brand,
(p.category_id IS NOT NULL) AS has_cat,
(p.net_content_canonical IS NOT NULL) AS has_net,
(p.country_of_origin IS NOT NULL AND p.country_of_origin <> '') AS has_country,
(f.nutriments IS NOT NULL AND f.nutriments::text <> '{}') AS has_nutri,
(f.ingredients_text IS NOT NULL AND f.ingredients_text <> '') AS has_ing,
EXISTS (SELECT 1 FROM product_image pi WHERE pi.product_id = p.id) AS has_img
FROM product p
LEFT JOIN brand b ON b.id = p.brand_id
LEFT JOIN category c ON c.id = p.category_id
LEFT JOIN food_detail f ON f.product_id = p.id ` + where +
" ORDER BY p.updated_at DESC LIMIT $" + strconv.Itoa(len(args)-1) + " OFFSET $" + strconv.Itoa(len(args))
rows, err := s.pool.Query(ctx, sql, args...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
out := []ProductRow{}
for rows.Next() {
var r ProductRow
var hasBrand, hasCat, hasNet, hasCountry, hasNutri, hasIng, hasImg bool
var updated time.Time
if err := rows.Scan(&r.ID, &r.GTIN, &r.Name, &r.Brand, &r.CategoryPath, &r.Status,
&r.QualityScore, &updated, &hasBrand, &hasCat, &hasNet, &hasCountry,
&hasNutri, &hasIng, &hasImg); err != nil {
return nil, 0, err
}
r.UpdatedAt = updated.Format(time.RFC3339)
present := map[string]bool{
"name": r.Name != "",
"gtin": r.GTIN != nil && *r.GTIN != "",
"brand": hasBrand,
"category": hasCat,
"net_content": hasNet,
"country_of_origin": hasCountry,
"nutriments": hasNutri,
"ingredients": hasIng,
"image": hasImg,
}
r.Missing = []string{}
for _, f := range CompletenessFields {
if !present[f] {
r.Missing = append(r.Missing, f)
}
}
out = append(out, r)
}
return out, total, rows.Err()
}
// ---------- detail ----------
// ProductImage is one image row.
type ProductImage struct {
ID string `json:"id"`
URL string `json:"url"`
Kind string `json:"kind"`
License *string `json:"license"`
}
// MSRP is one suggested-retail-price snapshot.
type MSRP struct {
ID string `json:"id"`
Amount float64 `json:"amount"`
Currency string `json:"currency"`
Region string `json:"region"`
EffectiveDate *string `json:"effective_date"`
SourceURL *string `json:"source_url"`
Note *string `json:"note"`
}
// ProductDetail is the full editable view of a product.
type ProductDetail struct {
ID string `json:"id"`
GTIN *string `json:"gtin"`
Name string `json:"name"`
BrandID *string `json:"brand_id"`
Brand *string `json:"brand"`
CategoryID *string `json:"category_id"`
CategoryPath *string `json:"category_path"`
NetContentValue *float64 `json:"net_content_value"`
NetContentUnit *string `json:"net_content_unit"`
CountryOfOrigin *string `json:"country_of_origin"`
Status string `json:"status"`
QualityScore float64 `json:"quality_score"`
IngredientsText *string `json:"ingredients_text"`
Allergens []string `json:"allergens"`
Additives []string `json:"additives"`
Nutriments map[string]any `json:"nutriments"`
NutritionBasis *string `json:"nutrition_basis"`
ServingSize *string `json:"serving_size"`
NutriScore *string `json:"nutri_score"`
Images []ProductImage `json:"images"`
MSRP []MSRP `json:"msrp"`
Missing []string `json:"missing"`
UpdatedAt string `json:"updated_at"`
}
// GetProduct returns the full editable detail for one product.
func (s *Store) GetProduct(ctx context.Context, id string) (*ProductDetail, error) {
var d ProductDetail
var nutriments []byte
var updated time.Time
err := s.pool.QueryRow(ctx, `
SELECT p.id, p.gtin, p.name, p.brand_id, b.name, p.category_id, c.path::text,
p.net_content_value, p.net_content_unit, p.country_of_origin, p.status,
p.quality_score, p.updated_at,
f.ingredients_text, f.allergens, f.additives, f.nutriments,
f.nutrition_basis, f.serving_size, f.nutri_score
FROM product p
LEFT JOIN brand b ON b.id = p.brand_id
LEFT JOIN category c ON c.id = p.category_id
LEFT JOIN food_detail f ON f.product_id = p.id
WHERE p.id = $1`, id).Scan(
&d.ID, &d.GTIN, &d.Name, &d.BrandID, &d.Brand, &d.CategoryID, &d.CategoryPath,
&d.NetContentValue, &d.NetContentUnit, &d.CountryOfOrigin, &d.Status,
&d.QualityScore, &updated,
&d.IngredientsText, &d.Allergens, &d.Additives, &nutriments,
&d.NutritionBasis, &d.ServingSize, &d.NutriScore,
)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
d.UpdatedAt = updated.Format(time.RFC3339)
if len(nutriments) > 0 {
_ = json.Unmarshal(nutriments, &d.Nutriments)
}
if d.Allergens == nil {
d.Allergens = []string{}
}
if d.Additives == nil {
d.Additives = []string{}
}
imgs, err := s.listImages(ctx, id)
if err != nil {
return nil, err
}
d.Images = imgs
msrps, err := s.listMSRP(ctx, id)
if err != nil {
return nil, err
}
d.MSRP = msrps
d.Missing = missingFromDetail(&d)
return &d, nil
}
func missingFromDetail(d *ProductDetail) []string {
present := map[string]bool{
"name": d.Name != "",
"gtin": d.GTIN != nil && *d.GTIN != "",
"brand": d.BrandID != nil,
"category": d.CategoryID != nil,
"net_content": d.NetContentValue != nil,
"country_of_origin": d.CountryOfOrigin != nil && *d.CountryOfOrigin != "",
"nutriments": len(d.Nutriments) > 0,
"ingredients": d.IngredientsText != nil && *d.IngredientsText != "",
"image": len(d.Images) > 0,
}
missing := []string{}
for _, f := range CompletenessFields {
if !present[f] {
missing = append(missing, f)
}
}
return missing
}
func (s *Store) listImages(ctx context.Context, productID string) ([]ProductImage, error) {
rows, err := s.pool.Query(ctx,
"SELECT id, url, kind, license FROM product_image WHERE product_id = $1 ORDER BY id", productID)
if err != nil {
return nil, err
}
defer rows.Close()
out := []ProductImage{}
for rows.Next() {
var im ProductImage
if err := rows.Scan(&im.ID, &im.URL, &im.Kind, &im.License); err != nil {
return nil, err
}
out = append(out, im)
}
return out, rows.Err()
}
func (s *Store) listMSRP(ctx context.Context, productID string) ([]MSRP, error) {
rows, err := s.pool.Query(ctx, `
SELECT id, amount, currency, region, effective_date::text, source_url, note
FROM product_msrp WHERE product_id = $1 ORDER BY effective_date DESC NULLS LAST`, productID)
if err != nil {
return nil, err
}
defer rows.Close()
out := []MSRP{}
for rows.Next() {
var m MSRP
if err := rows.Scan(&m.ID, &m.Amount, &m.Currency, &m.Region, &m.EffectiveDate, &m.SourceURL, &m.Note); err != nil {
return nil, err
}
out = append(out, m)
}
return out, rows.Err()
}
+137
View File
@@ -0,0 +1,137 @@
package adminstore
import (
"context"
"math"
"time"
"github.com/jackc/pgx/v5"
)
// Quality weights mirror ingestion/opengoods/etl/quality.py.
const (
wCompleteness = 0.4
wSourceTrust = 0.3
wAgreement = 0.2
wFreshness = 0.1
)
// queryer is satisfied by both *pgxpool.Pool and pgx.Tx.
type queryer interface {
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
}
func agreementFromSources(n int) float64 {
switch {
case n <= 1:
return 0.5
case n == 2:
return 0.8
default:
return 1.0
}
}
func freshnessFromAge(ageDays *float64) float64 {
if ageDays == nil {
return 0.5
}
d := *ageDays
switch {
case d <= 30:
return 1.0
case d <= 180:
return 0.8
case d <= 365:
return 0.6
case d <= 730:
return 0.4
default:
return 0.2
}
}
func (s *Store) computeQuality(ctx context.Context, q queryer, productID string) (float64, error) {
var name, country *string
var gtin *string
var brandID, categoryID *string
var netCanonical *float64
var ingredients *string
var hasNutri, hasImage bool
err := q.QueryRow(ctx, `
SELECT p.name, p.gtin, p.brand_id, p.category_id, p.net_content_canonical,
p.country_of_origin, f.ingredients_text,
(f.nutriments IS NOT NULL AND f.nutriments::text <> '{}'),
EXISTS (SELECT 1 FROM product_image pi WHERE pi.product_id = p.id)
FROM product p LEFT JOIN food_detail f ON f.product_id = p.id
WHERE p.id = $1`, productID).Scan(
&name, &gtin, &brandID, &categoryID, &netCanonical, &country,
&ingredients, &hasNutri, &hasImage)
if err != nil {
return 0, err
}
present := 0
bump := func(ok bool) {
if ok {
present++
}
}
bump(name != nil && *name != "")
bump(gtin != nil && *gtin != "")
bump(brandID != nil)
bump(categoryID != nil)
bump(netCanonical != nil)
bump(country != nil && *country != "")
bump(hasNutri)
bump(ingredients != nil && *ingredients != "")
bump(hasImage)
completeness := float64(present) / float64(len(CompletenessFields))
var sourceCount int
var sourceTrust *float64
var lastFetched *time.Time
err = q.QueryRow(ctx, `
SELECT count(DISTINCT ps.source_id), COALESCE(max(s.trust_weight),0), max(ps.fetched_at)
FROM product_source ps LEFT JOIN source s ON s.id = ps.source_id
WHERE ps.product_id = $1`, productID).Scan(&sourceCount, &sourceTrust, &lastFetched)
if err != nil {
return 0, err
}
trust := 0.0
if sourceTrust != nil {
trust = *sourceTrust
}
var ageDays *float64
if lastFetched != nil {
d := time.Since(*lastFetched).Hours() / 24.0
if d < 0 {
d = 0
}
ageDays = &d
}
raw := wCompleteness*completeness + wSourceTrust*trust +
wAgreement*agreementFromSources(sourceCount) + wFreshness*freshnessFromAge(ageDays)
raw = math.Max(0, math.Min(1, raw))
return math.Round(raw*1000) / 1000, nil
}
func (s *Store) recomputeQualityTx(ctx context.Context, tx pgx.Tx, productID string) (float64, error) {
v, err := s.computeQuality(ctx, tx, productID)
if err != nil {
return 0, err
}
_, err = tx.Exec(ctx, "UPDATE product SET quality_score=$1 WHERE id=$2", v, productID)
return v, err
}
func (s *Store) recomputeQuality(ctx context.Context, productID string) (float64, error) {
v, err := s.computeQuality(ctx, s.pool, productID)
if err != nil {
return 0, err
}
_, err = s.pool.Exec(ctx, "UPDATE product SET quality_score=$1 WHERE id=$2", v, productID)
return v, err
}
+30
View File
@@ -0,0 +1,30 @@
package adminstore
import "testing"
func TestAgreementFromSources(t *testing.T) {
cases := map[int]float64{0: 0.5, 1: 0.5, 2: 0.8, 3: 1.0, 9: 1.0}
for n, want := range cases {
if got := agreementFromSources(n); got != want {
t.Errorf("agreementFromSources(%d) = %v, want %v", n, got, want)
}
}
}
func TestFreshnessFromAge(t *testing.T) {
mk := func(d float64) *float64 { return &d }
if got := freshnessFromAge(nil); got != 0.5 {
t.Errorf("nil age = %v, want 0.5", got)
}
cases := []struct {
days float64
want float64
}{
{10, 1.0}, {30, 1.0}, {100, 0.8}, {300, 0.6}, {500, 0.4}, {1000, 0.2},
}
for _, c := range cases {
if got := freshnessFromAge(mk(c.days)); got != c.want {
t.Errorf("freshnessFromAge(%v) = %v, want %v", c.days, got, c.want)
}
}
}
+421
View File
@@ -0,0 +1,421 @@
package adminstore
import (
"context"
"encoding/json"
"errors"
"strconv"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// ErrConflict is returned when a submission has already been reviewed.
var ErrConflict = errors.New("conflict")
// SubmissionImage is one proposed image URL inside a contribution.
type SubmissionImage struct {
URL string `json:"url"`
Kind string `json:"kind"`
}
// SubmissionInput is the public contribution payload (no login required).
type SubmissionInput struct {
GTIN *string `json:"gtin"`
Name string `json:"name"`
BrandName *string `json:"brand_name"`
CategoryID *string `json:"category_id"`
NetContentValue *float64 `json:"net_content_value"`
NetContentUnit *string `json:"net_content_unit"`
CountryOfOrigin *string `json:"country_of_origin"`
IngredientsText *string `json:"ingredients_text"`
Nutriments map[string]any `json:"nutriments"`
NutritionBasis *string `json:"nutrition_basis"`
ServingSize *string `json:"serving_size"`
NutriScore *string `json:"nutri_score"`
Images []SubmissionImage `json:"images"`
MSRP []MSRPInput `json:"msrp"`
SubmitterName *string `json:"submitter_name"`
SubmitterContact *string `json:"submitter_contact"`
Note *string `json:"note"`
}
// SubmissionRow is a queue-list row for the admin review table.
type SubmissionRow struct {
ID string `json:"id"`
GTIN *string `json:"gtin"`
Name string `json:"name"`
Status string `json:"status"`
SubmitterName *string `json:"submitter_name"`
Matched bool `json:"matched"`
CreatedAt string `json:"created_at"`
ReviewedAt *string `json:"reviewed_at"`
}
// SubmissionDetail is the full review view of one contribution.
type SubmissionDetail struct {
ID string `json:"id"`
Status string `json:"status"`
GTIN *string `json:"gtin"`
Name string `json:"name"`
SubmitterName *string `json:"submitter_name"`
SubmitterContact *string `json:"submitter_contact"`
Note *string `json:"note"`
ReviewNote *string `json:"review_note"`
ReviewedBy *string `json:"reviewed_by"`
ReviewedAt *string `json:"reviewed_at"`
CreatedAt string `json:"created_at"`
TargetProductID *string `json:"target_product_id"`
ResultProductID *string `json:"result_product_id"`
Payload SubmissionInput `json:"payload"`
ExistingProduct *ProductDetail `json:"existing_product,omitempty"`
}
// CreateSubmission validates and stores a public contribution as pending.
func (s *Store) CreateSubmission(ctx context.Context, in SubmissionInput, remoteIP string) (string, error) {
in.Name = strings.TrimSpace(in.Name)
if in.Name == "" {
return "", errors.New("商品名称不能为空")
}
if in.GTIN != nil {
g := strings.TrimSpace(*in.GTIN)
if g == "" {
in.GTIN = nil
} else {
in.GTIN = &g
}
}
// Link to an existing product when the barcode already exists (supplement).
var target *string
if in.GTIN != nil {
var pid string
err := s.pool.QueryRow(ctx, "SELECT id FROM product WHERE gtin = $1", *in.GTIN).Scan(&pid)
if err == nil {
target = &pid
} else if !errors.Is(err, pgx.ErrNoRows) {
return "", err
}
}
payload, err := json.Marshal(in)
if err != nil {
return "", err
}
var id string
err = s.pool.QueryRow(ctx, `
INSERT INTO submission (gtin, name, payload, target_product_id, submitter_name, submitter_contact, note, remote_ip)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8) RETURNING id`,
in.GTIN, in.Name, payload, target, in.SubmitterName, in.SubmitterContact, in.Note, remoteIP).Scan(&id)
return id, err
}
// ListSubmissions returns submissions filtered by status (empty = all).
func (s *Store) ListSubmissions(ctx context.Context, status string, limit, offset int) ([]SubmissionRow, int, error) {
args := []any{}
where := "WHERE 1=1"
if status != "" {
args = append(args, status)
where += " AND status = $1"
}
var total int
if err := s.pool.QueryRow(ctx, "SELECT count(*) FROM submission "+where, args...).Scan(&total); err != nil {
return nil, 0, err
}
args = append(args, limit, offset)
sql := `
SELECT id, gtin, name, status, submitter_name, (target_product_id IS NOT NULL),
created_at, reviewed_at
FROM submission ` + where +
" ORDER BY (status='pending') DESC, created_at DESC LIMIT $" +
strconv.Itoa(len(args)-1) + " OFFSET $" + strconv.Itoa(len(args))
rows, err := s.pool.Query(ctx, sql, args...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
out := []SubmissionRow{}
for rows.Next() {
var r SubmissionRow
var created time.Time
var reviewed *time.Time
if err := rows.Scan(&r.ID, &r.GTIN, &r.Name, &r.Status, &r.SubmitterName, &r.Matched, &created, &reviewed); err != nil {
return nil, 0, err
}
r.CreatedAt = created.Format(time.RFC3339)
if reviewed != nil {
t := reviewed.Format(time.RFC3339)
r.ReviewedAt = &t
}
out = append(out, r)
}
return out, total, rows.Err()
}
// PendingSubmissionCount returns the number of submissions awaiting review.
func (s *Store) PendingSubmissionCount(ctx context.Context) (int, error) {
var n int
err := s.pool.QueryRow(ctx, "SELECT count(*) FROM submission WHERE status='pending'").Scan(&n)
return n, err
}
// GetSubmission returns the full review detail for one submission.
func (s *Store) GetSubmission(ctx context.Context, id string) (*SubmissionDetail, error) {
var d SubmissionDetail
var payload []byte
var created time.Time
var reviewed *time.Time
err := s.pool.QueryRow(ctx, `
SELECT id, status, gtin, name, submitter_name, submitter_contact, note,
review_note, reviewed_by, reviewed_at, created_at, target_product_id, result_product_id, payload
FROM submission WHERE id = $1`, id).Scan(
&d.ID, &d.Status, &d.GTIN, &d.Name, &d.SubmitterName, &d.SubmitterContact, &d.Note,
&d.ReviewNote, &d.ReviewedBy, &reviewed, &created, &d.TargetProductID, &d.ResultProductID, &payload,
)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
d.CreatedAt = created.Format(time.RFC3339)
if reviewed != nil {
t := reviewed.Format(time.RFC3339)
d.ReviewedAt = &t
}
if len(payload) > 0 {
_ = json.Unmarshal(payload, &d.Payload)
}
if d.TargetProductID != nil {
if ep, err := s.GetProduct(ctx, *d.TargetProductID); err == nil {
d.ExistingProduct = ep
}
}
return &d, nil
}
// RejectSubmission marks a pending submission as rejected with a reviewer note.
func (s *Store) RejectSubmission(ctx context.Context, id, actor, note string) error {
ct, err := s.pool.Exec(ctx, `
UPDATE submission SET status='rejected', review_note=$2, reviewed_by=$3, reviewed_at=now()
WHERE id=$1 AND status='pending'`, id, note, actor)
if err != nil {
return err
}
if ct.RowsAffected() == 0 {
// Distinguish missing vs already-reviewed.
var st string
if e := s.pool.QueryRow(ctx, "SELECT status FROM submission WHERE id=$1", id).Scan(&st); errors.Is(e, pgx.ErrNoRows) {
return ErrNotFound
}
return ErrConflict
}
_ = s.writeAudit(ctx, actor, "reject_submission", "submission", &id, []string{}, nil, map[string]string{"review_note": note})
return nil
}
// ApproveSubmission applies a pending contribution to the product store
// (creating or supplementing a product), records community provenance + audit,
// recomputes quality, and marks the submission approved.
func (s *Store) ApproveSubmission(ctx context.Context, id, actor string) (*ProductDetail, error) {
sub, err := s.GetSubmission(ctx, id)
if err != nil {
return nil, err
}
if sub.Status != "pending" {
return nil, ErrConflict
}
in := sub.Payload
tx, err := s.pool.Begin(ctx)
if err != nil {
return nil, err
}
defer tx.Rollback(ctx)
communityID, err := s.sourceIDTx(ctx, tx, "community")
if err != nil {
return nil, err
}
// Resolve the target product (existing supplement vs new create).
productID := ""
if sub.TargetProductID != nil {
productID = *sub.TargetProductID
} else if in.GTIN != nil {
var pid string
if e := tx.QueryRow(ctx, "SELECT id FROM product WHERE gtin=$1", *in.GTIN).Scan(&pid); e == nil {
productID = pid
} else if !errors.Is(e, pgx.ErrNoRows) {
return nil, e
}
}
var brandID *string
if in.BrandName != nil && strings.TrimSpace(*in.BrandName) != "" {
bid, err := s.ensureBrand(ctx, tx, strings.TrimSpace(*in.BrandName))
if err != nil {
return nil, err
}
brandID = &bid
}
var gpc *string
if in.CategoryID != nil && *in.CategoryID != "" {
if err := tx.QueryRow(ctx, "SELECT gpc_brick_code FROM category WHERE id=$1", *in.CategoryID).Scan(&gpc); err != nil && !errors.Is(err, pgx.ErrNoRows) {
return nil, err
}
}
canonical, err := s.netCanonical(ctx, tx, in.NetContentValue, in.NetContentUnit)
if err != nil {
return nil, err
}
fields := submissionFields(in)
if productID == "" {
// Create a new product from the contribution.
err = tx.QueryRow(ctx, `
INSERT INTO product (gtin, name, brand_id, category_id, gpc_brick_code,
net_content_value, net_content_unit, net_content_canonical, country_of_origin, status)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,'active') RETURNING id`,
in.GTIN, in.Name, brandID, in.CategoryID, gpc,
in.NetContentValue, in.NetContentUnit, canonical, in.CountryOfOrigin).Scan(&productID)
if err != nil {
return nil, err
}
} else {
// Supplement an existing product: only overwrite fields the
// contribution actually provides (COALESCE keeps current values).
_, err = tx.Exec(ctx, `
UPDATE product SET
name=COALESCE(NULLIF($2,''), name),
brand_id=COALESCE($3, brand_id),
category_id=COALESCE($4, category_id),
gpc_brick_code=COALESCE($5, gpc_brick_code),
net_content_value=COALESCE($6, net_content_value),
net_content_unit=COALESCE($7, net_content_unit),
net_content_canonical=COALESCE($8, net_content_canonical),
country_of_origin=COALESCE($9, country_of_origin),
gtin=COALESCE($10, gtin)
WHERE id=$1`,
productID, in.Name, brandID, in.CategoryID, gpc,
in.NetContentValue, in.NetContentUnit, canonical, in.CountryOfOrigin, in.GTIN)
if err != nil {
return nil, err
}
}
// food_detail: upsert, preserving existing values where not provided.
var nutriJSON []byte
if len(in.Nutriments) > 0 {
nutriJSON, _ = json.Marshal(in.Nutriments)
}
_, err = tx.Exec(ctx, `
INSERT INTO food_detail (product_id, ingredients_text, nutriments, nutrition_basis, serving_size, nutri_score)
VALUES ($1,$2,$3,$4,$5,$6)
ON CONFLICT (product_id) DO UPDATE SET
ingredients_text=COALESCE(EXCLUDED.ingredients_text, food_detail.ingredients_text),
nutriments=COALESCE(EXCLUDED.nutriments, food_detail.nutriments),
nutrition_basis=COALESCE(EXCLUDED.nutrition_basis, food_detail.nutrition_basis),
serving_size=COALESCE(EXCLUDED.serving_size, food_detail.serving_size),
nutri_score=COALESCE(EXCLUDED.nutri_score, food_detail.nutri_score)`,
productID, in.IngredientsText, nutriJSON, in.NutritionBasis, in.ServingSize, in.NutriScore)
if err != nil {
return nil, err
}
for _, im := range in.Images {
url := strings.TrimSpace(im.URL)
if url == "" {
continue
}
kind := im.Kind
if kind != "front" && kind != "ingredients" && kind != "nutrition" {
kind = "other"
}
if _, err := tx.Exec(ctx, `
INSERT INTO product_image (product_id, url, kind, source_id) VALUES ($1,$2,$3,$4)`,
productID, url, kind, communityID); err != nil {
return nil, err
}
}
for _, m := range in.MSRP {
if m.Amount <= 0 {
continue
}
cur := m.Currency
if cur == "" {
cur = "CNY"
}
region := m.Region
if region == "" {
region = "CN"
}
if _, err := tx.Exec(ctx, `
INSERT INTO product_msrp (product_id, amount, currency, region, source_id, source_url, effective_date, note)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8)`,
productID, m.Amount, cur, region, communityID, m.SourceURL, m.EffectiveDate, m.Note); err != nil {
return nil, err
}
}
if _, err := s.recomputeQualityTx(ctx, tx, productID); err != nil {
return nil, err
}
if _, err := tx.Exec(ctx, `
UPDATE submission SET status='approved', reviewed_by=$2, reviewed_at=now(), result_product_id=$3
WHERE id=$1`, id, actor, productID); err != nil {
return nil, err
}
// Field-level provenance for the contributed fields (community source).
if len(fields) > 0 {
if _, err := tx.Exec(ctx, `
INSERT INTO product_source (product_id, source_id, url, fields, fetched_at, raw)
VALUES ($1,$2,NULL,$3,now(),NULL)`, productID, communityID, fields); err != nil {
return nil, err
}
}
if err := tx.Commit(ctx); err != nil {
return nil, err
}
_ = s.writeAudit(ctx, actor, "approve_submission", "product", &productID, fields,
map[string]string{"submission_id": id}, map[string]string{"product_id": productID})
return s.GetProduct(ctx, productID)
}
func (s *Store) sourceIDTx(ctx context.Context, tx pgx.Tx, name string) (string, error) {
var id string
err := tx.QueryRow(ctx, "SELECT id FROM source WHERE name=$1", name).Scan(&id)
return id, err
}
// submissionFields lists the product fields a contribution provides values for.
func submissionFields(in SubmissionInput) []string {
fields := []string{"name"}
add := func(name string, present bool) {
if present {
fields = append(fields, name)
}
}
add("gtin", in.GTIN != nil && *in.GTIN != "")
add("brand", in.BrandName != nil && strings.TrimSpace(*in.BrandName) != "")
add("category", in.CategoryID != nil && *in.CategoryID != "")
add("net_content", in.NetContentValue != nil)
add("country_of_origin", in.CountryOfOrigin != nil && *in.CountryOfOrigin != "")
add("ingredients", in.IngredientsText != nil && *in.IngredientsText != "")
add("nutriments", len(in.Nutriments) > 0)
add("image", len(in.Images) > 0)
return fields
}
+415
View File
@@ -0,0 +1,415 @@
package adminstore
import (
"context"
"encoding/json"
"errors"
"strings"
"github.com/jackc/pgx/v5"
)
// ProductInput is the editable payload accepted from the admin UI.
type ProductInput struct {
GTIN *string `json:"gtin"`
Name string `json:"name"`
BrandID *string `json:"brand_id"`
BrandName *string `json:"brand_name"`
CategoryID *string `json:"category_id"`
NetContentValue *float64 `json:"net_content_value"`
NetContentUnit *string `json:"net_content_unit"`
CountryOfOrigin *string `json:"country_of_origin"`
Status string `json:"status"`
IngredientsText *string `json:"ingredients_text"`
Allergens []string `json:"allergens"`
Additives []string `json:"additives"`
Nutriments map[string]any `json:"nutriments"`
NutritionBasis *string `json:"nutrition_basis"`
ServingSize *string `json:"serving_size"`
NutriScore *string `json:"nutri_score"`
}
func normBrand(name string) string { return strings.Join(strings.Fields(strings.ToLower(name)), " ") }
func (s *Store) ensureBrand(ctx context.Context, tx pgx.Tx, name string) (string, error) {
var id string
err := tx.QueryRow(ctx, `
INSERT INTO brand (name, normalized_name) VALUES ($1, $2)
ON CONFLICT (normalized_name) DO UPDATE SET name = brand.name
RETURNING id`, name, normBrand(name)).Scan(&id)
return id, err
}
func (s *Store) manualSourceID(ctx context.Context, tx pgx.Tx) (string, error) {
var id string
err := tx.QueryRow(ctx, "SELECT id FROM source WHERE name = 'manual'").Scan(&id)
return id, err
}
// netCanonical converts value+unit to the canonical base unit via the unit table.
func (s *Store) netCanonical(ctx context.Context, tx pgx.Tx, value *float64, unit *string) (*float64, error) {
if value == nil || unit == nil || *unit == "" {
return nil, nil
}
var factor *float64
err := tx.QueryRow(ctx, "SELECT to_canonical_factor FROM unit WHERE code = $1", *unit).Scan(&factor)
if errors.Is(err, pgx.ErrNoRows) || factor == nil {
return nil, nil
}
if err != nil {
return nil, err
}
c := *value * *factor
return &c, nil
}
// UpdateProduct applies an edit, records provenance + audit, and recomputes quality.
func (s *Store) UpdateProduct(ctx context.Context, id, actor string, in ProductInput) (*ProductDetail, error) {
before, err := s.GetProduct(ctx, id)
if err != nil {
return nil, err
}
tx, err := s.pool.Begin(ctx)
if err != nil {
return nil, err
}
defer tx.Rollback(ctx)
// Resolve brand (create-by-name takes precedence over id).
brandID := in.BrandID
if in.BrandName != nil && strings.TrimSpace(*in.BrandName) != "" {
bid, err := s.ensureBrand(ctx, tx, strings.TrimSpace(*in.BrandName))
if err != nil {
return nil, err
}
brandID = &bid
}
// Resolve category gpc brick code.
var gpc *string
if in.CategoryID != nil && *in.CategoryID != "" {
if err := tx.QueryRow(ctx, "SELECT gpc_brick_code FROM category WHERE id = $1", *in.CategoryID).Scan(&gpc); err != nil && !errors.Is(err, pgx.ErrNoRows) {
return nil, err
}
}
canonical, err := s.netCanonical(ctx, tx, in.NetContentValue, in.NetContentUnit)
if err != nil {
return nil, err
}
status := in.Status
if status == "" {
status = before.Status
}
_, err = tx.Exec(ctx, `
UPDATE product SET gtin=$1, name=$2, brand_id=$3, category_id=$4, gpc_brick_code=$5,
net_content_value=$6, net_content_unit=$7, net_content_canonical=$8,
country_of_origin=$9, status=$10
WHERE id=$11`,
in.GTIN, in.Name, brandID, in.CategoryID, gpc,
in.NetContentValue, in.NetContentUnit, canonical,
in.CountryOfOrigin, status, id)
if err != nil {
return nil, err
}
var nutriJSON []byte
if in.Nutriments != nil {
nutriJSON, _ = json.Marshal(in.Nutriments)
}
allergens := in.Allergens
if allergens == nil {
allergens = []string{}
}
additives := in.Additives
if additives == nil {
additives = []string{}
}
_, err = tx.Exec(ctx, `
INSERT INTO food_detail (product_id, ingredients_text, allergens, additives,
nutriments, nutrition_basis, serving_size, nutri_score)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8)
ON CONFLICT (product_id) DO UPDATE SET
ingredients_text=EXCLUDED.ingredients_text,
allergens=EXCLUDED.allergens,
additives=EXCLUDED.additives,
nutriments=EXCLUDED.nutriments,
nutrition_basis=EXCLUDED.nutrition_basis,
serving_size=EXCLUDED.serving_size,
nutri_score=EXCLUDED.nutri_score`,
id, in.IngredientsText, allergens, additives,
nutriJSON, in.NutritionBasis, in.ServingSize, in.NutriScore)
if err != nil {
return nil, err
}
if _, err := s.recomputeQualityTx(ctx, tx, id); err != nil {
return nil, err
}
if err := tx.Commit(ctx); err != nil {
return nil, err
}
after, err := s.GetProduct(ctx, id)
if err != nil {
return nil, err
}
changed := diffFields(before, after)
if len(changed) > 0 {
if err := s.recordProvenance(ctx, id, changed); err != nil {
return nil, err
}
}
if err := s.writeAudit(ctx, actor, "update", "product", &id, changed, before, after); err != nil {
return nil, err
}
return after, nil
}
func strEq(a, b *string) bool {
if a == nil && b == nil {
return true
}
if a == nil || b == nil {
return false
}
return *a == *b
}
func floatEq(a, b *float64) bool {
if a == nil && b == nil {
return true
}
if a == nil || b == nil {
return false
}
return *a == *b
}
func diffFields(a, b *ProductDetail) []string {
changed := []string{}
add := func(name string, eq bool) {
if !eq {
changed = append(changed, name)
}
}
add("gtin", strEq(a.GTIN, b.GTIN))
add("name", a.Name == b.Name)
add("brand", strEq(a.BrandID, b.BrandID))
add("category", strEq(a.CategoryID, b.CategoryID))
add("net_content", floatEq(a.NetContentValue, b.NetContentValue) && strEq(a.NetContentUnit, b.NetContentUnit))
add("country_of_origin", strEq(a.CountryOfOrigin, b.CountryOfOrigin))
add("status", a.Status == b.Status)
add("ingredients", strEq(a.IngredientsText, b.IngredientsText))
ja, _ := json.Marshal(a.Nutriments)
jb, _ := json.Marshal(b.Nutriments)
add("nutriments", string(ja) == string(jb))
add("nutrition_basis", strEq(a.NutritionBasis, b.NutritionBasis))
add("serving_size", strEq(a.ServingSize, b.ServingSize))
add("nutri_score", strEq(a.NutriScore, b.NutriScore))
return changed
}
func (s *Store) recordProvenance(ctx context.Context, productID string, fields []string) error {
var srcID string
if err := s.pool.QueryRow(ctx, "SELECT id FROM source WHERE name = 'manual'").Scan(&srcID); err != nil {
return err
}
_, err := s.pool.Exec(ctx, `
INSERT INTO product_source (product_id, source_id, url, fields, fetched_at, raw)
VALUES ($1, $2, NULL, $3, now(), NULL)`, productID, srcID, fields)
return err
}
func (s *Store) writeAudit(ctx context.Context, actor, action, entity string, entityID *string, fields []string, before, after any) error {
bj, _ := json.Marshal(before)
aj, _ := json.Marshal(after)
if fields == nil {
fields = []string{}
}
_, err := s.pool.Exec(ctx, `
INSERT INTO audit_log (actor, action, entity, entity_id, fields, before, after)
VALUES ($1,$2,$3,$4,$5,$6,$7)`, actor, action, entity, entityID, fields, bj, aj)
return err
}
// ---------- images ----------
// AddImage inserts an image URL (manual source) and recomputes quality.
func (s *Store) AddImage(ctx context.Context, productID, actor, url, kind string) (*ProductImage, error) {
if kind == "" {
kind = "other"
}
var srcID string
if err := s.pool.QueryRow(ctx, "SELECT id FROM source WHERE name = 'manual'").Scan(&srcID); err != nil {
return nil, err
}
var im ProductImage
err := s.pool.QueryRow(ctx, `
INSERT INTO product_image (product_id, url, kind, license, source_id)
VALUES ($1,$2,$3,NULL,$4) RETURNING id, url, kind, license`,
productID, url, kind, srcID).Scan(&im.ID, &im.URL, &im.Kind, &im.License)
if err != nil {
return nil, err
}
if _, err := s.recomputeQuality(ctx, productID); err != nil {
return nil, err
}
_ = s.writeAudit(ctx, actor, "add_image", "product", &productID, []string{"image"}, nil, im)
return &im, nil
}
// DeleteImage removes an image and recomputes quality.
func (s *Store) DeleteImage(ctx context.Context, productID, imageID, actor string) error {
ct, err := s.pool.Exec(ctx, "DELETE FROM product_image WHERE id=$1 AND product_id=$2", imageID, productID)
if err != nil {
return err
}
if ct.RowsAffected() == 0 {
return ErrNotFound
}
if _, err := s.recomputeQuality(ctx, productID); err != nil {
return err
}
_ = s.writeAudit(ctx, actor, "delete_image", "product", &productID, []string{"image"}, map[string]string{"image_id": imageID}, nil)
return nil
}
// ---------- msrp ----------
// MSRPInput is the payload for adding an MSRP snapshot.
type MSRPInput struct {
Amount float64 `json:"amount"`
Currency string `json:"currency"`
Region string `json:"region"`
EffectiveDate *string `json:"effective_date"`
SourceURL *string `json:"source_url"`
Note *string `json:"note"`
}
// AddMSRP inserts a suggested-retail-price snapshot.
func (s *Store) AddMSRP(ctx context.Context, productID, actor string, in MSRPInput) (*MSRP, error) {
if in.Currency == "" {
in.Currency = "CNY"
}
if in.Region == "" {
in.Region = "CN"
}
var srcID string
_ = s.pool.QueryRow(ctx, "SELECT id FROM source WHERE name = 'manual'").Scan(&srcID)
var m MSRP
err := s.pool.QueryRow(ctx, `
INSERT INTO product_msrp (product_id, amount, currency, region, source_id, source_url, effective_date, note)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8)
RETURNING id, amount, currency, region, effective_date::text, source_url, note`,
productID, in.Amount, in.Currency, in.Region, srcID, in.SourceURL, in.EffectiveDate, in.Note).
Scan(&m.ID, &m.Amount, &m.Currency, &m.Region, &m.EffectiveDate, &m.SourceURL, &m.Note)
if err != nil {
return nil, err
}
_ = s.writeAudit(ctx, actor, "add_msrp", "product", &productID, []string{"msrp"}, nil, m)
return &m, nil
}
// DeleteMSRP removes an MSRP snapshot.
func (s *Store) DeleteMSRP(ctx context.Context, productID, msrpID, actor string) error {
ct, err := s.pool.Exec(ctx, "DELETE FROM product_msrp WHERE id=$1 AND product_id=$2", msrpID, productID)
if err != nil {
return err
}
if ct.RowsAffected() == 0 {
return ErrNotFound
}
_ = s.writeAudit(ctx, actor, "delete_msrp", "product", &productID, []string{"msrp"}, map[string]string{"msrp_id": msrpID}, nil)
return nil
}
// ---------- dictionaries ----------
// Brand is a brand option for the edit form.
type Brand struct {
ID string `json:"id"`
Name string `json:"name"`
}
// ListBrands returns all brands ordered by name.
func (s *Store) ListBrands(ctx context.Context) ([]Brand, error) {
rows, err := s.pool.Query(ctx, "SELECT id, name FROM brand ORDER BY name")
if err != nil {
return nil, err
}
defer rows.Close()
out := []Brand{}
for rows.Next() {
var b Brand
if err := rows.Scan(&b.ID, &b.Name); err != nil {
return nil, err
}
out = append(out, b)
}
return out, rows.Err()
}
// Category is a category option for the edit form.
type Category struct {
ID string `json:"id"`
NameZH string `json:"name_zh"`
NameEN *string `json:"name_en"`
Path string `json:"path"`
Level int `json:"level"`
}
// ListCategories returns the full category tree.
func (s *Store) ListCategories(ctx context.Context) ([]Category, error) {
rows, err := s.pool.Query(ctx,
"SELECT id, name_zh, name_en, path::text, level FROM category ORDER BY path")
if err != nil {
return nil, err
}
defer rows.Close()
out := []Category{}
for rows.Next() {
var c Category
if err := rows.Scan(&c.ID, &c.NameZH, &c.NameEN, &c.Path, &c.Level); err != nil {
return nil, err
}
out = append(out, c)
}
return out, rows.Err()
}
// ---------- audit ----------
// AuditEntry is one audit-log row for the history view.
type AuditEntry struct {
ID string `json:"id"`
Actor string `json:"actor"`
Action string `json:"action"`
Fields []string `json:"fields"`
CreatedAt string `json:"created_at"`
}
// ListAudit returns audit history for one product, newest first.
func (s *Store) ListAudit(ctx context.Context, productID string, limit int) ([]AuditEntry, error) {
rows, err := s.pool.Query(ctx, `
SELECT id, actor, action, fields, created_at::text
FROM audit_log WHERE entity='product' AND entity_id=$1
ORDER BY created_at DESC LIMIT $2`, productID, limit)
if err != nil {
return nil, err
}
defer rows.Close()
out := []AuditEntry{}
for rows.Next() {
var e AuditEntry
if err := rows.Scan(&e.ID, &e.Actor, &e.Action, &e.Fields, &e.CreatedAt); err != nil {
return nil, err
}
out = append(out, e)
}
return out, rows.Err()
}
+7
View File
@@ -0,0 +1,7 @@
<!doctype html>
<html lang="zh">
<head><meta charset="utf-8" /><title>OpenGoods 管理后台</title></head>
<body>
<p>管理后台前端尚未构建。Docker 构建会在此处放入真正的前端产物。</p>
</body>
</html>
+21
View File
@@ -0,0 +1,21 @@
// Package adminweb embeds the built admin SPA (Vite dist). During Docker builds
// the real dist/ is produced by the node stage and copied in before go build;
// the committed placeholder keeps the package compilable for `go build ./...`.
package adminweb
import (
"embed"
"io/fs"
)
//go:embed all:dist
var distFS embed.FS
// Dist returns the embedded SPA filesystem rooted at dist/.
func Dist() fs.FS {
sub, err := fs.Sub(distFS, "dist")
if err != nil {
panic(err)
}
return sub
}
+126
View File
@@ -0,0 +1,126 @@
// Package auth provides minimal single-account authentication for the admin
// console: a bcrypt-verified login and a stdlib HMAC-SHA256 signed token
// (JWT-compatible) plus a chi middleware that guards write routes.
package auth
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"errors"
"net/http"
"strings"
"time"
"golang.org/x/crypto/bcrypt"
)
// Authenticator holds the single admin credential and token signing secret.
type Authenticator struct {
username string
passwordHash []byte
secret []byte
ttl time.Duration
}
// New builds an Authenticator. passwordHash must be a bcrypt hash.
func New(username string, passwordHash, secret []byte, ttl time.Duration) *Authenticator {
return &Authenticator{username: username, passwordHash: passwordHash, secret: secret, ttl: ttl}
}
// ErrInvalidCredentials is returned when login fails.
var ErrInvalidCredentials = errors.New("invalid credentials")
// Login verifies the username/password and returns a signed token on success.
func (a *Authenticator) Login(username, password string) (string, error) {
if username != a.username {
// Still run bcrypt to keep timing roughly constant.
_ = bcrypt.CompareHashAndPassword(a.passwordHash, []byte(password))
return "", ErrInvalidCredentials
}
if err := bcrypt.CompareHashAndPassword(a.passwordHash, []byte(password)); err != nil {
return "", ErrInvalidCredentials
}
return a.issue(username)
}
type claims struct {
Sub string `json:"sub"`
Exp int64 `json:"exp"`
}
func b64(b []byte) string { return base64.RawURLEncoding.EncodeToString(b) }
func (a *Authenticator) sign(signingInput string) string {
mac := hmac.New(sha256.New, a.secret)
mac.Write([]byte(signingInput))
return b64(mac.Sum(nil))
}
func (a *Authenticator) issue(sub string) (string, error) {
header := b64([]byte(`{"alg":"HS256","typ":"JWT"}`))
payloadJSON, err := json.Marshal(claims{Sub: sub, Exp: time.Now().Add(a.ttl).Unix()})
if err != nil {
return "", err
}
payload := b64(payloadJSON)
signingInput := header + "." + payload
return signingInput + "." + a.sign(signingInput), nil
}
// Verify checks a token's signature and expiry, returning the subject.
func (a *Authenticator) Verify(token string) (string, error) {
parts := strings.Split(token, ".")
if len(parts) != 3 {
return "", errors.New("malformed token")
}
signingInput := parts[0] + "." + parts[1]
if !hmac.Equal([]byte(a.sign(signingInput)), []byte(parts[2])) {
return "", errors.New("bad signature")
}
payload, err := base64.RawURLEncoding.DecodeString(parts[1])
if err != nil {
return "", err
}
var c claims
if err := json.Unmarshal(payload, &c); err != nil {
return "", err
}
if time.Now().Unix() >= c.Exp {
return "", errors.New("token expired")
}
return c.Sub, nil
}
type ctxKey int
const userKey ctxKey = 0
// UserFrom returns the authenticated subject from the request context.
func UserFrom(ctx context.Context) string {
if v, ok := ctx.Value(userKey).(string); ok {
return v
}
return ""
}
// Middleware rejects requests without a valid Bearer token.
func (a *Authenticator) Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := r.Header.Get("Authorization")
token := strings.TrimPrefix(h, "Bearer ")
if token == h || token == "" {
http.Error(w, `{"error":{"code":"unauthorized","message":"missing token"}}`, http.StatusUnauthorized)
return
}
sub, err := a.Verify(token)
if err != nil {
http.Error(w, `{"error":{"code":"unauthorized","message":"invalid token"}}`, http.StatusUnauthorized)
return
}
ctx := context.WithValue(r.Context(), userKey, sub)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
+62
View File
@@ -0,0 +1,62 @@
package auth
import (
"testing"
"time"
"golang.org/x/crypto/bcrypt"
)
func newTestAuth(t *testing.T, ttl time.Duration) *Authenticator {
t.Helper()
hash, err := bcrypt.GenerateFromPassword([]byte("s3cret"), bcrypt.MinCost)
if err != nil {
t.Fatalf("hash: %v", err)
}
return New("admin", hash, []byte("test-secret"), ttl)
}
func TestLoginAndVerify(t *testing.T) {
a := newTestAuth(t, time.Hour)
token, err := a.Login("admin", "s3cret")
if err != nil {
t.Fatalf("login: %v", err)
}
sub, err := a.Verify(token)
if err != nil {
t.Fatalf("verify: %v", err)
}
if sub != "admin" {
t.Fatalf("sub = %q, want admin", sub)
}
}
func TestLoginWrongCredentials(t *testing.T) {
a := newTestAuth(t, time.Hour)
if _, err := a.Login("admin", "nope"); err == nil {
t.Fatal("expected error for wrong password")
}
if _, err := a.Login("other", "s3cret"); err == nil {
t.Fatal("expected error for wrong username")
}
}
func TestVerifyRejectsTampered(t *testing.T) {
a := newTestAuth(t, time.Hour)
token, _ := a.Login("admin", "s3cret")
if _, err := a.Verify(token + "x"); err == nil {
t.Fatal("expected bad signature error")
}
if _, err := a.Verify("not.a.token"); err == nil {
t.Fatal("expected malformed/decoding error")
}
}
func TestVerifyRejectsExpired(t *testing.T) {
a := newTestAuth(t, -time.Minute)
token, _ := a.Login("admin", "s3cret")
if _, err := a.Verify(token); err == nil {
t.Fatal("expected expired token error")
}
}
+30
View File
@@ -0,0 +1,30 @@
package config
import (
"os"
)
// Config holds runtime configuration for the OpenGoods API server.
// Values are read from environment variables with sensible defaults so the
// server can boot in a local Docker Compose setup without extra configuration.
type Config struct {
Addr string
DatabaseURL string
RedisURL string
}
// Load reads configuration from the environment.
func Load() Config {
return Config{
Addr: getenv("OPENGOODS_ADDR", ":8080"),
DatabaseURL: getenv("OPENGOODS_DATABASE_URL", "postgres://opengoods:opengoods@localhost:5432/opengoods?sslmode=disable"),
RedisURL: getenv("OPENGOODS_REDIS_URL", "redis://localhost:6379/0"),
}
}
func getenv(key, fallback string) string {
if v, ok := os.LookupEnv(key); ok && v != "" {
return v
}
return fallback
}
+236
View File
@@ -0,0 +1,236 @@
// Package handler wires up the public, read-only OpenGoods HTTP API.
// The OpenGoods service is a public-good product information API: it only
// collects and serves product facts. It exposes no purchase, checkout, or
// commerce endpoints by design.
package handler
import (
"encoding/json"
"errors"
"io/fs"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/baicai2026-baicai/goods/api/internal/store"
)
// APIVersion is the current public API version prefix.
const APIVersion = "v1"
const (
defaultPageSize = 20
maxPageSize = 100
)
// Handler holds dependencies shared by the HTTP routes.
type Handler struct {
store *store.Store
spa fs.FS
}
// New constructs a Handler backed by the given store. spa may be nil (JSON-only).
func New(s *store.Store, spa fs.FS) *Handler {
return &Handler{store: s, spa: spa}
}
// Router builds the top-level HTTP handler with middleware and routes mounted.
func (h *Handler) Router() http.Handler {
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.Recoverer)
r.Get("/healthz", h.Healthz)
r.Route("/api/"+APIVersion, func(r chi.Router) {
r.Route("/products", func(r chi.Router) {
r.Get("/barcode/{gtin}", h.ProductByBarcode)
r.Get("/search", h.SearchProducts)
r.Get("/{id}", h.ProductByID)
r.Get("/{id}/nutriments", h.ProductNutriments)
r.Get("/{id}/msrp", h.ProductMSRP)
})
r.Get("/brands", h.ListBrands)
r.Get("/categories", h.ListCategories)
r.Get("/sources/{id}", h.SourceByID)
})
// Public SPA (homepage + search + contribute). API routes above take
// precedence; everything else falls back to the embedded single-page app.
if h.spa != nil {
r.Handle("/*", http.HandlerFunc(h.serveSPA))
}
return r
}
func (h *Handler) serveSPA(w http.ResponseWriter, r *http.Request) {
rel := strings.TrimPrefix(r.URL.Path, "/")
if rel == "" {
rel = "index.html"
}
if f, err := h.spa.Open(rel); err == nil {
f.Close()
http.FileServer(http.FS(h.spa)).ServeHTTP(w, r)
return
}
// SPA fallback: serve index.html for client-side routes.
data, err := fs.ReadFile(h.spa, "index.html")
if err != nil {
http.NotFound(w, r)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write(data)
}
// Healthz reports liveness of the service.
func (h *Handler) Healthz(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}
// ProductByBarcode returns a product by its GTIN.
func (h *Handler) ProductByBarcode(w http.ResponseWriter, r *http.Request) {
p, err := h.store.ProductByGTIN(r.Context(), chi.URLParam(r, "gtin"))
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, p)
}
// ProductByID returns a product by its UUID.
func (h *Handler) ProductByID(w http.ResponseWriter, r *http.Request) {
p, err := h.store.ProductByID(r.Context(), chi.URLParam(r, "id"))
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, p)
}
// SearchProducts runs a fuzzy name search with optional category filter + paging.
func (h *Handler) SearchProducts(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query().Get("q")
category := r.URL.Query().Get("category")
page, size := pageParams(r)
items, total, err := h.store.SearchProducts(r.Context(), q, category, size, (page-1)*size)
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items,
"page": page,
"size": size,
"total": total,
})
}
// ProductNutriments returns just the nutrition facts of a product.
func (h *Handler) ProductNutriments(w http.ResponseWriter, r *http.Request) {
n, err := h.store.Nutriments(r.Context(), chi.URLParam(r, "id"))
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, n)
}
// ProductMSRP returns official suggested retail price snapshots (no purchase link).
func (h *Handler) ProductMSRP(w http.ResponseWriter, r *http.Request) {
items, err := h.store.ListMSRP(r.Context(), chi.URLParam(r, "id"))
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items,
"disclaimer": "厂商建议零售价历史快照,仅供参考,不构成购买建议,本服务不提供任何购买入口。",
})
}
// ListBrands returns a paginated list of brands.
func (h *Handler) ListBrands(w http.ResponseWriter, r *http.Request) {
page, size := pageParams(r)
items, total, err := h.store.ListBrands(r.Context(), size, (page-1)*size)
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items, "page": page, "size": size, "total": total,
})
}
// ListCategories returns the full category tree.
func (h *Handler) ListCategories(w http.ResponseWriter, r *http.Request) {
items, err := h.store.ListCategories(r.Context())
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, map[string]any{"items": items})
}
// SourceByID returns a single data source.
func (h *Handler) SourceByID(w http.ResponseWriter, r *http.Request) {
src, err := h.store.SourceByID(r.Context(), chi.URLParam(r, "id"))
if h.handleErr(w, r, err) {
return
}
writeJSON(w, http.StatusOK, src)
}
// handleErr writes an appropriate error response; returns true if it handled one.
func (h *Handler) handleErr(w http.ResponseWriter, r *http.Request, err error) bool {
if err == nil {
return false
}
if errors.Is(err, store.ErrNotFound) {
writeError(w, r, http.StatusNotFound, "not_found", "resource not found")
return true
}
writeError(w, r, http.StatusInternalServerError, "internal_error", "internal server error")
return true
}
func pageParams(r *http.Request) (page, size int) {
page = atoiDefault(r.URL.Query().Get("page"), 1)
if page < 1 {
page = 1
}
size = atoiDefault(r.URL.Query().Get("size"), defaultPageSize)
if size < 1 {
size = defaultPageSize
}
if size > maxPageSize {
size = maxPageSize
}
return page, size
}
func atoiDefault(s string, fallback int) int {
if s == "" {
return fallback
}
v, err := strconv.Atoi(s)
if err != nil {
return fallback
}
return v
}
func writeJSON(w http.ResponseWriter, status int, body any) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(body)
}
func writeError(w http.ResponseWriter, r *http.Request, status int, code, message string) {
writeJSON(w, status, map[string]any{
"error": map[string]string{
"code": code,
"message": message,
"request_id": middleware.GetReqID(r.Context()),
},
})
}
+134
View File
@@ -0,0 +1,134 @@
package handler
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/baicai2026-baicai/goods/api/internal/store"
)
// newTestHandler connects to the test database, skipping if unavailable or
// unmigrated. It inserts a known product (cleaned up via t.Cleanup) so the
// endpoint assertions are deterministic.
func newTestHandler(t *testing.T) (*Handler, string) {
t.Helper()
dsn := os.Getenv("OPENGOODS_DATABASE_URL")
if dsn == "" {
dsn = "postgres://opengoods:opengoods@localhost:5432/opengoods?sslmode=disable"
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
t.Skipf("no database: %v", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
t.Skipf("database not reachable: %v", err)
}
var hasProduct bool
if err := pool.QueryRow(ctx, "SELECT to_regclass('public.product') IS NOT NULL").Scan(&hasProduct); err != nil || !hasProduct {
pool.Close()
t.Skip("migrations not applied")
}
gtin := "4006381333931"
_, err = pool.Exec(context.Background(), `
INSERT INTO product (gtin, name, category_id, net_content_value, net_content_unit)
VALUES ($1, 'Test Cola', (SELECT id FROM category WHERE path='food.beverages.carbonated'), 330, 'ml')
ON CONFLICT (gtin) WHERE gtin IS NOT NULL DO UPDATE SET name = EXCLUDED.name`, gtin)
if err != nil {
pool.Close()
t.Fatalf("seed insert failed: %v", err)
}
var pid string
_ = pool.QueryRow(context.Background(), "SELECT id FROM product WHERE gtin=$1", gtin).Scan(&pid)
_, _ = pool.Exec(context.Background(), `
INSERT INTO food_detail (product_id, nutrition_basis, nutriments)
VALUES ($1, 'per_100ml', '{"energy_kcal": 42}'::jsonb)
ON CONFLICT (product_id) DO UPDATE SET nutriments = EXCLUDED.nutriments`, pid)
t.Cleanup(func() {
_, _ = pool.Exec(context.Background(), "DELETE FROM product WHERE gtin=$1", gtin)
pool.Close()
})
return New(store.New(pool), nil), gtin
}
func doGET(t *testing.T, h *Handler, path string) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequest(http.MethodGet, path, nil)
rec := httptest.NewRecorder()
h.Router().ServeHTTP(rec, req)
return rec
}
func TestProductByBarcode(t *testing.T) {
h, gtin := newTestHandler(t)
rec := doGET(t, h, "/api/"+APIVersion+"/products/barcode/"+gtin)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
}
var p store.Product
if err := json.NewDecoder(rec.Body).Decode(&p); err != nil {
t.Fatal(err)
}
if p.Name != "Test Cola" || p.GTIN == nil || *p.GTIN != gtin {
t.Fatalf("unexpected product: %+v", p)
}
if p.CategoryPath == nil || *p.CategoryPath != "food.beverages.carbonated" {
t.Fatalf("category not joined: %+v", p.CategoryPath)
}
}
func TestProductByBarcodeNotFound(t *testing.T) {
h, _ := newTestHandler(t)
rec := doGET(t, h, "/api/"+APIVersion+"/products/barcode/0000000000000")
if rec.Code != http.StatusNotFound {
t.Fatalf("expected 404, got %d", rec.Code)
}
}
func TestSearchProducts(t *testing.T) {
h, _ := newTestHandler(t)
rec := doGET(t, h, "/api/"+APIVersion+"/products/search?q=Cola&category=food.beverages")
if rec.Code != http.StatusOK {
t.Fatalf("status = %d", rec.Code)
}
var body struct {
Items []store.ProductSummary `json:"items"`
Total int `json:"total"`
}
if err := json.NewDecoder(rec.Body).Decode(&body); err != nil {
t.Fatal(err)
}
if body.Total < 1 {
t.Fatalf("expected at least 1 result, got %d", body.Total)
}
}
func TestListCategories(t *testing.T) {
h, _ := newTestHandler(t)
rec := doGET(t, h, "/api/"+APIVersion+"/categories")
if rec.Code != http.StatusOK {
t.Fatalf("status = %d", rec.Code)
}
var body struct {
Items []store.Category `json:"items"`
}
if err := json.NewDecoder(rec.Body).Decode(&body); err != nil {
t.Fatal(err)
}
if len(body.Items) < 20 {
t.Fatalf("expected seeded categories, got %d", len(body.Items))
}
}
+48
View File
@@ -0,0 +1,48 @@
package handler
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
func TestHealthz(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/healthz", nil)
rec := httptest.NewRecorder()
New(nil, nil).Router().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status %d, got %d", http.StatusOK, rec.Code)
}
var body map[string]string
if err := json.NewDecoder(rec.Body).Decode(&body); err != nil {
t.Fatalf("failed to decode body: %v", err)
}
if body["status"] != "ok" {
t.Fatalf("expected status ok, got %q", body["status"])
}
}
func TestPageParams(t *testing.T) {
cases := []struct {
query string
wantPage, wantSz int
}{
{"", 1, defaultPageSize},
{"page=3&size=10", 3, 10},
{"page=0&size=-5", 1, defaultPageSize},
{"size=1000", 1, maxPageSize},
{"page=abc", 1, defaultPageSize},
}
for _, c := range cases {
req := httptest.NewRequest(http.MethodGet, "/?"+c.query, nil)
page, size := pageParams(req)
if page != c.wantPage || size != c.wantSz {
t.Errorf("query %q: got page=%d size=%d, want page=%d size=%d",
c.query, page, size, c.wantPage, c.wantSz)
}
}
}
+10
View File
@@ -0,0 +1,10 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<title>OpenGoods</title>
</head>
<body>
<div id="root">OpenGoods public site placeholder. Built assets are injected during Docker build.</div>
</body>
</html>
+22
View File
@@ -0,0 +1,22 @@
// Package publicweb embeds the built public SPA (Vite dist). During Docker
// builds the real dist/ is produced by the node stage and copied in before go
// build; the committed placeholder keeps the package compilable for
// `go build ./...`.
package publicweb
import (
"embed"
"io/fs"
)
//go:embed all:dist
var distFS embed.FS
// Dist returns the embedded SPA filesystem rooted at dist/.
func Dist() fs.FS {
sub, err := fs.Sub(distFS, "dist")
if err != nil {
panic(err)
}
return sub
}
+278
View File
@@ -0,0 +1,278 @@
// Package store is the read-only data access layer for the OpenGoods API.
// It only issues SELECT queries; all writes happen in the Python ingestion path.
package store
import (
"context"
"errors"
"strconv"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// ErrNotFound is returned when a requested row does not exist.
var ErrNotFound = errors.New("not found")
// Store wraps a PostgreSQL connection pool.
type Store struct {
pool *pgxpool.Pool
}
// New constructs a Store from an existing pgx pool.
func New(pool *pgxpool.Pool) *Store {
return &Store{pool: pool}
}
// Ping verifies database connectivity.
func (s *Store) Ping(ctx context.Context) error {
return s.pool.Ping(ctx)
}
// Product is the full public view of a product.
type Product struct {
ID string `json:"id"`
GTIN *string `json:"gtin"`
Name string `json:"name"`
Brand *string `json:"brand"`
CategoryPath *string `json:"category_path"`
GPCBrickCode *string `json:"gpc_brick_code"`
NetContentValue *float64 `json:"net_content_value"`
NetContentUnit *string `json:"net_content_unit"`
CountryOfOrigin *string `json:"country_of_origin"`
QualityScore float64 `json:"quality_score"`
Nutriments map[string]any `json:"nutriments,omitempty"`
NutritionBasis *string `json:"nutrition_basis,omitempty"`
NutriScore *string `json:"nutri_score,omitempty"`
Ingredients *string `json:"ingredients_text,omitempty"`
Allergens []string `json:"allergens,omitempty"`
Additives []string `json:"additives,omitempty"`
}
// ProductSummary is a lightweight row used in search/listing responses.
type ProductSummary struct {
ID string `json:"id"`
GTIN *string `json:"gtin"`
Name string `json:"name"`
Brand *string `json:"brand"`
CategoryPath *string `json:"category_path"`
}
const productSelect = `
SELECT p.id, p.gtin, p.name, b.name, c.path::text, p.gpc_brick_code,
p.net_content_value, p.net_content_unit, p.country_of_origin, p.quality_score,
f.nutriments, f.nutrition_basis, f.nutri_score, f.ingredients_text,
f.allergens, f.additives
FROM product p
LEFT JOIN brand b ON b.id = p.brand_id
LEFT JOIN category c ON c.id = p.category_id
LEFT JOIN food_detail f ON f.product_id = p.id
`
func scanProduct(row pgx.Row) (*Product, error) {
var p Product
err := row.Scan(
&p.ID, &p.GTIN, &p.Name, &p.Brand, &p.CategoryPath, &p.GPCBrickCode,
&p.NetContentValue, &p.NetContentUnit, &p.CountryOfOrigin, &p.QualityScore,
&p.Nutriments, &p.NutritionBasis, &p.NutriScore, &p.Ingredients,
&p.Allergens, &p.Additives,
)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
return &p, nil
}
// ProductByGTIN looks up an active product by its barcode.
func (s *Store) ProductByGTIN(ctx context.Context, gtin string) (*Product, error) {
row := s.pool.QueryRow(ctx, productSelect+" WHERE p.gtin = $1 AND p.status = 'active'", gtin)
return scanProduct(row)
}
// ProductByID looks up a product by its UUID.
func (s *Store) ProductByID(ctx context.Context, id string) (*Product, error) {
row := s.pool.QueryRow(ctx, productSelect+" WHERE p.id = $1", id)
return scanProduct(row)
}
// SearchProducts performs a fuzzy name search with optional category subtree filter.
func (s *Store) SearchProducts(ctx context.Context, q, category string, limit, offset int) ([]ProductSummary, int, error) {
args := []any{}
where := "WHERE p.status = 'active'"
if q != "" {
args = append(args, q)
where += " AND p.name ILIKE '%' || $1 || '%'"
}
if category != "" {
args = append(args, category)
where += " AND c.path <@ $" + strconv.Itoa(len(args)) + "::ltree"
}
countSQL := "SELECT count(*) FROM product p LEFT JOIN category c ON c.id = p.category_id " + where
var total int
if err := s.pool.QueryRow(ctx, countSQL, args...).Scan(&total); err != nil {
return nil, 0, err
}
args = append(args, limit, offset)
listSQL := `
SELECT p.id, p.gtin, p.name, b.name, c.path::text
FROM product p
LEFT JOIN brand b ON b.id = p.brand_id
LEFT JOIN category c ON c.id = p.category_id ` + where +
" ORDER BY p.name LIMIT $" + strconv.Itoa(len(args)-1) + " OFFSET $" + strconv.Itoa(len(args))
rows, err := s.pool.Query(ctx, listSQL, args...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
out := []ProductSummary{}
for rows.Next() {
var ps ProductSummary
if err := rows.Scan(&ps.ID, &ps.GTIN, &ps.Name, &ps.Brand, &ps.CategoryPath); err != nil {
return nil, 0, err
}
out = append(out, ps)
}
return out, total, rows.Err()
}
// Nutriments returns just the nutrition payload for a product.
type Nutriments struct {
ProductID string `json:"product_id"`
Basis *string `json:"basis"`
NutriScore *string `json:"nutri_score"`
Values map[string]any `json:"values"`
}
// Nutriments fetches the nutrition facts of a product.
func (s *Store) Nutriments(ctx context.Context, id string) (*Nutriments, error) {
var n Nutriments
n.ProductID = id
err := s.pool.QueryRow(ctx,
"SELECT nutriments, nutrition_basis, nutri_score FROM food_detail WHERE product_id = $1", id,
).Scan(&n.Values, &n.Basis, &n.NutriScore)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
return &n, nil
}
// MSRP is an official suggested retail price snapshot (never a purchase link).
type MSRP struct {
Amount float64 `json:"amount"`
Currency string `json:"currency"`
Region string `json:"region"`
EffectiveDate *string `json:"effective_date"`
SourceURL *string `json:"source_url"`
Note *string `json:"note"`
}
// ListMSRP returns all MSRP snapshots for a product.
func (s *Store) ListMSRP(ctx context.Context, id string) ([]MSRP, error) {
rows, err := s.pool.Query(ctx,
`SELECT amount, currency, region, effective_date::text, source_url, note
FROM product_msrp WHERE product_id = $1 ORDER BY effective_date DESC NULLS LAST`, id)
if err != nil {
return nil, err
}
defer rows.Close()
out := []MSRP{}
for rows.Next() {
var m MSRP
if err := rows.Scan(&m.Amount, &m.Currency, &m.Region, &m.EffectiveDate, &m.SourceURL, &m.Note); err != nil {
return nil, err
}
out = append(out, m)
}
return out, rows.Err()
}
// Brand is a public brand entry.
type Brand struct {
ID string `json:"id"`
Name string `json:"name"`
}
// ListBrands returns brands ordered by name.
func (s *Store) ListBrands(ctx context.Context, limit, offset int) ([]Brand, int, error) {
var total int
if err := s.pool.QueryRow(ctx, "SELECT count(*) FROM brand").Scan(&total); err != nil {
return nil, 0, err
}
rows, err := s.pool.Query(ctx, "SELECT id, name FROM brand ORDER BY name LIMIT $1 OFFSET $2", limit, offset)
if err != nil {
return nil, 0, err
}
defer rows.Close()
out := []Brand{}
for rows.Next() {
var b Brand
if err := rows.Scan(&b.ID, &b.Name); err != nil {
return nil, 0, err
}
out = append(out, b)
}
return out, total, rows.Err()
}
// Category is a node in the self-built category tree.
type Category struct {
ID string `json:"id"`
NameZH string `json:"name_zh"`
NameEN *string `json:"name_en"`
Path string `json:"path"`
GPCBrickCode *string `json:"gpc_brick_code"`
Level int `json:"level"`
}
// ListCategories returns the full category tree ordered by path.
func (s *Store) ListCategories(ctx context.Context) ([]Category, error) {
rows, err := s.pool.Query(ctx,
"SELECT id, name_zh, name_en, path::text, gpc_brick_code, level FROM category ORDER BY path")
if err != nil {
return nil, err
}
defer rows.Close()
out := []Category{}
for rows.Next() {
var c Category
if err := rows.Scan(&c.ID, &c.NameZH, &c.NameEN, &c.Path, &c.GPCBrickCode, &c.Level); err != nil {
return nil, err
}
out = append(out, c)
}
return out, rows.Err()
}
// Source describes a data source with its license and trust weight.
type Source struct {
ID string `json:"id"`
Name string `json:"name"`
Homepage *string `json:"homepage"`
License *string `json:"license"`
TrustWeight float64 `json:"trust_weight"`
}
// SourceByID fetches a single data source.
func (s *Store) SourceByID(ctx context.Context, id string) (*Source, error) {
var src Source
err := s.pool.QueryRow(ctx,
"SELECT id, name, homepage, license, trust_weight FROM source WHERE id = $1", id,
).Scan(&src.ID, &src.Name, &src.Homepage, &src.License, &src.TrustWeight)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
if err != nil {
return nil, err
}
return &src, nil
}
+79
View File
@@ -0,0 +1,79 @@
name: goods
services:
postgres:
image: postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
interval: 5s
timeout: 5s
retries: 10
redis:
image: redis:7-alpine
restart: unless-stopped
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 10
minio:
image: minio/minio:latest
restart: unless-stopped
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
volumes:
- miniodata:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 10
api:
build:
context: .
dockerfile: api/Dockerfile.prod
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
environment:
OPENGOODS_ADDR: ":8080"
OPENGOODS_DATABASE_URL: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable"
OPENGOODS_REDIS_URL: "redis://redis:6379/0"
ports:
- "127.0.0.1:8120:8080"
admin:
build:
context: .
dockerfile: api/Dockerfile.admin
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
environment:
GOODS_ADMIN_ADDR: ":8080"
OPENGOODS_DATABASE_URL: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable"
GOODS_ADMIN_BASE_PATH: "/ping"
GOODS_ADMIN_USER: "${GOODS_ADMIN_USER}"
GOODS_ADMIN_PASSWORD: "${GOODS_ADMIN_PASSWORD}"
GOODS_ADMIN_JWT_SECRET: "${GOODS_ADMIN_JWT_SECRET}"
ports:
- "127.0.0.1:8121:8080"
volumes:
pgdata:
miniodata:
+61
View File
@@ -0,0 +1,61 @@
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: opengoods
POSTGRES_PASSWORD: opengoods
POSTGRES_DB: opengoods
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U opengoods"]
interval: 5s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
ports:
- "6379:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 5
minio:
image: minio/minio:latest
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: opengoods
MINIO_ROOT_PASSWORD: opengoods123
ports:
- "9000:9000"
- "9001:9001"
volumes:
- miniodata:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
api:
build: ./api
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
environment:
OPENGOODS_ADDR: ":8080"
OPENGOODS_DATABASE_URL: "postgres://opengoods:opengoods@postgres:5432/opengoods?sslmode=disable"
OPENGOODS_REDIS_URL: "redis://redis:6379/0"
ports:
- "8080:8080"
volumes:
pgdata:
miniodata:
+36
View File
@@ -0,0 +1,36 @@
# 数据契约 (Data Contract) v0.1
本契约是 Go(API) 与 Python(ingestion) 两端共享的"事实约定",避免两端对字段含义理解不一致。
> 写入责任:**仅 Python (ingestion) 通过 ETL 写入数据库**Go (API) **只读**。所有写入必须经过单位归一化与字段级溯源。
## 1. 边界原则
- 系统只采集与提供**客观商品信息**;不包含任何购买/交易语义的字段或端点。
- 价格仅收录**官方建议零售价 (MSRP)** 的静态快照,必须带 `currency`/`region`/`source`/`effective_date`
## 2. 固定枚举
| 字段 | 取值 |
|------|------|
| `product.status` | `active` / `merged` / `deprecated` |
| `food_detail.nutrition_basis` | `per_100g` / `per_100ml` / `per_serving` |
| `unit.dimension` | `mass` / `volume` / `energy` / `count` / `ratio` / `length` / `duration` |
| `source.license` | `ODbL` / `CC0` / `proprietary` / ... |
| `product_image.kind` | `front` / `ingredients` / `nutrition` / `other` |
## 3. 单位规则
- 数值字段同时保存**原始值 + 单位**与**归一化值 + 基准单位**canonical)。
- 质量 → `g`,体积 → `ml`,能量 → `kJ`(同时保留 `kcal`)。
- 归一化逻辑由 `ingestion/opengoods/units.py` 提供(纯函数,含测试),换算因子是唯一事实来源。
- 营养成分统一折算到品类模板规定的基准(`per_100g` / `per_100ml`)。
## 4. 标识与可空性
- `product.gtin`8/12/13/14 位数字,可空(无条码商品),非空时全局唯一。
- `product.quality_score` ∈ [0, 1]。
- 货币用 ISO 4217`CNY` 等),国家/地区用简短代码(`CN` 等)。
## 5. 溯源 (Provenance)
- 每条数据通过 `product_source` 记录来源、URL、贡献字段、抓取时间与原始快照。
- 对外 API 在 `sources` 中透明返回来源与其许可。
## 6. 版本
- 本契约随 schema 演进版本化;任何 schema 变更需同步更新:迁移(SQL) + 本契约 + `docs/openapi.yaml`
+34
View File
@@ -0,0 +1,34 @@
# 生产部署 (Docker)
`docker-compose.prod.yml` 部署,与本地 `docker-compose.yml` 的区别:
-`api` 映射宿主端口,且绑定 `127.0.0.1:8120`(由外层 nginx 反代 + HTTPS);`postgres`/`redis`/`minio` 不对外暴露端口,仅容器内网互通。
- 所有服务 `restart: unless-stopped`
- 凭据从 `.env` 注入(见 `.env.example`),不写入仓库。
- `api` 使用 `api/Dockerfile.prod`:运行镜像用 `scratch`(从构建镜像拷贝 ca-certs),适用于 `gcr.io/distroless` 不可达的环境;Go 模块走 `goproxy.cn`
- `admin`(运营后台):带登录的写入服务 + 内嵌前端,绑定 `127.0.0.1:8121`,由 nginx 反代到公开站点的 `/ping` 路径。镜像 `api/Dockerfile.admin`(node 构建前端 → 内嵌进 Go 二进制 → scratch 运行)。仅 `admin` 可写库(人工编辑以 `source=manual` 记录字段级溯源 + `audit_log` 留痕),公开 `api` 仍只读。
## 步骤
```bash
cp .env.example .env # 填入真实随机密码
docker compose -f docker-compose.prod.yml up -d --build
# 迁移(migrate 容器接入同一网络,DSN 指向 postgres 服务)
set -a; . ./.env; set +a
DBURL="postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable"
docker run --rm --network goods_default -v "$PWD/migrations:/migrations" \
migrate/migrate -path=/migrations -database "$DBURL" up
curl -s http://127.0.0.1:8120/healthz # {"status":"ok"}
```
nginx 反代(子域 + HTTPS):80 端口 301 跳转到 443443 `proxy_pass http://127.0.0.1:8120`,证书用 acme.sh 签发并配 `--reloadcmd "nginx -s reload"` 自动续期。
运营后台 `/ping`(同域复用证书):在 443 server 块内加一段
```nginx
location /ping { proxy_pass http://127.0.0.1:8121; }
```
后台凭据见 `.env``GOODS_ADMIN_USER` / `GOODS_ADMIN_PASSWORD` / `GOODS_ADMIN_JWT_SECRET`。新增迁移 `0005_admin``audit_log` 表 + `manual` 来源)随 `migrate ... up` 自动应用。
+11
View File
@@ -0,0 +1,11 @@
# 免责声明 (Disclaimer)
天工·商品标签 (OpenGoods) 是一个**公益信息平台**。
- 本站**仅提供商品参数信息,不提供任何购买、下单、比价或导购服务**,不包含任何购买入口或交易链接。
- 商品参数(成分、营养、规格等)来自多个数据来源并标注出处,可能存在误差或滞后;**请以商品实物标签为准**。
- 价格字段仅为**官方建议零售价 (MSRP) 的历史快照**,标注来源与时间,实际售价以零售商为准,**不构成消费或购买建议**。
- 本站不提供医疗、健康或功效宣称。
- 数据按各来源许可使用(详见各条数据的 `sources` 字段与来源说明);权利方可通过公开渠道申请更正或下架。
> The OpenGoods service only collects and serves product information for public benefit. It provides **no purchase, checkout, price-comparison, or shopping-guide functionality**.
+38
View File
@@ -0,0 +1,38 @@
# ETL: Open Food Facts 导入 (M2)
把 Open Food Facts (OFF, ODbL 许可) 的食品数据采集、转换并入库。只有 Python 采集侧写库,每条记录都以 `openfoodfacts` 为来源记录**字段级溯源**。
## 流程
```
OFF API / dump(jsonl[.gz])
→ adapters/openfoodfacts.py # 读取(限速 + User-Agent) / 解析 dump
→ etl/transform.py # 字段映射 + 单位归一 + 营养 per_100g + 分类映射(关键词)
→ etl/load.py # psycopg upsert(product/food_detail/product_image) + product_source 溯源
```
## 运行
先确保本地依赖与迁移就绪:`docker compose up -d postgres` + `migrate ... up`
```bash
# 用 OFF API 拉指定条码(客户端限速, 默认 4s/次)
python -m opengoods.jobs.seed_off --barcodes 3017624010701 5449000000996
# 用下载好的 OFF dump 批量导入(可 .gz), 限制条数
python -m opengoods.jobs.seed_off --dump products.jsonl.gz --limit 1000
```
DSN 默认读 `OPENGOODS_DATABASE_URL`
## 字段映射要点
| OFF | OpenGoods | 处理 |
|-----|-----------|------|
| `code` | `product.gtin` | GTIN-8/12/13/14 校验位验证, 不合法则不作为 gtin |
| `product_name_zh/_/_en` | `product.name` | 优先中文 |
| `brands` | `brand` | 取第一个, normalized_name 去重 |
| `quantity` | `net_content_*` | 解析 "500 g"/"1,5 L" → 经 `units.py` 归一(原始+归一双存) |
| `nutriments.*_100g` | `food_detail.nutriments` | per_100g; 能量 kJ/kcal 双存, 缺一自动换算 |
| `allergens_tags`/`additives_tags` | `allergens`/`additives` | 去 `en:` 前缀 |
| `nutriscore_grade` | `nutri_score` | 大写单字母 |
| `categories*`/name | `category_id` | 关键词映射到自建品类树(起步版, 后续换 OFF 分类→GPC 映射表) |
| `image_front_url` | `product_image` | 标 CC-BY-SA 许可 |
> 全量 dump 约数 GBCI 与单测用 fixture 离线验证 transform,DB 集成测试在无库时自动跳过。
+76
View File
@@ -0,0 +1,76 @@
# 采集管理 (M4)
M4 在 M2Open Food Facts 首次导入)基础上,补齐"持续运营"所需的采集能力:
增量更新、第二数据源补全(GS1)、去重合并与字段级冲突解决、数据质量评分,
以及把这些串起来的定时调度。全部为 Python 侧(`ingestion/`),只写库、可单测。
## 组成
| 能力 | 模块 | 说明 |
|------|------|------|
| 增量采集 | `adapters/openfoodfacts.py: fetch_modified_since()` | 按 `last_modified_t` 拉取自上次水位后变更的商品 |
| 采集水位 | `etl/state.py` + `ingest_state` 表 | 每个源持久化 `last_modified_t`,只前进不回退 |
| GS1 补全 | `adapters/gs1.py` + `etl/supplement.py` | 用权威条码源补**缺失**字段(品牌/厂商/GPC/产地/净含量),不覆盖已有值 |
| 去重合并 | `etl/dedup.py` | 非 GTIN 重复(同名+品牌+净含量)合并到质量最高的主记录 |
| 冲突解决 | `etl/merge.py` | 多源同字段按"源权重 > 新鲜度"择优,保留字段级溯源 |
| 质量评分 | `etl/quality.py` | 0~1 分,落到 `product.quality_score` |
| 定时调度 | `jobs/schedule.py` | 固定周期跑"增量 + 去重"一轮,零额外依赖 |
## 质量评分
锁定公式(各分量均归一到 0~1):
```
quality = 0.4 * 完整度 + 0.3 * 源权重 + 0.2 * 多源一致 + 0.1 * 新鲜度
```
- **完整度**`name/gtin/brand/category/net_content/country/nutriments/ingredients/image` 9 项的命中比例。
- **源权重**:贡献该商品的源中最高 `source.trust_weight`OFF=0.7GS1=0.9)。
- **多源一致**:源数量代理——单源 0.5、两源 0.8、三源及以上 1.0(单源无法互证)。
- **新鲜度**:最近一次 `product_source.fetched_at` 的时间衰减(≤30d=1.0 … >730d=0.2)。
`load_record()``merge_products()` 写入后都会调 `update_quality()` 重算。
## 增量水位
`ingest_state`(迁移 `0004`)每源一行,记录 `last_modified_t``last_run_at``stats`
`set_watermark()``GREATEST(...)` 保证水位只前进,避免乱序/中断的运行回退进度。
## 运行
前置:`docker compose up -d postgres` 且迁移已 `up`(含 `0004`)。DSN 默认读 `OPENGOODS_DATABASE_URL`
```bash
# 增量更新 OFF(从持久化水位开始;--since 可覆盖)
python -m opengoods.jobs.update_off --max-pages 5
python -m opengoods.jobs.update_off --since 1700000000
# 去重合并(--dry-run 只报告不写库)
python -m opengoods.jobs.dedup --dry-run
python -m opengoods.jobs.dedup --actor nightly
# 定时调度:单轮 / 周期循环(增量 + 去重)
python -m opengoods.jobs.schedule --once
python -m opengoods.jobs.schedule --interval 3600
```
## GS1 补全
GS1 为付费、分区域的授权数据,适配器支持两种模式:
- **离线**(默认):从本地 JSON 映射 `{gtin: {...}}` 查(`GS1Adapter.from_file(path)`),
供测试与内网环境使用。
- **在线**:传 `base_url` + `client`+ `api_key`),`GET {base_url}/{gtin}`,按
Verified-by-GS1 风格字段解析。
补全只填**空缺**字段并在 `product_source` 记字段级溯源,源标记为 `gs1`
## 测试
```bash
cd ingestion && pip install -e ".[dev]"
ruff check . && ruff format --check . && pytest -q
```
纯函数测试(质量/冲突/增量分页)始终运行;依赖库的测试(水位/质量落库/去重/GS1 补全)
在无数据库或未应用 M4 迁移时自动跳过。
-115
View File
@@ -1,115 +0,0 @@
# 天工·商品标签 (OpenGoods) — 最终规划 (Final)
> 公益网站/服务:采集全网商品信息,提供商品参数查询 API。
> **核心原则:只采集 + 只提供信息,绝不涉及任何购买/下单/比价导购。**
> 本文档为前几版(v0.1 → v2.0)的最终收敛版,所有关键决策已锁定。详细设计见 v2.0 附件。
---
## 0. 项目标识
- **中文名**:天工·商品标签(呼应《天工开物》)
- **英文名**OpenGoods
- **定位**:开放、中立、可溯源的"商品参数百科 + 开放 API"
---
## 1. 已锁定的全部决策
| 维度 | 决策 |
|------|------|
| 首批品类 | **食品快消** |
| 价格 | 只收 **官方标准零售价 (MSRP)**:静态字段,带 currency/region/source/effective_date + 免责;**不收实时电商价、无购买入口** |
| 技术栈 | **Go**(对外只读 API/核心服务) + **Python**(采集/ETL/爬虫),经 **PostgreSQL + Redis/队列** 解耦 |
| 种子数据 | **Open Food Facts 食品 dump** 先导入,最快有真实数据 |
| 数据许可 | 对外数据库用 **ODbL + 署名**;CC0 来源(USDA)自由混入;每条数据按来源标注许可 |
| 商品分类 | **GS1 GPC 四层标准码 (Segment→Family→Class→Brick)** 为骨架 + **自建中文品类树** 映射 + 保留来源原始分类 |
| 单位管理 | 量纲字典;**原始值 + 归一化值双存**;营养统一折算到 `per_100g/per_100ml`;能量 **双存 kJ+kcal**;用十进制(NUMERIC)防误差 |
| 质量评分 | `0.4*完整度 + 0.3*来源权威 + 0.2*多源一致 + 0.1*新鲜度` |
| 众包 | **一期不做众包,先纯采集**;二期再开放贡献/纠错(带审核与版本化) |
| Go 框架 | `chi` + 标准库 `net/http`(轻量) |
| 迁移工具 | `golang-migrate`(纯 SQL,两端共享 schema |
| 部署 | 初期 Docker Composepostgres+redis+minio+go-api+python-worker)→ 后期 K8s |
| 地域 | 先用 OFF 全球食品库起步,后接 GS1-China 补强中国数据 |
---
## 2. 架构(定稿)
```
数据源: OFF dump / OFF API / USDA(CC0) / GS1-China
▼ Python: 采集 adapters → ETL(清洗/单位归一/分类映射/去重/质量评分)
│ 写入
┌────▼─────────┐ 图片 ┌──────────┐
│ PostgreSQL │◀───────▶│ MinIO/S3 │
│ (商品档案主库)│ └──────────┘
└────▲─────────┘
│ 只读 (+Redis 缓存/限流)
▼ Go: 公开 REST API + OpenAPI 文档
各种软件 / 开发者 (无任何交易端点)
```
两端不直接互调,通过共享 PostgreSQL schema + 《数据契约文档》协作。
---
## 3. 仓库结构(写代码时落地)
```
goods/ (OpenGoods 天工·商品标签)
├── README.md
├── LICENSE # 代码: Apache-2.0/MIT; 数据: ODbL 说明
├── docker-compose.yml
├── docs/{data-contract.md, openapi.yaml, disclaimer.md}
├── migrations/ # golang-migrate 共享 SQL
├── api/ # Go 只读 API (chi)
│ ├── cmd/server/main.go
│ └── internal/{handler,store,model,middleware}/
└── ingestion/ # Python 采集 + ETL
├── adapters/{openfoodfacts,usda,gs1}.py
├── etl/{normalize_units,map_category,dedup,quality}.py
└── jobs/{seed_off_dump,scheduler}.py
```
---
## 4. 最终可执行任务清单(按里程碑)
**M0 — 工程地基**~35d
- [ ] Go module + Python 项目骨架
- [ ] docker-composepostgres+redis+minio
- [ ] CIGo build/vet/testPython ruff/pytest
- [ ] `docs/data-contract.md``docs/disclaimer.md`(不提供购买声明)初版
**M1 — 数据模型 + 分类 + 单位**~46d
- [ ] migrationsproduct / food_detail / product_msrp / product_source / brand / manufacturer / category / category_schema / unit / attribute_definition / merge_log
- [ ] 导入 GS1 GPC 骨架 + 建自建中文品类树 + 映射表
- [ ] 单位字典 + 归一化规则
- [ ] 索引:gtin 唯一、name trigram、JSONB GIN、category ltree
**M2 — 种子数据 (Python)**~58d
- [ ] 下载 OFF 食品 dump → 字段映射(成分/营养/过敏原/图片)
- [ ] 单位归一 + 分类映射入库
- [ ] USDA(CC0) 营养补全(可选)
**M3 — MVP API (Go)**~58d
- [ ] 端点:barcode / id / search / nutriments / msrp / brands / categories / sources / healthz
- [ ] 统一响应信封、分页、`fields=` 裁剪、错误码
- [ ] Redis 缓存 + IP 限流 + OpenAPI 文档
**M4 — 采集管线 (Python)**~812d
- [ ] adapterOFF API 增量 + GS1 条码补全
- [ ] ETL:清洗/归一/去重合并/冲突解决/质量评分/字段级溯源
- [ ] 调度(定时增量更新)
**M5 — 开放与规模化**~1015d
- [ ] 搜索引擎(PG 全文 → OpenSearch)、CDN
- [ ] 免费 API Key(防滥用+统计)
- [ ] 众包贡献后台(提交/审核/版本/信誉)
- [ ] 开发者文档站 + 开放数据许可与免责声明上线
> 关键路径:M0→M1→M2→M3(最快拿到可查询 MVP);M4/M5 后续并行迭代。
---
## 5. 下一步
规划已全部定稿。你之前说"先不写代码",所以我**停在这里待命**。
等你说"开始",我从 **M0 工程地基** 动手,搭好骨架后开 PR 给你看(也可指定先只做某几个里程碑,例如 M0+M1)。
-373
View File
@@ -1,373 +0,0 @@
# 商品档案公益 API 系统 — 深化规划 (v2.0)
> 在 v1.0 定稿基础上,全面展开 8 个方向,并新增 **商品分类体系** 与 **单位管理体系** 两章。
> 不变原则:**只采集 + 只提供信息,绝不涉及购买/交易行为。** 全文仍为规划,未写代码。
**目录**
- A. 商品分类体系(新增)
- B. 单位管理体系(新增)
- 1. 数据库详细设计
- 2. API 详细契约
- 3. 数据治理(去重/冲突/质量评分/溯源)
- 4. 采集合规细则
- 5. 部署与运维
- 6. 众包贡献流程
- 7. 项目治理(域名/许可/免责)
- 8. 时间与里程碑估算
---
## A. 商品分类体系(Taxonomy
商品分类是整个档案库的骨架,直接影响搜索、参数模板、去重。建议**对齐国际标准 + 自建可读品类树**双轨。
### A.1 采用 GS1 GPC 作为标准骨架
GS1 **GPCGlobal Product Classification** 是四层、规则化的全球商品分类,8 位数字编码:
```
Segment(段) → Family(族) → Class(类) → Brick(砖)
47000000 47100000 47101800 10000xxx
清洁/卫生 清洁用品 ... 具体品类(GTIN挂这里)
```
- 全球 44 个 Segment,食品快消主要落在 **Food/Beverage/Tobacco****Cleaning/Hygiene** 等段。
- **Brick** 是最细粒度,商品(GTIN)挂在 brick 上;每个 brick 可带 ≤25 个属性,正好对应我们的"品类参数模板"。
- 好处:与 GS1/电商/数据池天然对齐,便于将来对接 OFF、USDA、GS1-China。
### A.2 三层映射策略
| 层 | 用途 | 来源 |
|----|------|------|
| **标准码 (gpc_brick_code)** | 机器对齐、跨源映射 | GS1 GPC |
| **自建品类树 (category)** | 人类可读、网站导航、中文友好 | 自建,映射到 GPC |
| **来源原始分类 (source_category)** | 保留溯源 | OFF categories / USDA / GS1 |
> OFF 有自己的 categories taxonomy(标签式、多语言),导入时做 `OFF category → 自建 category → GPC brick` 的映射表,未命中的进人工/众包校对队列。
### A.3 品类参数模板(Category Schema
每个叶子品类定义"应有哪些参数",用于:① 数据完整度评分 ② 录入/校验约束 ③ API 返回结构提示。
```jsonc
// category_schema 示例: 包装水
{
"category_id": "beverage/packaged_water",
"gpc_brick_code": "10000159",
"required_attributes": ["net_content", "shelf_life"],
"recommended_attributes": ["ph", "tds", "water_type"],
"nutriment_basis": "per_100ml"
}
```
### A.4 分类落地要点
- 分类树存为**邻接表 + 物化路径**(`path` 列,便于子树查询)。
- 多对一:一个商品归一个主品类(primary),可挂多个辅助标签(labels)。
- 分类可演进:用 `category_version` 管理重命名/合并,旧 ID 重定向不破坏 API。
---
## B. 单位管理体系(Units
食品参数单位混乱(g/kg/ml/L/份/%/kcal/kJ…),必须有统一的**单位字典 + 量纲 + 归一化**机制,否则无法比较和检索。
### B.1 量纲与基准单位
| 量纲 (dimension) | 基准单位 (canonical) | 常见单位 |
|------------------|----------------------|----------|
| 质量 mass | g | mg, g, kg, 斤, oz, lb |
| 体积 volume | ml | ml, L, cl, fl oz |
| 能量 energy | kJ | kJ, kcal(同时存两者) |
| 数量 count | 个 | 个/瓶/包/片/粒 |
| 比例 ratio | %(或无量纲) | %, ‰, mg/100g |
| 长度 length | mm | mm, cm, m, in |
| 时间(保质期) duration | 天 | 天/月/年 |
### B.2 单位字典 `unit`
```jsonc
{
"code": "kg",
"dimension": "mass",
"to_canonical_factor": 1000, // 1 kg = 1000 g
"canonical": "g",
"aliases": ["千克", "公斤", "kgs"],
"display": "kg"
}
```
### B.3 归一化规则
- **入库双存**:原始值/单位 `{value, unit}` + 归一化值 `{canonical_value, canonical_unit}`,原始保留供溯源与展示。
- **营养基准统一**:全部折算到 `per_100g``per_100ml`(按品类模板决定),并保留 `serving_size` 原值。
- **能量双单位**:同时存 kJ + kcal1 kcal ≈ 4.184 kJ),缺一个则自动换算并标记 `derived=true`
- **不可换算**:count(个/瓶)等不跨量纲换算;只做单位别名归一。
- **精度与舍入**:用十进制(`NUMERIC`)避免浮点误差;记录有效数字。
- **冲突处理**:单位无法识别 → 入"待清洗队列",不丢数据。
### B.4 单位与 API
- API 默认返回**原始单位 + 归一化值**两套;可加 `?unit_system=metric|original` 控制展示。
- 搜索/过滤一律基于 canonical 值(如"热量<200kcal/100g")。
---
## 1. 数据库详细设计
PostgreSQL。核心:关系表 + JSONB 灵活属性 + 结构化营养子表。以下为 DDL 草案(写代码时落到 `migrations/`)。
### 1.1 ER 概览
```
brand 1───* product *───1 category ───* category_schema
manufacturer 1───* product
product 1───1 food_detail
product 1───* product_msrp
product 1───* product_source (溯源, 字段级)
product *───* attribute (via product_attribute, 或 JSONB)
unit (字典) attribute_definition (参数字典)
contribution / merge_log / source (治理与登记)
```
### 1.2 关键建表草案(节选)
```sql
CREATE TABLE product (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
gtin VARCHAR(14) UNIQUE, -- 可空(无条码商品)
name TEXT NOT NULL,
brand_id UUID REFERENCES brand(id),
manufacturer_id UUID REFERENCES manufacturer(id),
category_id UUID REFERENCES category(id),
gpc_brick_code VARCHAR(8),
net_content_value NUMERIC,
net_content_unit VARCHAR(16),
net_content_canonical NUMERIC, -- 归一化(g/ml)
country_of_origin VARCHAR(64),
shelf_life_days INT,
storage TEXT,
attributes JSONB DEFAULT '{}', -- 灵活参数
quality_score NUMERIC(4,3) DEFAULT 0,
status VARCHAR(16) DEFAULT 'active',
created_at TIMESTAMPTZ DEFAULT now(),
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE food_detail (
product_id UUID PRIMARY KEY REFERENCES product(id) ON DELETE CASCADE,
ingredients_text TEXT,
ingredients JSONB, -- [{name,rank}]
allergens TEXT[],
additives TEXT[],
nutriments JSONB, -- 见单位章, 归一到 per_100g/ml
nutrition_basis VARCHAR(16),
serving_size VARCHAR(32),
nutri_score CHAR(1),
labels TEXT[]
);
CREATE TABLE product_msrp (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID REFERENCES product(id) ON DELETE CASCADE,
amount NUMERIC(12,2) NOT NULL,
currency CHAR(3) NOT NULL, -- ISO 4217
region VARCHAR(8) DEFAULT 'CN',
source_id UUID REFERENCES source(id),
source_url TEXT,
effective_date DATE,
note TEXT,
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE product_source ( -- 字段级溯源
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID REFERENCES product(id) ON DELETE CASCADE,
source_id UUID REFERENCES source(id),
url TEXT,
fields TEXT[], -- 该来源贡献了哪些字段
fetched_at TIMESTAMPTZ,
raw JSONB -- 原始快照
);
```
### 1.3 索引策略
- `product.gtin` 唯一索引;`product.name``pg_trgm` GIN(模糊搜索)。
- `product.attributes``food_detail.nutriments`**JSONB GIN** 索引(参数检索)。
- `category.path``ltree` 或前缀索引(子树查询)。
- 全文检索:初期 `tsvector`(name+brand+ingredients) GIN;规模化后迁 OpenSearch。
- 时间列 `updated_at` 索引(增量同步)。
---
## 2. API 详细契约
只读、版本化、统一信封。下面给核心端点的示例。
### 2.1 按条码查询(最常用)
```
GET /api/v1/products/barcode/3017624010701?fields=name,brand,nutriments,msrp
```
```jsonc
{
"data": {
"id": "…", "gtin": "3017624010701",
"name": "示例牌 巧克力榛子酱 400g", "brand": "示例牌",
"category": "食品/酱料/巧克力酱",
"net_content": {"value":400,"unit":"g","canonical":{"value":400,"unit":"g"}},
"food": {
"nutriments": {"energy_kcal":539,"energy_kj":2255,"fat_g":30.9,"sugars_g":56.3,"salt_g":0.107},
"nutrition_basis":"per_100g", "allergens":["坚果","乳","大豆"]
},
"msrp": {"amount":29.90,"currency":"CNY","region":"CN","effective_date":"2026-01-01",
"note":"官方建议零售价, 本站不提供购买"}
},
"meta": {"version":"v1"},
"sources": [{"source":"Open Food Facts","url":"…","fetched_at":"…","license":"ODbL"}]
}
```
### 2.2 搜索
```
GET /api/v1/products/search?q=巧克力&brand=示例牌&category=酱料&allergen_free=花生&page=1&size=20&fields=…
```
返回 `data:[…]` + `meta:{page,size,total,total_pages}`
### 2.3 端点清单 & 错误码
| 端点 | 说明 |
|------|------|
| `GET /products/barcode/{gtin}` | 条码查 |
| `GET /products/{id}` | ID 查 |
| `GET /products/search` | 搜索/过滤/分页 |
| `GET /products/{id}/nutriments` | 仅营养 |
| `GET /products/{id}/msrp` | 仅官方价(含免责) |
| `GET /brands` `GET /categories` | 品牌 / 品类树 |
| `GET /sources/{id}` | 数据来源透明说明 |
| `GET /healthz` `GET /openapi.json` | 健康检查 / 机读文档 |
错误码:`400`(参数错) `404`(未找到, 返回 `{error:{code:"not_found"}}`) `429`(限流, 带 `Retry-After`) `5xx`(服务端)。统一错误信封 `{error:{code,message,request_id}}`
### 2.4 跨切面
- **版本化** `/v1/`;破坏性变更升 `/v2/`,旧版保留过渡期。
- **限流**:匿名 IP 默认 60 req/min(可调);免费 API Key 提配额。响应头 `X-RateLimit-*`
- **缓存**`Cache-Control` + ETagCDN + Redis;条码查命中率高。
- **CORS**:开放 GET(公益 API)。
- **分页**`page/size`(上限 100);大结果集用 `search_after` 游标(OpenSearch 阶段)。
---
## 3. 数据治理
### 3.1 实体去重 / 匹配
1. **强匹配**:相同 `gtin` → 同一商品(条码是天然主键)。
2. **弱匹配**(无 gtin 或 gtin 缺失):`(规范化品牌 + 规范化型号/名称 + 净含量)` 相似度(trigram/编辑距离)+ 阈值;命中候选进**人工/众包确认**,不自动硬合并。
3. **合并**:保留一条 canonical,其余标 `status=merged` 并写 `merge_log`(可回滚)。
### 3.2 多源字段冲突解决
- 每个字段记录来源 + 时间 + 来源可信度权重。
- 冲突时:① 按**来源可信度**(GS1官方 > 厂商官网 > OFF众包 > 第三方)② 同级取**最新** ③ 数值类可取多数/中位数。
- 保留所有来源值于 `product_source.raw`,对外 `sources` 字段透明展示"该字段来自谁"。
### 3.3 质量评分公式(0~1
```
quality_score = 0.4*完整度 + 0.3*来源权威度 + 0.2*多源一致性 + 0.1*新鲜度
完整度 = 命中品类模板 required/recommended 字段的比例
权威度 = 贡献字段的来源权重加权
一致性 = 多源同字段一致的比例
新鲜度 = 最近更新时间衰减
```
低分商品在搜索中降权,并进入"待补全"队列(可派给众包)。
### 3.4 溯源(Provenance
字段级溯源:每条数据可回答"这个营养值/价格来自哪个来源、什么时间、什么许可"。这是公益项目可信度的核心,也用于许可合规标注。
---
## 4. 采集合规细则
### 4.1 通用护栏
- 严格遵守 `robots.txt` 与各源服务条款;礼貌限速(OFF 读 ≤15 req/min/IP);错峰;明确 `User-Agent` 标识本项目身份与联系方式。
- 只采**客观参数**;不照搬受版权保护的营销文案/评测原文(链接来源即可)。
- 增量优先:用 `last_modified`/dump 差异做增量,避免重复抓取。
### 4.2 各源接入步骤
| 源 | 步骤 | 许可 |
|----|------|------|
| **OFF dump**(首批种子) | 下载 `en.openfoodfacts.org.products.csv.gz`(~0.9G压缩) → 解析 → 映射字段 → 入库 | ODbL(衍生库需 ODbL+署名) |
| **OFF API**(增量) | 按 gtin 拉取/按更新时间增量;遵守限速 | 同上 |
| **USDA FoodData Central** | 申请免费 API key;或下载 Branded/Foundation JSON;补全营养 | CC0(最宽松) |
| **GS1 / 中国商品信息服务平台** | 条码→品牌/规格/厂商;API ≤1000 GTIN/次(需授权) | 受限,按授权使用 |
| **厂商官网** | 逐站 adapter,遵守 robots,取官方规格表/MSRP | 取客观参数 |
### 4.3 许可合规
- OFF=ODbL(传染性,衍生数据库须同样开放+署名 OFF);USDA=CC0。
- 对外数据库整体采用 **ODbL + 署名**;每条数据按 `sources[].license` 标注其来源许可,避免冲突。
---
## 5. 部署与运维
### 5.1 演进路径
- **初期**Docker Compose 一键起 `postgres + redis + minio + go-api + python-worker`,单机即可跑通 MVP。
- **成长期**:API 多副本 + 读副本数据库 + CDN;worker 横向扩展。
- **规模化**K8sAPI Deployment + HPA、worker Job/CronJob)、OpenSearch 集群、对象存储用云 S3。
### 5.2 可观测性
- 指标:Prometheus(QPS、延迟、缓存命中、限流计数、采集成功率)。
- 日志:结构化日志 + request_id 贯穿。
- 链路:OpenTelemetryAPI → DB)。
- 告警:错误率/延迟/采集失败/磁盘。
### 5.3 备份与可靠性
- Postgres 每日全量 + WAL 归档;定期恢复演练。
- 对象存储多版本/冗余。
- 采集 worker 幂等 + 重试 + 死信队列。
### 5.4 成本(量级估算,公益项目控成本)
- MVP:单台小型云主机(2C4G)+ 对象存储即可(月成本很低)。
- OFF 食品子集约数百万条,PG 单实例可承载;图片走对象存储 + CDN(按流量)。
- 详细预算待定(取决于云厂商与访问量),可后续出一版成本表。
---
## 6. 众包贡献流程
公益库靠社区补全/纠错。流程:
1. **提交**:用户对某商品提交新增/修改(带可选来源链接、照片)。
2. **校验**:单位/格式/品类模板校验 + 反垃圾(限频、信誉分、验证码)。
3. **审核**:低风险字段自动接受并标 `source=community`;高风险(价格、品牌)进人工/资深用户审核队列。
4. **版本化**:每次修改存历史版本,可 diff、可回滚(类似 wiki)。
5. **信誉系统**:贡献被采纳提升信誉;高信誉用户审核权更大。
6. **溯源透明**:众包数据与官方数据在 `sources` 中明确区分。
> 注意:众包内容也要遵守"只客观信息、不导购",并保留权利方下架通道。
---
## 7. 项目治理(域名/许可/免责)
- **品牌/域名**:建议中性、表意清晰的名字(如 *商品档案 / OpenGoods* 之类),后续选定。
- **代码许可**:开源(如 MIT/Apache-2.0),鼓励复用。
- **数据许可****ODbL + 署名**(因含 OFF);API 文档明示再利用条款。
- **隐私**:不收集个人数据(PII),只处理商品信息;众包账号信息最小化。
- **免责声明(站点显著位置)**
- "本站为公益信息平台,**仅提供商品参数信息,不提供任何购买/交易服务**。"
- "价格为官方建议零售价历史快照,实际售价以零售商为准,**不构成消费或购买建议**。"
- "数据来自多来源并标注出处,可能存在误差;欢迎纠错,权利方可申请更正/下架。"
- **下架/纠错渠道**:公开邮箱/表单,承诺响应时限。
---
## 8. 时间与里程碑估算
> 仅为相对工作量估算(以"理想工作日"计,非承诺排期);实际取决于投入人力与数据源接入难度。
| 里程碑 | 内容 | 估算 | 依赖 | 主要风险 |
|--------|------|------|------|----------|
| **M0 地基** | Go/Python 骨架、Compose、CI、数据契约 | 35 d | — | 低 |
| **M1 数据模型** | 迁移、分类树、单位字典、参数模板 | 4–6 d | M0 | 分类/单位建模需打磨 |
| **M2 种子数据** | OFF dump 导入 + 单位归一 + 分类映射 | 5–8 d | M1 | dump 体量大、字段映射脏 |
| **M3 MVP API(Go)** | 端点 + 缓存/限流 + OpenAPI | 58 d | M1,M2 | 检索性能调优 |
| **M4 采集管线(Python)** | OFF/GS1 adapter + ETL + 去重 + 质量分 + 调度 | 8–12 d | M2 | 去重/冲突算法、合规 |
| **M5 开放/规模化** | 搜索引擎、CDN、API Key、众包后台、文档站 | 1015 d | M3,M4 | 众包审核与防滥用 |
关键路径:M0→M1→M2→M3(最快拿到可查询 MVP);M4/M5 可与后续并行迭代。
---
## 9. 待确认(本版新增点)
1. **分类标准**:认同以 **GS1 GPC** 为标准骨架 + 自建中文品类树映射吗?
2. **单位策略**:营养统一折算到 `per_100g/per_100ml`、能量双存 kJ+kcal,认同吗?
3. **质量评分权重**:上面的 0.4/0.3/0.2/0.1 权重是否合适,或你有偏好?
4. **众包**:第一阶段就要做众包贡献,还是先纯采集、后期再开放众包?
5. **项目命名/域名**:有想好的名字吗?没有的话我可以提几个候选。
> 确认后我把 v2.0 收敛为可执行的工程任务清单。需要动手写代码时你说一声,我从 M0 开始搭骨架开 PR。
-408
View File
@@ -1,408 +0,0 @@
# 天工·商品标签 (OpenGoods) — 深化规划 (v3.0)
> 在最终版基础上,展开全部 10 个进阶方向。仍为规划,未写代码。
> 原则不变:**只采集 + 只提供信息,绝不涉及购买/交易。**
**目录**
1. 完整 OpenAPI 规范草案
2. 完整数据库 DDL
3. 数据契约文档
4. OFF 字段映射表
5. 中国合规专项
6. 测试与数据质量保障
7. 安全与反滥用
8. 商品图片处理
9. 可用性与 SLA
10. 竞品 / 同类项目分析
---
## 1. 完整 OpenAPI 规范草案(节选骨架,写代码时落到 `docs/openapi.yaml`
```yaml
openapi: 3.1.0
info:
title: OpenGoods API (天工·商品标签)
version: "1.0.0"
description: >
公益商品参数查询 API。只提供信息,不提供购买/交易。
数据采用 ODbL 许可并署名来源。
license: {name: ODbL-1.0, url: https://opendatacommons.org/licenses/odbl/}
servers:
- {url: https://api.opengoods.org/api/v1}
paths:
/products/barcode/{gtin}:
get:
summary: 按条码查询商品档案
parameters:
- {name: gtin, in: path, required: true, schema: {type: string, pattern: '^[0-9]{8,14}$'}}
- {name: fields, in: query, schema: {type: string}, description: 逗号分隔字段裁剪}
responses:
'200': {description: OK, content: {application/json: {schema: {$ref: '#/components/schemas/ProductEnvelope'}}}}
'404': {description: 未找到, content: {application/json: {schema: {$ref: '#/components/schemas/Error'}}}}
'429': {description: 限流, headers: {Retry-After: {schema: {type: integer}}}}
/products/{id}:
get: { summary: 按ID查询, parameters: [{name: id, in: path, required: true, schema: {type: string, format: uuid}}], responses: {'200': {description: OK}} }
/products/search:
get:
summary: 搜索/过滤/分页
parameters:
- {name: q, in: query, schema: {type: string}}
- {name: brand, in: query, schema: {type: string}}
- {name: category, in: query, schema: {type: string}}
- {name: allergen_free, in: query, schema: {type: string}}
- {name: page, in: query, schema: {type: integer, default: 1}}
- {name: size, in: query, schema: {type: integer, default: 20, maximum: 100}}
responses: {'200': {description: OK, content: {application/json: {schema: {$ref: '#/components/schemas/SearchEnvelope'}}}}}
/products/{id}/nutriments: {get: {summary: 仅营养}}
/products/{id}/msrp: {get: {summary: 仅官方零售价(含免责)}}
/brands: {get: {summary: 品牌列表}}
/categories: {get: {summary: 品类树}}
/sources/{id}:{get: {summary: 数据来源透明说明}}
/healthz: {get: {summary: 健康检查}}
components:
schemas:
ProductEnvelope:
type: object
properties:
data: {$ref: '#/components/schemas/Product'}
meta: {type: object}
sources: {type: array, items: {$ref: '#/components/schemas/SourceRef'}}
Product:
type: object
properties:
id: {type: string, format: uuid}
gtin: {type: string}
name: {type: string}
brand: {type: string}
category: {type: string}
net_content: {$ref: '#/components/schemas/Quantity'}
food: {$ref: '#/components/schemas/FoodDetail'}
msrp: {$ref: '#/components/schemas/Msrp'}
quality_score: {type: number}
Quantity:
type: object
properties: {value: {type: number}, unit: {type: string}, canonical: {type: object}}
FoodDetail:
type: object
properties:
ingredients_text: {type: string}
allergens: {type: array, items: {type: string}}
additives: {type: array, items: {type: string}}
nutriments: {type: object}
nutrition_basis: {type: string, enum: [per_100g, per_100ml, per_serving]}
nutri_score: {type: string}
Msrp:
type: object
properties:
amount: {type: number}
currency: {type: string}
region: {type: string}
effective_date: {type: string, format: date}
note: {type: string, default: "官方建议零售价, 本站不提供购买"}
SourceRef:
type: object
properties: {source: {type: string}, url: {type: string}, fetched_at: {type: string}, license: {type: string}}
Error:
type: object
properties: {error: {type: object, properties: {code: {type: string}, message: {type: string}, request_id: {type: string}}}}
```
> 该 `openapi.yaml` 既是契约也是文档源:Go 端用它做路由校验/生成 Swagger UI,客户端可由它生成 SDK。
---
## 2. 完整数据库 DDL(全部表)
```sql
-- 扩展
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS ltree;
-- gen_random_uuid() 由 pgcrypto 提供
CREATE TABLE source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL, -- Open Food Facts / USDA / GS1-China ...
homepage TEXT,
license TEXT, -- ODbL / CC0 / proprietary
trust_weight NUMERIC(3,2) DEFAULT 0.5, -- 来源可信度(冲突解决用)
notes TEXT
);
CREATE TABLE brand (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
normalized_name TEXT, -- 规范化(去空格/大小写/全半角)用于匹配
aliases TEXT[],
UNIQUE(normalized_name)
);
CREATE TABLE manufacturer (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
normalized_name TEXT,
country VARCHAR(64),
UNIQUE(normalized_name)
);
CREATE TABLE category (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name_zh TEXT NOT NULL,
name_en TEXT,
parent_id UUID REFERENCES category(id),
path LTREE, -- 物化路径, 子树查询
gpc_brick_code VARCHAR(8), -- 映射到 GS1 GPC
level INT,
UNIQUE(path)
);
CREATE TABLE category_schema ( -- 品类参数模板
category_id UUID PRIMARY KEY REFERENCES category(id),
required_attributes TEXT[],
recommended_attributes TEXT[],
nutriment_basis VARCHAR(16)
);
CREATE TABLE unit ( -- 单位字典
code VARCHAR(16) PRIMARY KEY,
dimension VARCHAR(16) NOT NULL, -- mass/volume/energy/count/ratio/length/duration
canonical VARCHAR(16) NOT NULL,
to_canonical_factor NUMERIC, -- code -> canonical 的换算因子
aliases TEXT[],
display TEXT
);
CREATE TABLE attribute_definition ( -- 参数字典(标准名/别名/单位)
key VARCHAR(64) PRIMARY KEY,
label_zh TEXT, label_en TEXT,
dimension VARCHAR(16),
default_unit VARCHAR(16) REFERENCES unit(code),
aliases TEXT[]
);
CREATE TABLE product (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
gtin VARCHAR(14) UNIQUE,
name TEXT NOT NULL,
brand_id UUID REFERENCES brand(id),
manufacturer_id UUID REFERENCES manufacturer(id),
category_id UUID REFERENCES category(id),
gpc_brick_code VARCHAR(8),
net_content_value NUMERIC,
net_content_unit VARCHAR(16),
net_content_canonical NUMERIC,
country_of_origin VARCHAR(64),
shelf_life_days INT,
storage TEXT,
attributes JSONB DEFAULT '{}',
quality_score NUMERIC(4,3) DEFAULT 0,
status VARCHAR(16) DEFAULT 'active', -- active/merged/deprecated
canonical_id UUID REFERENCES product(id), -- 被合并到哪个
search_tsv TSVECTOR,
created_at TIMESTAMPTZ DEFAULT now(),
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE food_detail (
product_id UUID PRIMARY KEY REFERENCES product(id) ON DELETE CASCADE,
ingredients_text TEXT,
ingredients JSONB,
allergens TEXT[],
additives TEXT[],
nutriments JSONB,
nutrition_basis VARCHAR(16),
serving_size VARCHAR(32),
nutri_score CHAR(1),
labels TEXT[]
);
CREATE TABLE product_msrp (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID REFERENCES product(id) ON DELETE CASCADE,
amount NUMERIC(12,2) NOT NULL,
currency CHAR(3) NOT NULL,
region VARCHAR(8) DEFAULT 'CN',
source_id UUID REFERENCES source(id),
source_url TEXT,
effective_date DATE,
note TEXT,
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE product_image (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID REFERENCES product(id) ON DELETE CASCADE,
url TEXT, -- 对象存储 URL
kind VARCHAR(16), -- front/ingredients/nutrition
license TEXT,
source_id UUID REFERENCES source(id)
);
CREATE TABLE product_source ( -- 字段级溯源
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID REFERENCES product(id) ON DELETE CASCADE,
source_id UUID REFERENCES source(id),
url TEXT,
fields TEXT[],
fetched_at TIMESTAMPTZ,
raw JSONB
);
CREATE TABLE merge_log ( -- 合并/回滚
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kept_id UUID, merged_id UUID,
reason TEXT, by TEXT, created_at TIMESTAMPTZ DEFAULT now()
);
-- 索引
CREATE UNIQUE INDEX idx_product_gtin ON product(gtin) WHERE gtin IS NOT NULL;
CREATE INDEX idx_product_name_trgm ON product USING gin (name gin_trgm_ops);
CREATE INDEX idx_product_attrs ON product USING gin (attributes);
CREATE INDEX idx_food_nutriments ON food_detail USING gin (nutriments);
CREATE INDEX idx_product_tsv ON product USING gin (search_tsv);
CREATE INDEX idx_category_path ON category USING gist (path);
CREATE INDEX idx_product_updated ON product(updated_at);
```
---
## 3. 数据契约文档(`docs/data-contract.md` 概要)
两端共享的"事实约定",避免 Go/Python 理解不一致:
- **枚举固定**`status`(active/merged/deprecated)、`nutrition_basis`(per_100g/per_100ml/per_serving)、`dimension``source.license``image.kind`
- **字段含义与可空性**:逐字段写明(如 `gtin` 可空、唯一;`quality_score` ∈ [0,1])。
- **单位规则**:原始 + canonical 双存;能量双存 kJ/kcal;换算因子来自 `unit` 表。
- **写入责任**:仅 Python(ingestion) 写库;Go 只读。所有写入走 ETL,保证归一化与溯源。
- **版本**:契约本身版本化;schema 变更需同步更新契约 + 迁移 + OpenAPI。
- **示例**:附 1 条完整 product JSON 作为"黄金样例",两端测试都对它断言。
---
## 4. OFFOpen Food Facts)字段映射表(导入直接用)
| OFF 字段 (CSV) | OpenGoods 字段 | 处理 |
|----------------|----------------|------|
| `code` | `product.gtin` | 校验 8/12/13/14 位 + 校验位 |
| `product_name` / `product_name_zh` | `product.name` | 优先中文, 回退英文 |
| `brands` | `brand.name` | 拆分取首个, 规范化, upsert brand |
| `categories` / `categories_tags` | `source_category``category` | 走 OFF→自建→GPC 映射表 |
| `quantity` | `net_content_*` | 解析数值+单位 → 归一化 |
| `countries` | `country_of_origin` | 取销售国/产地 |
| `ingredients_text` | `food_detail.ingredients_text` | 原文保留 |
| `allergens_tags` | `food_detail.allergens` | 标签清洗为中文 |
| `additives_tags` | `food_detail.additives` | E-number 解析 |
| `energy-kj_100g` / `energy-kcal_100g` | `nutriments.energy_kj/kcal` | 缺一个则换算, 标 derived |
| `fat_100g` `saturated-fat_100g` `carbohydrates_100g` `sugars_100g` `proteins_100g` `salt_100g` | `nutriments.*` | 归一 per_100g |
| `nutriscore_grade` | `food_detail.nutri_score` | AE |
| `serving_size` | `food_detail.serving_size` | 原值 |
| `image_url` / `image_front_url` 等 | `product_image.url` | 下载转存对象存储, 记 CC-BY-SA |
| `last_modified_t` | `product_source.fetched_at` | 增量基准 |
| (整行) | `product_source.raw` | 存原始快照 |
> 许可:OFF 数据=ODbL(衍生库需同样开放+署名);图片=CC-BY-SA。映射时全程记 `source_id=OFF`。
---
## 5. 中国合规专项(公益网站落地关键)
> 以下为工程与运营层面的合规要点梳理,**非法律意见**;正式上线前建议咨询专业法务。
### 5.1 网站备案
- 服务器在中国大陆 → 需 **ICP 备案**(公益网站可走非经营性 ICP 备案);部分地区/类目可能涉 **公安联网备案**
- 若用境外/港澳服务器可免 ICP,但访问速度与合规另作权衡。
### 5.2 数据合规(网络安全法 / 数据安全法 / 个人信息保护法)
- 本项目**只处理商品信息、不收集个人信息(PII)**,PIPL 风险低;众包阶段涉及用户账号时再做最小化收集 + 隐私政策。
- 《数据安全法》要求数据收集合法正当;做好数据分级与安全保护义务。
### 5.3 网络爬虫法律边界(重点)
依据中央网信办公开文章与司法实践,判断标准是**客观结果**——是否妨碍目标网站正常运行 / 危害合法权益:
- **守 robots.txt**、礼貌限速、错峰,**不得对目标站造成 DDoS 式压力**(否则可能触及破坏计算机信息系统罪等)。
- **不抓取非公开/需登录/绕过反爬**的数据(可能涉非法获取计算机信息系统数据罪)。
- 只采**客观公开的商品参数**;不抓取受版权保护内容、不抓个人信息。
- 优先用**官方开放数据/API/数据 dump**OFF dump、USDA、GS1 授权)——从源头规避爬虫风险。
### 5.4 食品信息合规
- 展示食品参数时注明"信息仅供参考,以实物标签为准";营养/成分以官方/厂商标签为准。
- 不做医疗/功效宣称;不构成消费建议。
### 5.5 价格与"不导购"
- 价格仅为**官方建议零售价历史快照**,显著标注;**全站无购买/下单/跳转购买链接**,避免被认定为经营性电商导购。
---
## 6. 测试与数据质量保障
### 6.1 代码测试
- **Go**handler 单元测试 + store 层用 `testcontainers`/临时 PG 集成测试 + API 契约测试(对 openapi.yaml 校验响应)。
- **Python**:ETL 纯函数单测(单位归一、分类映射、去重打分)+ adapter 用录制的样例数据测试(不打真实站点)。
- **CI**PR 必跑 lint + test;覆盖率门槛(如 ETL 核心 ≥80%)。
### 6.2 数据质量
- **入库校验**:gtin 校验位、单位可识别、营养数值合理区间、必填字段(按品类模板)。
- **质量评分**:见 v2.0 公式,低分进"待补全"队列。
- **数据回归**:黄金样例集 + 定期跑"数据健康检查"(孤儿记录、单位异常、重复 gtin、营养越界)。
- **可观测**:导入报表(新增/更新/拒绝条数、拒绝原因 top)。
---
## 7. 安全与反滥用
- **API 防刷**IP 限流 + 可选 API Key 分级配额;异常流量识别(突发高频降级/挑战)。
- **缓存挡压**:热点条码走 CDN/Redis,降低数据库压力,也抗刷。
- **输入校验**:所有参数严格校验(gtin 正则、size 上限),防注入(参数化查询,禁拼 SQL)。
- **密钥管理**:DB/对象存储/第三方 key 走环境变量/密钥管理,不入库不入仓。
- **最小权限**:Go 端用**只读** DB 账号;写权限仅 ingestion。
- **采集端被封应对**:合规限速 + 失败退避 + 死信队列 + 切换为官方 dump/API。
- **DDoS**CDN + 速率限制 + 云厂商防护;公益服务以可降级(只读缓存)保命。
- **依赖安全**Go `govulncheck`、Python `pip-audit`CI 中扫描。
---
## 8. 商品图片处理
- **版权**OFF 图片为 CC-BY-SA,须署名 + 同样开放;逐图记 `license` 与来源。
- **存储**:对象存储(MinIO/S3),路径按 `gtin/kind`;原图 + 生成多档缩略图(thumb/medium)。
- **处理管线**:下载 → 校验(类型/大小) → 去重(感知哈希避免重复) → 压缩 → 生成缩略图 → 记录。
- **分发**CDN 加速;API 只返回图片 URL,不内嵌二进制。
- **合规**:不展示含个人信息的图;提供权利方下架通道。
- **降级**:图片缺失返回占位;图片服务故障不影响参数 API。
---
## 9. 可用性与 SLA
| 项 | 目标(建议) |
|----|-----------|
| API 可用性 | 99.5%(公益项目务实目标,先保只读可用) |
| 读延迟 | p95 < 200ms(缓存命中 < 50ms |
| 数据新鲜度 | 增量同步 T+1(每日) |
| 降级策略 | DB 故障 → 只读缓存兜底;图片/搜索故障不影响核心参数查询 |
| 灾备 | 每日备份 + 异地副本;恢复演练季度一次 |
| 维护窗口 | 采集/重建索引放低峰;API 滚动发布不停服 |
> 公益项目优先"省成本 + 稳定只读";写入(采集)可异步、可补偿,读路径要稳。
---
## 10. 竞品 / 同类项目分析
| 项目 | 性质 | 数据 | 借鉴点 | 与我们差异 |
|------|------|------|--------|------------|
| **Open Food Facts** | 公益食品库 | ODbL, 海量, 可贡献, 有 dump/API | 字段模型、众包、Nutri-Score、API 设计 | 我们多语言架构(Go API)、聚焦中文/GPC、收 MSRP |
| **USDA FoodData Central** | 政府营养库 | CC0, 权威营养 | 营养数据补全、公共领域许可 | 偏美国/营养, 无条码生态 |
| **GS1 / Verified by GS1** | 官方条码登记 | 受限, 权威, 2亿+ | 条码→品牌/规格权威源 | 非开放、需授权 |
| **Wikidata** | 通用知识库 | CC0, 有 GTIN 属性(P3962) | 实体链接、结构化、开放 | 非商品专用、参数不规整 |
| **schema.org Product/gtin** | 数据标准 | 标准而非数据 | 用其词汇做对外结构化(SEO/互操作) | 仅规范, 需我们填数据 |
| **brocade.io / 各条码库** | 开放/商业条码库 | 参差 | 条码补全兜底 | 数据量/质量有限或收费 |
**结论与定位**
- 我们不是再造 OFF,而是做**面向中文世界、与 GS1 GPC 对齐、聚焦"商品参数标签"**的公益 API
- **站在巨人肩上**OFF/USDA 做种子与营养,GS1 做条码权威,Wikidata/schema.org 做实体与互操作标准;
- 差异化:中文优先、品类参数模板规整、官方 MSRP、字段级溯源、Go 高并发只读 API。
---
## 11. 小结
v3.0 已把工程落地与公益合规的关键面全部展开。规划层面已相当完整。
你之前说先不写代码,我**继续待命**:可以再深化任何一块,或等你说"开始",从 M0 搭骨架开 PR。
-40
View File
@@ -1,40 +0,0 @@
# 天工·商品标签 (OpenGoods) — 规划文档归档
本目录归档了项目从立项到方案定稿的全部规划文档。
## 项目一句话
公益网站/服务:**采集全网商品信息,对外提供商品参数查询 API**。
核心原则:**只采集 + 只提供信息,绝不涉及任何购买/下单/比价导购。**
## 当前文档(最新,建议优先阅读)
| 文档 | 内容 |
|------|------|
| [00-final-plan.md](./00-final-plan.md) | **最终规划**:锁定的全部决策 + 架构 + 仓库结构 + M0~M5 可执行任务清单 |
| [01-detailed-design-v2.0.md](./01-detailed-design-v2.0.md) | **详细设计**:商品分类体系(GS1 GPC)、单位管理、数据库设计、API 契约、数据治理、采集合规、部署运维、众包、里程碑估算 |
| [02-advanced-topics-v3.0.md](./02-advanced-topics-v3.0.md) | **进阶专题**:完整 OpenAPI、全表 DDL、数据契约、OFF 字段映射、中国合规专项、测试、安全反滥用、图片处理、SLA、竞品分析 |
## 演进历史(History
| 文档 | 阶段 |
|------|------|
| [history/v0.1-initial-plan.md](./history/v0.1-initial-plan.md) | 初版总体规划 |
| [history/v0.2-go-python-foodfmcg.md](./history/v0.2-go-python-foodfmcg.md) | 确定 Go+Python 架构、聚焦食品快消、数据源调研 |
| [history/v1.0-locked-decisions.md](./history/v1.0-locked-decisions.md) | 决策定稿(默认值) |
## 已锁定的关键决策(速览)
| 维度 | 决策 |
|------|------|
| 首批品类 | 食品快消 |
| 价格 | 只收官方标准零售价 (MSRP),静态字段,无购买入口 |
| 技术栈 | Go(对外只读 API + Python(采集/ETL),经 PostgreSQL + Redis/队列解耦 |
| 种子数据 | Open Food Facts 食品 dump |
| 数据许可 | 对外 ODbL + 署名;CC0 来源(USDA)自由混入 |
| 商品分类 | GS1 GPC 四层标准码 + 自建中文品类树映射 |
| 单位管理 | 原始值+归一化双存;营养统一 per_100g/ml;能量双存 kJ+kcal |
| 质量评分 | 0.4 完整度 + 0.3 来源权威 + 0.2 多源一致 + 0.1 新鲜度 |
| 众包 | 一期不做,先纯采集;二期开放 |
| Go 框架 / 迁移 / 部署 | chi + 标准库 / golang-migrate / Docker Compose |
> 注:文档中"中国合规专项"为工程与运营层面梳理,**非法律意见**;正式上线前请咨询专业法务。
-216
View File
@@ -1,216 +0,0 @@
# 商品档案公益 API 系统 — 规划方案 (v0.1)
> 一个公益性质的网站/服务:**采集全网商品信息**,对外提供**商品参数查询 API**。
> 核心原则:**只收集信息、只提供信息,不涉及任何购买、下单、比价导购等交易行为。**
---
## 1. 项目定位与原则
| 维度 | 说明 |
|------|------|
| 定位 | 公益的"商品参数百科 / 商品档案库",类似商品界的 Wikipedia + 开放 API |
| 提供什么 | 商品的客观参数(规格、型号、成分、能效、尺寸、条码等) |
| **不提供什么** | 价格交易、加购物车、下单、跳转购买链接、联盟分佣、比价导购 |
| 服务对象 | 第三方软件 / 开发者,通过 API 查询商品参数 |
| 数据态度 | 客观、中立、可溯源(每条数据标注来源与采集时间) |
> ⚠️ 关于价格:建议**默认不收录价格**。价格属于交易属性,且实时性强、争议大。如果一定要做,只做"历史参考价"且明确标注来源时间,绝不提供购买入口。**这一点需要你确认。**
---
## 2. 系统总体架构
```
┌─────────────────────────────┐
│ 数据来源 (Sources) │
│ 官网/厂商 / 开放数据 / GS1 │
│ 条码库 / 用户贡献 / 监管公开 │
└──────────────┬──────────────┘
┌──────────────────────────▼──────────────────────────┐
│ 采集层 Ingestion (Workers) │
│ 爬虫调度 + 适配器 + 限速 + robots 合规 + 去重 │
└──────────────────────────┬──────────────────────────┘
│ 原始数据 (raw)
┌──────────────────────────▼──────────────────────────┐
│ 清洗/标准化 ETL (Normalize & Dedup) │
│ 字段映射 / 单位归一 / 实体匹配 / 质量评分 │
└──────────────────────────┬──────────────────────────┘
│ 结构化商品档案
┌──────────────────────────▼──────────────────────────┐
│ 存储层 Storage │
│ PostgreSQL(主) + 对象存储(图片) + 搜索引擎(检索) │
└──────────────────────────┬──────────────────────────┘
┌──────────────────────────▼──────────────────────────┐
│ 公开 API 服务 (FastAPI) │
│ REST/JSON + 文档 + 限流 + 缓存 + API Key(可选) │
└──────────────────────────┬──────────────────────────┘
┌──────────────▼──────────────┐
│ 消费者:各种软件/开发者 │
└─────────────────────────────┘
```
分为四个相对独立的子系统:
1. **采集子系统**(爬虫/适配器,离线运行)
2. **数据处理子系统**(清洗、标准化、去重、质量评分)
3. **存储子系统**(关系库 + 搜索 + 对象存储)
4. **API 子系统**(对外只读公开 API + 文档站)
---
## 3. 核心数据模型(商品档案 Schema)
商品的本质是"一个实体 + 一组可扩展的参数"。建议采用 **核心字段 + 灵活属性(KV)** 的混合模型,以适配不同品类(手机、食品、家电、化妆品……参数差异极大)。
### 3.1 核心实体
```jsonc
// Product 商品档案
{
"id": "uuid", // 内部唯一ID
"gtin": "6901234567892", // 全球贸易项目代码(条码), 可空
"name": "示例牌 1.5L 纯净水",
"brand": "示例牌", // -> Brand 实体
"manufacturer": "示例食品有限公司",
"category": "饮料/包装水", // -> Category 树
"model": "型号/SKU标识",
"description": "客观描述, 非营销文案",
"images": ["对象存储URL", ...],
"attributes": [ // 灵活参数(见下)
{"key": "容量", "value": "1.5", "unit": "L"},
{"key": "保质期", "value": "12", "unit": "月"}
],
"identifiers": { // 其他标识
"ean": "...", "upc": "...", "asin": "...", "mpn": "..."
},
"sources": [ // 数据溯源(每个字段可标来源)
{"source_id": "...", "url": "...", "fetched_at": "2026-06-08T...", "field": "容量"}
],
"quality_score": 0.87, // 数据质量/可信度评分
"status": "active|merged|deprecated",
"created_at": "...", "updated_at": "..."
}
```
### 3.2 灵活属性 (EAV / JSONB)
- 不同品类参数差异巨大,核心表存通用字段,品类专属参数存 `attributes`PostgreSQL `JSONB`,可建 GIN 索引)。
- 配合**品类参数模板**Category Schema)约束某品类应有哪些参数,保证质量。
### 3.3 辅助实体
- `Brand`(品牌)、`Manufacturer`(厂商)、`Category`(品类树)、`Source`(数据来源登记)、`AttributeDefinition`(参数字典:标准名/别名/单位)。
- 实体去重/合并需要 `merge` 机制(同一商品多来源 → 合并为一条,保留溯源)。
---
## 4. 数据采集策略(最关键、也最需合规)
### 4.1 来源优先级(从"最合规"到"需谨慎"
1. **官方开放数据 / 标准库**:GS1 条码库、各国监管公开数据(能效标识、食品备案、药品/化妆品备案等)。✅ 最佳
2. **厂商官网 / 官方规格表**:参数最权威。需遵守 robots.txt。
3. **厂商/平台开放 API**:若有官方 API 走 API。
4. **用户/社区贡献**:众包补全与纠错(带审核)。
5. **第三方网页抓取**:⚠️ 合规风险最高,需严格遵守 robots、限速、只取客观参数、标注来源。
### 4.2 采集器设计
- **适配器模式**:每个来源一个 adapter(解析规则独立、可热插拔)。
- **调度**:任务队列(Celery / RQ / arq+ 定时(cron+ 增量更新。
- **合规护栏**:尊重 `robots.txt`、礼貌限速、`User-Agent` 标识身份、错峰、缓存避免重复抓取。
- **去重与匹配**:以 GTIN/条码为主键,无条码时用 (品牌+型号+关键参数) 做模糊匹配。
### 4.3 数据质量
- 每个字段记录来源 + 时间;多来源冲突时按来源可信度加权。
- 质量评分 `quality_score`:字段完整度 + 来源权威度 + 一致性。
---
## 5. 公开 API 设计(只读、RESTful
基础原则:**只读、无副作用、无购买入口、稳定版本化、有文档**。
```
GET /api/v1/products/{id} # 按内部ID查询商品档案
GET /api/v1/products/barcode/{gtin} # 按条码(GTIN/EAN/UPC)查询 ★最常用
GET /api/v1/products/search # 搜索: ?q=&brand=&category=&page=&size=
GET /api/v1/products/{id}/attributes # 仅取参数
GET /api/v1/brands / categories # 品牌/品类树
GET /api/v1/sources/{id} # 数据来源说明(透明溯源)
GET /healthz / /api/v1/openapi.json # 健康检查 / 机读文档
```
设计要点:
- **版本化** `/api/v1/`,破坏性变更升 `v2`
- **分页 + 字段筛选**`fields=` 减少传输)。
- **限流**:匿名按 IP 限流;可选 API Key 提升配额(免费,仅用于防滥用与统计)。
- **缓存**:CDN + 服务端缓存(商品参数变化慢,缓存命中率高)。
- **响应统一**JSON,含 `data` / `meta`(分页) / `sources`(溯源)。
- **开放协议**:数据采用开放许可(如 CC BY / ODbL),鼓励署名引用。
- **自动文档**FastAPI 自带 Swagger UI / ReDoc。
---
## 6. 技术选型建议
| 层 | 选型 | 理由 |
|----|------|------|
| API 框架 | **Python + FastAPI** | 与仓库定位一致、异步性能好、自带 OpenAPI 文档 |
| 主数据库 | **PostgreSQL** (JSONB) | 关系 + 灵活属性兼得,GIN 索引支持检索 |
| 搜索 | **OpenSearch / Elasticsearch / 或 PG 全文** | 商品名/参数全文与分面检索 |
| 缓存 | **Redis** | 热点缓存 + 限流计数 + 任务队列后端 |
| 采集任务 | **arq / Celery / RQ** | 异步调度爬虫与 ETL |
| 爬虫 | **httpx + selectolax/BeautifulSoup**,动态页用 **Playwright** | 轻量为主,必要时浏览器渲染 |
| 对象存储 | **S3 兼容 (MinIO / 云)** | 存商品图片 |
| 部署 | **Docker + Compose**(初期) → K8s(规模化) | 渐进式 |
| 文档站 | FastAPI 文档 + 静态站(MkDocs) | 开发者文档 |
> 如果你更偏好 Node.js / Go 也可以,我按你的偏好调整。仓库描述像是 FastAPI,所以我默认 Python。
---
## 7. 合规与法律(公益项目尤其重要)
- **爬取合规**:遵守 robots.txt、服务条款、合理限速;只采集**客观商品参数**,不抓取受版权保护的营销文案/评测原文(可链接来源)。
- **数据来源透明**:每条数据可溯源,标注来源与时间,尊重原始来源。
- **隐私**:只处理商品信息,不涉及个人数据(无 PII)。
- **商标/品牌**:品牌名仅用于客观标识商品,不做背书或贬损。
- **明确边界**:网站显著声明"仅提供信息、不提供购买、不构成消费建议"。
- **数据开放许可**:选择 CC BY 4.0 或 ODbL,明确他人使用条款。
- **下架机制**:提供来源方/权利方的纠错与下架联系渠道。
> 建议这块后续找法务/合规确认,我可以先把"合规护栏"写进采集器与站点声明。
---
## 8. 建议的实施路线图(分阶段,先跑起来再扩展)
**阶段 0 — 地基(先做)**
- 仓库工程化:FastAPI 项目骨架、PostgreSQL、Docker Compose、CI、代码规范。
- 定义核心数据模型 + 迁移。
**阶段 1 — MVP API**
- 实现只读 API:按条码/ID 查询 + 搜索 + 自动文档。
- 导入一批种子数据(用开放数据/手工录入几百条),让 API 真正可用。
**阶段 2 — 采集管线**
- 1~2 个合规来源的 adapter + 调度 + 清洗/去重/入库。
- 质量评分与溯源。
**阶段 3 — 规模化与开放**
- 搜索引擎、缓存/CDN、限流、API Key、众包贡献与审核后台。
- 开放数据许可与开发者文档站上线。
---
## 9. 需要你拍板的关键问题
1. **是否收录价格?**(建议默认不收录,只做客观参数)
2. **首批聚焦哪些品类?**(如:食品/快消、3C 数码、家电…… 建议先聚焦 1~2 个品类把模型打磨好)
3. **技术栈确认 Python + FastAPI 吗?**(还是有其他偏好)
4. **数据来源**:你手上有没有已知的合规数据源 / 官方 API / 已有数据集?
5. **首要交付**:先搭"可查询的 MVP API + 一批种子数据",还是先做"采集管线"?(建议先 MVP API
---
> 这是 v0.1 规划草案,确认大方向后我会细化成可执行的工程任务,并开始搭建仓库骨架 + MVP。
@@ -1,203 +0,0 @@
# 商品档案公益 API 系统 — 规划方案 (v0.2)
> 公益网站/服务:**采集全网商品信息**,对外提供**商品参数查询 API**。
> 原则:**只收集 + 只提供信息,不涉及任何购买/下单/比价导购**。
> 本版根据你的反馈定稿四件事:① 收录**官方标准零售价(MSRP)** ② 首批聚焦**食品快消** ③ **Go(系统) + Python(采集)** 多语言架构 ④ 附**开放数据源清单**。
---
## 0. 你已确认的决策
| # | 决策 | 说明 |
|---|------|------|
| 1 | **价格 = 官方标准零售价 (MSRP)** | 厂商指导价/官方建议零售价,属**静态属性**,带来源+时间+币种标注;**不收录实时电商售价、不提供购买入口** |
| 2 | **首批品类 = 食品快消 (Food & FMCG)** | 先把食品的数据模型打磨好(成分、营养、过敏原、规格、保质期…) |
| 3 | **技术栈 = Go + Python** | Go 写对外 API/核心服务;Python 写采集/ETL/爬虫;通过 PostgreSQL + 消息队列解耦 |
| 4 | **数据源 = 暂无,后期提供** | 本版先给出可立即接入的开放数据源清单 |
---
## 1. Go + Python 多语言架构(核心)
这是一个很经典且合理的组合。两端**不直接互相调用**,而是通过**共享数据库 + 消息队列**解耦,各自独立部署、独立扩展。
```
┌────────────────────── Python 侧 (采集/数据) ──────────────────────┐
│ │
数据源 ─▶│ 采集 Workers (爬虫/适配器) ─▶ ETL 清洗/标准化/去重 ─▶ 入库 │
│ httpx / Playwright / scrapy pandas / 规则引擎 │
└───────────────────────────┬────────────────────────────────────────┘
│ 写入
┌───────▼────────┐ ┌──────────────┐
│ PostgreSQL │◀──────▶│ 对象存储 S3 │ (商品图)
│ (商品档案主库) │ └──────────────┘
└───────▲────────┘
│ 只读
┌───────────────────────────┴────────────────────────────────────────┐
│ Go 侧 (对外服务) │
│ 公开 API (REST/JSON) + Redis 缓存/限流 + 搜索网关 + OpenAPI │
│ Gin/Echo/Chi/标准库 │
└───────────────────────────┬────────────────────────────────────────┘
各种软件 / 开发者消费
```
### 1.1 职责划分
| 子系统 | 语言 | 职责 |
|--------|------|------|
| **公开 API 服务** | **Go** | 对外只读 API、限流、缓存、鉴权(可选 API Key)、检索网关、高并发承载 |
| **采集 Workers** | **Python** | 每个数据源一个 adapter,抓取/调用 API、遵守 robots、限速、产出原始数据 |
| **ETL / 数据处理** | **Python** | 清洗、字段映射、单位归一、实体去重与合并、质量评分 |
| **调度 / 队列** | Python(worker) + Redis/消息队列 | 定时任务、增量更新、任务分发 |
| **存储** | PostgreSQL + Redis + S3 | 主库 / 缓存+限流 / 图片 |
| **检索** | 初期 PG 全文 → 后期 OpenSearch | 商品名/参数搜索与分面 |
### 1.2 为什么这样分?
- **Go 做 API**:编译型、单二进制部署、并发模型适合高 QPS 的只读公益 API,运维简单。
- **Python 做采集**:爬虫/解析/数据处理生态最强(scrapy、playwright、pandas),迭代快。
- **解耦点 = 数据库**:Go 端**只读**主库(或读副本),Python 端负责写入。两端通过稳定的表结构约定协作,互不阻塞;将来任一端换语言/重写都不影响另一端。
- **契约**:用数据库 schema + 一份内部「数据契约文档」固定字段含义,避免两端理解不一致。
---
## 2. 食品快消数据模型(细化)
食品参数差异大,沿用 **核心字段 + JSONB 灵活属性 + 营养结构化子表**。字段设计大量参考 Open Food Facts(成熟的食品开放库)。
### 2.1 商品主表 `product`
```jsonc
{
"id": "uuid",
"gtin": "6901234567892", // 条码(主键标识), EAN-13/UPC/EAN-8
"name": "示例牌 巧克力榛子酱 400g",
"brand": "示例牌", // -> brand
"manufacturer": "示例食品有限公司", // 生产商
"category": "食品/酱料/巧克力酱", // -> category 树 (可对齐 GS1 GPC / OFF categories)
"net_content": {"value": 400, "unit": "g"}, // 净含量
"country_of_origin": "中国",
"shelf_life": {"value": 12, "unit": "月"}, // 保质期
"storage": "常温避光保存",
"images": ["S3_URL", ...],
"msrp": { ... }, // 官方标准零售价, 见 2.3
"food": { ... }, // 食品专属结构化字段, 见 2.2
"attributes": [ {"key":"","value":"","unit":""} ], // 其余灵活参数(JSONB)
"identifiers": {"ean":"", "upc":"", "off_id":""},
"sources": [ {"source":"", "url":"", "fetched_at":"", "fields":["msrp"]} ],
"quality_score": 0.0,
"status": "active|merged|deprecated",
"created_at": "", "updated_at": ""
}
```
### 2.2 食品专属字段 `food`(结构化)
```jsonc
{
"ingredients_text": "白砂糖, 棕榈油, 榛子(13%), ...", // 配料表原文
"ingredients": [ {"name":"白砂糖","rank":1}, ... ], // 解析后(可选)
"allergens": ["坚果", "大豆", "乳"], // 过敏原
"additives": ["E322 卵磷脂"], // 添加剂
"nutriments": { // 营养成分(每100g/100ml)
"energy_kj": 2252, "energy_kcal": 539,
"fat_g": 30.9, "saturated_fat_g": 10.6,
"carbohydrates_g": 57.5, "sugars_g": 56.3,
"protein_g": 6.3, "salt_g": 0.107
},
"nutrition_basis": "per_100g", // per_100g | per_100ml | per_serving
"serving_size": "15g",
"is_vegetarian": null, "is_vegan": null, // 可空
"nutri_score": "C", // 若引用 OFF
"labels": ["无添加", "清真"] // 认证/标签
}
```
### 2.3 官方标准零售价 `msrp`(重点)
```jsonc
{
"amount": 29.90,
"currency": "CNY",
"type": "msrp", // 仅 msrp/官方指导价; 不存实时电商成交价
"region": "CN", // 适用地区(价格随地区不同)
"source": "厂商官网/官方价目表",
"source_url": "https://...",
"effective_date": "2026-01-01", // 价格生效/采集时间
"note": "官方建议零售价, 实际售价以零售商为准; 本站不提供购买"
}
```
> 设计要点:价格是**带时间戳的历史快照**而非实时报价;明确 `type=msrp`、标注地区与来源;响应里附免责说明。**坚决不出现购买/跳转链接。**
### 2.4 辅助实体
`brand` / `manufacturer` / `category`(品类树) / `source`(数据来源登记) / `attribute_definition`(参数字典: 标准名·别名·单位) / `merge_log`(实体合并记录, 保留溯源)。
---
## 3. 可立即接入的开放数据源清单(食品快消)
按"合规性 / 可用性"排序。这些可作为**种子数据 + 采集 adapter 的首批对象**。
| 数据源 | 内容 | 许可 | 接入方式 | 备注 |
|--------|------|------|----------|------|
| **Open Food Facts** ⭐ | 全球食品(成分/营养/过敏原/Nutri-Score/图片) | **ODbL**(数据)+DbCL+CC-BY-SA(图) | REST API + **每夜全量 dump**(CSV/MongoDB, ~9GB) | 食品首选;可贡献回写;限速 15 req/min/IP(读) |
| **USDA FoodData Central** ⭐ | 美国食品营养成分(含 Branded 品牌库) | **CC0(公共领域)** | REST API(需免费 key) + JSON/CSV 下载 | 营养数据权威;商业可用 |
| **GS1 / Verified by GS1**(中国商品信息服务平台) | 条码→品牌/规格/厂商(官方登记) | 受限(需企业/接口授权) | 网页查询 + API(≤1000 GTIN/次) | **条码→商品**最权威来源;2亿+条;中国数据首选 |
| **brocade.io** | 开放 GTIN/条码产品库 | 开源/开放 | 免费 REST(免鉴权读) | 数据量有限,可作补充 |
| **3023data 等条码接口** | 中国物品编码+UPC+ISBN | 商业(0.005~0.02元/次) | REST API | **付费**,作兜底补全,非首选 |
| 各国**监管公开数据** | 食品备案/标签/能效等 | 多为公开 | 各平台 | 后续按需逐个评估合规 |
**参考用开源项目(架构/数据模型借鉴,非数据源)**
- Open Food Facts Server (Product Opener) — 食品库的完整实现,可学其字段与流程
- UnoPIM / PCMT / brocade.io — 开源 PIM / 商品主数据系统,借鉴建模与去重
> 建议:**先用 Open Food Facts 全量 dump 作种子数据**(直接有海量真实食品),再用 GS1/USDA 做补全与校验。这样 MVP 阶段就有真实可查的数据。
---
## 4. 公开 API 契约(Go 实现,只读)
```
GET /api/v1/products/barcode/{gtin} # ★最常用: 条码查档案
GET /api/v1/products/{id} # 内部ID查
GET /api/v1/products/search # ?q=&brand=&category=&allergen_free=&page=&size=&fields=
GET /api/v1/products/{id}/nutriments # 仅营养
GET /api/v1/products/{id}/msrp # 仅官方零售价(含来源/时间/免责)
GET /api/v1/brands | /categories # 品牌 / 品类树
GET /api/v1/sources/{id} # 数据来源透明说明
GET /healthz | /api/v1/openapi.json # 健康检查 / 机读文档
```
约定:版本化 `/v1/`;统一响应 `{data, meta(分页), sources(溯源)}`;分页 + `fields=` 裁剪;匿名按 IP 限流,可选免费 API Key 提配额;CDN+Redis 缓存(参数变化慢,命中率高);数据采用开放许可(CC BY / ODbL,注意 OFF 的 ODbL 传染性);**无任何购买/交易端点**。
---
## 5. 合规与边界(公益项目重点)
- **数据源许可要分清**:OFF 是 **ODbL**(衍生数据库需同样开放+署名),USDA 是 **CC0**(最宽松)。混用时要按最严格许可对外标注,避免许可冲突。
- 爬取守 robots.txt / 服务条款,礼貌限速,标明 User-Agent 身份。
- 只采**客观参数**;营销文案/评测原文不照搬(链接来源即可)。
- 无个人数据(PII),只处理商品信息。
- 站点显著声明:**仅提供信息、不提供购买、不构成消费建议**;价格为官方指导价历史快照。
- 提供权利方**纠错/下架**联系渠道。
---
## 6. 里程碑(仍不写代码,仅规划,供确认)
| 阶段 | 目标 | 关键产出 |
|------|------|----------|
| **M0 工程地基** | 仓库骨架 | Go API 骨架 + Python 采集骨架 + PostgreSQL + Docker Compose + CI + 数据契约文档 |
| **M1 数据模型** | 食品 schema | 主表/食品字段/MSRP/辅助实体 的迁移与字典 |
| **M2 种子数据** | 有真实数据 | 导入 Open Food Facts dump(食品子集) + USDA 营养补全 |
| **M3 MVP API (Go)** | 可查询 | 条码/ID/搜索/营养/MSRP + OpenAPI 文档 + 限流缓存 |
| **M4 采集管线 (Python)** | 自动更新 | 1~2 个 adapter(OFF API / GS1) + ETL + 去重 + 质量评分 + 调度 |
| **M5 开放与规模化** | 上线 | 搜索引擎 + CDN + API Key + 众包纠错后台 + 开发者文档站 + 开放数据许可 |
---
## 7. 待你确认/补充
1. **价格范围**:确认只收「官方指导价 (MSRP)」、不碰实时电商价?(建议是)
2. **OFF 的 ODbL 许可**:可接受(意味着我们对外的数据库也要用 ODbL 并署名 OFF)?还是更想用 CC0 来源(USDA)为主以保持宽松?
3. **种子数据**:同意先导入 Open Food Facts 食品 dump 作为启动数据吗?
4. **Go Web 框架偏好**Gin / Echo / Chi / 标准库 net/http,有偏好吗?(无偏好我默认 Chi 或标准库,轻量)
5. **地域范围**:首批面向中国市场商品,还是中外都收?(影响优先用 GS1-China 还是 OFF 全球库)
> 你确认后,我把它定为 v1.0 规划,并据此拆成可执行的工程任务清单(仍按你的节奏,需要我动手写代码时再开始)。
@@ -1,107 +0,0 @@
# 商品档案公益 API 系统 — 规划方案 (v1.0 定稿)
> 公益网站/服务:采集全网商品信息,提供商品参数查询 API。**只收集 + 只提供信息,不涉及任何购买行为。**
---
## 1. 已锁定的决策
| # | 决策 | 结论 |
|---|------|------|
| 1 | 品类 | 首批 **食品快消** |
| 2 | 价格 | 只收 **官方标准零售价 (MSRP)**,静态字段,带来源/时间/地区/币种 + 免责说明;**不收实时电商价、不提供购买入口** |
| 3 | 技术栈 | **Go**(对外API/核心服务) + **Python**(采集/ETL/爬虫),经 PostgreSQL + Redis/队列解耦 |
| 4 | 种子数据 | ✅ **先导入 Open Food Facts 食品 dump**,最快拥有真实数据 |
| 5 | 数据许可 | 因采用 OFF → 对外数据库用 **ODbL** 并署名来源;CC0 来源(USDA)可自由混入 |
### 1.1 我先用的默认值(如不同意请指出,否则按此执行)
- **Go Web 框架**`chi` + 标准库 `net/http`(轻量、稳定、易维护)。
- **地域范围**:先用 OFF **全球食品库**起步,后续接 **GS1-China** 补强中国市场数据。
- **数据库迁移工具**Go 侧用 `golang-migrate`(纯 SQL 迁移,两端共享同一套 schema)。
- **部署**:初期 Docker Compose 一键起全套(Postgres/Redis/Go API/Python worker)。
---
## 2. 目标架构(定稿)
```
数据源(OFF dump / OFF API / USDA / GS1)
▼ Python: 采集 adapters → ETL(清洗/归一/去重/质量评分)
┌────▼─────────┐ 图片 ┌──────────┐
│ PostgreSQL │◀───────▶│ S3/MinIO │
│ (商品档案主库)│ └──────────┘
└────▲─────────┘
│ 只读 (+Redis缓存/限流)
▼ Go: 公开 REST API + OpenAPI 文档
各种软件 / 开发者
```
- **解耦契约**:两端通过共享 PostgreSQL schema + 一份《数据契约文档》协作,互不直接调用。
- **Go 端只读主库**(或读副本);**Python 端负责写入**。
---
## 3. 仓库结构(计划,写代码时落地)
```
goods/
├── README.md
├── docker-compose.yml # postgres + redis + minio + api + worker
├── docs/
│ ├── data-contract.md # 两端共享的字段契约
│ └── openapi.yaml # API 契约
├── migrations/ # 共享 SQL 迁移 (golang-migrate)
├── api/ # Go: 对外只读 API
│ ├── cmd/server/main.go
│ ├── internal/{handler,store,model,middleware}/
│ └── go.mod
└── ingestion/ # Python: 采集 + ETL
├── pyproject.toml
├── adapters/{openfoodfacts,usda,gs1}.py
├── etl/{normalize,dedup,quality}.py
└── jobs/{seed_off_dump,scheduler}.py
```
## 4. 数据模型 & API 契约
(沿用 v0.2`product` 主表 + `food` 食品字段 + `msrp` 价格 + 辅助实体;API 以 `GET /products/barcode/{gtin}` 为核心,全只读、无交易端点。详见 v0.2 附件。)
---
## 5. 可执行任务拆分(按里程碑,写代码时逐项落地)
**M0 — 工程地基**
- [ ] 初始化 Go module (`api/`) + Python 项目 (`ingestion/`)
- [ ] `docker-compose.yml`Postgres + Redis + MinIO
- [ ] CIGo: build/vet/testPython: ruff/pytest
- [ ] `docs/data-contract.md` 初版
**M1 — 数据模型**
- [ ] `migrations/`product / food / msrp / brand / manufacturer / category / source / attribute_definition / merge_log
- [ ] JSONB + GIN 索引;gtin 唯一索引
**M2 — 种子数据 (Python)**
- [ ] 下载 OFF 食品 dumpCSV
- [ ] `seed_off_dump`:字段映射 → 入库(含营养/成分/过敏原/图片URL)
- [ ] USDA(CC0) 营养补全(可选)
**M3 — MVP API (Go)**
- [ ] 路由 + handlerbarcode / id / search / nutriments / msrp / brands / categories / sources
- [ ] 统一响应、分页、`fields=` 裁剪、错误处理
- [ ] Redis 缓存 + IP 限流;`/healthz` + OpenAPI 文档
**M4 — 采集管线 (Python)**
- [ ] adapterOFF API(增量更新)+ GS1(条码补全)
- [ ] ETL:清洗/单位归一/去重合并/质量评分/溯源
- [ ] 调度(定时增量更新)
**M5 — 开放与规模化**
- [ ] 搜索引擎(PG 全文 → OpenSearch)、CDN 缓存
- [ ] 免费 API Key(防滥用+统计)、众包纠错后台
- [ ] 开发者文档站 + 开放数据许可声明 + 站点"不提供购买"声明
---
## 6. 下一步
规划已定稿。**你说先不写代码,所以我暂停在这里**。等你说"开始",我就从 **M0 工程地基** 动手,搭好骨架后开 PR 给你看。也可以先只做某个里程碑(比如先 M0+M1 把骨架和数据模型立起来)。
+9
View File
@@ -0,0 +1,9 @@
"""OpenGoods (天工·商品标签) ingestion package.
Collects public product information from open data sources (e.g. Open Food
Facts) and normalizes it into the OpenGoods database. This package only
collects and processes product facts; it performs no purchase or commerce
actions.
"""
__version__ = "0.1.0"
+6
View File
@@ -0,0 +1,6 @@
"""Source adapters.
Each open data source (Open Food Facts, USDA FoodData Central, GS1, ...) gets
its own adapter that fetches raw records and yields them for the ETL layer.
Adapters must respect each source's robots.txt, rate limits and license.
"""
+17
View File
@@ -0,0 +1,17 @@
"""Base adapter protocol shared by all source adapters."""
from __future__ import annotations
from collections.abc import Iterator
from typing import Protocol
class SourceAdapter(Protocol):
"""A source adapter fetches raw product records from one data source."""
#: Stable identifier of the source, e.g. "openfoodfacts".
source_name: str
def fetch(self) -> Iterator[dict]:
"""Yield raw product records as dictionaries."""
...
+124
View File
@@ -0,0 +1,124 @@
"""GS1 barcode supplement adapter.
GS1 (e.g. *Verified by GS1* / GS1 China) is the authoritative registry that maps
a GTIN to its brand owner, product description and GPC category. We use it to
*supplement* — fill gaps in — records gathered from crowd sources like Open Food
Facts, never to overwrite existing values.
Real GS1 access is credentialed and region-specific, so this adapter supports
two modes:
* **offline** (default): look barcodes up in a local JSON mapping file. This is
what tests and air-gapped runs use.
* **online**: GET ``{base_url}/{gtin}`` with an API key header, then normalize
the response. Enabled by passing ``base_url`` + ``client``.
Either way :meth:`fetch_barcode` returns a normalized *supplement* dict (or
``None``); :mod:`opengoods.etl.supplement` applies it to the database.
"""
from __future__ import annotations
import json
from collections.abc import Iterator
from pathlib import Path
import httpx
SOURCE_NAME = "gs1"
GS1_HOMEPAGE = "https://www.gs1.org"
GS1_LICENSE = "proprietary"
# GS1 is the authoritative barcode registry -> high trust.
GS1_TRUST = 0.9
# Keys of a normalized supplement record.
_SUPPLEMENT_KEYS = (
"gtin",
"name",
"brand",
"manufacturer",
"gpc_brick_code",
"country_of_origin",
"net_content_value",
"net_content_unit",
)
def _normalize(code: str, data: dict) -> dict:
"""Project a raw mapping/record onto the supplement schema (non-empty only)."""
rec: dict = {"gtin": code}
for key in _SUPPLEMENT_KEYS:
if key == "gtin":
continue
value = data.get(key)
if value not in (None, "", []):
rec[key] = value
return rec
def _parse_api(code: str, payload: dict) -> dict:
"""Best-effort mapping of a Verified-by-GS1 style payload to our schema."""
item = payload
if isinstance(payload.get("gtinRecords"), list) and payload["gtinRecords"]:
item = payload["gtinRecords"][0]
return _normalize(
code,
{
"name": item.get("productDescription") or item.get("description"),
"brand": item.get("brandName"),
"manufacturer": item.get("companyName") or item.get("licenseeName"),
"gpc_brick_code": item.get("gpcCategoryCode"),
"country_of_origin": item.get("countryOfSaleCode") or item.get("countryCode"),
"net_content_value": item.get("netContent"),
"net_content_unit": item.get("netContentUnit"),
},
)
class GS1Adapter:
"""Look up GTIN supplements from a local mapping or a GS1-style API."""
source_name = SOURCE_NAME
def __init__(
self,
mapping: dict | None = None,
*,
client: httpx.Client | None = None,
base_url: str | None = None,
api_key: str | None = None,
) -> None:
self._mapping = mapping or {}
self._client = client
self._base_url = base_url.rstrip("/") if base_url else None
self._api_key = api_key
@classmethod
def from_file(cls, path: str | Path) -> GS1Adapter:
"""Build an offline adapter from a JSON ``{gtin: {...}}`` mapping file."""
data = json.loads(Path(path).read_text(encoding="utf-8"))
return cls(mapping=data)
def fetch_barcode(self, code: str) -> dict | None:
"""Return a normalized supplement dict for ``code`` (or ``None``)."""
if self._base_url and self._client is not None:
headers = {"apikey": self._api_key} if self._api_key else {}
resp = self._client.get(f"{self._base_url}/{code}", headers=headers)
if resp.status_code == 404:
return None
resp.raise_for_status()
rec = _parse_api(code, resp.json())
else:
data = self._mapping.get(code)
if not data:
return None
rec = _normalize(code, data)
# A record with only the GTIN carries no supplement.
return rec if len(rec) > 1 else None
def fetch(self, barcodes: list[str]) -> Iterator[dict]:
"""Yield supplement records for the given barcodes."""
for code in barcodes:
rec = self.fetch_barcode(code)
if rec is not None:
yield rec
@@ -0,0 +1,135 @@
"""Open Food Facts (OFF) source adapter.
Fetches raw product records either from the OFF read API (one product per
barcode) or from a downloaded JSONL dump file. OFF data is licensed under the
Open Database License (ODbL); product images are CC-BY-SA. We record OFF as the
source for every field we ingest.
The adapter is read-only and rate-limited to stay well within OFF's API limits
(<= ~15 req/min/IP for product reads) and to be a good citizen.
"""
from __future__ import annotations
import json
import time
from collections.abc import Iterator
from pathlib import Path
import httpx
SOURCE_NAME = "openfoodfacts"
OFF_LICENSE = "ODbL"
USER_AGENT = "OpenGoods/0.1 (+https://github.com/baicai2026-baicai/goods) public-good product API"
# Conservative client-side spacing between API calls (seconds).
_DEFAULT_MIN_INTERVAL = 4.0
_API_URL = "https://world.openfoodfacts.org/api/v2/product/{barcode}.json"
_SEARCH_URL = "https://world.openfoodfacts.org/api/v2/search"
# Fields requested from the search API so a returned product can be transformed
# without an extra per-barcode round trip.
_SEARCH_FIELDS = (
"code,product_name,product_name_en,product_name_zh,brands,quantity,"
"categories,categories_tags,countries,ingredients_text,allergens_tags,"
"additives_tags,nutriments,nutriscore_grade,serving_size,"
"image_front_url,image_url,last_modified_t"
)
class OpenFoodFactsAdapter:
"""Read product records from the OFF API."""
source_name = SOURCE_NAME
def __init__(
self,
client: httpx.Client | None = None,
min_interval: float = _DEFAULT_MIN_INTERVAL,
) -> None:
self._client = client or httpx.Client(headers={"User-Agent": USER_AGENT}, timeout=30.0)
self._min_interval = min_interval
self._last_call = 0.0
def _throttle(self) -> None:
elapsed = time.monotonic() - self._last_call
wait = self._min_interval - elapsed
if wait > 0:
time.sleep(wait)
self._last_call = time.monotonic()
def fetch_barcode(self, barcode: str) -> dict | None:
"""Fetch a single product by barcode; return the raw `product` dict."""
self._throttle()
resp = self._client.get(_API_URL.format(barcode=barcode))
resp.raise_for_status()
payload = resp.json()
if payload.get("status") != 1:
return None
return payload["product"]
def fetch(self, barcodes: list[str]) -> Iterator[dict]:
"""Yield raw product records for the given barcodes."""
for code in barcodes:
record = self.fetch_barcode(code)
if record is not None:
yield record
def fetch_modified_since(
self,
since_t: int,
*,
page_size: int = 100,
max_pages: int = 10,
) -> Iterator[dict]:
"""Yield products modified after ``since_t`` (unix ``last_modified_t``).
Uses the OFF search API sorted by ``last_modified_t`` (most recent
first) and paginates until it reaches products at or before the
watermark, an empty/short page, or ``max_pages``. This is the
incremental ingestion path: callers persist the highest
``last_modified_t`` they processed as the next watermark.
"""
for page in range(1, max_pages + 1):
self._throttle()
resp = self._client.get(
_SEARCH_URL,
params={
"fields": _SEARCH_FIELDS,
"sort_by": "last_modified_t",
"page": page,
"page_size": page_size,
},
)
resp.raise_for_status()
products = resp.json().get("products") or []
if not products:
return
reached_old = False
for prod in products:
if int(prod.get("last_modified_t") or 0) <= since_t:
reached_old = True
break
yield prod
if reached_old or len(products) < page_size:
return
def read_dump(path: str | Path) -> Iterator[dict]:
"""Yield raw product records from an OFF JSONL dump file.
Each line is one product JSON object (the format of OFF's .jsonl export).
Supports plain or .gz files.
"""
p = Path(path)
if p.suffix == ".gz":
import gzip
opener = lambda: gzip.open(p, "rt", encoding="utf-8") # noqa: E731
else:
opener = lambda: open(p, encoding="utf-8") # noqa: E731
with opener() as fh:
for line in fh:
line = line.strip()
if line:
yield json.loads(line)
+1
View File
@@ -0,0 +1 @@
"""ETL: clean, normalize, dedup and score raw records before loading."""
+139
View File
@@ -0,0 +1,139 @@
"""Duplicate detection and product merging.
Barcodes (GTIN) are already unique at the schema level, so duplicates here are
non-GTIN records that describe the same product (same normalized name + brand +
net content). For each duplicate group we keep the highest-quality product as
canonical and merge the rest into it: child rows (provenance, images, MSRP) are
re-pointed to the canonical product, the merged product is marked ``merged``
with ``canonical_id`` set, and a row is written to ``merge_log``.
"""
from __future__ import annotations
from typing import Any
import psycopg
from opengoods.etl.quality import update_quality
def _norm(text: str | None) -> str:
return " ".join((text or "").lower().split())
def product_signature(name: str | None, brand: str | None, net_canonical: Any | None) -> str | None:
"""Stable signature for non-GTIN dedup, or ``None`` if too sparse to match."""
n = _norm(name)
if not n:
return None
net = "" if net_canonical is None else str(net_canonical)
return f"{n}|{_norm(brand)}|{net}"
def choose_canonical(members: list[dict]) -> dict:
"""Pick the canonical product: best quality, then oldest, then lowest id."""
return min(
members,
key=lambda m: (
-float(m.get("quality_score") or 0.0),
m.get("created_at"),
str(m.get("id")),
),
)
def find_duplicate_groups(conn: psycopg.Connection) -> list[list[dict]]:
"""Return groups (size >= 2) of active products sharing a signature."""
rows = conn.execute(
"""
SELECT p.id, p.name, b.normalized_name, p.net_content_canonical,
p.quality_score, p.created_at
FROM product p
LEFT JOIN brand b ON b.id = p.brand_id
WHERE p.status = 'active'
"""
).fetchall()
groups: dict[str, list[dict]] = {}
for r in rows:
sig = product_signature(r[1], r[2], r[3])
if sig is None:
continue
member = {
"id": r[0],
"name": r[1],
"quality_score": r[4],
"created_at": r[5],
}
groups.setdefault(sig, []).append(member)
return [m for m in groups.values() if len(m) >= 2]
def merge_products(
conn: psycopg.Connection,
kept_id: str,
merged_id: str,
reason: str = "auto-dedup",
actor: str = "ingestion",
) -> None:
"""Merge ``merged_id`` into ``kept_id`` (re-point children, mark merged)."""
if kept_id == merged_id:
return
# Re-point provenance, images and MSRP to the canonical product.
conn.execute(
"UPDATE product_source SET product_id = %s WHERE product_id = %s",
(kept_id, merged_id),
)
conn.execute(
"UPDATE product_image SET product_id = %s WHERE product_id = %s",
(kept_id, merged_id),
)
conn.execute(
"UPDATE product_msrp SET product_id = %s WHERE product_id = %s",
(kept_id, merged_id),
)
# food_detail has product_id as PK, so it can only move if the canonical
# product does not already have one.
kept_has_food = conn.execute(
"SELECT 1 FROM food_detail WHERE product_id = %s", (kept_id,)
).fetchone()
if not kept_has_food:
conn.execute(
"UPDATE food_detail SET product_id = %s WHERE product_id = %s",
(kept_id, merged_id),
)
conn.execute(
"UPDATE product SET status = 'merged', canonical_id = %s WHERE id = %s",
(kept_id, merged_id),
)
conn.execute(
"""
INSERT INTO merge_log (kept_id, merged_id, reason, actor)
VALUES (%s, %s, %s, %s)
""",
(kept_id, merged_id, reason, actor),
)
# The canonical product gained sources, so its quality may have changed.
update_quality(conn, kept_id)
def dedup_all(
conn: psycopg.Connection, actor: str = "ingestion", dry_run: bool = False
) -> dict[str, int]:
"""Merge every duplicate group. Returns counts of groups and merges."""
groups = find_duplicate_groups(conn)
merged = 0
for members in groups:
canonical = choose_canonical(members)
for m in members:
if m["id"] == canonical["id"]:
continue
if not dry_run:
merge_products(conn, canonical["id"], m["id"], actor=actor)
merged += 1
return {"groups": len(groups), "merged": merged}
+213
View File
@@ -0,0 +1,213 @@
"""Load transformed product records into the OpenGoods PostgreSQL database.
Only the ingestion side writes to the database. Every load records OFF as the
source with field-level provenance in `product_source`.
"""
from __future__ import annotations
import json
import os
from typing import Any
import psycopg
from psycopg.types.json import Jsonb
from opengoods.adapters.openfoodfacts import OFF_LICENSE, SOURCE_NAME
from opengoods.etl.quality import update_quality
OFF_HOMEPAGE = "https://world.openfoodfacts.org"
def default_dsn() -> str:
return os.environ.get(
"OPENGOODS_DATABASE_URL",
"postgres://opengoods:opengoods@localhost:5432/opengoods?sslmode=disable",
)
def _normalize_brand(name: str) -> str:
return " ".join(name.lower().split())
def ensure_source_named(
conn: psycopg.Connection,
name: str,
homepage: str,
license: str,
trust_weight: float,
) -> str:
"""Upsert a source row by name and return its id."""
row = conn.execute(
"""
INSERT INTO source (name, homepage, license, trust_weight)
VALUES (%s, %s, %s, %s)
ON CONFLICT (name) DO UPDATE SET homepage = EXCLUDED.homepage
RETURNING id
""",
(name, homepage, license, trust_weight),
).fetchone()
return row[0]
def ensure_source(conn: psycopg.Connection) -> str:
"""Upsert the Open Food Facts source row and return its id."""
return ensure_source_named(conn, SOURCE_NAME, OFF_HOMEPAGE, OFF_LICENSE, 0.7)
def _ensure_brand(conn: psycopg.Connection, name: str | None) -> str | None:
if not name:
return None
row = conn.execute(
"""
INSERT INTO brand (name, normalized_name)
VALUES (%s, %s)
ON CONFLICT (normalized_name) DO UPDATE SET name = brand.name
RETURNING id
""",
(name, _normalize_brand(name)),
).fetchone()
return row[0]
def _category_id(conn: psycopg.Connection, path: str | None) -> tuple[str | None, str | None]:
if not path:
return None, None
row = conn.execute(
"SELECT id, gpc_brick_code FROM category WHERE path = %s::ltree", (path,)
).fetchone()
return (row[0], row[1]) if row else (None, None)
def load_record(conn: psycopg.Connection, rec: dict[str, Any], source_id: str, raw: dict) -> str:
"""Upsert one transformed record; return the product id."""
brand_id = _ensure_brand(conn, rec.get("brand"))
category_id, gpc_brick = _category_id(conn, rec.get("category_path"))
fields = ["name", "brand", "net_content", "category", "country_of_origin"]
if rec.get("gtin"):
prod = conn.execute(
"""
INSERT INTO product (gtin, name, brand_id, category_id, gpc_brick_code,
net_content_value, net_content_unit, net_content_canonical,
country_of_origin, attributes)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
ON CONFLICT (gtin) WHERE gtin IS NOT NULL DO UPDATE SET
name = EXCLUDED.name,
brand_id = COALESCE(EXCLUDED.brand_id, product.brand_id),
category_id = COALESCE(EXCLUDED.category_id, product.category_id),
gpc_brick_code = COALESCE(EXCLUDED.gpc_brick_code, product.gpc_brick_code),
net_content_value = EXCLUDED.net_content_value,
net_content_unit = EXCLUDED.net_content_unit,
net_content_canonical = EXCLUDED.net_content_canonical,
country_of_origin = EXCLUDED.country_of_origin
RETURNING id
""",
(
rec["gtin"],
rec["name"],
brand_id,
category_id,
gpc_brick,
rec.get("net_content_value"),
rec.get("net_content_unit"),
rec.get("net_content_canonical"),
rec.get("country_of_origin"),
Jsonb({}),
),
).fetchone()
else:
prod = conn.execute(
"""
INSERT INTO product (name, brand_id, category_id, gpc_brick_code,
net_content_value, net_content_unit, net_content_canonical,
country_of_origin, attributes)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
RETURNING id
""",
(
rec["name"],
brand_id,
category_id,
gpc_brick,
rec.get("net_content_value"),
rec.get("net_content_unit"),
rec.get("net_content_canonical"),
rec.get("country_of_origin"),
Jsonb({}),
),
).fetchone()
product_id = prod[0]
food = rec.get("food") or {}
conn.execute(
"""
INSERT INTO food_detail (product_id, ingredients_text, allergens, additives,
nutriments, nutrition_basis, serving_size, nutri_score)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s)
ON CONFLICT (product_id) DO UPDATE SET
ingredients_text = EXCLUDED.ingredients_text,
allergens = EXCLUDED.allergens,
additives = EXCLUDED.additives,
nutriments = EXCLUDED.nutriments,
nutrition_basis = EXCLUDED.nutrition_basis,
serving_size = EXCLUDED.serving_size,
nutri_score = EXCLUDED.nutri_score
""",
(
product_id,
food.get("ingredients_text"),
food.get("allergens") or [],
food.get("additives") or [],
Jsonb(food.get("nutriments") or {}),
food.get("nutrition_basis"),
food.get("serving_size"),
food.get("nutri_score"),
),
)
if rec.get("image_url"):
conn.execute(
"""
INSERT INTO product_image (product_id, url, kind, license, source_id)
VALUES (%s,%s,'front',%s,%s)
""",
(product_id, rec["image_url"], "CC-BY-SA", source_id),
)
fields.append("image")
conn.execute(
"""
INSERT INTO product_source (product_id, source_id, url, fields, fetched_at, raw)
VALUES (%s,%s,%s,%s, now(), %s)
""",
(
product_id,
source_id,
f"{OFF_HOMEPAGE}/product/{rec.get('gtin') or ''}",
fields,
Jsonb(_jsonable(raw)),
),
)
# Recompute the data-quality score now that all facts + provenance exist.
update_quality(conn, product_id)
return product_id
def _jsonable(raw: dict) -> dict:
"""Drop values that are not JSON-serializable from a raw record."""
try:
json.dumps(raw)
return raw
except (TypeError, ValueError):
return {k: v for k, v in raw.items() if _is_jsonable(v)}
def _is_jsonable(v: object) -> bool:
try:
json.dumps(v)
return True
except (TypeError, ValueError):
return False
+104
View File
@@ -0,0 +1,104 @@
"""Field-level conflict resolution for multi-source records.
When more than one source describes the same product, each field may have
several candidate values. We pick a winner per field by source trust first,
then recency, ignoring empty values, and keep a provenance trail of which
source won each field.
These are pure functions (no DB / no network) so they are easy to unit-test;
the DB-level record merge lives in :mod:`opengoods.etl.dedup`.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime
@dataclass(frozen=True)
class Candidate:
"""One source's proposed value for a field."""
value: object
source: str
trust: float = 0.5
fetched_at: datetime | None = None
@dataclass
class FieldResolution:
"""The winning value for a field plus the source it came from."""
value: object
source: str | None = None
@dataclass
class MergedRecord:
"""A merged record with per-field provenance (field name -> source)."""
values: dict[str, object] = field(default_factory=dict)
provenance: dict[str, str] = field(default_factory=dict)
def _is_empty(value: object) -> bool:
if value is None:
return True
if isinstance(value, str):
return value.strip() == ""
if isinstance(value, (list, dict, tuple, set)):
return len(value) == 0
return False
def _sort_key(c: Candidate) -> tuple[float, float]:
ts = c.fetched_at.timestamp() if c.fetched_at is not None else float("-inf")
return (c.trust, ts)
def resolve_field(candidates: list[Candidate]) -> FieldResolution | None:
"""Pick the best non-empty candidate for one field.
Ranking: highest source trust, then most recent ``fetched_at``. Returns
``None`` when there is no usable (non-empty) candidate.
"""
usable = [c for c in candidates if not _is_empty(c.value)]
if not usable:
return None
winner = max(usable, key=_sort_key)
return FieldResolution(value=winner.value, source=winner.source)
def merge_records(records: list[dict], *, fields: list[str] | None = None) -> MergedRecord:
"""Merge several ``{field: Candidate|value}`` records into one.
Each input record maps field name -> :class:`Candidate` (preferred) or a
bare value (treated as trust 0.5, no timestamp). The result keeps, for each
field, the winning value and the name of the source that supplied it.
"""
keys: list[str]
if fields is not None:
keys = list(fields)
else:
seen: dict[str, None] = {}
for rec in records:
for k in rec:
seen.setdefault(k, None)
keys = list(seen)
merged = MergedRecord()
for key in keys:
candidates: list[Candidate] = []
for rec in records:
if key not in rec:
continue
cand = rec[key]
if not isinstance(cand, Candidate):
cand = Candidate(value=cand, source="unknown")
candidates.append(cand)
resolution = resolve_field(candidates)
if resolution is not None:
merged.values[key] = resolution.value
if resolution.source is not None:
merged.provenance[key] = resolution.source
return merged
+176
View File
@@ -0,0 +1,176 @@
"""Product data-quality scoring.
The quality score is a 0..1 number combining four signals, per the locked
project decision:
quality = 0.4 * completeness
+ 0.3 * source_trust
+ 0.2 * multi_source_agreement
+ 0.1 * freshness
Each component is itself normalized to 0..1. The pure helpers below are
unit-testable; :func:`compute_quality` / :func:`update_quality` read the signals
for a product out of the database and persist the result on ``product``.
"""
from __future__ import annotations
from datetime import UTC, datetime
import psycopg
W_COMPLETENESS = 0.4
W_SOURCE_TRUST = 0.3
W_AGREEMENT = 0.2
W_FRESHNESS = 0.1
# Fields that count towards completeness (weighted equally).
COMPLETENESS_FIELDS = (
"name",
"gtin",
"brand",
"category",
"net_content",
"country_of_origin",
"nutriments",
"ingredients",
"image",
)
def completeness(present: set[str]) -> float:
"""Fraction of :data:`COMPLETENESS_FIELDS` that are present for a product."""
if not COMPLETENESS_FIELDS:
return 0.0
hits = sum(1 for f in COMPLETENESS_FIELDS if f in present)
return hits / len(COMPLETENESS_FIELDS)
def agreement_from_sources(source_count: int) -> float:
"""Multi-source corroboration proxy from the number of distinct sources.
A single source cannot be corroborated, so it scores a neutral 0.5; more
independent sources that describe the same product raise confidence.
"""
if source_count <= 1:
return 0.5
if source_count == 2:
return 0.8
return 1.0
def freshness_from_age(age_days: float | None) -> float:
"""Recency score from the age (in days) of the most recent source fetch."""
if age_days is None:
return 0.5
if age_days <= 30:
return 1.0
if age_days <= 180:
return 0.8
if age_days <= 365:
return 0.6
if age_days <= 730:
return 0.4
return 0.2
def score(
*,
completeness_score: float,
source_trust: float,
agreement: float,
freshness: float,
) -> float:
"""Combine the four normalized components into a 0..1 quality score."""
raw = (
W_COMPLETENESS * completeness_score
+ W_SOURCE_TRUST * source_trust
+ W_AGREEMENT * agreement
+ W_FRESHNESS * freshness
)
return round(max(0.0, min(1.0, raw)), 3)
def _present_fields(prod: dict, has_image: bool) -> set[str]:
present: set[str] = set()
if prod.get("name"):
present.add("name")
if prod.get("gtin"):
present.add("gtin")
if prod.get("brand_id"):
present.add("brand")
if prod.get("category_id"):
present.add("category")
if prod.get("net_content_canonical") is not None:
present.add("net_content")
if prod.get("country_of_origin"):
present.add("country_of_origin")
if prod.get("nutriments"):
present.add("nutriments")
if prod.get("ingredients_text"):
present.add("ingredients")
if has_image:
present.add("image")
return present
def compute_quality(conn: psycopg.Connection, product_id: str) -> float:
"""Compute (but do not persist) the quality score for one product."""
row = conn.execute(
"""
SELECT p.name, p.gtin, p.brand_id, p.category_id, p.net_content_canonical,
p.country_of_origin, f.nutriments, f.ingredients_text,
EXISTS (SELECT 1 FROM product_image pi WHERE pi.product_id = p.id)
FROM product p
LEFT JOIN food_detail f ON f.product_id = p.id
WHERE p.id = %s
""",
(product_id,),
).fetchone()
if row is None:
return 0.0
prod = {
"name": row[0],
"gtin": row[1],
"brand_id": row[2],
"category_id": row[3],
"net_content_canonical": row[4],
"country_of_origin": row[5],
"nutriments": row[6],
"ingredients_text": row[7],
}
has_image = bool(row[8])
src = conn.execute(
"""
SELECT count(DISTINCT ps.source_id), COALESCE(max(s.trust_weight), 0), max(ps.fetched_at)
FROM product_source ps
LEFT JOIN source s ON s.id = ps.source_id
WHERE ps.product_id = %s
""",
(product_id,),
).fetchone()
source_count = int(src[0] or 0)
source_trust = float(src[1] or 0.0)
last_fetched: datetime | None = src[2]
age_days: float | None = None
if last_fetched is not None:
now = datetime.now(UTC)
if last_fetched.tzinfo is None:
last_fetched = last_fetched.replace(tzinfo=UTC)
age_days = max(0.0, (now - last_fetched).total_seconds() / 86400.0)
return score(
completeness_score=completeness(_present_fields(prod, has_image)),
source_trust=source_trust,
agreement=agreement_from_sources(source_count),
freshness=freshness_from_age(age_days),
)
def update_quality(conn: psycopg.Connection, product_id: str) -> float:
"""Compute the quality score and write it to ``product.quality_score``."""
value = compute_quality(conn, product_id)
conn.execute("UPDATE product SET quality_score = %s WHERE id = %s", (value, product_id))
return value
+45
View File
@@ -0,0 +1,45 @@
"""Persistent ingestion watermark stored in the ``ingest_state`` table.
The incremental updater uses this to remember how far it got for each source
(e.g. Open Food Facts exposes a ``last_modified_t`` unix timestamp on every
product) so repeated runs only fetch what changed.
"""
from __future__ import annotations
from typing import Any
import psycopg
from psycopg.types.json import Jsonb
def get_watermark(conn: psycopg.Connection, source: str) -> int:
"""Return the last processed ``last_modified_t`` for *source* (0 if none)."""
row = conn.execute(
"SELECT last_modified_t FROM ingest_state WHERE source = %s", (source,)
).fetchone()
return int(row[0]) if row else 0
def set_watermark(
conn: psycopg.Connection,
source: str,
last_modified_t: int,
stats: dict[str, Any] | None = None,
) -> None:
"""Upsert the watermark and run metadata for *source*.
The watermark only ever moves forward: a lower ``last_modified_t`` is
ignored so an out-of-order or partial run cannot rewind progress.
"""
conn.execute(
"""
INSERT INTO ingest_state (source, last_modified_t, last_run_at, stats)
VALUES (%s, %s, now(), %s)
ON CONFLICT (source) DO UPDATE SET
last_modified_t = GREATEST(ingest_state.last_modified_t, EXCLUDED.last_modified_t),
last_run_at = now(),
stats = EXCLUDED.stats
""",
(source, int(last_modified_t), Jsonb(stats or {})),
)
+133
View File
@@ -0,0 +1,133 @@
"""Apply GS1 (or other authoritative) supplements to existing products.
A supplement only fills *gaps*: a field is written only when the product does
not already have a value. Each applied supplement records field-level provenance
in ``product_source`` and refreshes the product's quality score.
"""
from __future__ import annotations
from decimal import Decimal, InvalidOperation
from typing import Any
import psycopg
from psycopg.types.json import Jsonb
from opengoods import units
from opengoods.adapters.gs1 import GS1_HOMEPAGE, GS1_LICENSE, GS1_TRUST, SOURCE_NAME
from opengoods.etl.load import _ensure_brand, _normalize_brand, ensure_source_named
from opengoods.etl.quality import update_quality
def ensure_gs1_source(conn: psycopg.Connection) -> str:
"""Upsert the GS1 source row and return its id."""
return ensure_source_named(conn, SOURCE_NAME, GS1_HOMEPAGE, GS1_LICENSE, GS1_TRUST)
def _ensure_manufacturer(conn: psycopg.Connection, name: str | None) -> str | None:
if not name:
return None
row = conn.execute(
"""
INSERT INTO manufacturer (name, normalized_name)
VALUES (%s, %s)
ON CONFLICT (normalized_name) DO UPDATE SET name = manufacturer.name
RETURNING id
""",
(name, _normalize_brand(name)),
).fetchone()
return row[0]
def _net_content(rec: dict) -> tuple[Decimal, str, Decimal | None] | None:
raw_value = rec.get("net_content_value")
unit = rec.get("net_content_unit")
if raw_value is None or not unit:
return None
try:
value = Decimal(str(raw_value))
except (InvalidOperation, ValueError):
return None
try:
canonical = units.normalize(value, unit).canonical_value
except units.UnitError:
canonical = None
return value, unit, canonical
def apply_supplement(conn: psycopg.Connection, rec: dict[str, Any], source_id: str) -> list[str]:
"""Fill missing fields of the GTIN-matched product from ``rec``.
Returns the list of field names actually filled (empty if the product is
unknown or already complete for the supplied fields).
"""
gtin = rec.get("gtin")
if not gtin:
return []
prod = conn.execute(
"""
SELECT id, brand_id, manufacturer_id, gpc_brick_code, country_of_origin,
net_content_value
FROM product
WHERE gtin = %s AND status = 'active'
""",
(gtin,),
).fetchone()
if prod is None:
return []
product_id, brand_id, manufacturer_id, gpc, country, net_value = prod
sets: list[str] = []
params: list[Any] = []
filled: list[str] = []
if brand_id is None and rec.get("brand"):
new_brand_id = _ensure_brand(conn, rec["brand"])
if new_brand_id is not None:
sets.append("brand_id = %s")
params.append(new_brand_id)
filled.append("brand")
if manufacturer_id is None and rec.get("manufacturer"):
new_mfr_id = _ensure_manufacturer(conn, rec["manufacturer"])
if new_mfr_id is not None:
sets.append("manufacturer_id = %s")
params.append(new_mfr_id)
filled.append("manufacturer")
if gpc is None and rec.get("gpc_brick_code"):
sets.append("gpc_brick_code = %s")
params.append(rec["gpc_brick_code"])
filled.append("gpc_brick_code")
if country is None and rec.get("country_of_origin"):
sets.append("country_of_origin = %s")
params.append(rec["country_of_origin"])
filled.append("country_of_origin")
if net_value is None:
net = _net_content(rec)
if net is not None:
value, unit, canonical = net
sets += [
"net_content_value = %s",
"net_content_unit = %s",
"net_content_canonical = %s",
]
params += [value, unit, canonical]
filled.append("net_content")
if not filled:
return []
params.append(product_id)
conn.execute(f"UPDATE product SET {', '.join(sets)} WHERE id = %s", params)
conn.execute(
"""
INSERT INTO product_source (product_id, source_id, url, fields, fetched_at, raw)
VALUES (%s, %s, %s, %s, now(), %s)
""",
(product_id, source_id, GS1_HOMEPAGE, filled, Jsonb(rec)),
)
update_quality(conn, product_id)
return filled
+159
View File
@@ -0,0 +1,159 @@
"""Transform raw Open Food Facts records into the OpenGoods internal shape.
Pure functions (no DB, no network) so they are easy to unit-test against
fixtures. The output dict mirrors the columns the loader writes.
"""
from __future__ import annotations
import re
from decimal import Decimal
from opengoods.units import UnitError, normalize
# OFF nutriment key -> our attribute key. Energy handled separately.
_NUTRIMENT_KEYS = {
"proteins_100g": "proteins",
"fat_100g": "fat",
"saturated-fat_100g": "saturated_fat",
"carbohydrates_100g": "carbohydrates",
"sugars_100g": "sugars",
"salt_100g": "salt",
}
# Very small keyword -> category path map (starter; replaced by a proper
# OFF taxonomy -> GPC mapping table later).
_CATEGORY_KEYWORDS: list[tuple[tuple[str, ...], str]] = [
(("water", "eau", "饮用水", "矿泉水"), "food.beverages.water"),
(("soda", "carbonated", "汽水", "碳酸"), "food.beverages.carbonated"),
(("juice", "jus", "果汁"), "food.beverages.juice"),
(("milk", "lait", "牛奶"), "food.dairy.milk"),
(("yogurt", "yoghurt", "yaourt", "酸奶"), "food.dairy.yogurt"),
(("cheese", "fromage", "奶酪", "干酪"), "food.dairy.cheese"),
(("bread", "pain", "面包"), "food.bakery.bread"),
(("biscuit", "cookie", "饼干"), "food.bakery.biscuits"),
(("chips", "crisps", "薯片", "膨化"), "food.snacks.chips"),
(("chocolate", "chocolat", "巧克力"), "food.snacks.chocolate"),
(("rice", "riz", "大米", "稻米"), "food.staple.rice"),
(("noodle", "pasta", "面条", "挂面"), "food.staple.noodles"),
(("oil", "huile", "食用油", "食油"), "food.staple.cooking_oil"),
(("soy sauce", "酱油"), "food.condiments.soy_sauce"),
(("salt", "sel", "食盐"), "food.condiments.salt"),
]
_QTY_RE = re.compile(r"(?P<value>\d+(?:[.,]\d+)?)\s*(?P<unit>[a-zA-Z\u4e00-\u9fff%]+)")
def is_valid_gtin(code: str) -> bool:
"""Validate a GTIN-8/12/13/14 using the standard check digit."""
if not code.isdigit() or len(code) not in (8, 12, 13, 14):
return False
digits = [int(c) for c in code]
check = digits[-1]
body = digits[:-1][::-1]
total = sum(d * (3 if i % 2 == 0 else 1) for i, d in enumerate(body))
return (10 - total % 10) % 10 == check
def parse_quantity(text: str) -> tuple[Decimal, str] | None:
"""Parse a free-text quantity like '500 g' or '1,5 L' -> (value, unit)."""
if not text:
return None
m = _QTY_RE.search(text)
if not m:
return None
value = Decimal(m.group("value").replace(",", "."))
return value, m.group("unit")
def map_category(raw: dict) -> str | None:
"""Best-effort map OFF categories/name to a self-built category path."""
haystack = " ".join(
str(raw.get(k, ""))
for k in ("categories", "categories_tags", "product_name", "product_name_en")
).lower()
for keywords, path in _CATEGORY_KEYWORDS:
if any(kw.lower() in haystack for kw in keywords):
return path
return None
def _clean_tags(tags: list[str] | None, prefix: str = "") -> list[str]:
out: list[str] = []
for t in tags or []:
v = t.split(":", 1)[-1] if ":" in t else t
v = v.strip().replace("-", " ")
if v:
out.append(v)
return out
def transform_nutriments(off_nutriments: dict) -> dict:
"""Build a nutriments dict on a per_100g basis with dual energy units."""
out: dict[str, object] = {}
for off_key, our_key in _NUTRIMENT_KEYS.items():
if off_key in off_nutriments and off_nutriments[off_key] is not None:
out[our_key] = float(off_nutriments[off_key])
kj = off_nutriments.get("energy-kj_100g")
kcal = off_nutriments.get("energy-kcal_100g")
if kj is None and kcal is not None:
kj = float(Decimal(str(kcal)) * Decimal("4.184"))
if kcal is None and kj is not None:
kcal = float(Decimal(str(kj)) / Decimal("4.184"))
if kj is not None:
out["energy_kj"] = round(float(kj), 3)
if kcal is not None:
out["energy_kcal"] = round(float(kcal), 3)
return out
def transform(raw: dict) -> dict | None:
"""Transform one raw OFF product record into an internal product dict.
Returns None if the record lacks a usable name.
"""
name = raw.get("product_name_zh") or raw.get("product_name") or raw.get("product_name_en")
if not name:
return None
code = str(raw.get("code", "")).strip()
gtin = code if code and is_valid_gtin(code) else None
brands = raw.get("brands") or ""
brand = brands.split(",")[0].strip() or None
net_value = net_unit = net_canonical = None
parsed = parse_quantity(raw.get("quantity", ""))
if parsed:
value, unit = parsed
try:
norm = normalize(value, unit)
net_value, net_unit, net_canonical = (
norm.value,
norm.unit,
norm.canonical_value,
)
except UnitError:
net_value, net_unit = value, unit
return {
"gtin": gtin,
"name": str(name).strip(),
"brand": brand,
"category_path": map_category(raw),
"net_content_value": net_value,
"net_content_unit": net_unit,
"net_content_canonical": net_canonical,
"country_of_origin": (raw.get("countries") or "").split(",")[0].strip() or None,
"food": {
"ingredients_text": raw.get("ingredients_text") or None,
"allergens": _clean_tags(raw.get("allergens_tags")),
"additives": _clean_tags(raw.get("additives_tags")),
"nutriments": transform_nutriments(raw.get("nutriments") or {}),
"nutrition_basis": "per_100g",
"serving_size": raw.get("serving_size") or None,
"nutri_score": (raw.get("nutriscore_grade") or "").upper()[:1] or None,
},
"image_url": raw.get("image_front_url") or raw.get("image_url") or None,
}
+1
View File
@@ -0,0 +1 @@
"""Jobs: seed import and scheduled incremental ingestion."""
+40
View File
@@ -0,0 +1,40 @@
"""Deduplicate products: merge non-GTIN duplicates into a canonical record.
Usage:
python -m opengoods.jobs.dedup --dry-run
python -m opengoods.jobs.dedup --actor nightly
"""
from __future__ import annotations
import argparse
import sys
import psycopg
from opengoods.etl.dedup import dedup_all
from opengoods.etl.load import default_dsn
def run(args: argparse.Namespace) -> int:
with psycopg.connect(args.dsn, autocommit=False) as conn:
summary = dedup_all(conn, actor=args.actor, dry_run=args.dry_run)
if args.dry_run:
conn.rollback()
else:
conn.commit()
mode = "dry-run" if args.dry_run else "applied"
print(f"{mode} groups={summary['groups']} merged={summary['merged']}")
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Deduplicate OpenGoods products")
parser.add_argument("--actor", default="ingestion", help="merge_log actor label")
parser.add_argument("--dry-run", action="store_true", help="report only, do not write")
parser.add_argument("--dsn", default=default_dsn(), help="PostgreSQL DSN")
return run(parser.parse_args(argv))
if __name__ == "__main__":
sys.exit(main())
+66
View File
@@ -0,0 +1,66 @@
"""Lightweight recurring ingestion scheduler.
Runs one ingestion cycle (incremental OFF update, then dedup) on a fixed
interval. Dependency-free: a plain sleep loop rather than a cron/APScheduler
dependency, so it is trivial to run in a container or under systemd/supervisor.
Usage:
python -m opengoods.jobs.schedule --once # single cycle, then exit
python -m opengoods.jobs.schedule --interval 3600 # every hour
"""
from __future__ import annotations
import argparse
import sys
import time
from datetime import UTC, datetime
from opengoods.etl.load import default_dsn
from opengoods.jobs import dedup as dedup_job
from opengoods.jobs import update_off as update_job
def _cycle(args: argparse.Namespace) -> None:
ts = datetime.now(UTC).isoformat(timespec="seconds")
print(f"[{ts}] cycle start")
update_job.run(
argparse.Namespace(
since=None,
page_size=args.page_size,
max_pages=args.max_pages,
min_interval=args.min_interval,
dsn=args.dsn,
)
)
if not args.skip_dedup:
dedup_job.run(argparse.Namespace(actor="scheduler", dry_run=False, dsn=args.dsn))
def run(args: argparse.Namespace) -> int:
_cycle(args)
if args.once:
return 0
while True:
time.sleep(args.interval)
try:
_cycle(args)
except Exception as exc: # noqa: BLE001 - keep the loop alive across failures
print(f"cycle error: {exc}", file=sys.stderr)
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Recurring OpenGoods ingestion")
parser.add_argument("--interval", type=int, default=3600, help="seconds between cycles")
parser.add_argument("--once", action="store_true", help="run a single cycle and exit")
parser.add_argument("--skip-dedup", action="store_true", help="run update only")
parser.add_argument("--page-size", type=int, default=100, help="search page size")
parser.add_argument("--max-pages", type=int, default=10, help="max pages to scan")
parser.add_argument("--min-interval", type=float, default=4.0, help="API throttle seconds")
parser.add_argument("--dsn", default=default_dsn(), help="PostgreSQL DSN")
return run(parser.parse_args(argv))
if __name__ == "__main__":
sys.exit(main())
+66
View File
@@ -0,0 +1,66 @@
"""Seed the database with Open Food Facts data.
Usage:
# from a list of barcodes via the OFF API
python -m opengoods.jobs.seed_off --barcodes 3017624010701 5449000000996
# from a downloaded OFF JSONL dump (optionally .gz), limited to N records
python -m opengoods.jobs.seed_off --dump products.jsonl.gz --limit 1000
The OFF read API is rate-limited client-side; for large imports use a dump.
"""
from __future__ import annotations
import argparse
import sys
from collections.abc import Iterator
import psycopg
from opengoods.adapters.openfoodfacts import OpenFoodFactsAdapter, read_dump
from opengoods.etl.load import default_dsn, ensure_source, load_record
from opengoods.etl.transform import transform
def _raw_records(args: argparse.Namespace) -> Iterator[dict]:
if args.dump:
records = read_dump(args.dump)
else:
adapter = OpenFoodFactsAdapter(min_interval=args.min_interval)
records = adapter.fetch(args.barcodes)
for i, rec in enumerate(records):
if args.limit and i >= args.limit:
break
yield rec
def run(args: argparse.Namespace) -> int:
loaded = skipped = 0
with psycopg.connect(args.dsn, autocommit=False) as conn:
source_id = ensure_source(conn)
for raw in _raw_records(args):
rec = transform(raw)
if rec is None:
skipped += 1
continue
load_record(conn, rec, source_id, raw)
loaded += 1
conn.commit()
print(f"loaded={loaded} skipped={skipped}")
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Seed OpenGoods from Open Food Facts")
src = parser.add_mutually_exclusive_group(required=True)
src.add_argument("--barcodes", nargs="+", help="barcodes to fetch via the OFF API")
src.add_argument("--dump", help="path to an OFF JSONL dump (.jsonl or .jsonl.gz)")
parser.add_argument("--limit", type=int, default=0, help="max records to load (0 = all)")
parser.add_argument("--min-interval", type=float, default=4.0, help="API throttle seconds")
parser.add_argument("--dsn", default=default_dsn(), help="PostgreSQL DSN")
return run(parser.parse_args(argv))
if __name__ == "__main__":
sys.exit(main())
+65
View File
@@ -0,0 +1,65 @@
"""Incremental Open Food Facts update.
Fetches products modified since the persisted watermark, loads them, then
advances the watermark to the newest ``last_modified_t`` processed so the next
run only sees what changed.
Usage:
python -m opengoods.jobs.update_off --max-pages 5
python -m opengoods.jobs.update_off --since 1700000000 # override watermark
"""
from __future__ import annotations
import argparse
import sys
import psycopg
from opengoods.adapters.openfoodfacts import SOURCE_NAME, OpenFoodFactsAdapter
from opengoods.etl.load import default_dsn, ensure_source, load_record
from opengoods.etl.state import get_watermark, set_watermark
from opengoods.etl.transform import transform
def run(args: argparse.Namespace) -> int:
adapter = OpenFoodFactsAdapter(min_interval=args.min_interval)
loaded = skipped = 0
high_watermark = 0
with psycopg.connect(args.dsn, autocommit=False) as conn:
source_id = ensure_source(conn)
since = args.since if args.since is not None else get_watermark(conn, SOURCE_NAME)
high_watermark = since
for raw in adapter.fetch_modified_since(
since, page_size=args.page_size, max_pages=args.max_pages
):
high_watermark = max(high_watermark, int(raw.get("last_modified_t") or 0))
rec = transform(raw)
if rec is None:
skipped += 1
continue
load_record(conn, rec, source_id, raw)
loaded += 1
set_watermark(
conn,
SOURCE_NAME,
high_watermark,
stats={"loaded": loaded, "skipped": skipped, "since": since},
)
conn.commit()
print(f"since={since} loaded={loaded} skipped={skipped} watermark={high_watermark}")
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Incremental OFF update")
parser.add_argument("--since", type=int, default=None, help="override watermark (unix ts)")
parser.add_argument("--page-size", type=int, default=100, help="search page size")
parser.add_argument("--max-pages", type=int, default=10, help="max pages to scan")
parser.add_argument("--min-interval", type=float, default=4.0, help="API throttle seconds")
parser.add_argument("--dsn", default=default_dsn(), help="PostgreSQL DSN")
return run(parser.parse_args(argv))
if __name__ == "__main__":
sys.exit(main())
+97
View File
@@ -0,0 +1,97 @@
"""Unit normalization for OpenGoods.
Product parameters arrive in many units (g/kg/ml/L, kcal/kJ, ...). To make
values comparable and searchable we store both the original value and a
normalized value expressed in a canonical unit per dimension.
This module is intentionally dependency-free and pure so it is easy to test.
"""
from __future__ import annotations
from dataclasses import dataclass
from decimal import Decimal
# Conversion factor maps each unit to its canonical unit within a dimension.
# canonical_value = value * factor
_FACTORS: dict[str, tuple[str, str, Decimal]] = {
# mass -> g
"mg": ("mass", "g", Decimal("0.001")),
"g": ("mass", "g", Decimal("1")),
"kg": ("mass", "g", Decimal("1000")),
# volume -> ml
"ml": ("volume", "ml", Decimal("1")),
"cl": ("volume", "ml", Decimal("10")),
"l": ("volume", "ml", Decimal("1000")),
# energy -> kJ
"kj": ("energy", "kJ", Decimal("1")),
"kcal": ("energy", "kJ", Decimal("4.184")),
}
# Alias map normalizes common spellings/locales to a canonical unit code.
_ALIASES: dict[str, str] = {
"kgs": "kg",
"千克": "kg",
"公斤": "kg",
"": "g",
"毫升": "ml",
"": "l",
"L": "l",
"litre": "l",
"liter": "l",
"kj": "kj",
"kJ": "kj",
"千焦": "kj",
"千卡": "kcal",
"大卡": "kcal",
}
class UnitError(ValueError):
"""Raised when a unit cannot be recognized."""
@dataclass(frozen=True)
class Normalized:
"""Result of normalizing a (value, unit) pair to its canonical unit."""
value: Decimal
unit: str
dimension: str
canonical_value: Decimal
canonical_unit: str
def canonical_unit_code(unit: str) -> str:
"""Resolve a raw unit string to a known canonical unit code."""
cleaned = unit.strip()
cleaned = _ALIASES.get(cleaned, cleaned).lower()
if cleaned not in _FACTORS:
raise UnitError(f"unknown unit: {unit!r}")
return cleaned
def normalize(value: Decimal | float | int | str, unit: str) -> Normalized:
"""Normalize a value+unit to its canonical unit within its dimension."""
code = canonical_unit_code(unit)
dimension, canonical, factor = _FACTORS[code]
dec = value if isinstance(value, Decimal) else Decimal(str(value))
return Normalized(
value=dec,
unit=code,
dimension=dimension,
canonical_value=dec * factor,
canonical_unit=canonical,
)
def kcal_to_kj(kcal: Decimal | float | int | str) -> Decimal:
"""Convert energy in kcal to kJ (1 kcal = 4.184 kJ)."""
dec = kcal if isinstance(kcal, Decimal) else Decimal(str(kcal))
return dec * Decimal("4.184")
def kj_to_kcal(kj: Decimal | float | int | str) -> Decimal:
"""Convert energy in kJ to kcal."""
dec = kj if isinstance(kj, Decimal) else Decimal(str(kj))
return dec / Decimal("4.184")
+32
View File
@@ -0,0 +1,32 @@
[project]
name = "opengoods-ingestion"
version = "0.1.0"
description = "OpenGoods (天工·商品标签) ingestion & ETL: collect product data and load it into the OpenGoods database."
requires-python = ">=3.11"
dependencies = [
"httpx>=0.27",
"psycopg[binary]>=3.2",
]
[project.optional-dependencies]
dev = [
"ruff>=0.6",
"pytest>=8.0",
]
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
include = ["opengoods*"]
[tool.ruff]
line-length = 100
target-version = "py311"
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B"]
[tool.pytest.ini_options]
testpaths = ["tests"]
View File
+31
View File
@@ -0,0 +1,31 @@
"""Shared test fixtures.
`db_conn` yields a psycopg connection inside a transaction that is rolled back
after each test, so DB tests stay isolated and leave no residue. Tests are
skipped automatically when no database is reachable or M4 migrations are not
applied (e.g. local runs without docker).
"""
from __future__ import annotations
import psycopg
import pytest
from opengoods.etl.load import default_dsn
@pytest.fixture()
def db_conn():
try:
conn = psycopg.connect(default_dsn(), connect_timeout=3)
except psycopg.OperationalError as exc: # pragma: no cover - env dependent
pytest.skip(f"no database available: {exc}")
has_state = conn.execute("SELECT to_regclass('public.ingest_state') IS NOT NULL").fetchone()[0]
if not has_state:
conn.close()
pytest.skip("M4 migrations not applied")
try:
yield conn
finally:
conn.rollback()
conn.close()
+14
View File
@@ -0,0 +1,14 @@
{
"06901234567892": {
"name": "示例矿泉水 550ml",
"brand": "示例品牌",
"manufacturer": "示例饮品有限公司",
"gpc_brick_code": "10000224",
"country_of_origin": "China",
"net_content_value": 550,
"net_content_unit": "ml"
},
"00000000000000": {
"brand": ""
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"code": "3017624010701",
"product_name": "Nutella",
"product_name_en": "Nutella hazelnut spread",
"brands": "Ferrero, Nutella",
"quantity": "400 g",
"countries": "France, China",
"categories": "Spreads, Hazelnut spreads, Chocolate spreads",
"categories_tags": ["en:spreads", "en:chocolate-spreads"],
"ingredients_text": "Sugar, palm oil, hazelnuts, cocoa, skimmed milk powder",
"allergens_tags": ["en:milk", "en:nuts"],
"additives_tags": ["en:e322"],
"serving_size": "15 g",
"nutriscore_grade": "e",
"image_front_url": "https://images.openfoodfacts.org/images/products/301/762/401/0701/front_en.jpg",
"nutriments": {
"energy-kj_100g": 2252,
"energy-kcal_100g": 539,
"fat_100g": 30.9,
"saturated-fat_100g": 10.6,
"carbohydrates_100g": 57.5,
"sugars_100g": 56.3,
"proteins_100g": 6.3,
"salt_100g": 0.107
}
}
+62
View File
@@ -0,0 +1,62 @@
from opengoods.etl.dedup import choose_canonical, dedup_all, product_signature
from opengoods.etl.load import _ensure_brand, ensure_source
def test_product_signature_normalization():
a = product_signature(" Spring Water ", "Acme", 500)
b = product_signature("spring water", "acme", 500)
assert a == b
assert product_signature("", "x", 1) is None
def test_choose_canonical_prefers_quality():
members = [
{"id": "a", "quality_score": 0.2, "created_at": 1},
{"id": "b", "quality_score": 0.9, "created_at": 2},
]
assert choose_canonical(members)["id"] == "b"
def test_dedup_merges_duplicates(db_conn):
brand_id = _ensure_brand(db_conn, "DupBrand")
def mk(quality):
return db_conn.execute(
"""
INSERT INTO product (name, brand_id, net_content_canonical, quality_score)
VALUES (%s, %s, %s, %s) RETURNING id
""",
("Dup Snack", brand_id, 100, quality),
).fetchone()[0]
keep = mk(0.9)
drop = mk(0.2)
src = ensure_source(db_conn)
db_conn.execute(
"INSERT INTO product_source (product_id, source_id, fields) VALUES (%s, %s, %s)",
(drop, src, ["name"]),
)
summary = dedup_all(db_conn)
assert summary == {"groups": 1, "merged": 1}
keep_status = db_conn.execute("SELECT status FROM product WHERE id = %s", (keep,)).fetchone()[0]
drop_status, canonical_id = db_conn.execute(
"SELECT status, canonical_id FROM product WHERE id = %s", (drop,)
).fetchone()
assert keep_status == "active"
assert drop_status == "merged"
assert str(canonical_id) == str(keep)
# The merged product's source row was re-pointed to the canonical product.
reattached = db_conn.execute(
"SELECT count(*) FROM product_source WHERE product_id = %s", (keep,)
).fetchone()[0]
assert reattached == 1
logged = db_conn.execute(
"SELECT count(*) FROM merge_log WHERE kept_id = %s AND merged_id = %s",
(keep, drop),
).fetchone()[0]
assert logged == 1
+53
View File
@@ -0,0 +1,53 @@
from pathlib import Path
from opengoods.adapters.gs1 import GS1Adapter
from opengoods.etl.supplement import apply_supplement, ensure_gs1_source
MAPPING = Path(__file__).parent / "fixtures" / "gs1_mapping.json"
GTIN = "06901234567892"
def test_gs1_adapter_offline_lookup():
adapter = GS1Adapter.from_file(MAPPING)
rec = adapter.fetch_barcode(GTIN)
assert rec["brand"] == "示例品牌"
assert rec["net_content_value"] == 550
assert rec["net_content_unit"] == "ml"
# An entry that only has empty values yields no supplement.
assert adapter.fetch_barcode("00000000000000") is None
# Unknown barcode -> None.
assert adapter.fetch_barcode("99999999999999") is None
def test_gs1_supplement_fills_only_gaps(db_conn):
pid = db_conn.execute(
"INSERT INTO product (gtin, name) VALUES (%s, %s) RETURNING id", (GTIN, "")
).fetchone()[0]
adapter = GS1Adapter.from_file(MAPPING)
rec = adapter.fetch_barcode(GTIN)
source_id = ensure_gs1_source(db_conn)
filled = apply_supplement(db_conn, rec, source_id)
assert {"brand", "country_of_origin", "net_content"} <= set(filled)
brand_id, country, net_value, net_unit = db_conn.execute(
"""
SELECT brand_id, country_of_origin, net_content_value, net_content_unit
FROM product WHERE id = %s
""",
(pid,),
).fetchone()
assert brand_id is not None
assert country == "China"
assert float(net_value) == 550.0
assert net_unit == "ml"
fields = db_conn.execute(
"SELECT fields FROM product_source WHERE product_id = %s AND source_id = %s",
(pid, source_id),
).fetchone()[0]
assert "brand" in fields
# Re-applying does nothing because the gaps are now filled.
assert apply_supplement(db_conn, rec, source_id) == []
+61
View File
@@ -0,0 +1,61 @@
"""Integration test for the DB loader.
Skipped automatically when no database is reachable (e.g. local runs without
docker, or CI jobs without a postgres service). Requires migrations applied.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from opengoods.etl.load import default_dsn, ensure_source, load_record
from opengoods.etl.transform import transform
psycopg = pytest.importorskip("psycopg")
FIXTURE = json.loads((Path(__file__).parent / "fixtures" / "off_product.json").read_text())
@pytest.fixture()
def conn():
try:
c = psycopg.connect(default_dsn(), connect_timeout=3)
except psycopg.OperationalError as exc: # pragma: no cover - env dependent
pytest.skip(f"no database available: {exc}")
# ensure schema present
has_product = c.execute("SELECT to_regclass('public.product') IS NOT NULL").fetchone()[0]
if not has_product:
c.close()
pytest.skip("migrations not applied")
yield c
c.rollback()
c.close()
def test_load_record_roundtrip(conn):
source_id = ensure_source(conn)
rec = transform(FIXTURE)
product_id = load_record(conn, rec, source_id, FIXTURE)
row = conn.execute(
"SELECT name, gtin, net_content_unit FROM product WHERE id = %s", (product_id,)
).fetchone()
assert row[0] == "Nutella"
assert row[1] == "3017624010701"
assert row[2] == "g"
nutri = conn.execute(
"SELECT nutriments ->> 'energy_kcal' FROM food_detail WHERE product_id = %s",
(product_id,),
).fetchone()
assert nutri[0] == "539.0"
prov = conn.execute(
"SELECT count(*) FROM product_source WHERE product_id = %s", (product_id,)
).fetchone()
assert prov[0] >= 1
conn.rollback() # keep the test DB clean
+62
View File
@@ -0,0 +1,62 @@
from datetime import UTC, datetime
from opengoods.etl.merge import Candidate, merge_records, resolve_field
def _ts(y, m, d):
return datetime(y, m, d, tzinfo=UTC)
def test_resolve_field_prefers_trust_then_recency():
cands = [
Candidate(value="A", source="off", trust=0.7, fetched_at=_ts(2024, 1, 1)),
Candidate(value="B", source="gs1", trust=0.9, fetched_at=_ts(2023, 1, 1)),
]
res = resolve_field(cands)
assert res is not None
assert res.value == "B"
assert res.source == "gs1"
def test_resolve_field_recency_tiebreak_on_equal_trust():
cands = [
Candidate(value="old", source="a", trust=0.7, fetched_at=_ts(2023, 1, 1)),
Candidate(value="new", source="b", trust=0.7, fetched_at=_ts(2024, 6, 1)),
]
assert resolve_field(cands).value == "new"
def test_resolve_field_skips_empty():
cands = [
Candidate(value="", source="a", trust=0.99),
Candidate(value=None, source="b", trust=0.99),
Candidate(value="kept", source="c", trust=0.1),
]
assert resolve_field(cands).value == "kept"
assert resolve_field([Candidate(value="", source="a")]) is None
def test_merge_records_provenance():
records = [
{
"name": Candidate("Water", "off", 0.7, _ts(2024, 1, 1)),
"brand": Candidate("", "off", 0.7),
},
{
"brand": Candidate("Acme", "gs1", 0.9, _ts(2024, 2, 1)),
"gtin": Candidate("123", "gs1", 0.9),
},
]
merged = merge_records(records)
assert merged.values["name"] == "Water"
assert merged.values["brand"] == "Acme"
assert merged.values["gtin"] == "123"
assert merged.provenance["brand"] == "gs1"
assert merged.provenance["name"] == "off"
def test_merge_records_accepts_bare_values():
merged = merge_records([{"x": 1}, {"x": 2}])
# both bare -> trust tie, no timestamps -> first max() wins deterministically
assert merged.values["x"] in (1, 2)
assert merged.provenance["x"] == "unknown"
+44
View File
@@ -0,0 +1,44 @@
import httpx
from opengoods.adapters.openfoodfacts import OpenFoodFactsAdapter
def _product(code, lm):
return {"code": code, "product_name": f"P{code}", "last_modified_t": lm}
def _adapter(pages):
"""Build an adapter whose search endpoint serves the given pages."""
def handler(request: httpx.Request) -> httpx.Response:
page = int(request.url.params.get("page", "1"))
products = pages.get(page, [])
return httpx.Response(200, json={"products": products, "page": page})
client = httpx.Client(transport=httpx.MockTransport(handler))
return OpenFoodFactsAdapter(client=client, min_interval=0)
def test_incremental_yields_only_newer_and_stops_at_watermark():
pages = {
1: [_product("1", 300), _product("2", 250), _product("3", 100)],
}
adapter = _adapter(pages)
got = list(adapter.fetch_modified_since(200, page_size=3, max_pages=5))
codes = [p["code"] for p in got]
assert codes == ["1", "2"] # 100 <= 200 stops iteration
def test_incremental_paginates_until_short_page():
pages = {
1: [_product("1", 900), _product("2", 800)],
2: [_product("3", 700)], # short page -> stop after
}
adapter = _adapter(pages)
got = list(adapter.fetch_modified_since(0, page_size=2, max_pages=5))
assert [p["code"] for p in got] == ["1", "2", "3"]
def test_incremental_empty_first_page():
adapter = _adapter({1: []})
assert list(adapter.fetch_modified_since(0, page_size=10, max_pages=3)) == []
+38
View File
@@ -0,0 +1,38 @@
from opengoods.etl.quality import (
COMPLETENESS_FIELDS,
agreement_from_sources,
completeness,
freshness_from_age,
score,
)
def test_completeness_bounds():
assert completeness(set()) == 0.0
assert completeness(set(COMPLETENESS_FIELDS)) == 1.0
half = set(list(COMPLETENESS_FIELDS)[: len(COMPLETENESS_FIELDS) // 2])
assert 0.0 < completeness(half) < 1.0
def test_agreement_from_sources():
assert agreement_from_sources(0) == 0.5
assert agreement_from_sources(1) == 0.5
assert agreement_from_sources(2) == 0.8
assert agreement_from_sources(5) == 1.0
def test_freshness_from_age():
assert freshness_from_age(None) == 0.5
assert freshness_from_age(1) == 1.0
assert freshness_from_age(100) == 0.8
assert freshness_from_age(300) == 0.6
assert freshness_from_age(700) == 0.4
assert freshness_from_age(5000) == 0.2
def test_score_weighted_sum_and_bounds():
assert score(completeness_score=0, source_trust=0, agreement=0, freshness=0) == 0.0
assert score(completeness_score=1, source_trust=1, agreement=1, freshness=1) == 1.0
# 0.4*1 + 0.3*0.5 + 0.2*0.5 + 0.1*1 = 0.75
got = score(completeness_score=1.0, source_trust=0.5, agreement=0.5, freshness=1.0)
assert got == 0.75
+21
View File
@@ -0,0 +1,21 @@
import json
from pathlib import Path
from opengoods.etl.load import ensure_source, load_record
from opengoods.etl.quality import compute_quality
from opengoods.etl.transform import transform
FIXTURE = json.loads((Path(__file__).parent / "fixtures" / "off_product.json").read_text())
def test_quality_score_set_on_load(db_conn):
source_id = ensure_source(db_conn)
rec = transform(FIXTURE)
pid = load_record(db_conn, rec, source_id, FIXTURE)
stored = float(
db_conn.execute("SELECT quality_score FROM product WHERE id = %s", (pid,)).fetchone()[0]
)
assert 0.0 < stored <= 1.0
# The persisted value matches a fresh recomputation.
assert abs(stored - compute_quality(db_conn, pid)) < 1e-9
+16
View File
@@ -0,0 +1,16 @@
from opengoods.etl.state import get_watermark, set_watermark
def test_watermark_roundtrip_and_monotonic(db_conn):
src = "test-source"
assert get_watermark(db_conn, src) == 0
set_watermark(db_conn, src, 100, stats={"loaded": 1})
assert get_watermark(db_conn, src) == 100
# A lower watermark must not rewind progress.
set_watermark(db_conn, src, 50)
assert get_watermark(db_conn, src) == 100
set_watermark(db_conn, src, 150)
assert get_watermark(db_conn, src) == 150
+67
View File
@@ -0,0 +1,67 @@
import json
from decimal import Decimal
from pathlib import Path
import pytest
from opengoods.etl.transform import (
is_valid_gtin,
map_category,
parse_quantity,
transform,
transform_nutriments,
)
FIXTURE = json.loads((Path(__file__).parent / "fixtures" / "off_product.json").read_text())
def test_is_valid_gtin():
assert is_valid_gtin("3017624010701") # real EAN-13
assert is_valid_gtin("5449000000996") # Coca-Cola
assert not is_valid_gtin("3017624010700") # bad check digit
assert not is_valid_gtin("123")
assert not is_valid_gtin("notanumber")
def test_parse_quantity():
assert parse_quantity("400 g") == (Decimal("400"), "g")
assert parse_quantity("1,5 L") == (Decimal("1.5"), "L")
assert parse_quantity("") is None
assert parse_quantity("family size") is None
def test_map_category():
assert map_category({"product_name": "Spring Water"}) == "food.beverages.water"
assert map_category({"categories": "Dark chocolate"}) == "food.snacks.chocolate"
assert map_category({"product_name": "Mystery"}) is None
def test_transform_nutriments_dual_energy():
out = transform_nutriments(FIXTURE["nutriments"])
assert out["energy_kj"] == 2252.0
assert out["energy_kcal"] == 539.0
assert out["fat"] == 30.9
assert out["salt"] == 0.107
def test_transform_nutriments_fills_missing_energy():
out = transform_nutriments({"energy-kcal_100g": 100})
assert out["energy_kj"] == pytest.approx(418.4)
def test_transform_full_record():
rec = transform(FIXTURE)
assert rec is not None
assert rec["gtin"] == "3017624010701"
assert rec["name"] == "Nutella"
assert rec["brand"] == "Ferrero"
assert rec["net_content_unit"] == "g"
assert rec["net_content_canonical"] == Decimal("400")
assert rec["country_of_origin"] == "France"
assert rec["food"]["nutri_score"] == "E"
assert "milk" in rec["food"]["allergens"]
assert rec["image_url"].endswith(".jpg")
def test_transform_drops_unnamed():
assert transform({"code": "0000000000000"}) is None
+47
View File
@@ -0,0 +1,47 @@
from decimal import Decimal
import pytest
from opengoods.units import (
UnitError,
canonical_unit_code,
kcal_to_kj,
kj_to_kcal,
normalize,
)
def test_normalize_mass_kg_to_g():
result = normalize("1.5", "kg")
assert result.dimension == "mass"
assert result.canonical_unit == "g"
assert result.canonical_value == Decimal("1500.0")
def test_normalize_volume_litre_alias():
result = normalize(2, "")
assert result.dimension == "volume"
assert result.canonical_value == Decimal("2000")
assert result.canonical_unit == "ml"
def test_normalize_energy_kcal_to_kj():
result = normalize("539", "kcal")
assert result.dimension == "energy"
assert result.canonical_unit == "kJ"
assert result.canonical_value == Decimal("539") * Decimal("4.184")
def test_canonical_unit_code_alias():
assert canonical_unit_code("公斤") == "kg"
assert canonical_unit_code(" G ") == "g"
def test_unknown_unit_raises():
with pytest.raises(UnitError):
normalize(1, "parsec")
def test_energy_roundtrip():
assert kcal_to_kj(1) == Decimal("4.184")
assert kj_to_kcal(Decimal("4.184")) == Decimal("1")
+21
View File
@@ -0,0 +1,21 @@
DROP TRIGGER IF EXISTS trg_product_sync ON product;
DROP FUNCTION IF EXISTS product_sync_tsv();
DROP TABLE IF EXISTS merge_log;
DROP TABLE IF EXISTS product_source;
DROP TABLE IF EXISTS product_image;
DROP TABLE IF EXISTS product_msrp;
DROP TABLE IF EXISTS food_detail;
DROP TABLE IF EXISTS product;
DROP TABLE IF EXISTS attribute_definition;
DROP TABLE IF EXISTS unit;
DROP TABLE IF EXISTS category_schema;
DROP TABLE IF EXISTS category;
DROP TABLE IF EXISTS manufacturer;
DROP TABLE IF EXISTS brand;
DROP TABLE IF EXISTS source;
DROP EXTENSION IF EXISTS ltree;
DROP EXTENSION IF EXISTS pg_trgm;
-- keep pgcrypto (commonly shared); drop only if you are sure:
-- DROP EXTENSION IF EXISTS pgcrypto;
+182
View File
@@ -0,0 +1,182 @@
-- OpenGoods (天工·商品标签) initial schema.
-- Public-good product information store: facts only, no commerce.
CREATE EXTENSION IF NOT EXISTS pgcrypto; -- gen_random_uuid()
CREATE EXTENSION IF NOT EXISTS pg_trgm; -- fuzzy name search
CREATE EXTENSION IF NOT EXISTS ltree; -- category subtree queries
-- Data sources (Open Food Facts / USDA / GS1 ...) with trust + license.
CREATE TABLE source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
homepage TEXT,
license TEXT,
trust_weight NUMERIC(3,2) NOT NULL DEFAULT 0.5,
notes TEXT,
UNIQUE (name)
);
CREATE TABLE brand (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
normalized_name TEXT NOT NULL,
aliases TEXT[] NOT NULL DEFAULT '{}',
UNIQUE (normalized_name)
);
CREATE TABLE manufacturer (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
normalized_name TEXT NOT NULL,
country VARCHAR(64),
UNIQUE (normalized_name)
);
-- Self-built category tree, each node optionally mapped to a GS1 GPC brick.
CREATE TABLE category (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name_zh TEXT NOT NULL,
name_en TEXT,
parent_id UUID REFERENCES category(id),
path LTREE NOT NULL,
gpc_brick_code VARCHAR(10),
level INT NOT NULL DEFAULT 0,
UNIQUE (path)
);
-- Parameter template / constraints per category.
CREATE TABLE category_schema (
category_id UUID PRIMARY KEY REFERENCES category(id) ON DELETE CASCADE,
required_attributes TEXT[] NOT NULL DEFAULT '{}',
recommended_attributes TEXT[] NOT NULL DEFAULT '{}',
nutriment_basis VARCHAR(16)
);
-- Unit dictionary: each unit maps to a canonical unit within its dimension.
CREATE TABLE unit (
code VARCHAR(16) PRIMARY KEY,
dimension VARCHAR(16) NOT NULL,
canonical VARCHAR(16) NOT NULL,
to_canonical_factor NUMERIC,
aliases TEXT[] NOT NULL DEFAULT '{}',
display TEXT
);
-- Parameter dictionary: standard attribute keys with default unit.
CREATE TABLE attribute_definition (
key VARCHAR(64) PRIMARY KEY,
label_zh TEXT,
label_en TEXT,
dimension VARCHAR(16),
default_unit VARCHAR(16) REFERENCES unit(code),
aliases TEXT[] NOT NULL DEFAULT '{}'
);
CREATE TABLE product (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
gtin VARCHAR(14),
name TEXT NOT NULL,
brand_id UUID REFERENCES brand(id),
manufacturer_id UUID REFERENCES manufacturer(id),
category_id UUID REFERENCES category(id),
gpc_brick_code VARCHAR(10),
net_content_value NUMERIC,
net_content_unit VARCHAR(16),
net_content_canonical NUMERIC,
country_of_origin VARCHAR(64),
shelf_life_days INT,
storage TEXT,
attributes JSONB NOT NULL DEFAULT '{}',
quality_score NUMERIC(4,3) NOT NULL DEFAULT 0,
status VARCHAR(16) NOT NULL DEFAULT 'active',
canonical_id UUID REFERENCES product(id),
search_tsv TSVECTOR,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT product_status_chk CHECK (status IN ('active','merged','deprecated')),
CONSTRAINT product_quality_chk CHECK (quality_score >= 0 AND quality_score <= 1)
);
CREATE TABLE food_detail (
product_id UUID PRIMARY KEY REFERENCES product(id) ON DELETE CASCADE,
ingredients_text TEXT,
ingredients JSONB,
allergens TEXT[] NOT NULL DEFAULT '{}',
additives TEXT[] NOT NULL DEFAULT '{}',
nutriments JSONB,
nutrition_basis VARCHAR(16),
serving_size VARCHAR(32),
nutri_score CHAR(1),
labels TEXT[] NOT NULL DEFAULT '{}',
CONSTRAINT food_basis_chk CHECK (nutrition_basis IS NULL OR nutrition_basis IN ('per_100g','per_100ml','per_serving'))
);
-- Official manufacturer-suggested retail price snapshot (no purchase link).
CREATE TABLE product_msrp (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID NOT NULL REFERENCES product(id) ON DELETE CASCADE,
amount NUMERIC(12,2) NOT NULL,
currency CHAR(3) NOT NULL,
region VARCHAR(8) NOT NULL DEFAULT 'CN',
source_id UUID REFERENCES source(id),
source_url TEXT,
effective_date DATE,
note TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE product_image (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID NOT NULL REFERENCES product(id) ON DELETE CASCADE,
url TEXT NOT NULL,
kind VARCHAR(16) NOT NULL DEFAULT 'other',
license TEXT,
source_id UUID REFERENCES source(id),
CONSTRAINT image_kind_chk CHECK (kind IN ('front','ingredients','nutrition','other'))
);
-- Field-level provenance: which source provided which fields.
CREATE TABLE product_source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
product_id UUID NOT NULL REFERENCES product(id) ON DELETE CASCADE,
source_id UUID REFERENCES source(id),
url TEXT,
fields TEXT[] NOT NULL DEFAULT '{}',
fetched_at TIMESTAMPTZ,
raw JSONB
);
CREATE TABLE merge_log (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
kept_id UUID,
merged_id UUID,
reason TEXT,
actor TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Indexes
CREATE UNIQUE INDEX idx_product_gtin ON product (gtin) WHERE gtin IS NOT NULL;
CREATE INDEX idx_product_name_trgm ON product USING gin (name gin_trgm_ops);
CREATE INDEX idx_product_attrs ON product USING gin (attributes);
CREATE INDEX idx_product_tsv ON product USING gin (search_tsv);
CREATE INDEX idx_product_category ON product (category_id);
CREATE INDEX idx_product_brand ON product (brand_id);
CREATE INDEX idx_product_updated ON product (updated_at);
CREATE INDEX idx_food_nutriments ON food_detail USING gin (nutriments);
CREATE INDEX idx_category_path ON category USING gist (path);
CREATE INDEX idx_msrp_product ON product_msrp (product_id);
CREATE INDEX idx_psource_product ON product_source (product_id);
-- Keep search_tsv and updated_at in sync.
CREATE OR REPLACE FUNCTION product_sync_tsv() RETURNS trigger AS $$
BEGIN
NEW.search_tsv := to_tsvector('simple', coalesce(NEW.name, ''));
NEW.updated_at := now();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_product_sync
BEFORE INSERT OR UPDATE ON product
FOR EACH ROW EXECUTE FUNCTION product_sync_tsv();
+2
View File
@@ -0,0 +1,2 @@
DELETE FROM attribute_definition;
DELETE FROM unit;
+29
View File
@@ -0,0 +1,29 @@
-- Unit dictionary seed. Keep factors aligned with ingestion/opengoods/units.py.
INSERT INTO unit (code, dimension, canonical, to_canonical_factor, aliases, display) VALUES
('mg', 'mass', 'g', 0.001, ARRAY['毫克'], 'mg'),
('g', 'mass', 'g', 1, ARRAY['','gram','grams'], 'g'),
('kg', 'mass', 'g', 1000, ARRAY['kgs','千克','公斤'], 'kg'),
('ml', 'volume', 'ml', 1, ARRAY['毫升','milliliter'], 'mL'),
('cl', 'volume', 'ml', 10, ARRAY['厘升'], 'cL'),
('l', 'volume', 'ml', 1000, ARRAY['L','','litre','liter'], 'L'),
('kj', 'energy', 'kJ', 1, ARRAY['kJ','千焦'], 'kJ'),
('kcal', 'energy', 'kJ', 4.184, ARRAY['千卡','大卡'], 'kcal'),
('pct', 'ratio', 'pct', 1, ARRAY['%','percent','百分比'], '%'),
('unit', 'count', 'unit',1, ARRAY['','','pcs','piece'], ''),
('mm', 'length', 'mm', 1, ARRAY['毫米'], 'mm'),
('cm', 'length', 'mm', 10, ARRAY['厘米'], 'cm'),
('day', 'duration', 'day', 1, ARRAY['','','days'], 'day')
ON CONFLICT (code) DO NOTHING;
-- A few common food attribute definitions referencing the unit dictionary.
INSERT INTO attribute_definition (key, label_zh, label_en, dimension, default_unit, aliases) VALUES
('energy', '能量', 'Energy', 'energy', 'kj', ARRAY['energy_kj']),
('proteins', '蛋白质', 'Proteins', 'mass', 'g', ARRAY['protein']),
('fat', '脂肪', 'Fat', 'mass', 'g', ARRAY['fats']),
('saturated_fat', '饱和脂肪','Saturated fat','mass', 'g', ARRAY['saturated-fat']),
('carbohydrates', '碳水化合物','Carbohydrates','mass', 'g', ARRAY['carbs']),
('sugars', '', 'Sugars', 'mass', 'g', ARRAY['sugar']),
('salt', '', 'Salt', 'mass', 'g', ARRAY['sodium_salt']),
('net_content', '净含量', 'Net content', NULL, NULL, ARRAY['quantity'])
ON CONFLICT (key) DO NOTHING;

Some files were not shown because too many files have changed in this diff Show More