lixu/goods
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

bypos-collector: 持久化采集记录去重 + Goods 系统批量导入 #24

Open
lixu wants to merge 4 commits from devin/1782276014-persistent-history-import into main
pull from: devin/1782276014-persistent-history-import
merge into: lixu:main
Conversation 0 Commits 4 Files Changed 9
+667 -57
lixu commented 2026-06-24 12:45:08 +08:00
Owner
Copy Link
Copy Source

Summary

两大功能 + 之前的改进。

1. 跨次采集去重

采集器维护 collected.txt 持久化已查询条码集。启动时 loadCollected() 读入历史，采集后追加写入。下次运行同一条码自动跳过。

Web UI 显示「历史已采集: N 个条码」徽章。

2. Goods 系统批量导入

采集器端：POST /api/import 接收 {api_url, username, password}，登录 Goods 获取 JWT，读取 products.jsonl，POST 到 /api/import/bypos。

Goods API 端：POST /api/import/bypos（需认证）：

  • 逐条 savepoint 隔离 upsert：product 按 GTIN upsert + attributes jsonb merge
  • 自动创建 source / manufacturer
  • 插入 product_msrp(CNY)、product_barcode、product_source
  • recomputeQualityTx 重算质量分
  • 返回 {loaded, skipped, errored}

Web UI 新增 ④ 导入区域：填写 Goods 后台地址 + 管理员凭据一键导入。

之前的改进

  • sdogid 不硬编码（$BYPOS_SDOGID / -sdogid / Web 输入框）
  • 网络重试 3 次指数退避
  • CSV 补全 retmsg/source 列
  • 端口冲突自动选可用端口
  • Windows 日志文件 + 错误保持窗口
  • Go 单元测试
## Summary 两大功能 + 之前的改进。 ### 1. 跨次采集去重 采集器维护 `collected.txt` 持久化已查询条码集。启动时 `loadCollected()` 读入历史，采集后追加写入。下次运行同一条码自动跳过。 Web UI 显示「历史已采集: N 个条码」徽章。 ### 2. Goods 系统批量导入 **采集器端**：`POST /api/import` 接收 `{api_url, username, password}`，登录 Goods 获取 JWT，读取 `products.jsonl`，POST 到 `/api/import/bypos`。 **Goods API 端**：`POST /api/import/bypos`（需认证）： - 逐条 savepoint 隔离 upsert：product 按 GTIN upsert + attributes jsonb merge - 自动创建 source / manufacturer - 插入 product_msrp(CNY)、product_barcode、product_source - recomputeQualityTx 重算质量分 - 返回 `{loaded, skipped, errored}` Web UI 新增 ④ 导入区域：填写 Goods 后台地址 + 管理员凭据一键导入。 ### 之前的改进 - sdogid 不硬编码（$BYPOS_SDOGID / -sdogid / Web 输入框） - 网络重试 3 次指数退避 - CSV 补全 retmsg/source 列 - 端口冲突自动选可用端口 - Windows 日志文件 + 错误保持窗口 - Go 单元测试
lixu added 4 commits 2026-06-24 12:45:09 +08:00
bypos-collector: 移除硬编码 sdogid、添加重试、补全 CSV 导出、增加测试
CI / Go (api) (pull_request) Failing after 1s
Details
CI / Python (ingestion) (pull_request) Failing after 0s
Details
CI / Migrations (postgres) (pull_request) Failing after 2s
Details
21c895a008 
改进内容:
- sdogid 不再硬编码，优先从 $BYPOS_SDOGID 环境变量读取，其次 -sdogid 参数，
  也可在 Web 控制台输入框填写；三者都未设时启动警告、采集报错
- 网络错误自动重试（最多 3 次，指数退避 500ms/1s/2s）
- Web 控制台采集参数区增加 sdogid 输入框
- CSV 导出补全 retmsg 和 source 两列
- 新增 Go 单元测试（ean13Check、md5hex、sanitizeBarcodes、resolveSdogID 等）
- README 更新配置说明

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fix: Windows 闪退 — 增加日志文件输出、错误时保持窗口
CI / Go (api) (pull_request) Failing after 2s
Details
CI / Python (ingestion) (pull_request) Failing after 0s
Details
CI / Migrations (postgres) (pull_request) Failing after 2s
Details
5a641705dd 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fix: 端口占用时自动选择可用端口，不再闪退
CI / Go (api) (pull_request) Failing after 1s
Details
CI / Python (ingestion) (pull_request) Failing after 1s
Details
CI / Migrations (postgres) (pull_request) Failing after 1s
Details
cbb0968256 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
feat: 持久化采集记录去重 + Goods系统批量导入
CI / Go (api) (pull_request) Successful in 55s
Details
CI / Python (ingestion) (pull_request) Failing after 31s
Details
CI / Migrations (postgres) (pull_request) Failing after 32s
Details
b08495b6f6 
- collected.txt 跨文件/跨次记录已查询条码，避免重复采集
- Web UI 显示历史已采集条码计数
- Goods API 新增 POST /api/import/bypos 批量导入端点（含upsert）
- 采集器 Web UI 新增一键导入按钮（登录+上传JSONL）
- Windows exe 重新编译

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Some checks are pending
CI / Go (api) (pull_request) Successful in 55s
Details
CI / Python (ingestion) (pull_request) Failing after 31s
Details
CI / Migrations (postgres) (pull_request) Failing after 32s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin devin/1782276014-persistent-history-import:devin/1782276014-persistent-history-import
git checkout devin/1782276014-persistent-history-import
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lixu
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lixu/goods#24
Delete Branch "devin/1782276014-persistent-history-import"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?