672 lines
16 KiB
Markdown
672 lines
16 KiB
Markdown
# V2 技术架构 — Batch Auto-Import
|
||
|
||
日期：2026-05-22
|
||
状态：设计中
|
||
关联文档：
|
||
|
||
- `docs/2026-05-21-BATCH_AUTO_IMPORT_SPEC.md`
|
||
- `docs/2026-05-21-BATCH_AUTO_IMPORT_TDD_PLAN.md`
|
||
- `docs/openapi.yaml`
|
||
|
||
## 1. 文档目标
|
||
|
||
这份文档只回答 4 个问题：
|
||
|
||
1. V2 的后端组件如何分层
|
||
2. 运行态状态如何持久化并保证稳定推进
|
||
3. 结果页到底展示什么、从哪张表读
|
||
4. 如何把 run / item / retry / advisory / validation 收口成单一真相
|
||
|
||
## 2. 核心原则
|
||
|
||
V2 必须同时满足：
|
||
|
||
1. **可恢复**：控制面重启后 run/item 仍可查看，unfinished item 可继续推进
|
||
2. **可观测**：页面和 API 不依赖日志拼接，而依赖 canonical state store
|
||
3. **可解释**：warning/broken 必须给出可读原因
|
||
4. **可分层**：Probe、Provision、Confirm、Validate 各司其职
|
||
5. **可兼容**：针对第三方 OpenAI-compatible 上游，显式记录 transport + model capability
|
||
6. **可复用**：重复导入命中同 URL + 同模型家族时，优先复用已有 provider/account，而不是重复创建
|
||
|
||
## 3. Canonical runtime model
|
||
|
||
### 3.1 单一真相
|
||
|
||
V2 的单一真相是控制面自己的三类表：
|
||
|
||
- `import_runs`
|
||
- `import_run_items`
|
||
- `import_run_item_events`
|
||
|
||
现有表：
|
||
|
||
- `import_batches`
|
||
- `import_batch_items`
|
||
- `probe_results`
|
||
- `access_closure_records`
|
||
- `managed_resources`
|
||
|
||
在 V2 中只承担两类职责：
|
||
|
||
1. 资源关联和 legacy 追溯
|
||
2. 对现有 v1 行为的兼容
|
||
|
||
它们不再是结果页与 V2 API 的主数据源。
|
||
|
||
### 3.2 状态归属
|
||
|
||
- `run.state`：只属于 `import_runs`
|
||
- `item.current_stage / confirmation_status / access_status`：只属于 `import_run_items`
|
||
- `retry trail / advisory / stage transition`：只属于 `import_run_item_events`
|
||
|
||
## 4. 组件分层
|
||
|
||
```text
|
||
operator input
|
||
↓
|
||
Batch Import API / CLI
|
||
↓
|
||
BatchImportService
|
||
├── Reuse Preflight
|
||
├── Probe Layer
|
||
├── Capability Profiler
|
||
├── Provision Adapter
|
||
├── RunStateStore
|
||
└── ConfirmationQueue
|
||
↓
|
||
ConfirmationWorker
|
||
↓
|
||
ValidationService
|
||
↓
|
||
ResultProjection
|
||
↓
|
||
HTTP API / Result Pages / CLI wait output
|
||
```
|
||
|
||
### 4.1 Batch Import API / CLI
|
||
|
||
职责：
|
||
|
||
- 接收 `BatchImportRunRequest`
|
||
- 校验 `access_mode` 必填输入
|
||
- 创建 run 与 item
|
||
- 触发 `BatchImportService`
|
||
- 可选等待一个短暂窗口，但不负责长期 confirm
|
||
|
||
### 4.2 BatchImportService
|
||
|
||
职责：
|
||
|
||
- 执行 Reuse Preflight
|
||
- 执行 Stage 1 Probe
|
||
- 执行 Stage 2 Provision
|
||
- 把 item 推进到 `confirm`
|
||
- 将确认任务交给后台 worker
|
||
|
||
不负责：
|
||
|
||
- 把每个 item 长时间阻塞在请求线程里等待最终稳定
|
||
|
||
### 4.3 ConfirmationWorker
|
||
|
||
职责：
|
||
|
||
- 扫描需要确认的 item
|
||
- 吸收 probe race / warmup 窗口
|
||
- 将 item 从 `pending` 推进到 `confirmed/advisory/failed`
|
||
- 推进到 validate 阶段
|
||
|
||
### 4.4 ValidationService
|
||
|
||
职责：
|
||
|
||
- 唯一写入 `access_status`
|
||
- 使用宿主 gateway 真实 `/v1/chat/completions`
|
||
- 产生 `active/degraded/broken`
|
||
|
||
### 4.5 ResultProjection
|
||
|
||
职责：
|
||
|
||
- 将底层运行态投影为页面/API 视图
|
||
- 统一状态 badge / warning 文案 / item 摘要
|
||
|
||
## 5. State machine
|
||
|
||
### 5.1 Run state
|
||
|
||
`run.state`：
|
||
|
||
- `running`
|
||
- `completed`
|
||
- `completed_with_warnings`
|
||
- `failed`
|
||
- `cancelled`
|
||
|
||
Run 级 projection 规则：
|
||
|
||
- 只要存在 `broken` item，且未完成恢复，则 run 可能 `failed`
|
||
- 无 broken 但存在 advisory/degraded item，则 run 为 `completed_with_warnings`
|
||
- 全部 item 为 `confirmed/active`，则 run 为 `completed`
|
||
|
||
### 5.2 Item lifecycle
|
||
|
||
`item.current_stage`：
|
||
|
||
- `probe`
|
||
- `provision`
|
||
- `confirm`
|
||
- `validate`
|
||
- `done`
|
||
|
||
`item.confirmation_status`：
|
||
|
||
- `pending`
|
||
- `confirmed`
|
||
- `advisory`
|
||
- `failed`
|
||
|
||
`item.access_status`：
|
||
|
||
- `unknown`
|
||
- `active`
|
||
- `degraded`
|
||
- `broken`
|
||
|
||
`item.matched_account_state`：
|
||
|
||
- `none`
|
||
- `active`
|
||
- `disabled`
|
||
- `deprecated`
|
||
- `broken`
|
||
|
||
`item.account_resolution`：
|
||
|
||
- `created`
|
||
- `reused`
|
||
- `reactivated`
|
||
- `replaced`
|
||
|
||
### 5.3 State ownership
|
||
|
||
| 字段 | 写入者 |
|
||
|---|---|
|
||
| `current_stage` | service / worker / validation |
|
||
| `confirmation_status` | confirmation worker |
|
||
| `access_status` | validation service |
|
||
| `run.state` | result projection / run aggregator |
|
||
|
||
## 6. Request architecture
|
||
|
||
### 6.1 Canonical request
|
||
|
||
```text
|
||
BatchImportRunRequest
|
||
- host_id
|
||
- mode
|
||
- access_mode
|
||
- confirm_wait_timeout_sec
|
||
- subscription_users
|
||
- subscription_days
|
||
- probe_api_key
|
||
- entries[]
|
||
```
|
||
|
||
### 6.2 Access-mode matrix
|
||
|
||
| access_mode | 必填字段 | 用途 |
|
||
|---|---|---|
|
||
| `subscription` | `subscription_users`, `subscription_days` | 订阅绑定和闭环验证 |
|
||
| `self_service` | `probe_api_key` | gateway key 验证 |
|
||
|
||
这一步必须在入口层就校验，不能把不完整请求放进 worker 后再失败。
|
||
|
||
## 7. Capability architecture
|
||
|
||
### 7.1 两层 capability
|
||
|
||
V2 不再把 capability 当成“一个 key 一个总画像”，而是拆成：
|
||
|
||
1. `transport_profile`
|
||
2. `model_profiles[]`
|
||
|
||
### 7.2 为什么必须按模型维度记录
|
||
|
||
目标是满足“快速匹配兼容模型”的运营需求。
|
||
如果只有 upstream 级总画像，无法表达：
|
||
|
||
- 同 upstream 下模型 A 可 stream，模型 B 不可
|
||
- 同 upstream 下模型 A 支持 reasoning 字段，模型 B 不支持
|
||
- 同 upstream 下模型 A smoke 通过，模型 B 失败
|
||
|
||
### 7.3 Canonical JSON
|
||
|
||
```json
|
||
{
|
||
"transport_profile": {
|
||
"supports_openai_models": true,
|
||
"supports_openai_chat_completions": true,
|
||
"supports_openai_responses": false,
|
||
"supports_anthropic_messages": false,
|
||
"auth_style": "bearer",
|
||
"model_id_style": "vendor_prefixed",
|
||
"known_advisories": [
|
||
"responses_unsupported_but_chat_ok",
|
||
"initial_probe_race_expected"
|
||
]
|
||
},
|
||
"model_profiles": [
|
||
{
|
||
"raw_model_id": "kimi-k2.6",
|
||
"normalized_model_id": "kimi-k2.6",
|
||
"canonical_model_family": "kimi-k2.6",
|
||
"supports_stream": true,
|
||
"supports_tools": "unknown",
|
||
"supports_reasoning_fields": "unknown",
|
||
"smoke_chat_ok": true
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
### 7.4 Canonical model family 的作用
|
||
|
||
`normalized_model_id` 只能解决字符串归一化，不能稳定回答跨中转的“是否同一模型家族”。
|
||
|
||
V2 需要额外记录 `canonical_model_family`，用于识别这类情况：
|
||
|
||
- `kimi 2.6`
|
||
- `kimi-2.6`
|
||
- `kimi-k2.6`
|
||
|
||
它们可能属于同一个模型家族，应支持：
|
||
|
||
1. 重复导入时快速匹配
|
||
2. 已存在 provider 只 patch 别名映射，不重复 provision
|
||
3. 结果页解释“为何这次被直接复用”
|
||
|
||
## 8. State store schema
|
||
|
||
### 8.1 `import_runs`
|
||
|
||
```text
|
||
run_id TEXT PRIMARY KEY
|
||
mode TEXT NOT NULL
|
||
access_mode TEXT NOT NULL
|
||
state TEXT NOT NULL
|
||
total_items INTEGER NOT NULL
|
||
completed_items INTEGER NOT NULL
|
||
active_items INTEGER NOT NULL
|
||
degraded_items INTEGER NOT NULL
|
||
broken_items INTEGER NOT NULL
|
||
warning_items INTEGER NOT NULL
|
||
started_at DATETIME NOT NULL
|
||
updated_at DATETIME NOT NULL
|
||
finished_at DATETIME NULL
|
||
```
|
||
|
||
### 8.2 `import_run_items`
|
||
|
||
```text
|
||
item_id TEXT PRIMARY KEY
|
||
run_id TEXT NOT NULL
|
||
base_url TEXT NOT NULL
|
||
provider_id TEXT NOT NULL
|
||
api_key_fingerprint TEXT NOT NULL
|
||
requested_models_json TEXT NOT NULL
|
||
raw_models_json TEXT NOT NULL
|
||
normalized_models_json TEXT NOT NULL
|
||
canonical_model_families_json TEXT NOT NULL
|
||
resolved_smoke_model TEXT NULL
|
||
recommended_models_json TEXT NOT NULL
|
||
capability_profile_json TEXT NOT NULL
|
||
current_stage TEXT NOT NULL
|
||
confirmation_status TEXT NOT NULL
|
||
access_status TEXT NOT NULL
|
||
matched_account_state TEXT NOT NULL
|
||
account_resolution TEXT NOT NULL
|
||
provision_reused INTEGER NOT NULL
|
||
reused_from_provider_id TEXT NULL
|
||
reused_from_account_id INTEGER NULL
|
||
channel_id INTEGER NULL
|
||
account_id INTEGER NULL
|
||
retry_count INTEGER NOT NULL
|
||
confirmation_attempts INTEGER NOT NULL
|
||
last_retry_at DATETIME NULL
|
||
next_retry_at DATETIME NULL
|
||
lease_owner TEXT NULL
|
||
lease_until DATETIME NULL
|
||
advisory_messages_json TEXT NOT NULL
|
||
last_error_stage TEXT NULL
|
||
last_error TEXT NULL
|
||
legacy_batch_id INTEGER NULL
|
||
legacy_provider_id TEXT NULL
|
||
created_at DATETIME NOT NULL
|
||
updated_at DATETIME NOT NULL
|
||
```
|
||
|
||
关键约束：
|
||
|
||
- `resolved_smoke_model` 可为 `NULL`，因为 Stage 1 可能失败
|
||
- `channel_id/account_id` 可为 `NULL`，因为 Stage 2 可能未开始
|
||
- `access_status` 初始必须允许 `unknown`
|
||
- `api_key_fingerprint` 只存指纹，不存明文 key
|
||
- `canonical_model_families_json` 是 Reuse Preflight 的核心输入，而不是附带展示字段
|
||
- `matched_account_state` 用于结果页直接展示“重复已启用 / 已弃用待启用 / 已重新启用”
|
||
- `account_resolution` 用于解释这次 item 最终是创建、复用、快速启用还是替换
|
||
|
||
### 8.3 为什么必须有 key fingerprint
|
||
|
||
只靠 `provider_id` 不能稳定区分：
|
||
|
||
- 同一个 URL 下是否是同一把 key
|
||
- 是真正的重复导入，还是同 URL 的新账号
|
||
|
||
因此 V2 必须显式落：
|
||
|
||
- `api_key_fingerprint`
|
||
- `provision_reused`
|
||
- `reused_from_provider_id`
|
||
- `reused_from_account_id`
|
||
|
||
这样 Reuse Preflight 才能先按：
|
||
|
||
1. `host_id + provider_id`
|
||
2. `host_id + base_url + api_key_fingerprint`
|
||
3. `canonical_model_families`
|
||
|
||
做稳定判定。
|
||
|
||
### 8.4 已存在账号状态的标准化
|
||
|
||
对命中的既有账号，V2 需要统一投影为：
|
||
|
||
- `active`
|
||
- 当前账号已启用，可直接使用
|
||
- `disabled`
|
||
- 当前账号被停用，但可尝试快速启用
|
||
- `deprecated`
|
||
- 当前账号不推荐继续调度，但仍可由 operator 重新启用
|
||
- `broken`
|
||
- 当前账号不可直接复用
|
||
|
||
对应本次导入的处理结果：
|
||
|
||
- `created`
|
||
- `reused`
|
||
- `reactivated`
|
||
- `replaced`
|
||
|
||
### 8.5 `import_run_item_events`
|
||
|
||
```text
|
||
event_id TEXT PRIMARY KEY
|
||
run_id TEXT NOT NULL
|
||
item_id TEXT NOT NULL
|
||
event_type TEXT NOT NULL
|
||
stage TEXT NOT NULL
|
||
attempt INTEGER NOT NULL
|
||
message TEXT NOT NULL
|
||
payload_json TEXT NOT NULL
|
||
created_at DATETIME NOT NULL
|
||
```
|
||
|
||
事件类型示例：
|
||
|
||
- `stage_transition`
|
||
- `retry_scheduled`
|
||
- `advisory_added`
|
||
- `validation_result`
|
||
|
||
### 8.6 为什么必须有 event 表
|
||
|
||
仅靠 `retry_count` 不足以支撑结果页要求。
|
||
页面要能展示：
|
||
|
||
- 第几次重试
|
||
- 在哪个阶段重试
|
||
- 为什么进入 advisory
|
||
- warning/broken 最终解释
|
||
|
||
这必须依赖 event trail。
|
||
|
||
## 9. Confirmation worker design
|
||
|
||
### 9.1 轮询条件
|
||
|
||
worker 每次 Tick 只捞：
|
||
|
||
- `current_stage='confirm'`
|
||
- `confirmation_status='pending'`
|
||
- `next_retry_at IS NULL OR next_retry_at <= now`
|
||
- `lease_until IS NULL OR lease_until < now`
|
||
|
||
### 9.2 Lease 机制
|
||
|
||
为避免多个 worker 重复确认，同一 item 需要：
|
||
|
||
- `lease_owner`
|
||
- `lease_until`
|
||
|
||
worker 成功抢到 lease 后才能执行 confirm。
|
||
|
||
### 9.3 Retry policy
|
||
|
||
建议默认：
|
||
|
||
- probe race `403`：advisory，不立即失败
|
||
- `503 no available accounts`：短暂指数退避，最多 N 次
|
||
- definitive `401/403 unauthorized`：立即失败
|
||
|
||
每次 retry 都写 event。
|
||
|
||
### 9.4 Restart safety
|
||
|
||
V2 要求：
|
||
|
||
- worker 重启后自动接管过期 lease 的 item
|
||
- unfinished item 不需要人工恢复
|
||
- CLI 超时退出不影响后台继续推进
|
||
|
||
## 10. Validation architecture
|
||
|
||
### 10.1 唯一职责
|
||
|
||
ValidationService 只做一件事：
|
||
|
||
- 对已经完成 confirm 的 item 执行最终 gateway completion 验证
|
||
|
||
### 10.2 Outcome mapping
|
||
|
||
| confirmation_status | gateway result | access_status |
|
||
|---|---|---|
|
||
| `confirmed` | `200` | `active` |
|
||
| `advisory` | `200` | `active` |
|
||
| `confirmed`/`advisory` | transient but exhausted | `degraded` |
|
||
| `failed` | any | `broken` |
|
||
| any | definitive invalid path | `broken` |
|
||
|
||
## 11. Result projection
|
||
|
||
### 11.1 Projection fields
|
||
|
||
结果页/API 不再自己拼原始表字段，Projection 层统一输出：
|
||
|
||
- run summary
|
||
- item table row
|
||
- item detail
|
||
- warning explanation
|
||
- badge color mapping
|
||
|
||
### 11.2 Warning 文案模板
|
||
|
||
建议至少固化：
|
||
|
||
- `responses_unsupported_but_chat_ok`
|
||
- `该上游不支持 /v1/responses，系统已自动回退到 /v1/chat/completions`
|
||
- `initial_probe_race_expected`
|
||
- `账号创建后宿主异步探测尚未稳定，首次 /test 已按 advisory 处理`
|
||
- `gateway_warmup_retry_succeeded`
|
||
- `初次调度出现 no available accounts，短暂重试后已恢复`
|
||
- `provision_reused`
|
||
- `已检测到同 URL + 同模型家族 + 健康账号，系统直接复用已有 provider`
|
||
- `patch_only_new_aliases`
|
||
- `模型属于已覆盖家族，仅补充别名映射与定价，不重复创建资源`
|
||
- `duplicate_active_account`
|
||
- `该账号已存在且处于启用状态，本次未重复创建，直接复用`
|
||
- `deprecated_account_reactivated`
|
||
- `该账号此前处于弃用/停用状态，本次已快速启用并重新确认`
|
||
|
||
## 12. Result pages
|
||
|
||
### 12.1 页面列表
|
||
|
||
- `/batch-import/runs`
|
||
- `/batch-import/runs/{run_id}`
|
||
|
||
### 12.2 列表页字段
|
||
|
||
| 列 | 数据源 |
|
||
|---|---|
|
||
| `Run ID` | `import_runs.run_id` |
|
||
| `State` | projection |
|
||
| `Mode` | `import_runs.mode` |
|
||
| `Access Mode` | `import_runs.access_mode` |
|
||
| `Total/Active/Degraded/Broken/Warning` | `import_runs.*` |
|
||
| `Started/Finished` | `import_runs.*` |
|
||
|
||
筛选值使用 canonical 枚举：
|
||
|
||
- `running`
|
||
- `completed`
|
||
- `completed_with_warnings`
|
||
- `failed`
|
||
- `cancelled`
|
||
|
||
页面可把 `completed_with_warnings` 渲染成 badge 文案 `warning`。
|
||
|
||
### 12.3 详情页字段
|
||
|
||
Item 行至少包含：
|
||
|
||
- `item_id`
|
||
- `base_url`
|
||
- `provider_id`
|
||
- `api_key_fingerprint`
|
||
- `requested_models`
|
||
- `canonical_model_families`
|
||
- `resolved_smoke_model`
|
||
- `current_stage`
|
||
- `confirmation_status`
|
||
- `access_status`
|
||
- `matched_account_state`
|
||
- `account_resolution`
|
||
- `provision_reused`
|
||
- `retry_count`
|
||
- `last_error_stage`
|
||
- `last_error`
|
||
|
||
Item 详情至少包含：
|
||
|
||
- `raw_models`
|
||
- `normalized_models`
|
||
- `canonical_model_families`
|
||
- `recommended_models`
|
||
- `transport_profile`
|
||
- `model_profiles`
|
||
- `matched_account_state`
|
||
- `account_resolution`
|
||
- `reused_from_provider_id`
|
||
- `reused_from_account_id`
|
||
- `channel_id`
|
||
- `account_id`
|
||
- `advisory_messages`
|
||
- `event trail`
|
||
|
||
## 13. HTTP API
|
||
|
||
### 13.1 V2 canonical endpoints
|
||
|
||
```text
|
||
POST /api/batch-import/runs
|
||
GET /api/batch-import/runs
|
||
GET /api/batch-import/runs/{run_id}
|
||
GET /api/batch-import/runs/{run_id}/items
|
||
GET /api/batch-import/runs/{run_id}/items/{item_id}
|
||
```
|
||
|
||
### 13.2 Legacy endpoints
|
||
|
||
以下继续保留，但在 OpenAPI 中标为 `deprecated` 或 `legacy`：
|
||
|
||
- `/api/import-batches/{batchID}`
|
||
- `/api/import-batches/{batchID}/rollback`
|
||
|
||
## 14. Backend module mapping
|
||
|
||
建议模块边界如下：
|
||
|
||
### `internal/batch/run_state.go`
|
||
|
||
- run/item/event 仓储
|
||
- lease 管理
|
||
- run 聚合基础能力
|
||
|
||
### `internal/batch/status_projection.go`
|
||
|
||
- 页面/API 视图 projection
|
||
- state/badge/warning 文案映射
|
||
|
||
### `internal/batch/service.go`
|
||
|
||
- run 创建
|
||
- Stage 1 + Stage 2
|
||
- item 入队 confirm
|
||
|
||
### `internal/batch/confirmation.go`
|
||
|
||
- worker polling
|
||
- confirm logic
|
||
- retry scheduling
|
||
|
||
### `internal/batch/validation.go`
|
||
|
||
- final gateway validation
|
||
- final access_status write
|
||
|
||
### `internal/app/http_batch_import.go`
|
||
|
||
- create/list/detail APIs
|
||
|
||
### `internal/app/http_batch_runs.go`
|
||
|
||
- 结果页渲染
|
||
|
||
## 15. Implementation boundary
|
||
|
||
第一阶段实现可以暂不做：
|
||
|
||
- WebSocket 实时刷新
|
||
- 多 worker 分布式协调
|
||
- 跨 host 汇总看板
|
||
|
||
但以下不能再降级为“后续再说”：
|
||
|
||
- canonical runtime tables
|
||
- confirmation worker
|
||
- lease + retry scheduling
|
||
- event trail
|
||
- 结果 API
|
||
|
||
## 16. Architecture acceptance
|
||
|
||
该架构只有满足以下条件，才算真正达到 V2 目标：
|
||
|
||
1. 输入契约、状态枚举、API、页面字段完全一致
|
||
2. run/item/event 可持久化并支持重启恢复
|
||
3. 结果页只读 canonical state store
|
||
4. transport capability 与 model capability 都能表达
|
||
5. `confirmation_status` 与 `access_status` 责任边界清楚
|
||
6. 第三方兼容 upstream 的异步窗口不会直接把可用链路打成最终失败
|