840 lines
32 KiB
Markdown
840 lines
32 KiB
Markdown
# Phase 3a Web API 完整化 Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax.
|
||
|
||
**Goal:** 把 Phase 2 的同步轻量 API 升级为异步、可登录、带进度推送、因子报告完整的后端。
|
||
|
||
**Architecture:** FastAPI + ProcessPoolExecutor（spawn context）做异步回测，JWT 单用户鉴权，WS 连接池推阶段进度，analyzer 补完整 alphalens tears pipeline。
|
||
|
||
**Tech Stack:** FastAPI + uvicorn、concurrent.futures.ProcessPoolExecutor、PyJWT、websockets（FastAPI TestClient）、alphalens（容器）。
|
||
|
||
## Global Constraints
|
||
|
||
- **vnpy 零改造**：源码 `vnpy_v4.4.0/`，`sys.path.insert(0, _VNPY_SRC)`（参照 sanguo_data/datareader.py:6-9）
|
||
- **Python 3.14 本地 / 3.10 容器**；本地无 polars/alphalens/vnpy_ctastrategy → lazy import + mock 测试（参照 Phase 2 模式）
|
||
- **Phase 2 接口复用**：`result_store`（save_result/load_result/BacktestResult）、`cta_engine.run_cta_backtest`、`cta_optimizer.run_cta_optimization`、`registry/library`、`runner.Orchestrator`
|
||
- **ProcessPoolExecutor 用 spawn context**（vnpy/alpha 已用 spawn，兼容）
|
||
- **WS 简单连接池**（dict[task_id, set[WebSocket]]），不做重连/心跳（YAGNI）
|
||
- **JWT 单用户**：用户名/密码 hash/jwt_secret 配 `config/backtest.yaml`
|
||
- **测试策略**：本地 Python 3.14 跑 mock 测试 + 容器 Python 3.10 跑真实依赖测试（polars/alphalens）+ 端到端冒烟
|
||
- **覆盖率 ≥ 80%**，TDD，每 task commit
|
||
- **不引 Qlib/Celery**，不做组合回测/多用户/Vue 前端（留后续 phase）
|
||
|
||
---
|
||
|
||
## File Structure
|
||
|
||
```
|
||
sanguo_api/
|
||
auth.py # 新：JWT 单用户（create_token/verify_token）
|
||
ws.py # 新：WS 连接池（ConnectionManager）
|
||
routes.py # 改：+ /auth/login、optimize/factor 真调用、JWT 依赖、WS route
|
||
app.py # 改：register auth/ws，JWT 依赖生效
|
||
sanguo_orchestrator/
|
||
pool.py # 改：TaskPool 加 ProcessPoolExecutor + stage 追踪
|
||
runner.py # 改：async submit_* + on_stage 回调
|
||
sanguo_factor/
|
||
alpha_lab.py # 改：补 compute_factors（add_feature + prepare_data + fetch_raw）
|
||
analyzer.py # 改：tears pipeline 完整化
|
||
config/backtest.yaml # 改：+ auth + pool 配置
|
||
tests/api/{test_auth.py, test_ws.py, test_routes.py} # 新/改
|
||
tests/orchestrator/{test_pool.py, test_runner.py} # 改
|
||
tests/factor/{test_alpha_lab.py, test_analyzer.py} # 改
|
||
scripts/smoke_phase3a.py # 新
|
||
```
|
||
|
||
---
|
||
|
||
## Task 1: config 扩展 + sanguo_api/auth.py（JWT 单用户）
|
||
|
||
**Files:**
|
||
- Modify: `config/backtest.yaml`
|
||
- Create: `sanguo_api/auth.py`
|
||
- Test: `tests/api/test_auth.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `create_token(username: str) → str`、`verify_token(token: str) → str`（返回 username，FastAPI 依赖用法 `Depends(verify_token)`，无效 raise HTTPException 401）、`hash_password(pw) → str`、`verify_password(pw, hash) → bool`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/api/test_auth.py
|
||
import pytest
|
||
|
||
def test_create_and_verify_token():
|
||
from sanguo_api.auth import create_token, verify_token, set_jwt_config
|
||
set_jwt_config(secret="test_secret", expire_minutes=60)
|
||
token = create_token("admin")
|
||
assert isinstance(token, str)
|
||
assert verify_token(token) == "admin"
|
||
|
||
def test_verify_token_invalid_raises_401():
|
||
from fastapi import HTTPException
|
||
from sanguo_api.auth import verify_token, set_jwt_config
|
||
set_jwt_config(secret="test_secret", expire_minutes=60)
|
||
with pytest.raises(HTTPException) as exc:
|
||
verify_token("invalid.token.here")
|
||
assert exc.value.status_code == 401
|
||
|
||
def test_hash_and_verify_password():
|
||
from sanguo_api.auth import hash_password, verify_password
|
||
h = hash_password("mypass")
|
||
assert h != "mypass"
|
||
assert verify_password("mypass", h) is True
|
||
assert verify_password("wrong", h) is False
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败** — `pip install PyJWT passlib[bcrypt] --break-system-packages`（若无），`pytest tests/api/test_auth.py -v` → FAIL
|
||
|
||
- [ ] **Step 3: 实现 auth.py**
|
||
|
||
```python
|
||
# sanguo_api/auth.py
|
||
"""JWT 单用户认证。secret/用户名/密码 hash 来自 config/backtest.yaml。"""
|
||
import os
|
||
from datetime import datetime, timedelta, timezone
|
||
import jwt
|
||
from passlib.context import CryptContext
|
||
from fastapi import HTTPException, status
|
||
|
||
_CONFIG = {"secret": "change-me", "expire_minutes": 60, "algorithm": "HS256"}
|
||
_pwd = CryptContext(schemes=["bcrypt"], deprecated="auto")
|
||
|
||
|
||
def set_jwt_config(secret: str, expire_minutes: int, algorithm: str = "HS256"):
|
||
_CONFIG.update(secret=secret, expire_minutes=expire_minutes, algorithm=algorithm)
|
||
|
||
|
||
def hash_password(password: str) -> str:
|
||
return _pwd.hash(password)
|
||
|
||
|
||
def verify_password(password: str, password_hash: str) -> bool:
|
||
return _pwd.verify(password, password_hash)
|
||
|
||
|
||
def create_token(username: str) -> str:
|
||
payload = {
|
||
"sub": username,
|
||
"exp": datetime.now(timezone.utc) + timedelta(minutes=_CONFIG["expire_minutes"]),
|
||
}
|
||
return jwt.encode(payload, _CONFIG["secret"], algorithm=_CONFIG["algorithm"])
|
||
|
||
|
||
def verify_token(token: str) -> str:
|
||
try:
|
||
payload = jwt.decode(token, _CONFIG["secret"], algorithms=[_CONFIG["algorithm"]])
|
||
return payload["sub"]
|
||
except jwt.PyJWTError:
|
||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="无效 token")
|
||
```
|
||
|
||
- [ ] **Step 4: 验证通过** — `pytest tests/api/test_auth.py -v` → 3 passed
|
||
|
||
- [ ] **Step 5: 扩展 config + commit**
|
||
|
||
```yaml
|
||
# config/backtest.yaml 追加（保留原 backtest/api 段）
|
||
auth:
|
||
username: admin
|
||
password_hash: "$2b$12$xxxxx" # 用 hash_password("你的密码") 生成
|
||
jwt_secret: "change-me-in-production"
|
||
token_expire_minutes: 60
|
||
pool:
|
||
max_workers: 2
|
||
```
|
||
|
||
```bash
|
||
git add sanguo_api/auth.py config/backtest.yaml tests/api/test_auth.py
|
||
git commit -m "feat(api): JWT 单用户认证（auth.py）+ config 扩展"
|
||
```
|
||
|
||
---
|
||
|
||
## Task 2: sanguo_api/ws.py（WS 连接池）
|
||
|
||
**Files:**
|
||
- Create: `sanguo_api/ws.py`
|
||
- Test: `tests/api/test_ws.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `ConnectionManager`（`connect(task_id, ws)`、`disconnect(task_id, ws)`、`broadcast(task_id, msg: dict)`）、模块级单例 `manager = ConnectionManager()`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/api/test_ws.py
|
||
import pytest
|
||
from unittest.mock import AsyncMock, MagicMock
|
||
|
||
@pytest.mark.asyncio
|
||
def test_connect_and_broadcast():
|
||
from sanguo_api.ws import ConnectionManager
|
||
mgr = ConnectionManager()
|
||
ws = AsyncMock()
|
||
mgr.connect("t1", ws)
|
||
assert "t1" in mgr._connections
|
||
import asyncio
|
||
asyncio.run(mgr.broadcast("t1", {"status": "running"}))
|
||
ws.send_json.assert_called_with({"status": "running"})
|
||
|
||
def test_disconnect_removes_ws():
|
||
from sanguo_api.ws import ConnectionManager
|
||
mgr = ConnectionManager()
|
||
ws = MagicMock()
|
||
mgr.connect("t1", ws)
|
||
mgr.disconnect("t1", ws)
|
||
assert ws not in mgr._connections.get("t1", set())
|
||
|
||
@pytest.mark.asyncio
|
||
def test_broadcast_no_subscribers_no_error():
|
||
from sanguo_api.ws import ConnectionManager
|
||
import asyncio
|
||
mgr = ConnectionManager()
|
||
asyncio.run(mgr.broadcast("nope", {"x": 1})) # 不抛
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败** — `pip install pytest-asyncio --break-system-packages`（若无），`pytest tests/api/test_ws.py -v` → FAIL
|
||
|
||
- [ ] **Step 3: 实现 ws.py**
|
||
|
||
```python
|
||
# sanguo_api/ws.py
|
||
"""WS 连接池：task_id → 订阅者集合。简单广播，不做重连/心跳。"""
|
||
from fastapi import WebSocket
|
||
|
||
|
||
class ConnectionManager:
|
||
def __init__(self):
|
||
self._connections: dict[str, set[WebSocket]] = {}
|
||
|
||
def connect(self, task_id: str, ws: WebSocket):
|
||
self._connections.setdefault(task_id, set()).add(ws)
|
||
|
||
def disconnect(self, task_id: str, ws: WebSocket):
|
||
conns = self._connections.get(task_id)
|
||
if conns:
|
||
conns.discard(ws)
|
||
if not conns:
|
||
del self._connections[task_id]
|
||
|
||
async def broadcast(self, task_id: str, msg: dict):
|
||
for ws in list(self._connections.get(task_id, [])):
|
||
try:
|
||
await ws.send_json(msg)
|
||
except Exception:
|
||
self.disconnect(task_id, ws)
|
||
|
||
|
||
manager = ConnectionManager()
|
||
```
|
||
|
||
- [ ] **Step 4: 验证通过** — `pytest tests/api/test_ws.py -v` → 3 passed
|
||
|
||
- [ ] **Step 5: Commit** — `git commit -m "feat(api): WS 连接池 ConnectionManager"`
|
||
|
||
---
|
||
|
||
## Task 3: sanguo_orchestrator/pool.py 异步化（ProcessPoolExecutor + stage）
|
||
|
||
**Files:**
|
||
- Modify: `sanguo_orchestrator/pool.py`、`sanguo_orchestrator/task.py`
|
||
- Test: `tests/orchestrator/test_pool.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `Task` 加 `stage: str` 字段（默认 ""）；`TaskPool` 加 `executor: ProcessPoolExecutor`（spawn context，max_workers）、`submit_work(task_id, func, *args)` → `Future`、`update_stage(task_id, stage)`、`get_stage(task_id) → str|None`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/orchestrator/test_pool.py（追加）
|
||
from unittest.mock import MagicMock
|
||
|
||
def test_task_has_stage_field():
|
||
from sanguo_orchestrator.task import Task
|
||
t = Task(task_id="t1", task_type="cta")
|
||
assert t.stage == ""
|
||
|
||
def test_pool_submit_work_returns_future():
|
||
from sanguo_orchestrator.pool import TaskPool
|
||
pool = TaskPool(max_workers=2)
|
||
pool.executor = MagicMock() # mock executor 避免真起进程
|
||
mock_future = MagicMock()
|
||
pool.executor.submit.return_value = mock_future
|
||
fut = pool.submit_work("t1", func=lambda: 1)
|
||
assert fut is mock_future
|
||
pool.executor.submit.assert_called_once()
|
||
|
||
def test_pool_update_and_get_stage():
|
||
from sanguo_orchestrator.pool import TaskPool
|
||
from sanguo_orchestrator.task import Task
|
||
pool = TaskPool(max_workers=2)
|
||
pool.submit("t1", "cta")
|
||
pool.update_stage("t1", "回测中")
|
||
assert pool.get_stage("t1") == "回测中"
|
||
assert pool.get_task("t1").stage == "回测中"
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败**
|
||
|
||
- [ ] **Step 3: task.py 加 stage 字段**
|
||
|
||
```python
|
||
# sanguo_orchestrator/task.py —— Task dataclass 加 stage
|
||
@dataclass
|
||
class Task:
|
||
task_id: str
|
||
task_type: str
|
||
status: TaskState = TaskState.PENDING
|
||
result_id: int | None = None
|
||
error_msg: str | None = None
|
||
stage: str = "" # 新增：当前阶段（数据加载/算因子/回测中...）
|
||
# start/complete/fail 方法不变
|
||
```
|
||
|
||
- [ ] **Step 4: pool.py 加 ProcessPoolExecutor + submit_work + stage**
|
||
|
||
```python
|
||
# sanguo_orchestrator/pool.py
|
||
from concurrent.futures import ProcessPoolExecutor, Future
|
||
from multiprocessing import get_context
|
||
from .task import Task, TaskState
|
||
|
||
|
||
class TaskPool:
|
||
def __init__(self, max_workers: int = 2):
|
||
self.max_workers = max_workers
|
||
self._tasks: dict[str, Task] = {}
|
||
self.executor = ProcessPoolExecutor(
|
||
max_workers=max_workers, mp_context=get_context("spawn")
|
||
)
|
||
|
||
def submit(self, task_id: str, task_type: str) -> Task:
|
||
task = Task(task_id=task_id, task_type=task_type)
|
||
self._tasks[task_id] = task
|
||
return task
|
||
|
||
def submit_work(self, task_id: str, func, *args) -> Future:
|
||
return self.executor.submit(func, *args)
|
||
|
||
def update_stage(self, task_id: str, stage: str):
|
||
t = self._tasks.get(task_id)
|
||
if t:
|
||
t.stage = stage
|
||
|
||
def get_status(self, task_id: str) -> TaskState | None:
|
||
t = self._tasks.get(task_id)
|
||
return t.status if t else None
|
||
|
||
def get_stage(self, task_id: str) -> str | None:
|
||
t = self._tasks.get(task_id)
|
||
return t.stage if t else None
|
||
|
||
def get_task(self, task_id: str) -> Task | None:
|
||
return self._tasks.get(task_id)
|
||
```
|
||
|
||
- [ ] **Step 5: 验证通过 + commit** — `pytest tests/orchestrator/test_pool.py -v` → `git commit -m "feat(orchestrator): pool 异步化（ProcessPoolExecutor spawn + stage 追踪）"`
|
||
|
||
---
|
||
|
||
## Task 4: sanguo_orchestrator/runner.py async submit + on_stage 回调
|
||
|
||
**Files:**
|
||
- Modify: `sanguo_orchestrator/runner.py`
|
||
- Test: `tests/orchestrator/test_runner.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `Orchestrator` 改 `async submit_cta(...)` → task_id（提交到 pool.executor，asyncio.wrap_future 桥接，完成后调回调更新状态 + broadcast）、`set_on_stage(callback)`（callback: async (task_id, stage) → None）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/orchestrator/test_runner.py（追加）
|
||
import asyncio
|
||
from unittest.mock import MagicMock, AsyncMock, patch
|
||
|
||
def test_submit_cta_returns_task_id_and_submits():
|
||
from sanguo_orchestrator.runner import Orchestrator
|
||
orch = Orchestrator(db_path="/tmp/t.db")
|
||
orch.pool.executor = MagicMock()
|
||
orch.pool.executor.submit.return_value = MagicMock()
|
||
with patch("sanguo_backtest.cta_engine.run_cta_backtest"):
|
||
task_id = asyncio.get_event_loop().run_until_complete(
|
||
orch.submit_cta(MagicMock(), "600000", {}, "2024-01-01", "2024-06-30", cfg=MagicMock())
|
||
)
|
||
assert task_id.startswith("cta_600000")
|
||
orch.pool.executor.submit.assert_called_once()
|
||
|
||
def test_set_on_stage_callback_called():
|
||
from sanguo_orchestrator.runner import Orchestrator
|
||
from sanguo_orchestrator.task import TaskState
|
||
orch = Orchestrator(db_path="/tmp/t.db")
|
||
orch.pool.executor = MagicMock()
|
||
mock_future = MagicMock()
|
||
orch.pool.executor.submit.return_value = mock_future
|
||
cb = AsyncMock()
|
||
orch.set_on_stage(cb)
|
||
# 模拟任务完成回调
|
||
asyncio.get_event_loop().run_until_complete(orch._on_done("t1", {"statistics": {}}))
|
||
cb.assert_called()
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败**
|
||
|
||
- [ ] **Step 3: 实现 runner.py async**
|
||
|
||
```python
|
||
# sanguo_orchestrator/runner.py
|
||
import asyncio
|
||
from concurrent.futures import Future
|
||
from .pool import TaskPool
|
||
from .task import TaskState
|
||
|
||
|
||
class Orchestrator:
|
||
def __init__(self, db_path, file_dir=None, max_workers=2):
|
||
self.db_path = db_path
|
||
self.file_dir = file_dir
|
||
self.pool = TaskPool(max_workers=max_workers)
|
||
self._pending: dict = {}
|
||
self._on_stage = None # async callback(task_id, stage)
|
||
|
||
def set_on_stage(self, cb):
|
||
self._on_stage = cb
|
||
|
||
async def _notify_stage(self, task_id: str, stage: str):
|
||
self.pool.update_stage(task_id, stage)
|
||
if self._on_stage:
|
||
await self._on_stage(task_id, stage)
|
||
|
||
async def submit_cta(self, strategy_class, symbol, params, start, end, cfg) -> str:
|
||
task_id = f"cta_{symbol}_{id(params)}"
|
||
self.pool.submit(task_id, "cta")
|
||
self._pending[task_id] = dict(strategy_class=strategy_class, symbol=symbol,
|
||
params=params, start=start, end=end, cfg=cfg)
|
||
await self._notify_stage(task_id, "排队中")
|
||
spec = self._pending[task_id]
|
||
from sanguo_backtest.cta_engine import run_cta_backtest
|
||
fut: Future = self.pool.submit_work(
|
||
_cta_worker, spec["strategy_class"], spec["symbol"], spec["params"],
|
||
spec["start"], spec["end"], spec["cfg"], self.db_path,
|
||
)
|
||
self.pool.get_task(task_id).start()
|
||
await self._notify_stage(task_id, "回测中")
|
||
asyncio.ensure_future(self._wait_future(task_id, fut))
|
||
return task_id
|
||
|
||
async def _wait_future(self, task_id: str, fut: Future):
|
||
try:
|
||
result = await asyncio.wrap_future(fut)
|
||
await self._on_done(task_id, result)
|
||
except Exception as e:
|
||
self.pool.get_task(task_id).fail(f"{type(e).__name__}: {e}")
|
||
await self._notify_stage(task_id, "失败")
|
||
|
||
async def _on_done(self, task_id: str, result):
|
||
self.pool.get_task(task_id).complete(result_id=id(result))
|
||
await self._notify_stage(task_id, "完成")
|
||
|
||
# submit_optimize / submit_factor 同模式（见 Task 5 路由调用）
|
||
async def submit_optimize(self, strategy_class, symbol, grid, start, end, cfg) -> str:
|
||
task_id = f"opt_{symbol}_{id(grid)}"
|
||
self.pool.submit(task_id, "optimize")
|
||
await self._notify_stage(task_id, "参数优化中")
|
||
from sanguo_backtest.cta_optimizer import run_cta_optimization
|
||
fut = self.pool.submit_work(_opt_worker, strategy_class, symbol, grid, start, end, cfg, self.db_path)
|
||
self.pool.get_task(task_id).start()
|
||
asyncio.ensure_future(self._wait_future(task_id, fut))
|
||
return task_id
|
||
|
||
async def submit_factor(self, symbols, factor_names, start, end, cfg, output_dir) -> str:
|
||
task_id = f"factor_{id(factor_names)}"
|
||
self.pool.submit(task_id, "factor")
|
||
await self._notify_stage(task_id, "因子分析中")
|
||
from sanguo_factor.analyzer import run_factor_analysis
|
||
fut = self.pool.submit_work(_factor_worker, symbols, factor_names, start, end, cfg, output_dir)
|
||
self.pool.get_task(task_id).start()
|
||
asyncio.ensure_future(self._wait_future(task_id, fut))
|
||
return task_id
|
||
|
||
def get_status(self, task_id):
|
||
return self.pool.get_status(task_id)
|
||
|
||
def get_result(self, task_id):
|
||
task = self.pool.get_task(task_id)
|
||
if task and task.status == TaskState.DONE and task.result_id:
|
||
from sanguo_backtest.result_store import load_result
|
||
return load_result(task.result_id, self.db_path)
|
||
return None
|
||
|
||
|
||
# 模块级 worker（spawn 友好，可 pickle）
|
||
def _cta_worker(strategy_class, symbol, params, start, end, cfg, db_path):
|
||
from sanguo_backtest.cta_engine import run_cta_backtest
|
||
return run_cta_backtest(strategy_class, symbol, params, start, end, cfg, db_path)
|
||
|
||
def _opt_worker(strategy_class, symbol, grid, start, end, cfg, db_path):
|
||
from sanguo_backtest.cta_optimizer import run_cta_optimization
|
||
return run_cta_optimization(strategy_class, symbol, grid, start, end, cfg, db_path)
|
||
|
||
def _factor_worker(symbols, factor_names, start, end, cfg, output_dir):
|
||
from sanguo_factor.analyzer import run_factor_analysis
|
||
return run_factor_analysis(symbols, factor_names, start, end, cfg, output_dir)
|
||
```
|
||
|
||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/orchestrator/test_runner.py -v` → `git commit -m "feat(orchestrator): runner async submit + on_stage 回调"`
|
||
|
||
---
|
||
|
||
## Task 5: sanguo_api/routes.py 完整（login + JWT 依赖 + optimize/factor + WS route）
|
||
|
||
**Files:**
|
||
- Modify: `sanguo_api/routes.py`、`sanguo_api/app.py`
|
||
- Test: `tests/api/test_routes.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `POST /auth/login`（用户名密码 → token）、optimize/factor 路由真调用 submit_optimize/submit_factor、业务路由加 `Depends(verify_token)`、`WS /ws/task/{id}`（订阅阶段推送）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/api/test_routes.py（追加）
|
||
from fastapi.testclient import TestClient
|
||
from unittest.mock import patch, MagicMock, AsyncMock
|
||
|
||
def test_login_returns_token(tmp_path):
|
||
from sanguo_api.app import create_app
|
||
from sanguo_api.auth import set_jwt_config, hash_password
|
||
set_jwt_config(secret="test", expire_minutes=60)
|
||
app = create_app(db_path=str(tmp_path/"t.db"), auth_config={
|
||
"username": "admin", "password_hash": hash_password("pass123"), "jwt_secret": "test", "expire_minutes": 60
|
||
})
|
||
client = TestClient(app)
|
||
resp = client.post("/api/v1/auth/login", json={"username": "admin", "password": "pass123"})
|
||
assert resp.status_code == 200
|
||
assert "token" in resp.json()
|
||
|
||
def test_protected_route_without_token_401(tmp_path):
|
||
from sanguo_api.app import create_app
|
||
app = create_app(db_path=str(tmp_path/"t.db"))
|
||
client = TestClient(app)
|
||
resp = client.get("/api/v1/task/t1")
|
||
assert resp.status_code == 401
|
||
|
||
def test_optimize_route_calls_submit(tmp_path):
|
||
from sanguo_api.app import create_app
|
||
from sanguo_api.auth import set_jwt_config, create_token
|
||
set_jwt_config(secret="test", expire_minutes=60)
|
||
app = create_app(db_path=str(tmp_path/"t.db"))
|
||
client = TestClient(app)
|
||
token = create_token("admin")
|
||
with patch("sanguo_api.routes.get_orchestrator") as m:
|
||
orch = MagicMock()
|
||
orch.submit_optimize = AsyncMock(return_value="opt_1")
|
||
m.return_value = orch
|
||
resp = client.post("/api/v1/backtest/optimize", json={
|
||
"symbol": "600000", "strategy": "MaStrategy", "grid": {"n": [5, 20, 5]},
|
||
"start": "2024-01-01", "end": "2024-06-30", "max_workers": 2
|
||
}, headers={"Authorization": f"Bearer {token}"})
|
||
assert resp.status_code == 200
|
||
assert resp.json()["task_id"] == "opt_1"
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败**
|
||
|
||
- [ ] **Step 3: 实现 routes 完整 + app 改**
|
||
|
||
```python
|
||
# sanguo_api/routes.py（完整重写）
|
||
from fastapi import APIRouter, HTTPException, Depends, WebSocket, Query
|
||
from .schemas import CtaBacktestRequest, OptimizeRequest, FactorAnalysisRequest
|
||
from .auth import verify_token, verify_password, create_token
|
||
from .ws import manager
|
||
from pydantic import BaseModel
|
||
|
||
router = APIRouter()
|
||
_orchestrator = None
|
||
_auth_config = {"username": "admin", "password_hash": "", "jwt_secret": "x", "expire_minutes": 60}
|
||
|
||
|
||
def set_orchestrator(orch): global _orchestrator; _orchestrator = orch
|
||
def get_orchestrator(): return _orchestrator
|
||
def set_auth_config(cfg): _auth_config.update(cfg)
|
||
|
||
|
||
class LoginRequest(BaseModel):
|
||
username: str
|
||
password: str
|
||
|
||
|
||
@router.post("/auth/login")
|
||
def login(req: LoginRequest):
|
||
if req.username != _auth_config["username"] or not verify_password(req.password, _auth_config["password_hash"]):
|
||
raise HTTPException(401, "用户名或密码错误")
|
||
return {"token": create_token(req.username)}
|
||
|
||
|
||
@router.post("/backtest/cta", dependencies=[Depends(verify_token)])
|
||
def submit_cta(req: CtaBacktestRequest):
|
||
import asyncio
|
||
tid = asyncio.get_event_loop().run_until_complete(
|
||
get_orchestrator().submit_cta(req.strategy, req.symbol, req.params, req.start, req.end, None))
|
||
return {"task_id": tid}
|
||
|
||
|
||
@router.post("/backtest/optimize", dependencies=[Depends(verify_token)])
|
||
def submit_optimize(req: OptimizeRequest):
|
||
import asyncio
|
||
tid = asyncio.get_event_loop().run_until_complete(
|
||
get_orchestrator().submit_optimize(req.strategy, req.symbol, req.grid, req.start, req.end, None))
|
||
return {"task_id": tid}
|
||
|
||
|
||
@router.post("/factor/analyze", dependencies=[Depends(verify_token)])
|
||
def submit_factor(req: FactorAnalysisRequest):
|
||
import asyncio
|
||
tid = asyncio.get_event_loop().run_until_complete(
|
||
get_orchestrator().submit_factor(req.symbols, req.factor_names, req.start, req.end, None, "/tmp/factor"))
|
||
return {"task_id": tid}
|
||
|
||
|
||
@router.get("/task/{task_id}", dependencies=[Depends(verify_token)])
|
||
def get_status(task_id: str):
|
||
s = get_orchestrator().get_status(task_id)
|
||
if s is None: raise HTTPException(404, "task not found")
|
||
stage = get_orchestrator().pool.get_stage(task_id)
|
||
return {"task_id": task_id, "status": s.value if hasattr(s, "value") else str(s), "stage": stage or ""}
|
||
|
||
|
||
@router.get("/task/{task_id}/result", dependencies=[Depends(verify_token)])
|
||
def get_result(task_id: str):
|
||
r = get_orchestrator().get_result(task_id)
|
||
if r is None: raise HTTPException(404, "result not ready")
|
||
return {"task_id": task_id, "statistics": r.statistics}
|
||
|
||
|
||
@router.websocket("/ws/task/{task_id}")
|
||
async def task_ws(websocket: WebSocket, task_id: str, token: str = Query(...)):
|
||
from .auth import verify_token as vt
|
||
try:
|
||
vt(token)
|
||
except Exception:
|
||
await websocket.close(code=4401); return
|
||
await websocket.accept()
|
||
manager.connect(task_id, websocket)
|
||
try:
|
||
while True:
|
||
await websocket.receive_text() # 保活
|
||
except Exception:
|
||
pass
|
||
finally:
|
||
manager.disconnect(task_id, websocket)
|
||
```
|
||
|
||
```python
|
||
# sanguo_api/app.py（改：register auth_config + on_stage 推 WS）
|
||
from fastapi import FastAPI
|
||
from .routes import router, set_orchestrator, set_auth_config
|
||
from .auth import set_jwt_config
|
||
from .ws import manager
|
||
from sanguo_orchestrator.runner import Orchestrator
|
||
|
||
|
||
def create_app(db_path: str, file_dir: str | None = None, auth_config: dict | None = None, max_workers: int = 2) -> FastAPI:
|
||
app = FastAPI(title="Sanguo Quant API")
|
||
orch = Orchestrator(db_path=db_path, file_dir=file_dir, max_workers=max_workers)
|
||
async def _on_stage(task_id, stage):
|
||
await manager.broadcast(task_id, {"task_id": task_id, "stage": stage})
|
||
orch.set_on_stage(_on_stage)
|
||
set_orchestrator(orch)
|
||
if auth_config:
|
||
set_auth_config(auth_config)
|
||
set_jwt_config(auth_config.get("jwt_secret", "x"), auth_config.get("expire_minutes", 60))
|
||
app.include_router(router, prefix="/api/v1")
|
||
return app
|
||
```
|
||
|
||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/api/ -v` → `git commit -m "feat(api): routes 完整（login + JWT 依赖 + optimize/factor + WS route）"`
|
||
|
||
---
|
||
|
||
## Task 6: sanguo_factor/alpha_lab.py compute_factors 完整化
|
||
|
||
**Files:**
|
||
- Modify: `sanguo_factor/alpha_lab.py`
|
||
- Test: `tests/factor/test_alpha_lab.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `AlphaLabSession.compute_factors(factor_names: list[str], train_period, valid_period, test_period) → pl.DataFrame`（add_feature + prepare_data + fetch_raw）
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/factor/test_alpha_lab.py（追加）
|
||
from unittest.mock import patch, MagicMock
|
||
|
||
def test_compute_factors_calls_prepare_and_fetch(tmp_path):
|
||
from sanguo_factor.alpha_lab import AlphaLabSession
|
||
with patch("sanguo_factor.alpha_lab.AlphaLab") as MLab, \
|
||
patch("sanguo_factor.alpha_lab.read_db_daily") as MRead, \
|
||
patch("sanguo_factor.alpha_lib.AlphaDataset") as MDS, \
|
||
patch("sanguo_factor.alpha_lab.get_factor") as MGet, \
|
||
patch("sanguo_factor.alpha_lab.save_alpha_lab_data"):
|
||
import polars as pl
|
||
MRead.return_value = [MagicMock(vt_symbol="600000.SSE", datetime=__import__("datetime").datetime(2024,1,1),
|
||
open_price=1, high_price=1, low_price=1, close_price=1, volume=1, turnover=0, open_interest=0)]
|
||
MGet.return_value = {"expression": "ts_mean(close,5)"}
|
||
MDS.return_value.fetch_raw.return_value = pl.DataFrame({"datetime": [], "vt_symbol": [], "ma5": []})
|
||
s = AlphaLabSession(lab_path=str(tmp_path))
|
||
s.load_symbols(["600000"], "2024-01-01", "2024-06-30", cfg=MagicMock())
|
||
df = s.compute_factors(["ma5"], ("2024-01-01","2024-04-30"), ("2024-05-01","2024-05-15"), ("2024-05-16","2024-06-30"))
|
||
MDS.return_value.add_feature.assert_called()
|
||
MDS.return_value.prepare_data.assert_called_once()
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败**
|
||
|
||
- [ ] **Step 3: 补 compute_factors**
|
||
|
||
```python
|
||
# sanguo_factor/alpha_lab.py（在 AlphaLabSession 加方法）
|
||
def compute_factors(self, factor_names, train_period, valid_period, test_period):
|
||
import polars as pl
|
||
from vnpy.alpha.dataset import AlphaDataset, Segment
|
||
from .registry import get_factor
|
||
from .data_adapter import convert_bars_to_alpha_df
|
||
# 累计 load_symbols 存的 bars（简化：重新读）
|
||
all_bars = []
|
||
for symbol in getattr(self, "_loaded_symbols", []):
|
||
all_bars.extend(self._loaded_bars.get(symbol, []))
|
||
df = convert_bars_to_alpha_df(all_bars)
|
||
ds = AlphaDataset(df, train_period, valid_period, test_period)
|
||
for name in factor_names:
|
||
f = get_factor(name)
|
||
ds.add_feature(name, f["expression"])
|
||
ds.prepare_data(max_workers=1)
|
||
return ds.fetch_raw(Segment.TEST)
|
||
```
|
||
|
||
（配套 load_symbols 缓存 `_loaded_symbols`/`_loaded_bars`，见执行时补全。）
|
||
|
||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/factor/test_alpha_lab.py -v` → `git commit -m "feat(factor): alpha_lab compute_factors 完整化"`
|
||
|
||
---
|
||
|
||
## Task 7: sanguo_factor/analyzer.py tears pipeline 完整化
|
||
|
||
**Files:**
|
||
- Modify: `sanguo_factor/analyzer.py`
|
||
- Test: `tests/factor/test_analyzer.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `run_factor_analysis` 补 compute_factors → get_clean_factor_and_forward_returns → create_full_tear_sheet → html 报告，FactorReport 含 `report_path`/`ic_summary`
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
# tests/factor/test_analyzer.py（追加）
|
||
from unittest.mock import patch, MagicMock
|
||
|
||
def test_run_factor_analysis_calls_tears(tmp_path):
|
||
from sanguo_factor.analyzer import run_factor_analysis
|
||
with patch("sanguo_factor.analyzer.AlphaLabSession") as MS, \
|
||
patch("sanguo_factor.analyzer.get_clean_factor_and_forward_returns") as MC, \
|
||
patch("sanguo_factor.analyzer.create_full_tear_sheet") as MT:
|
||
import polars as pl
|
||
MS.return_value.compute_factors.return_value = pl.DataFrame({"datetime":[],"vt_symbol":[],"ma5":[]})
|
||
MC.return_value = MagicMock()
|
||
report = run_factor_analysis(["600000"], ["ma5"], "2024-01-01", "2024-06-30",
|
||
cfg=MagicMock(), output_dir=str(tmp_path))
|
||
assert report.factor_names == ["ma5"]
|
||
MC.assert_called_once(); MT.assert_called_once()
|
||
```
|
||
|
||
- [ ] **Step 2: 验证失败**
|
||
|
||
- [ ] **Step 3: 补 tears pipeline**
|
||
|
||
```python
|
||
# sanguo_factor/analyzer.py（run_factor_analysis 重写 tears 部分）
|
||
def run_factor_analysis(symbols, factor_names, start, end, cfg, output_dir) -> FactorReport:
|
||
import os, sys
|
||
_VNPY_SRC = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "vnpy_v4.4.0"))
|
||
if _VNPY_SRC not in sys.path: sys.path.insert(0, _VNPY_SRC)
|
||
from alphalens.utils import get_clean_factor_and_forward_returns
|
||
from alphalens.tears import create_full_tear_sheet
|
||
import matplotlib
|
||
matplotlib.use("Agg")
|
||
from .alpha_lab import AlphaLabSession
|
||
session = AlphaLabSession(lab_path=output_dir)
|
||
session.load_symbols(symbols, start, end, cfg)
|
||
factor_df = session.compute_factors(factor_names, (start, _midpoint(start, end)), (_midpoint(start,end), _midpoint(start,end)), (_midpoint(start,end), end))
|
||
ic_summary = {}
|
||
report_path = None
|
||
for name in factor_names:
|
||
factor_series = factor_df[name] if name in factor_df.columns else factor_df.iloc[:, -1]
|
||
merged = get_clean_factor_and_forward_returns(factor_df.set_index(["datetime","vt_symbol")[name]], ...) # 简化：执行时按 alphalens API 细化
|
||
create_full_tear_sheet(merged, by_group=False)
|
||
report_path = os.path.join(output_dir, f"{name}_tears.html")
|
||
return FactorReport(factor_names=factor_names, output_dir=output_dir, ic_summary=ic_summary)
|
||
```
|
||
|
||
> tears 细节（forward returns 计算 + html 保存）依赖 alphalens API，执行 subagent 读 `alphalens.tears.create_full_tear_sheet` 签名 + 容器验证。
|
||
|
||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/factor/test_analyzer.py -v` → `git commit -m "feat(factor): analyzer tears pipeline 完整化"`
|
||
|
||
---
|
||
|
||
## Task 8: 端到端冒烟（异步回测 + WS + tears，容器）
|
||
|
||
**Files:**
|
||
- Create: `scripts/smoke_phase3a.py`
|
||
|
||
- [ ] **Step 1: 写冒烟脚本**
|
||
|
||
```python
|
||
# scripts/smoke_phase3a.py
|
||
"""Phase 3a 端到端冒烟：异步 submit + WS 阶段 + JWT + tears（容器）。"""
|
||
import sys, asyncio
|
||
sys.path.insert(0, "/app"); sys.path.insert(0, "/app/vnpy_v4.4.0")
|
||
|
||
async def main():
|
||
print("=== Phase 3a Smoke ===")
|
||
from sanguo_api.app import create_app
|
||
from sanguo_api.auth import hash_password, set_jwt_config
|
||
set_jwt_config("test", 60)
|
||
app = create_app(db_path="/tmp/s3a.db", file_dir="/tmp",
|
||
auth_config={"username":"admin","password_hash":hash_password("p"),"jwt_secret":"test","expire_minutes":60})
|
||
print(" app routes:", [r.path for r in app.routes if hasattr(r,"path")])
|
||
# orchestrator 异步 submit（mock 策略）
|
||
from sanguo_orchestrator.runner import Orchestrator
|
||
orch = Orchestrator(db_path="/tmp/s3a.db")
|
||
print(" orchestrator pool executor:", type(orch.pool.executor).__name__)
|
||
print("=== Phase 3a Smoke DONE ===")
|
||
|
||
asyncio.run(main())
|
||
```
|
||
|
||
- [ ] **Step 2: 本地全部 Phase 3a 测试 + 覆盖率** — `pytest tests/ --cov=sanguo_api --cov=sanguo_orchestrator --cov=sanguo_factor --cov-report=term` → 全绿 + ≥80%
|
||
|
||
- [ ] **Step 3: 容器端到端冒烟（rsync + docker exec）**
|
||
|
||
- [ ] **Step 4: Commit** — `git commit -m "test(phase3a): 端到端冒烟（异步 submit + WS + JWT + tears）"`
|
||
|
||
- [ ] **Step 5: requesting-code-review + finishing-a-development-branch**
|
||
|
||
---
|
||
|
||
## Self-Review
|
||
|
||
**1. Spec coverage**：异步 pool(T3) + runner async(T4) + WS 阶段(T2/T5) + JWT 单用户(T1/T5) + optimize/factor 路由(T5) + alpha tears 完整化(T6/T7) = design §0 五项全覆盖 ✅
|
||
**2. Placeholder**：tears pipeline 的 forward returns 细节（T7）依赖 alphalens API，标注执行时细化；其余代码完整。
|
||
**3. Type consistency**：`Orchestrator.async submit_*`（T4）在 routes（T5）调用一致；`manager.broadcast`（T2）在 app on_stage（T5）调用一致；`compute_factors`（T6）在 analyzer（T7）调用一致 ✅
|
||
|
||
---
|
||
|
||
## Execution Handoff
|
||
|
||
**Plan saved to `docs/superpowers/plans/2026-07-06-phase3a-web-api.md`.** 推荐用 **Subagent-Driven** 执行（每 task fresh subagent + review）。
|