docs(phase3a): Web API 完整化实现计划(8 task,TDD)
This commit is contained in:
@@ -0,0 +1,839 @@
|
||||
# Phase 3a Web API 完整化 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** 把 Phase 2 的同步轻量 API 升级为异步、可登录、带进度推送、因子报告完整的后端。
|
||||
|
||||
**Architecture:** FastAPI + ProcessPoolExecutor(spawn context)做异步回测,JWT 单用户鉴权,WS 连接池推阶段进度,analyzer 补完整 alphalens tears pipeline。
|
||||
|
||||
**Tech Stack:** FastAPI + uvicorn、concurrent.futures.ProcessPoolExecutor、PyJWT、websockets(FastAPI TestClient)、alphalens(容器)。
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **vnpy 零改造**:源码 `vnpy_v4.4.0/`,`sys.path.insert(0, _VNPY_SRC)`(参照 sanguo_data/datareader.py:6-9)
|
||||
- **Python 3.14 本地 / 3.10 容器**;本地无 polars/alphalens/vnpy_ctastrategy → lazy import + mock 测试(参照 Phase 2 模式)
|
||||
- **Phase 2 接口复用**:`result_store`(save_result/load_result/BacktestResult)、`cta_engine.run_cta_backtest`、`cta_optimizer.run_cta_optimization`、`registry/library`、`runner.Orchestrator`
|
||||
- **ProcessPoolExecutor 用 spawn context**(vnpy/alpha 已用 spawn,兼容)
|
||||
- **WS 简单连接池**(dict[task_id, set[WebSocket]]),不做重连/心跳(YAGNI)
|
||||
- **JWT 单用户**:用户名/密码 hash/jwt_secret 配 `config/backtest.yaml`
|
||||
- **测试策略**:本地 Python 3.14 跑 mock 测试 + 容器 Python 3.10 跑真实依赖测试(polars/alphalens)+ 端到端冒烟
|
||||
- **覆盖率 ≥ 80%**,TDD,每 task commit
|
||||
- **不引 Qlib/Celery**,不做组合回测/多用户/Vue 前端(留后续 phase)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
sanguo_api/
|
||||
auth.py # 新:JWT 单用户(create_token/verify_token)
|
||||
ws.py # 新:WS 连接池(ConnectionManager)
|
||||
routes.py # 改:+ /auth/login、optimize/factor 真调用、JWT 依赖、WS route
|
||||
app.py # 改:register auth/ws,JWT 依赖生效
|
||||
sanguo_orchestrator/
|
||||
pool.py # 改:TaskPool 加 ProcessPoolExecutor + stage 追踪
|
||||
runner.py # 改:async submit_* + on_stage 回调
|
||||
sanguo_factor/
|
||||
alpha_lab.py # 改:补 compute_factors(add_feature + prepare_data + fetch_raw)
|
||||
analyzer.py # 改:tears pipeline 完整化
|
||||
config/backtest.yaml # 改:+ auth + pool 配置
|
||||
tests/api/{test_auth.py, test_ws.py, test_routes.py} # 新/改
|
||||
tests/orchestrator/{test_pool.py, test_runner.py} # 改
|
||||
tests/factor/{test_alpha_lab.py, test_analyzer.py} # 改
|
||||
scripts/smoke_phase3a.py # 新
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 1: config 扩展 + sanguo_api/auth.py(JWT 单用户)
|
||||
|
||||
**Files:**
|
||||
- Modify: `config/backtest.yaml`
|
||||
- Create: `sanguo_api/auth.py`
|
||||
- Test: `tests/api/test_auth.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `create_token(username: str) → str`、`verify_token(token: str) → str`(返回 username,FastAPI 依赖用法 `Depends(verify_token)`,无效 raise HTTPException 401)、`hash_password(pw) → str`、`verify_password(pw, hash) → bool`
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/api/test_auth.py
|
||||
import pytest
|
||||
|
||||
def test_create_and_verify_token():
|
||||
from sanguo_api.auth import create_token, verify_token, set_jwt_config
|
||||
set_jwt_config(secret="test_secret", expire_minutes=60)
|
||||
token = create_token("admin")
|
||||
assert isinstance(token, str)
|
||||
assert verify_token(token) == "admin"
|
||||
|
||||
def test_verify_token_invalid_raises_401():
|
||||
from fastapi import HTTPException
|
||||
from sanguo_api.auth import verify_token, set_jwt_config
|
||||
set_jwt_config(secret="test_secret", expire_minutes=60)
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
verify_token("invalid.token.here")
|
||||
assert exc.value.status_code == 401
|
||||
|
||||
def test_hash_and_verify_password():
|
||||
from sanguo_api.auth import hash_password, verify_password
|
||||
h = hash_password("mypass")
|
||||
assert h != "mypass"
|
||||
assert verify_password("mypass", h) is True
|
||||
assert verify_password("wrong", h) is False
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败** — `pip install PyJWT passlib[bcrypt] --break-system-packages`(若无),`pytest tests/api/test_auth.py -v` → FAIL
|
||||
|
||||
- [ ] **Step 3: 实现 auth.py**
|
||||
|
||||
```python
|
||||
# sanguo_api/auth.py
|
||||
"""JWT 单用户认证。secret/用户名/密码 hash 来自 config/backtest.yaml。"""
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import jwt
|
||||
from passlib.context import CryptContext
|
||||
from fastapi import HTTPException, status
|
||||
|
||||
_CONFIG = {"secret": "change-me", "expire_minutes": 60, "algorithm": "HS256"}
|
||||
_pwd = CryptContext(schemes=["bcrypt"], deprecated="auto")
|
||||
|
||||
|
||||
def set_jwt_config(secret: str, expire_minutes: int, algorithm: str = "HS256"):
|
||||
_CONFIG.update(secret=secret, expire_minutes=expire_minutes, algorithm=algorithm)
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
return _pwd.hash(password)
|
||||
|
||||
|
||||
def verify_password(password: str, password_hash: str) -> bool:
|
||||
return _pwd.verify(password, password_hash)
|
||||
|
||||
|
||||
def create_token(username: str) -> str:
|
||||
payload = {
|
||||
"sub": username,
|
||||
"exp": datetime.now(timezone.utc) + timedelta(minutes=_CONFIG["expire_minutes"]),
|
||||
}
|
||||
return jwt.encode(payload, _CONFIG["secret"], algorithm=_CONFIG["algorithm"])
|
||||
|
||||
|
||||
def verify_token(token: str) -> str:
|
||||
try:
|
||||
payload = jwt.decode(token, _CONFIG["secret"], algorithms=[_CONFIG["algorithm"]])
|
||||
return payload["sub"]
|
||||
except jwt.PyJWTError:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="无效 token")
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 验证通过** — `pytest tests/api/test_auth.py -v` → 3 passed
|
||||
|
||||
- [ ] **Step 5: 扩展 config + commit**
|
||||
|
||||
```yaml
|
||||
# config/backtest.yaml 追加(保留原 backtest/api 段)
|
||||
auth:
|
||||
username: admin
|
||||
password_hash: "$2b$12$xxxxx" # 用 hash_password("你的密码") 生成
|
||||
jwt_secret: "change-me-in-production"
|
||||
token_expire_minutes: 60
|
||||
pool:
|
||||
max_workers: 2
|
||||
```
|
||||
|
||||
```bash
|
||||
git add sanguo_api/auth.py config/backtest.yaml tests/api/test_auth.py
|
||||
git commit -m "feat(api): JWT 单用户认证(auth.py)+ config 扩展"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: sanguo_api/ws.py(WS 连接池)
|
||||
|
||||
**Files:**
|
||||
- Create: `sanguo_api/ws.py`
|
||||
- Test: `tests/api/test_ws.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `ConnectionManager`(`connect(task_id, ws)`、`disconnect(task_id, ws)`、`broadcast(task_id, msg: dict)`)、模块级单例 `manager = ConnectionManager()`
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/api/test_ws.py
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
@pytest.mark.asyncio
|
||||
def test_connect_and_broadcast():
|
||||
from sanguo_api.ws import ConnectionManager
|
||||
mgr = ConnectionManager()
|
||||
ws = AsyncMock()
|
||||
mgr.connect("t1", ws)
|
||||
assert "t1" in mgr._connections
|
||||
import asyncio
|
||||
asyncio.run(mgr.broadcast("t1", {"status": "running"}))
|
||||
ws.send_json.assert_called_with({"status": "running"})
|
||||
|
||||
def test_disconnect_removes_ws():
|
||||
from sanguo_api.ws import ConnectionManager
|
||||
mgr = ConnectionManager()
|
||||
ws = MagicMock()
|
||||
mgr.connect("t1", ws)
|
||||
mgr.disconnect("t1", ws)
|
||||
assert ws not in mgr._connections.get("t1", set())
|
||||
|
||||
@pytest.mark.asyncio
|
||||
def test_broadcast_no_subscribers_no_error():
|
||||
from sanguo_api.ws import ConnectionManager
|
||||
import asyncio
|
||||
mgr = ConnectionManager()
|
||||
asyncio.run(mgr.broadcast("nope", {"x": 1})) # 不抛
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败** — `pip install pytest-asyncio --break-system-packages`(若无),`pytest tests/api/test_ws.py -v` → FAIL
|
||||
|
||||
- [ ] **Step 3: 实现 ws.py**
|
||||
|
||||
```python
|
||||
# sanguo_api/ws.py
|
||||
"""WS 连接池:task_id → 订阅者集合。简单广播,不做重连/心跳。"""
|
||||
from fastapi import WebSocket
|
||||
|
||||
|
||||
class ConnectionManager:
|
||||
def __init__(self):
|
||||
self._connections: dict[str, set[WebSocket]] = {}
|
||||
|
||||
def connect(self, task_id: str, ws: WebSocket):
|
||||
self._connections.setdefault(task_id, set()).add(ws)
|
||||
|
||||
def disconnect(self, task_id: str, ws: WebSocket):
|
||||
conns = self._connections.get(task_id)
|
||||
if conns:
|
||||
conns.discard(ws)
|
||||
if not conns:
|
||||
del self._connections[task_id]
|
||||
|
||||
async def broadcast(self, task_id: str, msg: dict):
|
||||
for ws in list(self._connections.get(task_id, [])):
|
||||
try:
|
||||
await ws.send_json(msg)
|
||||
except Exception:
|
||||
self.disconnect(task_id, ws)
|
||||
|
||||
|
||||
manager = ConnectionManager()
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 验证通过** — `pytest tests/api/test_ws.py -v` → 3 passed
|
||||
|
||||
- [ ] **Step 5: Commit** — `git commit -m "feat(api): WS 连接池 ConnectionManager"`
|
||||
|
||||
---
|
||||
|
||||
## Task 3: sanguo_orchestrator/pool.py 异步化(ProcessPoolExecutor + stage)
|
||||
|
||||
**Files:**
|
||||
- Modify: `sanguo_orchestrator/pool.py`、`sanguo_orchestrator/task.py`
|
||||
- Test: `tests/orchestrator/test_pool.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `Task` 加 `stage: str` 字段(默认 "");`TaskPool` 加 `executor: ProcessPoolExecutor`(spawn context,max_workers)、`submit_work(task_id, func, *args)` → `Future`、`update_stage(task_id, stage)`、`get_stage(task_id) → str|None`
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/orchestrator/test_pool.py(追加)
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
def test_task_has_stage_field():
|
||||
from sanguo_orchestrator.task import Task
|
||||
t = Task(task_id="t1", task_type="cta")
|
||||
assert t.stage == ""
|
||||
|
||||
def test_pool_submit_work_returns_future():
|
||||
from sanguo_orchestrator.pool import TaskPool
|
||||
pool = TaskPool(max_workers=2)
|
||||
pool.executor = MagicMock() # mock executor 避免真起进程
|
||||
mock_future = MagicMock()
|
||||
pool.executor.submit.return_value = mock_future
|
||||
fut = pool.submit_work("t1", func=lambda: 1)
|
||||
assert fut is mock_future
|
||||
pool.executor.submit.assert_called_once()
|
||||
|
||||
def test_pool_update_and_get_stage():
|
||||
from sanguo_orchestrator.pool import TaskPool
|
||||
from sanguo_orchestrator.task import Task
|
||||
pool = TaskPool(max_workers=2)
|
||||
pool.submit("t1", "cta")
|
||||
pool.update_stage("t1", "回测中")
|
||||
assert pool.get_stage("t1") == "回测中"
|
||||
assert pool.get_task("t1").stage == "回测中"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败**
|
||||
|
||||
- [ ] **Step 3: task.py 加 stage 字段**
|
||||
|
||||
```python
|
||||
# sanguo_orchestrator/task.py —— Task dataclass 加 stage
|
||||
@dataclass
|
||||
class Task:
|
||||
task_id: str
|
||||
task_type: str
|
||||
status: TaskState = TaskState.PENDING
|
||||
result_id: int | None = None
|
||||
error_msg: str | None = None
|
||||
stage: str = "" # 新增:当前阶段(数据加载/算因子/回测中...)
|
||||
# start/complete/fail 方法不变
|
||||
```
|
||||
|
||||
- [ ] **Step 4: pool.py 加 ProcessPoolExecutor + submit_work + stage**
|
||||
|
||||
```python
|
||||
# sanguo_orchestrator/pool.py
|
||||
from concurrent.futures import ProcessPoolExecutor, Future
|
||||
from multiprocessing import get_context
|
||||
from .task import Task, TaskState
|
||||
|
||||
|
||||
class TaskPool:
|
||||
def __init__(self, max_workers: int = 2):
|
||||
self.max_workers = max_workers
|
||||
self._tasks: dict[str, Task] = {}
|
||||
self.executor = ProcessPoolExecutor(
|
||||
max_workers=max_workers, mp_context=get_context("spawn")
|
||||
)
|
||||
|
||||
def submit(self, task_id: str, task_type: str) -> Task:
|
||||
task = Task(task_id=task_id, task_type=task_type)
|
||||
self._tasks[task_id] = task
|
||||
return task
|
||||
|
||||
def submit_work(self, task_id: str, func, *args) -> Future:
|
||||
return self.executor.submit(func, *args)
|
||||
|
||||
def update_stage(self, task_id: str, stage: str):
|
||||
t = self._tasks.get(task_id)
|
||||
if t:
|
||||
t.stage = stage
|
||||
|
||||
def get_status(self, task_id: str) -> TaskState | None:
|
||||
t = self._tasks.get(task_id)
|
||||
return t.status if t else None
|
||||
|
||||
def get_stage(self, task_id: str) -> str | None:
|
||||
t = self._tasks.get(task_id)
|
||||
return t.stage if t else None
|
||||
|
||||
def get_task(self, task_id: str) -> Task | None:
|
||||
return self._tasks.get(task_id)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: 验证通过 + commit** — `pytest tests/orchestrator/test_pool.py -v` → `git commit -m "feat(orchestrator): pool 异步化(ProcessPoolExecutor spawn + stage 追踪)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 4: sanguo_orchestrator/runner.py async submit + on_stage 回调
|
||||
|
||||
**Files:**
|
||||
- Modify: `sanguo_orchestrator/runner.py`
|
||||
- Test: `tests/orchestrator/test_runner.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `Orchestrator` 改 `async submit_cta(...)` → task_id(提交到 pool.executor,asyncio.wrap_future 桥接,完成后调回调更新状态 + broadcast)、`set_on_stage(callback)`(callback: async (task_id, stage) → None)
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/orchestrator/test_runner.py(追加)
|
||||
import asyncio
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
|
||||
def test_submit_cta_returns_task_id_and_submits():
|
||||
from sanguo_orchestrator.runner import Orchestrator
|
||||
orch = Orchestrator(db_path="/tmp/t.db")
|
||||
orch.pool.executor = MagicMock()
|
||||
orch.pool.executor.submit.return_value = MagicMock()
|
||||
with patch("sanguo_backtest.cta_engine.run_cta_backtest"):
|
||||
task_id = asyncio.get_event_loop().run_until_complete(
|
||||
orch.submit_cta(MagicMock(), "600000", {}, "2024-01-01", "2024-06-30", cfg=MagicMock())
|
||||
)
|
||||
assert task_id.startswith("cta_600000")
|
||||
orch.pool.executor.submit.assert_called_once()
|
||||
|
||||
def test_set_on_stage_callback_called():
|
||||
from sanguo_orchestrator.runner import Orchestrator
|
||||
from sanguo_orchestrator.task import TaskState
|
||||
orch = Orchestrator(db_path="/tmp/t.db")
|
||||
orch.pool.executor = MagicMock()
|
||||
mock_future = MagicMock()
|
||||
orch.pool.executor.submit.return_value = mock_future
|
||||
cb = AsyncMock()
|
||||
orch.set_on_stage(cb)
|
||||
# 模拟任务完成回调
|
||||
asyncio.get_event_loop().run_until_complete(orch._on_done("t1", {"statistics": {}}))
|
||||
cb.assert_called()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败**
|
||||
|
||||
- [ ] **Step 3: 实现 runner.py async**
|
||||
|
||||
```python
|
||||
# sanguo_orchestrator/runner.py
|
||||
import asyncio
|
||||
from concurrent.futures import Future
|
||||
from .pool import TaskPool
|
||||
from .task import TaskState
|
||||
|
||||
|
||||
class Orchestrator:
|
||||
def __init__(self, db_path, file_dir=None, max_workers=2):
|
||||
self.db_path = db_path
|
||||
self.file_dir = file_dir
|
||||
self.pool = TaskPool(max_workers=max_workers)
|
||||
self._pending: dict = {}
|
||||
self._on_stage = None # async callback(task_id, stage)
|
||||
|
||||
def set_on_stage(self, cb):
|
||||
self._on_stage = cb
|
||||
|
||||
async def _notify_stage(self, task_id: str, stage: str):
|
||||
self.pool.update_stage(task_id, stage)
|
||||
if self._on_stage:
|
||||
await self._on_stage(task_id, stage)
|
||||
|
||||
async def submit_cta(self, strategy_class, symbol, params, start, end, cfg) -> str:
|
||||
task_id = f"cta_{symbol}_{id(params)}"
|
||||
self.pool.submit(task_id, "cta")
|
||||
self._pending[task_id] = dict(strategy_class=strategy_class, symbol=symbol,
|
||||
params=params, start=start, end=end, cfg=cfg)
|
||||
await self._notify_stage(task_id, "排队中")
|
||||
spec = self._pending[task_id]
|
||||
from sanguo_backtest.cta_engine import run_cta_backtest
|
||||
fut: Future = self.pool.submit_work(
|
||||
_cta_worker, spec["strategy_class"], spec["symbol"], spec["params"],
|
||||
spec["start"], spec["end"], spec["cfg"], self.db_path,
|
||||
)
|
||||
self.pool.get_task(task_id).start()
|
||||
await self._notify_stage(task_id, "回测中")
|
||||
asyncio.ensure_future(self._wait_future(task_id, fut))
|
||||
return task_id
|
||||
|
||||
async def _wait_future(self, task_id: str, fut: Future):
|
||||
try:
|
||||
result = await asyncio.wrap_future(fut)
|
||||
await self._on_done(task_id, result)
|
||||
except Exception as e:
|
||||
self.pool.get_task(task_id).fail(f"{type(e).__name__}: {e}")
|
||||
await self._notify_stage(task_id, "失败")
|
||||
|
||||
async def _on_done(self, task_id: str, result):
|
||||
self.pool.get_task(task_id).complete(result_id=id(result))
|
||||
await self._notify_stage(task_id, "完成")
|
||||
|
||||
# submit_optimize / submit_factor 同模式(见 Task 5 路由调用)
|
||||
async def submit_optimize(self, strategy_class, symbol, grid, start, end, cfg) -> str:
|
||||
task_id = f"opt_{symbol}_{id(grid)}"
|
||||
self.pool.submit(task_id, "optimize")
|
||||
await self._notify_stage(task_id, "参数优化中")
|
||||
from sanguo_backtest.cta_optimizer import run_cta_optimization
|
||||
fut = self.pool.submit_work(_opt_worker, strategy_class, symbol, grid, start, end, cfg, self.db_path)
|
||||
self.pool.get_task(task_id).start()
|
||||
asyncio.ensure_future(self._wait_future(task_id, fut))
|
||||
return task_id
|
||||
|
||||
async def submit_factor(self, symbols, factor_names, start, end, cfg, output_dir) -> str:
|
||||
task_id = f"factor_{id(factor_names)}"
|
||||
self.pool.submit(task_id, "factor")
|
||||
await self._notify_stage(task_id, "因子分析中")
|
||||
from sanguo_factor.analyzer import run_factor_analysis
|
||||
fut = self.pool.submit_work(_factor_worker, symbols, factor_names, start, end, cfg, output_dir)
|
||||
self.pool.get_task(task_id).start()
|
||||
asyncio.ensure_future(self._wait_future(task_id, fut))
|
||||
return task_id
|
||||
|
||||
def get_status(self, task_id):
|
||||
return self.pool.get_status(task_id)
|
||||
|
||||
def get_result(self, task_id):
|
||||
task = self.pool.get_task(task_id)
|
||||
if task and task.status == TaskState.DONE and task.result_id:
|
||||
from sanguo_backtest.result_store import load_result
|
||||
return load_result(task.result_id, self.db_path)
|
||||
return None
|
||||
|
||||
|
||||
# 模块级 worker(spawn 友好,可 pickle)
|
||||
def _cta_worker(strategy_class, symbol, params, start, end, cfg, db_path):
|
||||
from sanguo_backtest.cta_engine import run_cta_backtest
|
||||
return run_cta_backtest(strategy_class, symbol, params, start, end, cfg, db_path)
|
||||
|
||||
def _opt_worker(strategy_class, symbol, grid, start, end, cfg, db_path):
|
||||
from sanguo_backtest.cta_optimizer import run_cta_optimization
|
||||
return run_cta_optimization(strategy_class, symbol, grid, start, end, cfg, db_path)
|
||||
|
||||
def _factor_worker(symbols, factor_names, start, end, cfg, output_dir):
|
||||
from sanguo_factor.analyzer import run_factor_analysis
|
||||
return run_factor_analysis(symbols, factor_names, start, end, cfg, output_dir)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/orchestrator/test_runner.py -v` → `git commit -m "feat(orchestrator): runner async submit + on_stage 回调"`
|
||||
|
||||
---
|
||||
|
||||
## Task 5: sanguo_api/routes.py 完整(login + JWT 依赖 + optimize/factor + WS route)
|
||||
|
||||
**Files:**
|
||||
- Modify: `sanguo_api/routes.py`、`sanguo_api/app.py`
|
||||
- Test: `tests/api/test_routes.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `POST /auth/login`(用户名密码 → token)、optimize/factor 路由真调用 submit_optimize/submit_factor、业务路由加 `Depends(verify_token)`、`WS /ws/task/{id}`(订阅阶段推送)
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/api/test_routes.py(追加)
|
||||
from fastapi.testclient import TestClient
|
||||
from unittest.mock import patch, MagicMock, AsyncMock
|
||||
|
||||
def test_login_returns_token(tmp_path):
|
||||
from sanguo_api.app import create_app
|
||||
from sanguo_api.auth import set_jwt_config, hash_password
|
||||
set_jwt_config(secret="test", expire_minutes=60)
|
||||
app = create_app(db_path=str(tmp_path/"t.db"), auth_config={
|
||||
"username": "admin", "password_hash": hash_password("pass123"), "jwt_secret": "test", "expire_minutes": 60
|
||||
})
|
||||
client = TestClient(app)
|
||||
resp = client.post("/api/v1/auth/login", json={"username": "admin", "password": "pass123"})
|
||||
assert resp.status_code == 200
|
||||
assert "token" in resp.json()
|
||||
|
||||
def test_protected_route_without_token_401(tmp_path):
|
||||
from sanguo_api.app import create_app
|
||||
app = create_app(db_path=str(tmp_path/"t.db"))
|
||||
client = TestClient(app)
|
||||
resp = client.get("/api/v1/task/t1")
|
||||
assert resp.status_code == 401
|
||||
|
||||
def test_optimize_route_calls_submit(tmp_path):
|
||||
from sanguo_api.app import create_app
|
||||
from sanguo_api.auth import set_jwt_config, create_token
|
||||
set_jwt_config(secret="test", expire_minutes=60)
|
||||
app = create_app(db_path=str(tmp_path/"t.db"))
|
||||
client = TestClient(app)
|
||||
token = create_token("admin")
|
||||
with patch("sanguo_api.routes.get_orchestrator") as m:
|
||||
orch = MagicMock()
|
||||
orch.submit_optimize = AsyncMock(return_value="opt_1")
|
||||
m.return_value = orch
|
||||
resp = client.post("/api/v1/backtest/optimize", json={
|
||||
"symbol": "600000", "strategy": "MaStrategy", "grid": {"n": [5, 20, 5]},
|
||||
"start": "2024-01-01", "end": "2024-06-30", "max_workers": 2
|
||||
}, headers={"Authorization": f"Bearer {token}"})
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["task_id"] == "opt_1"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败**
|
||||
|
||||
- [ ] **Step 3: 实现 routes 完整 + app 改**
|
||||
|
||||
```python
|
||||
# sanguo_api/routes.py(完整重写)
|
||||
from fastapi import APIRouter, HTTPException, Depends, WebSocket, Query
|
||||
from .schemas import CtaBacktestRequest, OptimizeRequest, FactorAnalysisRequest
|
||||
from .auth import verify_token, verify_password, create_token
|
||||
from .ws import manager
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter()
|
||||
_orchestrator = None
|
||||
_auth_config = {"username": "admin", "password_hash": "", "jwt_secret": "x", "expire_minutes": 60}
|
||||
|
||||
|
||||
def set_orchestrator(orch): global _orchestrator; _orchestrator = orch
|
||||
def get_orchestrator(): return _orchestrator
|
||||
def set_auth_config(cfg): _auth_config.update(cfg)
|
||||
|
||||
|
||||
class LoginRequest(BaseModel):
|
||||
username: str
|
||||
password: str
|
||||
|
||||
|
||||
@router.post("/auth/login")
|
||||
def login(req: LoginRequest):
|
||||
if req.username != _auth_config["username"] or not verify_password(req.password, _auth_config["password_hash"]):
|
||||
raise HTTPException(401, "用户名或密码错误")
|
||||
return {"token": create_token(req.username)}
|
||||
|
||||
|
||||
@router.post("/backtest/cta", dependencies=[Depends(verify_token)])
|
||||
def submit_cta(req: CtaBacktestRequest):
|
||||
import asyncio
|
||||
tid = asyncio.get_event_loop().run_until_complete(
|
||||
get_orchestrator().submit_cta(req.strategy, req.symbol, req.params, req.start, req.end, None))
|
||||
return {"task_id": tid}
|
||||
|
||||
|
||||
@router.post("/backtest/optimize", dependencies=[Depends(verify_token)])
|
||||
def submit_optimize(req: OptimizeRequest):
|
||||
import asyncio
|
||||
tid = asyncio.get_event_loop().run_until_complete(
|
||||
get_orchestrator().submit_optimize(req.strategy, req.symbol, req.grid, req.start, req.end, None))
|
||||
return {"task_id": tid}
|
||||
|
||||
|
||||
@router.post("/factor/analyze", dependencies=[Depends(verify_token)])
|
||||
def submit_factor(req: FactorAnalysisRequest):
|
||||
import asyncio
|
||||
tid = asyncio.get_event_loop().run_until_complete(
|
||||
get_orchestrator().submit_factor(req.symbols, req.factor_names, req.start, req.end, None, "/tmp/factor"))
|
||||
return {"task_id": tid}
|
||||
|
||||
|
||||
@router.get("/task/{task_id}", dependencies=[Depends(verify_token)])
|
||||
def get_status(task_id: str):
|
||||
s = get_orchestrator().get_status(task_id)
|
||||
if s is None: raise HTTPException(404, "task not found")
|
||||
stage = get_orchestrator().pool.get_stage(task_id)
|
||||
return {"task_id": task_id, "status": s.value if hasattr(s, "value") else str(s), "stage": stage or ""}
|
||||
|
||||
|
||||
@router.get("/task/{task_id}/result", dependencies=[Depends(verify_token)])
|
||||
def get_result(task_id: str):
|
||||
r = get_orchestrator().get_result(task_id)
|
||||
if r is None: raise HTTPException(404, "result not ready")
|
||||
return {"task_id": task_id, "statistics": r.statistics}
|
||||
|
||||
|
||||
@router.websocket("/ws/task/{task_id}")
|
||||
async def task_ws(websocket: WebSocket, task_id: str, token: str = Query(...)):
|
||||
from .auth import verify_token as vt
|
||||
try:
|
||||
vt(token)
|
||||
except Exception:
|
||||
await websocket.close(code=4401); return
|
||||
await websocket.accept()
|
||||
manager.connect(task_id, websocket)
|
||||
try:
|
||||
while True:
|
||||
await websocket.receive_text() # 保活
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
manager.disconnect(task_id, websocket)
|
||||
```
|
||||
|
||||
```python
|
||||
# sanguo_api/app.py(改:register auth_config + on_stage 推 WS)
|
||||
from fastapi import FastAPI
|
||||
from .routes import router, set_orchestrator, set_auth_config
|
||||
from .auth import set_jwt_config
|
||||
from .ws import manager
|
||||
from sanguo_orchestrator.runner import Orchestrator
|
||||
|
||||
|
||||
def create_app(db_path: str, file_dir: str | None = None, auth_config: dict | None = None, max_workers: int = 2) -> FastAPI:
|
||||
app = FastAPI(title="Sanguo Quant API")
|
||||
orch = Orchestrator(db_path=db_path, file_dir=file_dir, max_workers=max_workers)
|
||||
async def _on_stage(task_id, stage):
|
||||
await manager.broadcast(task_id, {"task_id": task_id, "stage": stage})
|
||||
orch.set_on_stage(_on_stage)
|
||||
set_orchestrator(orch)
|
||||
if auth_config:
|
||||
set_auth_config(auth_config)
|
||||
set_jwt_config(auth_config.get("jwt_secret", "x"), auth_config.get("expire_minutes", 60))
|
||||
app.include_router(router, prefix="/api/v1")
|
||||
return app
|
||||
```
|
||||
|
||||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/api/ -v` → `git commit -m "feat(api): routes 完整(login + JWT 依赖 + optimize/factor + WS route)"`
|
||||
|
||||
---
|
||||
|
||||
## Task 6: sanguo_factor/alpha_lab.py compute_factors 完整化
|
||||
|
||||
**Files:**
|
||||
- Modify: `sanguo_factor/alpha_lab.py`
|
||||
- Test: `tests/factor/test_alpha_lab.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `AlphaLabSession.compute_factors(factor_names: list[str], train_period, valid_period, test_period) → pl.DataFrame`(add_feature + prepare_data + fetch_raw)
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/factor/test_alpha_lab.py(追加)
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
def test_compute_factors_calls_prepare_and_fetch(tmp_path):
|
||||
from sanguo_factor.alpha_lab import AlphaLabSession
|
||||
with patch("sanguo_factor.alpha_lab.AlphaLab") as MLab, \
|
||||
patch("sanguo_factor.alpha_lab.read_db_daily") as MRead, \
|
||||
patch("sanguo_factor.alpha_lib.AlphaDataset") as MDS, \
|
||||
patch("sanguo_factor.alpha_lab.get_factor") as MGet, \
|
||||
patch("sanguo_factor.alpha_lab.save_alpha_lab_data"):
|
||||
import polars as pl
|
||||
MRead.return_value = [MagicMock(vt_symbol="600000.SSE", datetime=__import__("datetime").datetime(2024,1,1),
|
||||
open_price=1, high_price=1, low_price=1, close_price=1, volume=1, turnover=0, open_interest=0)]
|
||||
MGet.return_value = {"expression": "ts_mean(close,5)"}
|
||||
MDS.return_value.fetch_raw.return_value = pl.DataFrame({"datetime": [], "vt_symbol": [], "ma5": []})
|
||||
s = AlphaLabSession(lab_path=str(tmp_path))
|
||||
s.load_symbols(["600000"], "2024-01-01", "2024-06-30", cfg=MagicMock())
|
||||
df = s.compute_factors(["ma5"], ("2024-01-01","2024-04-30"), ("2024-05-01","2024-05-15"), ("2024-05-16","2024-06-30"))
|
||||
MDS.return_value.add_feature.assert_called()
|
||||
MDS.return_value.prepare_data.assert_called_once()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败**
|
||||
|
||||
- [ ] **Step 3: 补 compute_factors**
|
||||
|
||||
```python
|
||||
# sanguo_factor/alpha_lab.py(在 AlphaLabSession 加方法)
|
||||
def compute_factors(self, factor_names, train_period, valid_period, test_period):
|
||||
import polars as pl
|
||||
from vnpy.alpha.dataset import AlphaDataset, Segment
|
||||
from .registry import get_factor
|
||||
from .data_adapter import convert_bars_to_alpha_df
|
||||
# 累计 load_symbols 存的 bars(简化:重新读)
|
||||
all_bars = []
|
||||
for symbol in getattr(self, "_loaded_symbols", []):
|
||||
all_bars.extend(self._loaded_bars.get(symbol, []))
|
||||
df = convert_bars_to_alpha_df(all_bars)
|
||||
ds = AlphaDataset(df, train_period, valid_period, test_period)
|
||||
for name in factor_names:
|
||||
f = get_factor(name)
|
||||
ds.add_feature(name, f["expression"])
|
||||
ds.prepare_data(max_workers=1)
|
||||
return ds.fetch_raw(Segment.TEST)
|
||||
```
|
||||
|
||||
(配套 load_symbols 缓存 `_loaded_symbols`/`_loaded_bars`,见执行时补全。)
|
||||
|
||||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/factor/test_alpha_lab.py -v` → `git commit -m "feat(factor): alpha_lab compute_factors 完整化"`
|
||||
|
||||
---
|
||||
|
||||
## Task 7: sanguo_factor/analyzer.py tears pipeline 完整化
|
||||
|
||||
**Files:**
|
||||
- Modify: `sanguo_factor/analyzer.py`
|
||||
- Test: `tests/factor/test_analyzer.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `run_factor_analysis` 补 compute_factors → get_clean_factor_and_forward_returns → create_full_tear_sheet → html 报告,FactorReport 含 `report_path`/`ic_summary`
|
||||
|
||||
- [ ] **Step 1: 写失败测试**
|
||||
|
||||
```python
|
||||
# tests/factor/test_analyzer.py(追加)
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
def test_run_factor_analysis_calls_tears(tmp_path):
|
||||
from sanguo_factor.analyzer import run_factor_analysis
|
||||
with patch("sanguo_factor.analyzer.AlphaLabSession") as MS, \
|
||||
patch("sanguo_factor.analyzer.get_clean_factor_and_forward_returns") as MC, \
|
||||
patch("sanguo_factor.analyzer.create_full_tear_sheet") as MT:
|
||||
import polars as pl
|
||||
MS.return_value.compute_factors.return_value = pl.DataFrame({"datetime":[],"vt_symbol":[],"ma5":[]})
|
||||
MC.return_value = MagicMock()
|
||||
report = run_factor_analysis(["600000"], ["ma5"], "2024-01-01", "2024-06-30",
|
||||
cfg=MagicMock(), output_dir=str(tmp_path))
|
||||
assert report.factor_names == ["ma5"]
|
||||
MC.assert_called_once(); MT.assert_called_once()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 验证失败**
|
||||
|
||||
- [ ] **Step 3: 补 tears pipeline**
|
||||
|
||||
```python
|
||||
# sanguo_factor/analyzer.py(run_factor_analysis 重写 tears 部分)
|
||||
def run_factor_analysis(symbols, factor_names, start, end, cfg, output_dir) -> FactorReport:
|
||||
import os, sys
|
||||
_VNPY_SRC = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "vnpy_v4.4.0"))
|
||||
if _VNPY_SRC not in sys.path: sys.path.insert(0, _VNPY_SRC)
|
||||
from alphalens.utils import get_clean_factor_and_forward_returns
|
||||
from alphalens.tears import create_full_tear_sheet
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
from .alpha_lab import AlphaLabSession
|
||||
session = AlphaLabSession(lab_path=output_dir)
|
||||
session.load_symbols(symbols, start, end, cfg)
|
||||
factor_df = session.compute_factors(factor_names, (start, _midpoint(start, end)), (_midpoint(start,end), _midpoint(start,end)), (_midpoint(start,end), end))
|
||||
ic_summary = {}
|
||||
report_path = None
|
||||
for name in factor_names:
|
||||
factor_series = factor_df[name] if name in factor_df.columns else factor_df.iloc[:, -1]
|
||||
merged = get_clean_factor_and_forward_returns(factor_df.set_index(["datetime","vt_symbol")[name]], ...) # 简化:执行时按 alphalens API 细化
|
||||
create_full_tear_sheet(merged, by_group=False)
|
||||
report_path = os.path.join(output_dir, f"{name}_tears.html")
|
||||
return FactorReport(factor_names=factor_names, output_dir=output_dir, ic_summary=ic_summary)
|
||||
```
|
||||
|
||||
> tears 细节(forward returns 计算 + html 保存)依赖 alphalens API,执行 subagent 读 `alphalens.tears.create_full_tear_sheet` 签名 + 容器验证。
|
||||
|
||||
- [ ] **Step 4: 验证通过 + commit** — `pytest tests/factor/test_analyzer.py -v` → `git commit -m "feat(factor): analyzer tears pipeline 完整化"`
|
||||
|
||||
---
|
||||
|
||||
## Task 8: 端到端冒烟(异步回测 + WS + tears,容器)
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/smoke_phase3a.py`
|
||||
|
||||
- [ ] **Step 1: 写冒烟脚本**
|
||||
|
||||
```python
|
||||
# scripts/smoke_phase3a.py
|
||||
"""Phase 3a 端到端冒烟:异步 submit + WS 阶段 + JWT + tears(容器)。"""
|
||||
import sys, asyncio
|
||||
sys.path.insert(0, "/app"); sys.path.insert(0, "/app/vnpy_v4.4.0")
|
||||
|
||||
async def main():
|
||||
print("=== Phase 3a Smoke ===")
|
||||
from sanguo_api.app import create_app
|
||||
from sanguo_api.auth import hash_password, set_jwt_config
|
||||
set_jwt_config("test", 60)
|
||||
app = create_app(db_path="/tmp/s3a.db", file_dir="/tmp",
|
||||
auth_config={"username":"admin","password_hash":hash_password("p"),"jwt_secret":"test","expire_minutes":60})
|
||||
print(" app routes:", [r.path for r in app.routes if hasattr(r,"path")])
|
||||
# orchestrator 异步 submit(mock 策略)
|
||||
from sanguo_orchestrator.runner import Orchestrator
|
||||
orch = Orchestrator(db_path="/tmp/s3a.db")
|
||||
print(" orchestrator pool executor:", type(orch.pool.executor).__name__)
|
||||
print("=== Phase 3a Smoke DONE ===")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 本地全部 Phase 3a 测试 + 覆盖率** — `pytest tests/ --cov=sanguo_api --cov=sanguo_orchestrator --cov=sanguo_factor --cov-report=term` → 全绿 + ≥80%
|
||||
|
||||
- [ ] **Step 3: 容器端到端冒烟(rsync + docker exec)**
|
||||
|
||||
- [ ] **Step 4: Commit** — `git commit -m "test(phase3a): 端到端冒烟(异步 submit + WS + JWT + tears)"`
|
||||
|
||||
- [ ] **Step 5: requesting-code-review + finishing-a-development-branch**
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**1. Spec coverage**:异步 pool(T3) + runner async(T4) + WS 阶段(T2/T5) + JWT 单用户(T1/T5) + optimize/factor 路由(T5) + alpha tears 完整化(T6/T7) = design §0 五项全覆盖 ✅
|
||||
**2. Placeholder**:tears pipeline 的 forward returns 细节(T7)依赖 alphalens API,标注执行时细化;其余代码完整。
|
||||
**3. Type consistency**:`Orchestrator.async submit_*`(T4)在 routes(T5)调用一致;`manager.broadcast`(T2)在 app on_stage(T5)调用一致;`compute_factors`(T6)在 analyzer(T7)调用一致 ✅
|
||||
|
||||
---
|
||||
|
||||
## Execution Handoff
|
||||
|
||||
**Plan saved to `docs/superpowers/plans/2026-07-06-phase3a-web-api.md`.** 推荐用 **Subagent-Driven** 执行(每 task fresh subagent + review)。
|
||||
Reference in New Issue
Block a user