claude_dev
c3e53fbef3
- scripts/data_platform/_archive/legacy/: 归档20个独立探针/诊断/旧降级脚本(零引用验证) - docs/archive/data/: 归档17个数据相关旧设计/plan/report(保留fusion spec作深读) - docs/data-platform/README.md: 数据层单一权威记录(8节:架构/布局/源/管线/铁律/API/缺口/待办) - 删除 _mootdx_depth_result.txt - Phase2待办: 15m灌库链+旧回填import链(有测试/wrapper依赖,VPS schtask确认后归档)
602 lines
30 KiB
Markdown
602 lines
30 KiB
Markdown
# LocalUnifiedProvider Implementation Plan (spec §6 使用层)
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** 实现 spec §6 使用层 `LocalUnifiedProvider`——读方案A 权威数据层(dbbardata/constituent_unified/valuation_baostock),零 online,治幸存者偏差,喂 all_weather 策略。
|
||
|
||
**Architecture:** 新建 `LocalUnifiedProvider(bullet_trade.DataProvider)`,内部按数据类路由方案A 权威表:日线读 `dbbardata('d')` raw + `bs_adjust_factor` 算前复权;成份股读 `constituent_unified` 并集(治偏差);估值读 `valuation_baostock` parquet + 市值读 static/valuation akshare parquet。Mac 测试用 `sqlite :memory:` + tmp parquet fixture,零 VPS 依赖。
|
||
|
||
**Tech Stack:** Python 3.10, pandas 2.3, sqlite3, pyarrow, pytest
|
||
|
||
## Global Constraints(spec + 用户铁律)
|
||
|
||
- **零 online**: provider 不 import baostock 调 online,纯读本地 DB/parquet(memory provider-local-data-only)。baostock 48000/天限频不波及使用层。
|
||
- **surgical**: 不改 `LocalParquetProvider`/`BaostockProvider`(旧链路保留,向后兼容)。
|
||
- **dbbardata 不破坏**: `UNIQUE(symbol,exchange,interval,datetime)`,只读不写。
|
||
- **复权**: dbbardata 存 raw,消费端按 `bs_adjust_factor.foreAdjustFactor` 算前复权(§14.7 最终目标,用户定不降级)。
|
||
- **constituent_unified 并集模型**: 表无 date 列,`get_index_stocks(date)` 返回 in_current∪was_removed 并集,date 参数无法精确时点过滤——治"纯当前幸存者"偏差,有轻微前视(使用说明标注)。
|
||
- **代码归一**: jq_code `600519.XSHG` ↔ dbbardata `symbol=600519, exchange=SSE`;`SSE→SH, SZSE→SZ`。
|
||
|
||
## 实测 schema(VPS 2026-07-23 probe,执行 agent 必读)
|
||
|
||
DB = `C:\sanguo_vnpy_v2\data\quant_trading.db`(VPS) / Mac 测试用 fixture 路径。
|
||
|
||
**dbbardata('d')** — 唯一行情表,raw 真实价:
|
||
```
|
||
列: symbol TEXT, exchange TEXT(SSE/SZSE), datetime TEXT(YYYY-MM-DD HH:MM:SS),
|
||
interval TEXT('d'), volume REAL, turnover REAL, open_interest REAL,
|
||
open_price REAL, high_price REAL, low_price REAL, close_price REAL
|
||
样本: 600519 10056行 2001-08-27~2026-07-22; 000005退市 8146行~2024-04-26; 510300 ETF 3439行
|
||
```
|
||
|
||
**constituent_unified** — 成份股并集(无 date!):
|
||
```
|
||
列: index_code TEXT(如 '000300'), code TEXT(纯6位如 '000001'), code_name TEXT,
|
||
source TEXT('baostock'/'akshare'), in_current INT(0/1), was_removed INT(0/1)
|
||
分布: 000300=940(300当前+640被踢) 000905=1803 000016=195 000852=1000(全当前,历史不可补)
|
||
399001=702 399005=145 399006=175 399330=150 932000=2000(全当前)
|
||
```
|
||
|
||
**bs_adjust_factor** — 复权因子:
|
||
```
|
||
列: code TEXT('sh.600519'), dividOperateDate TEXT(YYYY-MM-DD),
|
||
foreAdjustFactor REAL, backAdjustFactor REAL, adjustFactor REAL
|
||
语义: foreAdjustFactor 按除权日分段,最新事件=1.0,递减往历史。qfq[t]=raw[t]*factor[date[t]]。
|
||
600519 有 12 事件: 2020-06-24=0.856267 ... 2026-06-26=1.0
|
||
```
|
||
|
||
**valuation_baostock/<year>.parquet** — baostock 估值(1990-2026 全年份):
|
||
```
|
||
列: symbol(6位), exchange(SH/SZ), date(YYYY-MM-DD), peTTM, psTTM, pcfNcfTTM, pbMRQ, turn, pctChg, isST
|
||
注: 无 market_cap/total_share 列! 市值从 static/valuation akshare 补。
|
||
```
|
||
|
||
**static/valuation/<code>_valuation.parquet** — akshare 估值(市值/股本来源,5530 文件):
|
||
```
|
||
中文列(见 LocalParquetProvider._VAL_COL_MAP): 总市值→total_market_cap, 流通市值→circ_market_cap,
|
||
总股本→total_share, PE(TTM)→pe_ttm, 市净率→pb ...
|
||
```
|
||
|
||
**static/{balance,income,cashflow}/<code>_<table>.parquet** — akshare 三表(balance 221列/income 170列):
|
||
```
|
||
通用列: SECUCODE, REPORT_DATE, REPORT_TYPE; balance 有 TOTAL_ASSETS/TOTAL_LIABILITIES/TOTAL_PARENT_EQUITY;
|
||
income 有 BASIC_EPS/OPERATE_INCOME/PARENT_NETPROFIT/OPERATE_INCOME_YOY
|
||
```
|
||
|
||
---
|
||
|
||
## File Structure
|
||
|
||
- **Create:** `sanguo_portfolio/providers/local_unified_provider.py` — LocalUnifiedProvider 类(~400行)
|
||
- **Modify:** `sanguo_portfolio/providers/__init__.py` — 导出 LocalUnifiedProvider
|
||
- **Modify:** `sanguo_portfolio/runner_backtest.py` — `build_provider` 加 `unified` 选项(choices + 分支)
|
||
- **Create:** `tests/portfolio/test_local_unified_provider.py` — DataProvider 契约单测(fixture: sqlite + tmp parquet)
|
||
- **Create:** `tests/portfolio/conftest.py` 追加 — `local_unified_provider` fixture(若需要,否则在测试文件内建)
|
||
- **Create:** `docs/portfolio_local_unified_provider.md` — 使用说明(架构/数据源/接口/复权/治偏差/Mac测试/部署)
|
||
|
||
---
|
||
|
||
## Task 0: 代码转换 + DB 连接辅助 + 复权因子构造
|
||
|
||
**Files:**
|
||
- Create: `sanguo_portfolio/providers/local_unified_provider.py`(本 task 建文件骨架 + 模块级辅助函数)
|
||
- Test: `tests/portfolio/test_local_unified_provider.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `jq_to_dbbardata(jq_code) -> (symbol, exchange)` / `dbbardata_to_jq(symbol, exchange) -> jq_code`; `_connect(cfg) -> sqlite3.Connection`; `_build_qfq_factor(code, conn, dates) -> pd.Series(factor indexed by date)`
|
||
|
||
- [ ] **Step 1: 写失败测试 — 代码转换**
|
||
|
||
```python
|
||
# tests/portfolio/test_local_unified_provider.py
|
||
from sanguo_portfolio.providers.local_unified_provider import (
|
||
jq_to_dbbardata, dbbardata_to_jq, LocalUnifiedProvider,
|
||
)
|
||
|
||
def test_jq_to_dbbardata_roundtrip():
|
||
assert jq_to_dbbardata("600519.XSHG") == ("600519", "SSE")
|
||
assert jq_to_dbbardata("000001.XSHE") == ("000001", "SZSE")
|
||
assert jq_to_dbbardata("600519") == ("600519", "SSE") # 纯6位推断
|
||
assert dbbardata_to_jq("600519", "SSE") == "600519.XSHG"
|
||
assert dbbardata_to_jq("000001", "SZSE") == "000001.XSHE"
|
||
```
|
||
|
||
- [ ] **Step 2: 跑测试确认 FAIL** — `pytest tests/portfolio/test_local_unified_provider.py::test_jq_to_dbbardata_roundtrip -v`(ImportError)
|
||
|
||
- [ ] **Step 3: 实现模块骨架 + 代码转换**
|
||
|
||
```python
|
||
# sanguo_portfolio/providers/local_unified_provider.py
|
||
"""LocalUnifiedProvider: 读方案A 权威数据层, 零 online, 治幸存者偏差(spec §6)。
|
||
|
||
数据源(全本地 VPS C:\\sanguo_vnpy_v2\\data\\):
|
||
- 日线: dbbardata('d') raw + bs_adjust_factor 算前复权(§14.7)
|
||
- 成份股: constituent_unified 并集(治偏差,无 date 时点)
|
||
- 估值 pe/pb/ps/pcf: valuation_baostock/<year>.parquet(baostock 权威)
|
||
- 市值/股本: static/valuation akshare parquet(baostock valuation 无市值列)
|
||
- 三表: static/{balance,income,cashflow} akshare parquet
|
||
|
||
零 online: 不 import baostock 调 online。Mac 测试用 sqlite+parquet fixture。
|
||
"""
|
||
from __future__ import annotations
|
||
import logging, os, sqlite3
|
||
from datetime import datetime
|
||
from pathlib import Path
|
||
from typing import Any, Dict, List, Optional, Union
|
||
import pandas as pd
|
||
|
||
try:
|
||
from bullet_trade.data.providers.base import DataProvider # type: ignore
|
||
except ImportError:
|
||
class DataProvider: # type: ignore[no-redef]
|
||
name: str = "base"
|
||
|
||
logger = logging.getLogger(__name__)
|
||
_DEFAULT_DB = r"C:\sanguo_vnpy_v2\data\quant_trading.db"
|
||
_DEFAULT_DATA_DIR = r"C:\sanguo_vnpy_v2\data"
|
||
|
||
_JQ_SUFFIX_TO_EXC = {"XSHG": "SSE", "XSHE": "SZSE", "SH": "SSE", "SZ": "SZSE"}
|
||
_EXC_TO_JQ_SUFFIX = {"SSE": "XSHG", "SZSE": "XSHE"}
|
||
|
||
|
||
def jq_to_dbbardata(jq_code: str) -> tuple[str, str]:
|
||
"""600519.XSHG → ('600519', 'SSE')。纯6位按6开头=sh/0,3=sz 推断。"""
|
||
s = (jq_code or "").strip()
|
||
if "." not in s:
|
||
if len(s) == 6:
|
||
return s, ("SSE" if s.startswith("6") else "SZSE")
|
||
return s, "SSE"
|
||
code, suffix = s.split(".", 1)
|
||
return code, _JQ_SUFFIX_TO_EXC.get(suffix.upper(), "SSE")
|
||
|
||
|
||
def dbbardata_to_jq(symbol: str, exchange: str) -> str:
|
||
"""('600519','SSE') → '600519.XSHG'。"""
|
||
jq_suffix = _EXC_TO_JQ_SUFFIX.get(str(exchange).upper(), "XSHG")
|
||
return f"{symbol}.{jq_suffix}"
|
||
|
||
|
||
# 复权因子代码转换: 600519.XSHG → 'sh.600519'(bs_adjust_factor.code 格式)
|
||
def _jq_to_bs_code(jq_code: str) -> str:
|
||
sym, exc = jq_to_dbbardata(jq_code)
|
||
prefix = "sh" if exc == "SSE" else "sz"
|
||
return f"{prefix}.{sym}"
|
||
```
|
||
|
||
- [ ] **Step 4: 跑测试确认 PASS**
|
||
|
||
- [ ] **Step 5: 写失败测试 — 复权因子构造**
|
||
|
||
```python
|
||
def test_build_qfq_factor(tmp_path):
|
||
# fixture: 2 除权事件, 最新=1.0
|
||
import sqlite3
|
||
db = tmp_path / "t.db"
|
||
c = sqlite3.connect(str(db))
|
||
c.execute("CREATE TABLE bs_adjust_factor(code TEXT, dividOperateDate TEXT, foreAdjustFactor REAL, backAdjustFactor REAL, adjustFactor REAL)")
|
||
c.executemany("INSERT INTO bs_adjust_factor VALUES(?,?,?,?,?)", [
|
||
("sh.600519", "2024-06-19", 0.90, 0, 0),
|
||
("sh.600519", "2025-06-19", 1.00, 0, 0),
|
||
])
|
||
c.commit(); c.close()
|
||
from sanguo_portfolio.providers.local_unified_provider import _build_qfq_factor
|
||
dates = pd.to_datetime(["2023-01-01", "2024-07-01", "2025-07-01"])
|
||
f = _build_qfq_factor("sh.600519", sqlite3.connect(str(db)), dates)
|
||
# 2023(早于最早事件)=0.90; 2024-07(between)=0.90; 2025-07(最新后)=1.00
|
||
assert abs(f.iloc[0] - 0.90) < 1e-6
|
||
assert abs(f.iloc[1] - 0.90) < 1e-6
|
||
assert abs(f.iloc[2] - 1.00) < 1e-6
|
||
```
|
||
|
||
- [ ] **Step 6: 实现 `_build_qfq_factor`** — asof join 逻辑(每个 date 找 ≤ 的最大 dividOperateDate 的 foreAdjustFactor;早于所有事件用最早;晚于所有用最新):
|
||
|
||
```python
|
||
def _build_qfq_factor(bs_code: str, conn: sqlite3.Connection,
|
||
dates: pd.Series) -> pd.Series:
|
||
"""构造每个 date 的前复权因子(asof)。qfq[t]=raw[t]*factor[t]。"""
|
||
rows = conn.execute(
|
||
"SELECT dividOperateDate, foreAdjustFactor FROM bs_adjust_factor "
|
||
"WHERE code=? ORDER BY dividOperateDate", (bs_code,)).fetchall()
|
||
if not rows:
|
||
return pd.Series([1.0] * len(dates), index=dates)
|
||
ev_dates = pd.to_datetime([r[0] for r in rows])
|
||
factors = [float(r[1]) for r in rows]
|
||
out = []
|
||
for d in pd.to_datetime(dates):
|
||
# 找 <= d 的最大事件; 全部 > d 用最早(第一个); 全部 <= d 用最后一个
|
||
mask = ev_dates <= d
|
||
out.append(factors[mask.argmax()] if mask.any() else factors[0])
|
||
# mask.argmax() 给第一个 True 的索引;但我们要"<= d 的最大事件"= 最后一个 True
|
||
# 修正:取最后一个 True
|
||
out = []
|
||
for d in pd.to_datetime(dates):
|
||
mask = ev_dates <= d
|
||
idx = int(np.where(mask)[0][-1]) if mask.any() else 0
|
||
out.append(factors[idx])
|
||
return pd.Series(out, index=pd.to_datetime(dates))
|
||
```
|
||
(注意:`np` 需 `import numpy as np`。实现时简化为单次循环取最后一个 True 索引。)
|
||
|
||
- [ ] **Step 7: 跑测试确认 PASS**
|
||
- [ ] **Step 8: Commit** — `feat(portfolio): LocalUnifiedProvider 代码转换+复权因子(Task0)`
|
||
|
||
---
|
||
|
||
## Task 1: get_price(dbbardata raw + 前复权 + panel 长表)
|
||
|
||
**Files:** Modify `local_unified_provider.py` 加 `__init__` + `get_price`; Test 同文件。
|
||
|
||
**Interfaces:**
|
||
- Consumes: Task0 辅助函数 + `_connect`
|
||
- Produces: `LocalUnifiedProvider.get_price(security, start_date, end_date, frequency, fields, skip_paused, fq, count, panel, fill_paused) -> DataFrame`
|
||
|
||
策略契约(all_weather 实证):
|
||
- `get_price(hold_list, end_date, freq=daily, fields=[close,high_limit], count=1, panel=False)` — panel=False 长表需 time/code 列
|
||
- `get_price(stocks, freq=1d, fields=[close], count=n, panel=False)` — _trend_mean pivot(index=time,columns=code)
|
||
- `get_price(stock, freq=1m, fq="pre", count=1, panel=False)` — intraday(day 频率回测降级,1m 无数据返空)
|
||
|
||
- [ ] **Step 1: 写失败测试 — get_price daily 单股 + 复权**
|
||
|
||
```python
|
||
@pytest.fixture
|
||
def unified_provider(tmp_path):
|
||
"""造小样本 sqlite + parquet fixture。"""
|
||
db = tmp_path / "quant_trading.db"
|
||
c = sqlite3.connect(str(db))
|
||
c.execute("CREATE TABLE dbbardata(symbol,exchange,datetime,interval,volume,turnover,open_interest,open_price,high_price,low_price,close_price)")
|
||
rows = [
|
||
("600519","SSE","2024-06-18 00:00:00","d",1000,1e6,0,1000.0,1010.0,990.0,1000.0), # 除权前
|
||
("600519","SSE","2024-06-19 00:00:00","d",1000,1e6,0,900.0,910.0,890.0,900.0), # 除权日 raw 跳水
|
||
("600519","SSE","2024-06-20 00:00:00","d",1000,1e6,0,910.0,920.0,900.0,910.0),
|
||
]
|
||
c.executemany("INSERT INTO dbbardata VALUES(?,?,?,?,?,?,?,?,?,?,?)", rows)
|
||
c.execute("CREATE TABLE bs_adjust_factor(code,dividOperateDate,foreAdjustFactor,backAdjustFactor,adjustFactor)")
|
||
c.execute("INSERT INTO bs_adjust_factor VALUES('sh.600519','2024-06-19',0.9,0,0)") # 除权日 factor
|
||
c.commit(); c.close()
|
||
return LocalUnifiedProvider({"db_path": str(db), "data_dir": str(tmp_path)})
|
||
|
||
def test_get_price_raw_vs_qfq(unified_provider):
|
||
p = unified_provider
|
||
# raw: 除权日 900 跳水
|
||
df_raw = p.get_price("600519.XSHG", start_date="2024-06-18", end_date="2024-06-20", fq="raw")
|
||
assert len(df_raw) == 3
|
||
assert abs(df_raw.loc["2024-06-19", "close"] - 900.0) < 1e-6
|
||
# qfq: 06-18 = 1000*0.9 = 900; 06-19/20 = raw(factor=0.9 当 06-19 之后? 用最新段逻辑)
|
||
df_qfq = p.get_price("600519.XSHG", start_date="2024-06-18", end_date="2024-06-20", fq="qfq")
|
||
assert abs(df_qfq.loc["2024-06-18", "close"] - 900.0) < 1e-6 # 1000*0.9(早于事件用最早factor)
|
||
```
|
||
(复权断言:06-18 早于除权日 06-19 → 用 factor 0.9 → 1000*0.9=900;06-19/20 ≥ 事件日 → factor 取 06-19 的 0.9 → 900*0.9=810, 910*0.9=819。实现时按 `_build_qfq_factor` 语义校准断言。)
|
||
|
||
- [ ] **Step 2: 跑测试确认 FAIL**
|
||
|
||
- [ ] **Step 3: 实现 `__init__` + `get_price`**
|
||
|
||
```python
|
||
class LocalUnifiedProvider(DataProvider): # type: ignore[misc]
|
||
name: str = "sanguo_local_unified"
|
||
requires_live_data: bool = False
|
||
|
||
def __init__(self, config: Optional[Dict[str, Any]] = None) -> None:
|
||
cfg = config or {}
|
||
self.db_path: str = cfg.get("db_path", _DEFAULT_DB)
|
||
self.data_dir: str = cfg.get("data_dir", _DEFAULT_DATA_DIR)
|
||
self._conn: Optional[sqlite3.Connection] = None
|
||
self._val_bs_cache: Dict[int, pd.DataFrame] = {} # year -> valuation_baostock
|
||
|
||
def _connect(self) -> sqlite3.Connection:
|
||
if self._conn is None:
|
||
self._conn = sqlite3.connect(self.db_path, timeout=30)
|
||
self._conn.execute("PRAGMA busy_timeout = 30000")
|
||
return self._conn
|
||
|
||
def get_price(self, security, start_date=None, end_date=None, frequency="daily",
|
||
fields=None, skip_paused=False, fq="raw", count=None,
|
||
panel=True, fill_paused=True, **kwargs):
|
||
freq = str(frequency or "").lower()
|
||
if freq not in ("daily", "day", "1d", "d"):
|
||
return pd.DataFrame() # 1m/分钟 day 频率回测降级(数据层无 1m)
|
||
secs = [security] if isinstance(security, str) else list(security or [])
|
||
conn = self._connect()
|
||
start_str = self._to_date_str(start_date)
|
||
end_str = self._to_date_str(end_date) or datetime.now().strftime("%Y-%m-%d")
|
||
|
||
frames: Dict[str, pd.DataFrame] = {}
|
||
for jq_code in secs:
|
||
sym, exc = jq_to_dbbardata(jq_code)
|
||
q = "SELECT datetime, open_price, high_price, low_price, close_price, " \
|
||
"volume, turnover FROM dbbardata WHERE symbol=? AND exchange=? " \
|
||
"AND interval='d' AND datetime>=? AND datetime<=? ORDER BY datetime"
|
||
df = pd.read_sql(q, conn, params=(sym, exc, start_str + " 00:00:00", end_str + " 23:59:59"))
|
||
if df.empty:
|
||
frames[jq_code] = df; continue
|
||
df["datetime"] = pd.to_datetime(df["datetime"])
|
||
df = df.set_index("datetime")
|
||
df.index.name = None
|
||
if count:
|
||
df = df.tail(count)
|
||
# 复权
|
||
if fq in ("qfq", "pre", "前复权"):
|
||
factor = _build_qfq_factor(_jq_to_bs_code(jq_code), conn, df.index)
|
||
for col in ("open_price", "high_price", "low_price", "close_price"):
|
||
df[col] = df[col].values * factor.values
|
||
# 策略要 close/high_limit 字段名(jq 风格)
|
||
df = df.rename(columns={"open_price": "open", "high_price": "high",
|
||
"low_price": "low", "close_price": "close"})
|
||
# high_limit 不在 dbbardata, 留给 get_current_tick 语义;这里策略 prepare_stock_list 要 high_limit 列
|
||
# → 缺失列返 NaN(策略 hit = close==high_limit 不会命中,降级可接受)
|
||
if fields:
|
||
for f in fields:
|
||
if f not in df.columns:
|
||
df[f] = float("nan")
|
||
df = df[fields]
|
||
frames[jq_code] = df
|
||
|
||
if not frames or all(f.empty for f in frames.values()):
|
||
return pd.DataFrame()
|
||
if not panel:
|
||
parts = []
|
||
for jq_code, df in frames.items():
|
||
if df.empty:
|
||
continue
|
||
d = df.reset_index().rename(columns={"datetime": "time"})
|
||
d.insert(0, "code", jq_code)
|
||
parts.append(d)
|
||
return pd.concat(parts, ignore_index=True) if parts else pd.DataFrame()
|
||
if len(frames) == 1:
|
||
return next(iter(frames.values()))
|
||
return pd.concat(frames, axis=1)
|
||
```
|
||
|
||
- [ ] **Step 4: 跑测试确认 PASS**
|
||
- [ ] **Step 5: 写失败测试 — panel=False 多股长表 + count**
|
||
|
||
```python
|
||
def test_get_price_panel_false_multi(unified_provider):
|
||
df = unified_provider.get_price("600519.XSHG", end_date="2024-06-20", count=2, panel=False, fields=["close"])
|
||
assert "code" in df.columns and "time" in df.columns
|
||
assert len(df) == 2
|
||
```
|
||
|
||
- [ ] **Step 6: 实现(Step 3 已含 panel 分支),跑 PASS**
|
||
- [ ] **Step 7: Commit** — `feat(portfolio): LocalUnifiedProvider get_price+前复权(Task1)`
|
||
|
||
---
|
||
|
||
## Task 2: get_index_stocks + get_constituent(constituent_unified 并集,治偏差)
|
||
|
||
**Files:** Modify `local_unified_provider.py`; Test 同文件。
|
||
|
||
**Interfaces:**
|
||
- Produces: `get_index_stocks(index_symbol, date) -> List[str]` + `get_constituent(index, date) -> List[str]`(语义别名)
|
||
|
||
- [ ] **Step 1: 写失败测试**
|
||
|
||
```python
|
||
def test_get_index_stocks_union(tmp_path):
|
||
db = tmp_path / "t.db"; c = sqlite3.connect(str(db))
|
||
c.execute("CREATE TABLE constituent_unified(index_code TEXT,code TEXT,code_name TEXT,source TEXT,in_current INT,was_removed INT)")
|
||
c.executemany("INSERT INTO constituent_unified VALUES(?,?,?,?,?,?)", [
|
||
("000300", "600519", "贵州茅台", "baostock", 1, 0),
|
||
("000300", "000001", "平安银行", "baostock", 1, 0),
|
||
("000300", "600811", "退市股", "baostock", 0, 1), # 被踢
|
||
])
|
||
c.commit(); c.close()
|
||
p = LocalUnifiedProvider({"db_path": str(db), "data_dir": str(tmp_path)})
|
||
stocks = p.get_index_stocks("000300.XSHG", "2020-01-01")
|
||
assert set(stocks) == {"600519.XSHG", "000001.XSHE", "600811.SH"} # 并集含被踢
|
||
# date 参数不报错(并集模型忽略)
|
||
assert p.get_constituent("000300", None) == stocks # 别名
|
||
```
|
||
|
||
- [ ] **Step 2: 跑测试确认 FAIL**
|
||
|
||
- [ ] **Step 3: 实现** — 查 constituent_unified,index_code 匹配(去 `.XXXX` 后缀),返回 in_current=1 OR was_removed=1 的并集,code→jq_code:
|
||
|
||
```python
|
||
def get_index_stocks(self, index_symbol, date=None) -> List[str]:
|
||
idx = index_symbol.split(".")[0] if "." in str(index_symbol) else str(index_symbol)
|
||
conn = self._connect()
|
||
rows = conn.execute(
|
||
"SELECT code FROM constituent_unified WHERE index_code=? "
|
||
"AND (in_current=1 OR was_removed=1)", (idx,)).fetchall()
|
||
out = []
|
||
for (code,) in rows:
|
||
code = str(code).strip()
|
||
if len(code) != 6:
|
||
continue
|
||
exc = "SSE" if code.startswith("6") else "SZSE"
|
||
out.append(dbbardata_to_jq(code, exc))
|
||
return out
|
||
|
||
def get_constituent(self, index, date=None) -> List[str]:
|
||
"""spec §6 语义别名 = get_index_stocks。"""
|
||
return self.get_index_stocks(index, date)
|
||
```
|
||
|
||
- [ ] **Step 4: 跑测试 PASS**
|
||
- [ ] **Step 5: Commit** — `feat(portfolio): LocalUnifiedProvider 成份股并集治偏差(Task2)`
|
||
|
||
---
|
||
|
||
## Task 3: get_fundamentals_df(valuation_baostock + static akshare + 三表)
|
||
|
||
**Files:** Modify `local_unified_provider.py`; Test 同文件 + tmp parquet fixture。
|
||
|
||
**Interfaces:**
|
||
- Produces: `get_fundamentals_df(stocks, date) -> DataFrame` 列对齐 `_FUNDAMENTAL_COLUMNS`
|
||
|
||
数据源映射:
|
||
- `pe_ratio/pb_ratio/ps_ratio/pcf_ratio` ← valuation_baostock parquet(peTTM/pbMRQ/psTTM/pcfNcfTTM,baostock 权威)
|
||
- `market_cap/circulating_market_cap` ← static/valuation akshare parquet(total_market_cap/circ_market_cap,baston 无市值)
|
||
- 三表字段(eps/net_profit_margin/total_liability 等) ← static/{balance,income} akshare parquet(复用 LocalParquetProvider 读法)
|
||
|
||
- [ ] **Step 1: 写失败测试 — 估值字段从 valuation_baostock**
|
||
|
||
```python
|
||
def test_get_fundamentals_valuation(tmp_path):
|
||
# valuation_baostock/2024.parquet
|
||
vdir = tmp_path / "valuation_baostock"; vdir.mkdir()
|
||
pd.DataFrame({"symbol":["600519"],"exchange":["SH"],"date":["2024-09-30"],
|
||
"peTTM":[25.0],"psTTM":[15.0],"pcfNcfTTM":[20.0],"pbMRQ":[7.5],
|
||
"turn":[0.1],"pctChg":[1.0],"isST":[0]}).to_parquet(vdir/"2024.parquet")
|
||
# static/valuation akshare(市值)
|
||
sdir = tmp_path / "static" / "valuation"; sdir.mkdir(parents=True)
|
||
pd.DataFrame({"数据日期":["2024-09-30"],"总市值":[2e12],"流通市值":[2e12],"总股本":[1.256e9],
|
||
"PE(TTM)":[25],"市净率":[7.5]}).to_parquet(sdir/"600519.SH_valuation.parquet")
|
||
p = LocalUnifiedProvider({"db_path": str(tmp_path/"t.db"), "data_dir": str(tmp_path)})
|
||
df = p.get_fundamentals_df(["600519.XSHG"], date="2024-09-30")
|
||
assert abs(df.loc["600519.XSHG","pe_ratio"] - 25.0) < 1e-6 # baostock 权威
|
||
assert abs(df.loc["600519.XSHG","pb_ratio"] - 7.5) < 1e-6
|
||
assert abs(df.loc["600519.XSHG","market_cap"] - 2e4) < 1 # 2e12元→2e4亿
|
||
```
|
||
|
||
- [ ] **Step 2: 跑测试确认 FAIL**
|
||
|
||
- [ ] **Step 3: 实现** — 读 valuation_baostock parquet(year from date)+ static/valuation akshare;合并对齐 `_FUNDAMENTAL_COLUMNS`(复用 LocalParquetProvider 的 `_VAL_COL_MAP` / `to_yi` / 三表读法,import 复用):
|
||
|
||
```python
|
||
from .local_parquet_provider import (_VAL_COL_MAP, jq_to_file_code,
|
||
_to_float, _or_nan, _pct_to_decimal, _FUNDAMENTAL_COLUMNS)
|
||
from ..factors.valuation import to_yi
|
||
|
||
def get_fundamentals_df(self, stocks, date=None) -> pd.DataFrame:
|
||
if not stocks:
|
||
return pd.DataFrame(columns=_FUNDAMENTAL_COLUMNS)
|
||
date_str = self._to_date_str(date) or datetime.now().strftime("%Y-%m-%d")
|
||
rows = [self._build_fundamental_row(s, date_str) for s in stocks]
|
||
df = pd.DataFrame(rows, columns=_FUNDAMENTAL_COLUMNS)
|
||
if "code" in df.columns:
|
||
df = df.set_index("code", drop=False)
|
||
return df
|
||
|
||
def _read_valuation_baostock(self, year: int) -> pd.DataFrame:
|
||
if year in self._val_bs_cache:
|
||
return self._val_bs_cache[year]
|
||
p = os.path.join(self.data_dir, "valuation_baostock", f"{year}.parquet")
|
||
df = pd.read_parquet(p) if os.path.exists(p) else pd.DataFrame()
|
||
self._val_bs_cache[year] = df
|
||
return df
|
||
|
||
def _build_fundamental_row(self, jq_code, date_str) -> Dict[str, Any]:
|
||
sym, exc = jq_to_dbbardata(jq_code)
|
||
fc = jq_to_file_code(jq_code) # 600519.SH(static akshare 文件名)
|
||
row: Dict[str, Any] = {"code": jq_code}
|
||
# 1. pe/pb/ps/pcf ← valuation_baostock(baostock 权威)
|
||
year = int(date_str[:4])
|
||
vbs = self._read_valuation_baostock(year)
|
||
if not vbs.empty:
|
||
sub = vbs[(vbs["symbol"].astype(str) == sym) & (vbs["date"].astype(str) <= date_str)]
|
||
vrow = sub.iloc[-1] if not sub.empty else None
|
||
else:
|
||
vrow = None
|
||
def gbs(k):
|
||
return _to_float(vrow.get(k)) if vrow is not None else None
|
||
row["pe_ratio"] = _or_nan(gbs("peTTM"))
|
||
row["pb_ratio"] = _or_nan(gbs("pbMRQ"))
|
||
row["ps_ratio"] = _or_nan(gbs("psTTM"))
|
||
row["pcf_ratio"] = _or_nan(gbs("pcfNcfTTM"))
|
||
# 2. 市值/股本 + 三表 ← static akshare(复用 LocalParquetProvider 读法)
|
||
# 复用:直接实例化 LocalParquetProvider 读 static 部分,或内联读 static/valuation
|
||
ak_val = self._read_akshare_valuation(fc, date_str) # 返 renamed Series
|
||
mkt = _to_float(ak_val.get("total_market_cap")) if ak_val is not None else None
|
||
circ = _to_float(ak_val.get("circ_market_cap")) if ak_val is not None else None
|
||
row["market_cap"] = to_yi(mkt) if mkt else float("nan")
|
||
row["circulating_market_cap"] = to_yi(circ) if circ else float("nan")
|
||
# 3. 三表(income/balance)— 复用 LocalParquetProvider._read_quarter + 字段提取
|
||
# 简化:委托一个内部 LocalParquetProvider 实例读三表部分(eps/margin/liability)
|
||
lpp = self._get_lpp_helper()
|
||
inc = lpp._latest_row_before(lpp._read_quarter("income", fc), "REPORT_DATE", date_str)
|
||
bal = lpp._latest_row_before(lpp._read_quarter("balance", fc), "REPORT_DATE", date_str)
|
||
row["eps"] = _or_nan(_to_float(inc.get("BASIC_EPS")) if inc is not None else None)
|
||
# ... net_profit_margin/total_liability/roe 等(照 LocalParquetProvider._build_fundamental_row 逻辑)
|
||
return row
|
||
```
|
||
(实现时:`_get_lpp_helper()` 返一个复用的 `LocalParquetProvider(config)` 实例读 static 三表;`_read_akshare_valuation` 复用 LocalParquetProvider._read_valuation。DRY:不重写三表/akshare valuation 逻辑,委托 LocalParquetProvider。pe/pb 改 baostock 源覆盖 akshare 的。)
|
||
|
||
- [ ] **Step 4: 跑测试 PASS**
|
||
- [ ] **Step 5: 写测试 — 三表字段(eps/market_cap 全 _FUNDAMENTAL_COLUMNS 有值不 NaN)**
|
||
- [ ] **Step 6: 实现 + PASS**
|
||
- [ ] **Step 7: Commit** — `feat(portfolio): LocalUnifiedProvider fundamentals baostock估值+akshare市值(Task3)`
|
||
|
||
---
|
||
|
||
## Task 4: 辅助方法(trade_days/all_securities/security_info/current_tick/split_dividend)
|
||
|
||
**Files:** Modify `local_unified_provider.py`; Test 同文件。
|
||
|
||
- [ ] **Step 1-2: 写失败测试 + FAIL** — `get_trade_days(count=2)` 返 datetime list;`get_security_info` 返 display_name/start_date;`get_current_tick` 返 close+high_limit;`get_split_dividend` 返 bs_adjust_factor 事件;`get_all_securities` 返 dbbardata distinct symbol。
|
||
|
||
- [ ] **Step 3: 实现**:
|
||
- `get_trade_days`: 读 dbbardata 某 symbol(如 600519)distinct datetime,filter/count。
|
||
- `get_security_info`: dbbardata min/max datetime → start/end_date;display_name 从 constituent_unified code_name 或 code。
|
||
- `get_current_tick`: dbbardata 最近 close + valuation_baostock 最近 pctChg → high_limit=close×1.1(ST 0.05)。
|
||
- `get_split_dividend`: bs_adjust_factor → events(dividOperateDate + adjustFactor)。
|
||
- `get_all_securities`: dbbardata distinct symbol → DataFrame。
|
||
|
||
- [ ] **Step 4: 跑测试 PASS**
|
||
- [ ] **Step 5: Commit** — `feat(portfolio): LocalUnifiedProvider 辅助方法(Task4)`
|
||
|
||
---
|
||
|
||
## Task 5: 接线(__init__ 导出 + runner build_provider 加 unified)
|
||
|
||
**Files:** Modify `sanguo_portfolio/providers/__init__.py`; Modify `sanguo_portfolio/runner_backtest.py`。
|
||
|
||
- [ ] **Step 1: __init__.py 加导出**
|
||
```python
|
||
from .local_unified_provider import LocalUnifiedProvider
|
||
__all__ = ["SanguoMiniQmtProvider", "BaostockProvider", "LocalParquetProvider", "LocalUnifiedProvider"]
|
||
```
|
||
|
||
- [ ] **Step 2: runner_backtest build_provider 加 unified**
|
||
```python
|
||
# parse_args choices 加 "unified"; build_provider 加分支
|
||
p.add_argument("--provider", default="local", choices=["local", "baostock", "miniqmt", "unified"], ...)
|
||
# build_provider:
|
||
from .providers import LocalUnifiedProvider
|
||
if name == "unified":
|
||
return LocalUnifiedProvider(cfg)
|
||
```
|
||
|
||
- [ ] **Step 3: 跑 `pytest tests/portfolio/ -v` 全绿(回归)**
|
||
- [ ] **Step 4: Commit** — `feat(portfolio): 接线 LocalUnifiedProvider 到 runner(Task5)`
|
||
|
||
---
|
||
|
||
## Task 6: 使用说明 + VPS E2E 验证
|
||
|
||
**Files:** Create `docs/portfolio_local_unified_provider.md`; VPS 跑 `python -m sanguo_portfolio.runner_backtest --provider unified --start 2024-01-01 --end 2024-03-31 --max-pool 20`。
|
||
|
||
- [ ] **Step 1: 写使用说明** `docs/portfolio_local_unified_provider.md`(其他 session 直用)— 含:
|
||
- 一句话定位(读方案A权威层/零online/治偏差)
|
||
- 数据源映射表(每接口→哪张表/parquet)
|
||
- 接口清单(DataProvider 接口 + get_constituent)
|
||
- 复权说明(raw存储+消费端按bs_adjust_factor算qfq;fq参数 raw/qfq)
|
||
- **幸存者偏差说明**(constituent_unified 并集模型,治纯当前偏差,有轻微前视,date 参数忽略;中证1000/2000只快照永久gap)
|
||
- Mac 测试(fixture,零VPS依赖)
|
||
- 部署/运行(runner --provider unified;VPS 数据依赖 dbbardata/constituent_unified/valuation_baostock/static)
|
||
- 已知限制(high_limit 列 NaN→prepare_stock_list 涨停识别降级;1m 无数据;三表委托 LocalParquetProvider)
|
||
- 与旧 provider 关系(LocalParquetProvider/BaostockProvider 保留,unified 是方案A 后推荐)
|
||
|
||
- [ ] **Step 2: VPS E2E** — rsync 代码到 VPS,跑 `--provider unified --max-pool 20` 小样本回测,确认:
|
||
- get_price 读 dbbardata 出 K 线(含退市)
|
||
- get_index_stocks 出并集成份股
|
||
- get_fundamentals_df 出市值+pe/pb
|
||
- 回测不崩,有选股+指标输出
|
||
|
||
- [ ] **Step 3: Commit** — `docs(portfolio): LocalUnifiedProvider 使用说明+VPS E2E(Task6)`
|
||
|
||
---
|
||
|
||
## Self-Review(plan 自检)
|
||
|
||
1. **Spec 覆盖**: spec §6 接口(get_daily/get_constituent/get_fundamentals/...)— get_constituent 别名✓;get_price 覆盖 get_daily+get_etf_daily(都读 dbbardata,ETF 也在);get_fundamentals_df ✓;其余 §6 方法(industry/longhubang/instrument)数据层未就绪(P1),使用说明标注 NotImplementedError。✓
|
||
2. **方案A §14 一致**: dbbardata 唯一行情✓;constituent_unified 治偏差✓;valuation_baostock pe/pb✓;raw+factor 复权✓;零online✓。
|
||
3. **类型一致**: `_build_qfq_factor(code, conn, dates) -> Series` 在 Task0/Task1 调用签名一致✓。
|
||
4. **占位扫描**: Task3 的 `_get_lpp_helper/_read_akshare_valuation` 标了"复用 LocalParquetProvider",实现 agent 须内联或委托,不留空✓。
|
||
5. **风险**: get_price 的 high_limit 列缺失(NaN)→策略 prepare_stock_list 涨停识别降级,使用说明标注(Task6)✓。
|
||
|
||
## Execution Handoff
|
||
|
||
Plan complete and saved to `docs/superpowers/plans/2026-07-23-local-unified-provider.md`.
|