fix(data): dbbardata 日线双行根治(统一纯日期+helper,留ROWID max去重)
根因: dbbardata UNIQUE(symbol,exchange,interval,datetime) 按字符串字面比较, 多写入路径混用 'YYYY-MM-DD' 与 'YYYY-MM-DD 00:00:00' -> 同一交易日双行, INSERT OR REPLACE 不去重 -> 回测交易日翻倍/pivot duplicate/信号异常。 方案A (统一纯日期, 详见 Main Agent 诊断): - 新增 scripts/data_platform/dbbardata_utils.py: normalize_daily_dt(s) 取前 10 字符, None/短串安全 - 4 个日线写入脚本写入前调 helper: - bs_eod.py (sanguo-bs-eod 个股日线 baostock) - migrate_daily_baostock.py (历史迁移) - xt_eod.py (sanguo-xt-eod ETF/基金 xtata) - import_vnpy_daily_fast.py (NAS 日线 parquet 导入, 加防御) - TDD: tests/data_platform/test_dbbardata_utils.py 9 cases 全过 - 回归: tests/data_platform + tests/portfolio 199 passed 12 skipped peewee DateTimeField formats 含 '%Y-%m-%d' (阶段0 VPS 实测确认), 读纯日期不崩, 方案A 前提成立。 15min 干净, 不动 (分钟必须带时分)。只改日线 interval='d'。 数据层根治, 不在 provider 适配兜底 (用户铁律)。
This commit is contained in:
@@ -13,6 +13,8 @@ from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from dbbardata_utils import normalize_daily_dt
|
||||
|
||||
DB = Path(r"C:\sanguo_vnpy_v2\data\quant_trading.db")
|
||||
VAL_DIR = Path(r"C:\sanguo_vnpy_v2\data\valuation_baostock")
|
||||
EXC_MAP = {"SH": "SSE", "SZ": "SZSE"}
|
||||
@@ -52,7 +54,8 @@ for chunk in pd.read_sql("SELECT * FROM daily_baostock_full", c, chunksize=20000
|
||||
odb = pd.DataFrame({
|
||||
"symbol": chunk["symbol"].values,
|
||||
"exchange": chunk["exchange"].map(EXC_MAP).values,
|
||||
"datetime": (chunk["date"].astype(str) + " 00:00:00").values,
|
||||
# datetime 归一纯日期 (dbbardata 双行根治方案A)
|
||||
"datetime": chunk["date"].astype(str).map(normalize_daily_dt).values,
|
||||
"interval": "d",
|
||||
"volume": chunk["volume"].values,
|
||||
"turnover": chunk["amount"].values,
|
||||
|
||||
Reference in New Issue
Block a user