fix(data): dbbardata 日线双行根治(统一纯日期+helper,留ROWID max去重)

根因: dbbardata UNIQUE(symbol,exchange,interval,datetime) 按字符串字面比较,
多写入路径混用 'YYYY-MM-DD' 与 'YYYY-MM-DD 00:00:00' -> 同一交易日双行,
INSERT OR REPLACE 不去重 -> 回测交易日翻倍/pivot duplicate/信号异常。

方案A (统一纯日期, 详见 Main Agent 诊断):
- 新增 scripts/data_platform/dbbardata_utils.py: normalize_daily_dt(s)
  取前 10 字符, None/短串安全
- 4 个日线写入脚本写入前调 helper:
  - bs_eod.py (sanguo-bs-eod 个股日线 baostock)
  - migrate_daily_baostock.py (历史迁移)
  - xt_eod.py (sanguo-xt-eod ETF/基金 xtata)
  - import_vnpy_daily_fast.py (NAS 日线 parquet 导入, 加防御)
- TDD: tests/data_platform/test_dbbardata_utils.py 9 cases 全过
- 回归: tests/data_platform + tests/portfolio 199 passed 12 skipped

peewee DateTimeField formats 含 '%Y-%m-%d' (阶段0 VPS 实测确认),
读纯日期不崩, 方案A 前提成立。

15min 干净, 不动 (分钟必须带时分)。只改日线 interval='d'。

数据层根治, 不在 provider 适配兜底 (用户铁律)。
This commit is contained in:
2026-07-23 12:18:41 +08:00
parent 4c2af8ff00
commit 114a69e997
6 changed files with 92 additions and 7 deletions
+6 -2
View File
@@ -21,6 +21,8 @@ import time
from xtquant import xtdata as xd
import pandas as pd
from dbbardata_utils import normalize_daily_dt
DB = r"C:\sanguo_vnpy_v2\data\quant_trading.db"
LOOKBACK = int(__import__("os").environ.get("LOOKBACK_DAYS", "30"))
T0 = time.time()
@@ -88,8 +90,10 @@ def main():
db = pd.DataFrame({
"symbol": sym,
"exchange": exc_of(sym),
"datetime": [str(idx)[:4]+"-"+str(idx)[4:6]+"-"+str(idx)[6:8]+" 00:00:00"
for idx in df.index],
# datetime 归一纯日期 (dbbardata 双行根治方案A)
"datetime": [normalize_daily_dt(
f"{str(idx)[:4]}-{str(idx)[4:6]}-{str(idx)[6:8]}")
for idx in df.index],
"interval": "d",
"volume": (df["volume"].astype(float).values * 100),
"turnover": df["amount"].astype(float).values,