feat(data): 方案A 数据层落地(spec §14)— DB唯一表+治幸存者偏差+权威源
spec §14 方案A 数据层迁移完成 + E2E 验证(read_db_daily: 在市/退市治偏差/ETF 全OK):
- dbbardata('d') 1826万含退市(治回测幸存者偏差, INSERT OR REPLACE staging迁移, WHERE OHLC NOT NULL+COALESCE)
- constituent_unified 7110行/9指数(300/500/50 baostock全集 + 深证4指 akshare cni union + 中证1000/2000 snapshot)
- pe/pb 不进DB -> valuation_baostock/<year>.parquet 按年宽表(2003-2026)
- 废弃 daily_baostock_full/bs_index_constituent(rename _old 保留); 旧4 schtask disabled
- 新 schtask sanguo-bs-eod 18:05(baostock个股日线+15min+拆pe/pb DAILY_LIMIT 48000) + sanguo-xt-eod 18:40(ETF/基金xtata)
- 权威源: baostock个股日线+估值+15min+复权+300/500/50 / xtata ETF+基金+当天实时 / akshare三表+事件+深证中证成份股
- 全程备份+staging+_old保留可回滚; 脚本 audit/probe/migrate/merge/cleanup/fix_config/verify/bs_eod/xt_eod/wrapper/register_schtasks
- 待办(spec §6 使用层): LocalParquetProvider 接 constituent_unified+valuation_baostock + 实时拼接
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""dbbardata_probe.py — 只读探测 dbbardata 现状(schema + 抽样各 interval 覆盖)。
|
||||
|
||||
走 symbol 索引,避免全表 COUNT(亿级慢)。回答:
|
||||
- dbbardata 有哪些 interval(d/15m/5m)
|
||||
- 个股日线(interval='d' 类)含不含退市股(000005/000023/600811/600074)
|
||||
- 在市股(600519/000001)日线日期范围
|
||||
- ETF(510300/159915)有没有(判断 dbbardata 是否覆盖 ETF)
|
||||
用于决定迁移单元5(dbbardata 个股日线切 baostock 源)的工作量。
|
||||
"""
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
DB = Path(r"C:\sanguo_vnpy_v2\data\quant_trading.db")
|
||||
c = sqlite3.connect(str(DB), timeout=60)
|
||||
c.execute("PRAGMA busy_timeout = 60000")
|
||||
|
||||
cols = [r[1] for r in c.execute("PRAGMA table_info(dbbardata)")]
|
||||
print(f"DB: {DB}")
|
||||
print(f"dbbardata cols({len(cols)}): {cols}")
|
||||
|
||||
print("\n=== 抽样: 各 symbol 的 interval 覆盖 (走索引, 快) ===")
|
||||
samples = {
|
||||
"退市": ["000005", "000023", "600811", "600074"],
|
||||
"在市个股": ["600519", "000001"],
|
||||
"ETF/基金": ["510300", "159915", "510050"],
|
||||
}
|
||||
for label, syms in samples.items():
|
||||
print(f"\n[{label}]")
|
||||
for sym in syms:
|
||||
rows = c.execute(
|
||||
"SELECT interval, COUNT(*), MIN(datetime), MAX(datetime) "
|
||||
"FROM dbbardata WHERE symbol=? GROUP BY interval",
|
||||
(sym,),
|
||||
).fetchall()
|
||||
print(f" {sym}: {rows}")
|
||||
|
||||
c.close()
|
||||
Reference in New Issue
Block a user