feat(data): 方案A 数据层落地(spec §14)— DB唯一表+治幸存者偏差+权威源
spec §14 方案A 数据层迁移完成 + E2E 验证(read_db_daily: 在市/退市治偏差/ETF 全OK):
- dbbardata('d') 1826万含退市(治回测幸存者偏差, INSERT OR REPLACE staging迁移, WHERE OHLC NOT NULL+COALESCE)
- constituent_unified 7110行/9指数(300/500/50 baostock全集 + 深证4指 akshare cni union + 中证1000/2000 snapshot)
- pe/pb 不进DB -> valuation_baostock/<year>.parquet 按年宽表(2003-2026)
- 废弃 daily_baostock_full/bs_index_constituent(rename _old 保留); 旧4 schtask disabled
- 新 schtask sanguo-bs-eod 18:05(baostock个股日线+15min+拆pe/pb DAILY_LIMIT 48000) + sanguo-xt-eod 18:40(ETF/基金xtata)
- 权威源: baostock个股日线+估值+15min+复权+300/500/50 / xtata ETF+基金+当天实时 / akshare三表+事件+深证中证成份股
- 全程备份+staging+_old保留可回滚; 脚本 audit/probe/migrate/merge/cleanup/fix_config/verify/bs_eod/xt_eod/wrapper/register_schtasks
- 待办(spec §6 使用层): LocalParquetProvider 接 constituent_unified+valuation_baostock + 实时拼接
This commit is contained in:
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""probe_constituent.py — 单元3 前置: 探查成份股两源 schema (写 migrate 前必须看清)。
|
||||
|
||||
1. bs_index_constituent (DB, baostock 300/500/50 历史): cols/rows/distinct index_code/抽样
|
||||
2. data/index_const_hist/*.parquet (akshare cni 深证 union): 文件列表 + 每个 shape/cols/抽样
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import sqlite3
|
||||
|
||||
import pandas as pd
|
||||
|
||||
DB = r"C:\sanguo_vnpy_v2\data\quant_trading.db"
|
||||
HIST = r"C:\sanguo_vnpy_v2\data\index_const_hist"
|
||||
|
||||
c = sqlite3.connect(DB, timeout=60)
|
||||
c.execute("PRAGMA busy_timeout = 60000")
|
||||
cols = [r[1] for r in c.execute("PRAGMA table_info(bs_index_constituent)")]
|
||||
print("[bs_index_constituent] cols:", cols)
|
||||
print(" rows:", c.execute("SELECT COUNT(*) FROM bs_index_constituent").fetchone()[0])
|
||||
idx = [r[0] for r in c.execute(
|
||||
"SELECT DISTINCT index_code FROM bs_index_constituent ORDER BY index_code")]
|
||||
print(" distinct index_code:", idx)
|
||||
print(" distinct updateDate count:", c.execute(
|
||||
"SELECT COUNT(DISTINCT updateDate) FROM bs_index_constituent").fetchone()[0])
|
||||
print(" sample rows:", c.execute(
|
||||
"SELECT * FROM bs_index_constituent LIMIT 3").fetchall())
|
||||
c.close()
|
||||
|
||||
print("\n[index_const_hist parquets]")
|
||||
files = sorted(glob.glob(os.path.join(HIST, "*.parquet")))
|
||||
print("files:", [os.path.basename(f) for f in files])
|
||||
for f in files:
|
||||
df = pd.read_parquet(f)
|
||||
print(f" {os.path.basename(f)}: shape={df.shape} cols={list(df.columns)}")
|
||||
print(f" head:\n{df.head(2).to_string()}")
|
||||
Reference in New Issue
Block a user