refactor(data): 5m改同库设计——用户二次拍板2026-08-20:放弃独立库,5m直接写NAS副本主库dbbardata表(interval='5m'与镜像行d/15m唯一键正交=结构性互不干扰)——①bs_5m_eod缺省库改/volume1/stock/sanguo_vnpy_v2/data_backup/quant_trading.db(=同步目标=NAS回测provider读的库portfolio_worker.py:180),NAS就地5m回测/回放reader零改动,未来VPS扩容导interval='5m'反向merge一次迁移②同库安全性实证:同步链export不带id列+NAS侧INSERT OR IGNORE按UNIQUE去重→键永不相交双向碰不到;纯merge无文件覆盖NAS本地行不会被删;id由NAS AUTOINCREMENT自分配无跨库冲突③ensure_schema防重复索引:PRAGMA index_list/index_info识别已有同列唯一索引(NAS侧uq_dbbardata)跳过,免数GB无谓开销+双倍写放大④merge_increment补busy_timeout=60000(副本库从此有第二个写者,无它撞写锁当场locked失败,60s与5m侧对齐)⑤写锁竞争说明:双写者WAL+busy_timeout串行化,与每日增量merge(秒级)重叠单股失败次日LOOKBACK=7自愈;+2测试(缺省库契约钉死/副本已有索引不重复建);40+153绿(data_platform 155) [nas]
This commit is contained in:
@@ -1,9 +1,10 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""TDD for bs_5m_eod.py — NAS 独立 5 分钟线备份 (2026-08-20 用户拍板).
|
||||
"""TDD for bs_5m_eod.py — NAS 独立 5 分钟线备份 (2026-08-20 用户拍板, 同日二次拍板改同库).
|
||||
|
||||
定位: NAS 自己从 baostock 下载 5m, 不推 VPS; NAS 其余数据全是 VPS 同步镜像,
|
||||
本脚本是 NAS 唯一自主数据 —— 落独立库 BS_5M_DB(缺省 /volume1/stock/sanguo_5m/),
|
||||
绝不碰同步目录 /volume1/stock/sanguo_vnpy_v2/data_backup/quant_trading.db。
|
||||
定位: 5m 直接写 NAS 副本主库 dbbardata 表(与 VPS 同步镜像同库同表), 靠
|
||||
interval='5m' 与镜像行在唯一键上正交互不干扰 —— 同步链导出不带 id + NAS 侧
|
||||
INSERT OR IGNORE 去重(merge_increment.py), NAS 回测 reader 零改动可见 5m,
|
||||
未来 VPS 扩容导 interval='5m' 反向 merge 即迁移。
|
||||
|
||||
配额: NAS 出口 IP 独立核算 48k/天, 一趟仅 ~5.5k query(每股恰好 1 次调用,
|
||||
--full 全区间也一样); VPS 侧 bs_eod/bs_fund 用 VPS 自己的 IP 配额, 互不相干。
|
||||
@@ -51,6 +52,12 @@ def _m5_rows(bs_code, n=1, hhmm="0935"):
|
||||
|
||||
# ---------- schema ----------
|
||||
|
||||
def test_default_db_is_nas_replica_main_db():
|
||||
"""契约钉死: 缺省库 = NAS 副本主库(同步目标=回测源), 绝非旁路独立库。"""
|
||||
assert b5._DEFAULT_DB == \
|
||||
"/volume1/stock/sanguo_vnpy_v2/data_backup/quant_trading.db"
|
||||
|
||||
|
||||
def test_ensure_schema_creates_table_and_unique_index(tmp_path):
|
||||
db = tmp_path / "x.db"
|
||||
conn = sqlite3.connect(str(db))
|
||||
@@ -62,6 +69,29 @@ def test_ensure_schema_creates_table_and_unique_index(tmp_path):
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_ensure_schema_skips_duplicate_index_on_replica(tmp_path):
|
||||
"""NAS 副本主库已带 uq_dbbardata(merge_increment 建, 名字不同语义同):
|
||||
ensure_schema 不得再建同列唯一索引(数 GB 无谓开销+双倍写放大)。"""
|
||||
db = tmp_path / "replica.db"
|
||||
conn = sqlite3.connect(str(db))
|
||||
conn.execute(
|
||||
"CREATE TABLE dbbardata (id INTEGER PRIMARY KEY AUTOINCREMENT, "
|
||||
"symbol TEXT, exchange TEXT, datetime TEXT, interval TEXT, "
|
||||
"volume REAL, turnover REAL, open_interest REAL, "
|
||||
"open_price REAL, high_price REAL, low_price REAL, close_price REAL)")
|
||||
conn.execute(
|
||||
"CREATE UNIQUE INDEX uq_dbbardata "
|
||||
"ON dbbardata (symbol, exchange, interval, datetime)")
|
||||
conn.commit()
|
||||
b5.ensure_schema(conn)
|
||||
unique_idx = [r for r in conn.execute(
|
||||
"SELECT name FROM sqlite_master "
|
||||
"WHERE tbl_name='dbbardata' AND type='index' AND sql IS NOT NULL")]
|
||||
assert len(unique_idx) == 1 # 只有原有的 uq_dbbardata, 没有重复建
|
||||
assert unique_idx[0][0] == "uq_dbbardata"
|
||||
conn.close()
|
||||
|
||||
|
||||
# ---------- upsert_5m ----------
|
||||
|
||||
def test_upsert_5m_writes_dbbardata(tmp_db_conn):
|
||||
|
||||
Reference in New Issue
Block a user