fix(nas_sync): Phase3 full 修复(STEP降片+merge PRAGMA)+全量完成2.02亿行

- sync_dbbardata.sh: STEP 2000万→200万/片(首片2GB export+scp跨公网挂)
- merge_increment.py: 加 PRAGMA cache=500MB+WAL+NORMAL+synchronous
  (replica 2亿行 UNIQUE索引数GB,默认2MB cache致索引IO merge>120s超时)
- 全量完成: replica 2.019亿行(=dbbardata全量日线+15min+ETF)
- 踩坑链: ssh reset中断→timeout杀ssh留NAS merge残留持锁→since落后数据已在库
- 教训: timeout杀ssh不杀远程python;merge大表必加PRAGMA;断点续传since.txt救命

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-30 18:08:42 +08:00
parent c033cf6cb5
commit d76171434b
2 changed files with 5 additions and 1 deletions
+4
View File
@@ -30,6 +30,10 @@ def main():
conn = sqlite3.connect(args.db)
cur = conn.cursor()
# 大表(1.8亿行+)merge 提速: 默认2MB cache 致 UNIQUE索引(数GB)全磁盘IO, merge>120s超时
for _p in ("PRAGMA journal_mode=WAL", "PRAGMA synchronous=NORMAL",
"PRAGMA cache_size=-500000", "PRAGMA temp_store=MEMORY"):
cur.execute(_p)
cur.execute(SCHEMA)
cur.execute(
"CREATE UNIQUE INDEX IF NOT EXISTS uq_dbbardata "
+1 -1
View File
@@ -26,7 +26,7 @@ DB=$ROOT/data_backup/quant_trading.db
STAGE=$ROOT/data_backup/_staging
STATE=$ROOT/data_backup/since.txt
LOG=$ROOT/data_backup/sync.log
STEP=20000000 # 全量分片步长(行/片,~2GB/片
STEP=2000000 # 全量分片步长(行/片,~200MB/片;首版2000万/2GB致首片export+scp跨公网挂,降至此
mkdir -p "$STAGE"
LOG_TS="$(date '+%F %T')"