fix(nas_sync): Phase3 full 修复(STEP降片+merge PRAGMA)+全量完成2.02亿行
- sync_dbbardata.sh: STEP 2000万→200万/片(首片2GB export+scp跨公网挂) - merge_increment.py: 加 PRAGMA cache=500MB+WAL+NORMAL+synchronous (replica 2亿行 UNIQUE索引数GB,默认2MB cache致索引IO merge>120s超时) - 全量完成: replica 2.019亿行(=dbbardata全量日线+15min+ETF) - 踩坑链: ssh reset中断→timeout杀ssh留NAS merge残留持锁→since落后数据已在库 - 教训: timeout杀ssh不杀远程python;merge大表必加PRAGMA;断点续传since.txt救命 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -30,6 +30,10 @@ def main():
|
||||
|
||||
conn = sqlite3.connect(args.db)
|
||||
cur = conn.cursor()
|
||||
# 大表(1.8亿行+)merge 提速: 默认2MB cache 致 UNIQUE索引(数GB)全磁盘IO, merge>120s超时
|
||||
for _p in ("PRAGMA journal_mode=WAL", "PRAGMA synchronous=NORMAL",
|
||||
"PRAGMA cache_size=-500000", "PRAGMA temp_store=MEMORY"):
|
||||
cur.execute(_p)
|
||||
cur.execute(SCHEMA)
|
||||
cur.execute(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS uq_dbbardata "
|
||||
|
||||
@@ -26,7 +26,7 @@ DB=$ROOT/data_backup/quant_trading.db
|
||||
STAGE=$ROOT/data_backup/_staging
|
||||
STATE=$ROOT/data_backup/since.txt
|
||||
LOG=$ROOT/data_backup/sync.log
|
||||
STEP=20000000 # 全量分片步长(行/片,~2GB/片)
|
||||
STEP=2000000 # 全量分片步长(行/片,~200MB/片;首版2000万/2GB致首片export+scp跨公网挂,降至此)
|
||||
|
||||
mkdir -p "$STAGE"
|
||||
LOG_TS="$(date '+%F %T')"
|
||||
|
||||
Reference in New Issue
Block a user