claude_dev
86bec48b14
ak-eod每日--force全量重写valuation/financial_abstract(估值随股价每日变); bsdtar--newer慢盘stat不可行(11070文件>60s),改python tarfile全打包(实测9.5s/1.1GB)->scp->解包。 拆分:每日高频per-stock(valuation)+per-date(dragon_tiger等)+每周低频per-stock(northbound等)。
49 lines
2.0 KiB
Bash
49 lines
2.0 KiB
Bash
#!/bin/bash
|
||
# NAS 端：每日同步 VPS valuation + financial_abstract（ak-eod 每日 --force 全量重写这俩）。
|
||
#
|
||
# 设计（2026-07-31 实测）：
|
||
# - ak-eod 19:00 用 --force 每日全量重写 valuation/financial_abstract（估值随股价每日变）
|
||
# - bsdtar --newer 按 mtime 增量不可行（VPS 慢盘 stat 11070 文件 >60s）
|
||
# - python tarfile 全打包只 read 内容不 stat，实测 9.5s/1.1GB → scp 单 tar → 解包
|
||
#
|
||
# 调度：ak-eod 19:00 跑完（~21:30）后延迟跑（22:00 或挂 id=2 次日 3:00）。
|
||
# per-date 类（dragon_tiger 等）走 sync_parquet_daily.sh；weekly per-stock（northbound 等）走 sync_parquet.sh 周日全量。
|
||
set -u
|
||
|
||
KEY=/var/services/homes/admin/.ssh/id_ed25519_nas
|
||
VPS=Administrator@49.232.102.198
|
||
ROOT=/volume1/stock/sanguo_vnpy_v2
|
||
DATA=$ROOT/data
|
||
STAGE=$ROOT/data_backup/_staging
|
||
LOG=$ROOT/data_backup/valuation_daily.log
|
||
|
||
VPS_TAR='C:\sanguo_vnpy_v2\data\_sync_val.tar'
|
||
VPS_TAR_SCP='C:/sanguo_vnpy_v2/data/_sync_val.tar'
|
||
|
||
TS=$(date '+%F %T')
|
||
mkdir -p "$STAGE"
|
||
echo "=== $TS valuation+financial_abstract daily ===" >> "$LOG"
|
||
|
||
# VPS: python tarfile 打包（绕 bsdtar 命令解析/慢盘 stat 坑）
|
||
ssh -i "$KEY" -o StrictHostKeyChecking=no "$VPS" "C:\Python310\python.exe -X utf8 -" <<'PY' >> "$LOG" 2>&1
|
||
import tarfile, os
|
||
base = r'C:\sanguo_vnpy_v2\data'
|
||
out = base + r'\_sync_val.tar'
|
||
with tarfile.open(out, 'w') as t:
|
||
for d in ['valuation', 'financial_abstract']:
|
||
t.add(os.path.join(base, 'static', d), arcname='static/' + d)
|
||
print(f'PACK size={os.path.getsize(out)/1024/1024:.1f}MB')
|
||
PY
|
||
|
||
# scp tar 到 NAS + 解包
|
||
if scp -i "$KEY" -o StrictHostKeyChecking=no "$VPS:$VPS_TAR_SCP" "$STAGE/_sync_val.tar" >> "$LOG" 2>&1; then
|
||
tar -xf "$STAGE/_sync_val.tar" -C "$DATA" >> "$LOG" 2>&1
|
||
echo "[$(date '+%T')] valuation daily DONE" >> "$LOG"
|
||
else
|
||
echo "[$(date '+%T')] valuation daily FAIL (scp)" >> "$LOG"
|
||
fi
|
||
|
||
# 清理（NAS 本地 tar + VPS 临时 tar）
|
||
rm -f "$STAGE/_sync_val.tar"
|
||
ssh -i "$KEY" -o StrictHostKeyChecking=no "$VPS" "del \"$VPS_TAR\"" >> "$LOG" 2>&1
|