diff --git a/docs/superpowers/plans/2026-07-23-akshare-low-freq-schtask.md b/docs/superpowers/plans/2026-07-23-akshare-low-freq-schtask.md new file mode 100644 index 0000000..5b1a5ad --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-akshare-low-freq-schtask.md @@ -0,0 +1,168 @@ +# akshare 低频任务 schtask 部署 Plan (spec §14.5) + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development / executing-plans。本 plan 自包含(实测现状+脚本能力+VPS访问),fresh agent 可直接执行。 + +**Goal:** 部署 spec §14.5 akshare/index 低频 schtask — A 三表/估值增量 + B 成份股月度 + C 事件类 7 种。方案A 数据层(`dbbardata`/`constituent_unified`/`valuation_baostock`)的使用层配套,补全 `LocalUnifiedProvider` 依赖的静态数据源。 + +**Architecture:** 复用 `akshare_static_download.py`(16类/marker断点/4模式)+ `baostock_constituent_download.py`;改 `merge_constituent.py`/`migrate_constituent.py` 可重跑;按 akshare per-stock 全量慢 + 东财限流,拆多 schtask(日频/季频/月频)。 + +**Tech Stack:** Python 3.10, akshare, baostock, sqlite3, Windows schtasks + +## Global Constraints(铁律) + +- **akshare**: 单线程 0.8s sleep / 30s 超时 / 断路器(连30 failed exit) / marker 断点 / **unset proxy** — `akshare_static_download.py` 已内置;东财限流严,**per-stock 全量慢,夜间跑** +- **baostock**: 单进程单登录 48000/天,不并发(防黑名单) +- **staging→验证→合并**(成份股 B,用户铁律:下载质量不可控,不直接写主库) +- **VPS**: ssh alias = `49.232.102.198`(IP 即 alias,User Administrator,key id_ed25519);`C:\Python310\python.exe -X utf8`;schtasks `/create /ru SYSTEM /rl HIGHEST /sc daily|monthly`;Windows ssh 引号地狱 → 脚本 scp + ssh python 跑最稳 +- **dbbardata UNIQUE 不破坏**;constituent_unified 治偏差 +- 直连不走代理(schtask wrapper 开头 `unset http_proxy https_proxy all_proxy`) + +## 实测现状(2026-07-23 probe_akshare_status.py) + +- **static/**(`C:\sanguo_vnpy_v2\data\static\`): balance/income/cashflow/valuation/financial_abstract 各 **5530 parquet,停 2026-07-22 18:25**(sanguo-bs-akshare disabled 前最后一次);provider fundamentals 依赖,要续更 +- **events/**: **全 MISSING**(龙虎榜/北向/两融/解禁/大宗/可转债/研报从未采集) +- **constituent_unified**: 7110 行 = baostock 2938(000016:195/000300:940/000905:1803) + akshare_cni 1172(深证 399001:702/399005:145/399006:175/399330:150) + akshare_csindex 3000(000852:1000/932000:2000);**静态,月度更新 schtask 无** +- **schtask 状态**: sanguo-bs-akshare / sanguo-index / sanguo-index-hist **全无**(方案A `stop_all_data_schtasks.ps1` 清了) +- **akshare_static_download.py 16 类分 5 组**: + - PER_STOCK(5500股循环,unit=`{symbol}_{type}`): valuation/northbound/share_capital/balance/income/cashflow/financial_abstract + - TOP_HOLDERS(per-stock×period): top_holders + - PER_DATE(每交易日,unit=`{date}_{type}`): dragon_tiger/block_trade/margin_sse/restricted + - PER_PERIOD(报告期,unit=`{period}_{type}`): forecast/express + - ONE_SHOT: index_const/industry + - marker 是 **symbol 级**(非 period 级)→ 三表/估值要更新新数据必须 `--force`(否则 marker 跳过永不更新) +- **merge_constituent.py 不可重跑**: `ALTER TABLE bs_index_constituent RENAME TO bs_index_constituent_old` 只能一次(_old 已存在);无 DROP/REPLACE constituent_unified +- **existing wrappers**(参考模式): `bs_eod_wrapper.ps1` / `xt_eod_wrapper.ps1`(unset proxy + log + 调 python);`register_schtasks.ps1`(schtasks /create 模板) + +## schtask 清单(方案A §14.5 适配,按频率拆) + +| schtask | 频率 | 时间 | 脚本 | 内容 | +|---|---|---|---|---| +| `sanguo-ak-eod` | daily | 19:00 | ak_eod_wrapper.ps1 | valuation + financial_abstract `--force`(日频,5500×2×0.8s≈2.2h 夜间) | +| `sanguo-ak-quarter` | monthly(财报季 5/9/11 月+年报4月) | 周末 02:00 | ak_quarter_wrapper.ps1 | balance + income + cashflow + forecast + express `--force`(季频,5500×3×0.8≈2.2h) | +| `sanguo-ak-events` | daily | 19:30 | ak_events_wrapper.ps1 | dragon_tiger + block_trade + margin_sse + restricted `--start today --end today`(per-date 日频,4 unit 快) | +| `sanguo-ak-stock` | weekly | 周六 03:00 | ak_stock_wrapper.ps1 | northbound + share_capital + top_holders(per-stock 慢,周频) | +| `sanguo-index` | monthly | 19:50 | index_monthly_wrapper.ps1 | 成份股 3 源 + merge(B) | + +## File Structure + +- **Modify:** `scripts/data_platform/merge_constituent.py`(可重跑: DROP/REPLACE 替代 RENAME) +- **Modify:** `scripts/data_platform/migrate_constituent.py`(可重跑: staging 隔离 + DROP staging 重建) +- **Create:** `scripts/data_platform/ak_eod_wrapper.ps1` / `ak_quarter_wrapper.ps1` / `ak_events_wrapper.ps1` / `ak_stock_wrapper.ps1`(4 个 akshare wrapper) +- **Create:** `scripts/data_platform/index_monthly_wrapper.ps1`(B 成份股 3 源编排) +- **Create:** `scripts/data_platform/register_akshare_schtasks.ps1`(注册 5 schtask) +- **Create:** `scripts/data_platform/verify_akshare_e2e.py`(验证全部) +- **Test:** `tests/portfolio/test_merge_constituent_rerun.py`(B 改造 TDD) + +--- + +## Task A: 三表/估值增量(2 schtask) + +**Files:** Create 4 akshare wrapper + register; 复用 `akshare_static_download.py`(不改)。 + +- [ ] **A1: ak_eod_wrapper.ps1**(日频估值/财务摘要) +```powershell +# unset proxy + 调 akshare_static_download.py --types valuation,financial_abstract --force +$env:http_proxy=""; $env:https_proxy=""; $env:all_proxy="" +cd C:\sanguo_vnpy_v2 +C:\Python310\python.exe -X utf8 scripts\data_platform\akshare_static_download.py ` + --types valuation,financial_abstract --force ` + *>> C:\sanguo_vnpy_v2\data\ak_eod.log +``` +- [ ] **A2: ak_quarter_wrapper.ps1**(季频三表+预告/快报,财报季)— 同上 `--types balance,income,cashflow,forecast,express --force` +- [ ] **A3: 验证 A** — scp wrapper + 手动跑 `--limit 5` 确认 valuation parquet 更新今日: +```bash +scp scripts/data_platform/ak_eod_wrapper.ps1 49.232.102.198:'C:/sanguo_vnpy_v2/scripts/data_platform/' +ssh 49.232.102.198 'cd /d C:\sanguo_vnpy_v2 && C:\Python310\python.exe -X utf8 scripts\data_platform\akshare_static_download.py --types valuation --force --limit 3' +# 验 static/valuation/_valuation.parquet mtime = 今日 +``` +- [ ] **A4: 注册 schtask**(register_akshare_schtasks.ps1 含 sanguo-ak-eod daily 19:00 + sanguo-ak-quarter monthly) + +--- + +## Task B: 成份股月度(sanguo-index)— 代码改造 TDD + +**Files:** Modify `merge_constituent.py` + `migrate_constituent.py`; Create `index_monthly_wrapper.ps1`; Test `test_merge_constituent_rerun.py`。 + +- [ ] **B1: 写失败测试 — merge_constituent 可重跑** +```python +# tests/portfolio/test_merge_constituent_rerun.py +def test_merge_constituent_rerun_twice(tmp_path): + """merge_constituent 跑两次不崩(第二次 DROP 重建,不 RENAME 已 _old 的表)。""" + db = tmp_path / "t.db"; c = sqlite3.connect(str(db)) + # 造 constituent_unified + bs_index_constituent_old(已存在,_old 状态) + c.execute("CREATE TABLE constituent_unified(index_code,code,source,in_current,was_removed)") + c.execute("CREATE TABLE constituent_unified_staging(index_code,code,source,in_current,was_removed)") + c.execute("CREATE TABLE bs_index_constituent_old(code,date)") # _old 已存在 + c.executemany("INSERT INTO constituent_unified_staging VALUES(?,?,?,?,?)", + [("000300","600519","baostock",1,0)]) + c.commit(); c.close() + # 跑两次 + import scripts.data_platform.merge_constituent as m # 或函数级 import + m.merge(str(db)) # 第一次: staging→unified(DROP 旧 unified 重建) + m.merge(str(db)) # 第二次: 不崩, unified 仍 1 行 + c = sqlite3.connect(str(db)) + assert c.execute("SELECT COUNT(*) FROM constituent_unified").fetchone()[0] == 1 +``` + +- [ ] **B2: 改 merge_constituent.py 可重跑** — 把 `ALTER TABLE bs_index_constituent RENAME TO _old`(只能一次)改为:staging→`DROP TABLE IF EXISTS constituent_unified`→`CREATE constituent_unified AS SELECT FROM staging`。bs_index_constituent_old 已存在不碰。幂等。 + +- [ ] **B3: 改 migrate_constituent.py 可重跑** — staging 表 `DROP IF EXISTS constituent_unified_staging` 重建(每次重新聚合 baostock 988 时点 + akshare cni union + csindex),不依赖上次状态。 + +- [ ] **B4: index_monthly_wrapper.ps1**(3 源编排) +```powershell +$env:http_proxy=""; $env:https_proxy=""; $env:all_proxy="" +cd C:\sanguo_vnpy_v2 +# 1. baostock 300/500/50 最新快照(单进程) +C:\Python310\python.exe -X utf8 scripts\data_platform\baostock_constituent_download.py --start 2026-01-01 +# 2. akshare cni 深证 + csindex 中证(复用 P0 脚本 index_const_hist_download 或 akshare_static_download --types index_const) +C:\Python310\python.exe -X utf8 scripts\data_platform\akshare_constituent_download.py +# 3. migrate + merge(可重跑版) +C:\Python310\python.exe -X utf8 scripts\data_platform\migrate_constituent.py +C:\Python310\python.exe -X utf8 scripts\data_platform\merge_constituent.py +``` +(注:akshare_constituent_download.py 若不存在,从 `akshare_static_download.py --types index_const` 或 P0 的 `index_const_hist_download.py` 复用;执行 agent 确认现有脚本) + +- [ ] **B5: 验证 B** — 手动跑 wrapper,确认 `constituent_unified` 行数 ≥ 7110,source 含 baostock/akshare_cni/akshare_csindex,跑两次不崩。 +- [ ] **B6: 注册 sanguo-index monthly 19:50** + +--- + +## Task C: 事件类(per-date 日频 + per-stock 周频) + +**Files:** Create `ak_events_wrapper.ps1` + `ak_stock_wrapper.ps1`(已在 A 的 register 注册)。 + +- [ ] **C1: ak_events_wrapper.ps1**(per-date 日频)— `--types dragon_tiger,block_trade,margin_sse,restricted --start {today} --end {today}`。每日 4 类×1 unit,快。落 `events/{type}/{date}_{type}.parquet`。 +- [ ] **C2: ak_stock_wrapper.ps1**(per-stock 周频慢)— `--types northbound,share_capital,top_holders --force`。5500×3 慢,周六 03:00。 +- [ ] **C3: 可转债/研报**(spec §14.2 列但 akshare_static_download.py 16 类无对应 fetcher) — **评估**:若 akshare 有 `bond_zh_hs_cov_min`/`stock_research_info_em` 接口,加 fetcher;否则 N/A 标注使用说明。执行 agent 确认 akshare 接口可用性,不可用则跳过并在 verify 标注。 +- [ ] **C4: 验证 C** — 手动跑 events wrapper `--start today --end today`,确认 `events/dragon_tiger/{today}_dragon_tiger.parquet` 生成。 + +--- + +## Task D: 注册 + E2E + +- [ ] **D1: register_akshare_schtasks.ps1** — 注册 5 schtask(sanguo-ak-eod/ak-quarter/ak-events/ak-stock/index),`/create /ru SYSTEM /rl HIGHEST`,verify 段 `schtasks /query` 确认 Status=Ready。 +- [ ] **D2: verify_akshare_e2e.py** — 验证全部: + - static/valuation 最新 mtime = 近日 + - events/dragon_tiger 有 parquet + - constituent_unified ≥ 7110 + 3 source +- [ ] **D3: 更新 memory + 使用说明** — `data-fusion-design-finalized` 的"剩余待办 akshare schtask"标完成;`docs/portfolio_local_unified_provider.md` 事件类从 N/A 更新。 + +--- + +## Self-Review + +1. **spec §14.5 覆盖**: sanguo-bs-akshare(三表+事件)→ 拆 ak-eod/ak-quarter/ak-events/ak-stock(频率适配);sanguo-index 月度 → B。✓ +2. **provider 依赖**: valuation/balance/income/cashflow/financial_abstract(LocalUnifiedProvider.get_fundamentals_df)→ A 覆盖。✓ +3. **限流现实**: per-stock --force 全量慢,日频 valuation 2.2h / 季频三表 2.2h,夜间 + 周末 schtask。events per-date 日频快。✓ +4. **B 可重跑**: merge DROP 重建幂等,migrate staging 隔离。TDD 跑两次不崩。✓ +5. **风险**: akshare 东财封 IP(per-stock 全量)→ wrapper 内置断路器 + sleep;财报季触发 ak-quarter(`/sc monthly` 指定月或手动);可转债/研报接口待确认(C3)。 + +## Execution Handoff + +Plan saved to `docs/superpowers/plans/2026-07-23-akshare-low-freq-schtask.md`。compact 后新 session 派 Sub Agent 执行(参考 LocalUnifiedProvider 模式:Task A→B→C→D,每 task 验证 + commit)。 + +## 关联文档/memory + +- spec: `docs/superpowers/specs/2026-07-21-data-source-fusion-design.md` §14.5 +- memory: `data-fusion-design-finalized`(方案A)/ `local-unified-provider-complete`(使用层)/ `vps-local-data-layout` / `baostock-concurrent-blacklist` / `schtasks-system-bat-gotchas` +- VPS 访问: ssh `49.232.102.198`,见 memory `windows-vps-access` diff --git a/scripts/data_platform/probe_akshare_status.py b/scripts/data_platform/probe_akshare_status.py new file mode 100644 index 0000000..1287536 --- /dev/null +++ b/scripts/data_platform/probe_akshare_status.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""probe_akshare_status.py — 探查 akshare 低频任务现状(sanguo-bs-akshare + sanguo-index)。 + +看: 三表 static 各子目录文件数+最新mtime / 事件类数据有没有 / constituent_unified source 分布。 +""" +import datetime +import sqlite3 +from pathlib import Path + +BASE = Path(r"C:\sanguo_vnpy_v2\data") +print("===== data/ 子目录 =====") +if BASE.exists(): + for p in sorted(BASE.iterdir()): + if p.is_dir(): + n = sum(1 for _ in p.rglob("*") if _.is_file()) + print(f" {p.name}/ ({n} files)") + else: + print(f" {p.name}") + +print("\n===== static/ 各表文件数 + 最新 mtime =====") +for sub in ["balance", "income", "cashflow", "valuation", "financial_abstract"]: + d = BASE / "static" / sub + if not d.exists(): + print(f" static/{sub}: MISSING") + continue + files = list(d.glob("*.parquet")) + if not files: + print(f" static/{sub}: 0 parquet") + continue + mt = max(f.stat().st_mtime for f in files) + print(f" static/{sub}: {len(files)} parquet, latest mtime={datetime.datetime.fromtimestamp(mt):%Y-%m-%d %H:%M}") + +print("\n===== 事件类数据(龙虎榜/北向/两融/解禁/大宗/可转债/研报) =====") +for sub in ["longhubang", "north_flow", "margin", "blockade", "block_trade", "convertible_bond", "research"]: + d = BASE / "events" / sub + if d.exists(): + files = list(d.glob("*")) + print(f" events/{sub}: {len(files)} files") + else: + print(f" events/{sub}: MISSING") +# data 根下找可能的 events/其它事件目录 +for cand in ["events", "akshare_events", "longhubang", "north"]: + d = BASE / cand + if d.exists(): + print(f" {cand}/ exists") + +print("\n===== constituent_unified source 分布(看 akshare 部分啥时点) =====") +c = sqlite3.connect(str(BASE / "quant_trading.db")) +try: + print(" source 分布:", c.execute("SELECT source, COUNT(*) FROM constituent_unified GROUP BY source").fetchall()) + print(" per-index×source:") + for row in c.execute("SELECT index_code, source, COUNT(*) FROM constituent_unified GROUP BY index_code, source ORDER BY index_code"): + print(" ", row) +finally: + c.close() +print("\nPROBE DONE")