claude_dev
c3e53fbef3
- scripts/data_platform/_archive/legacy/: 归档20个独立探针/诊断/旧降级脚本(零引用验证) - docs/archive/data/: 归档17个数据相关旧设计/plan/report(保留fusion spec作深读) - docs/data-platform/README.md: 数据层单一权威记录(8节:架构/布局/源/管线/铁律/API/缺口/待办) - 删除 _mootdx_depth_result.txt - Phase2待办: 15m灌库链+旧回填import链(有测试/wrapper依赖,VPS schtask确认后归档)
169 lines
12 KiB
Markdown
169 lines
12 KiB
Markdown
# akshare 低频任务 schtask 部署 Plan (spec §14.5)
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development / executing-plans。本 plan 自包含(实测现状+脚本能力+VPS访问),fresh agent 可直接执行。
|
||
|
||
**Goal:** 部署 spec §14.5 akshare/index 低频 schtask — A 三表/估值增量 + B 成份股月度 + C 事件类 7 种。方案A 数据层(`dbbardata`/`constituent_unified`/`valuation_baostock`)的使用层配套,补全 `LocalUnifiedProvider` 依赖的静态数据源。
|
||
|
||
**Architecture:** 复用 `akshare_static_download.py`(16类/marker断点/4模式)+ `baostock_constituent_download.py`;改 `merge_constituent.py`/`migrate_constituent.py` 可重跑;按 akshare per-stock 全量慢 + 东财限流,拆多 schtask(日频/季频/月频)。
|
||
|
||
**Tech Stack:** Python 3.10, akshare, baostock, sqlite3, Windows schtasks
|
||
|
||
## Global Constraints(铁律)
|
||
|
||
- **akshare**: 单线程 0.8s sleep / 30s 超时 / 断路器(连30 failed exit) / marker 断点 / **unset proxy** — `akshare_static_download.py` 已内置;东财限流严,**per-stock 全量慢,夜间跑**
|
||
- **baostock**: 单进程单登录 48000/天,不并发(防黑名单)
|
||
- **staging→验证→合并**(成份股 B,用户铁律:下载质量不可控,不直接写主库)
|
||
- **VPS**: ssh alias = `49.232.102.198`(IP 即 alias,User Administrator,key id_ed25519);`C:\Python310\python.exe -X utf8`;schtasks `/create /ru SYSTEM /rl HIGHEST /sc daily|monthly`;Windows ssh 引号地狱 → 脚本 scp + ssh python 跑最稳
|
||
- **dbbardata UNIQUE 不破坏**;constituent_unified 治偏差
|
||
- 直连不走代理(schtask wrapper 开头 `unset http_proxy https_proxy all_proxy`)
|
||
|
||
## 实测现状(2026-07-23 probe_akshare_status.py)
|
||
|
||
- **static/**(`C:\sanguo_vnpy_v2\data\static\`): balance/income/cashflow/valuation/financial_abstract 各 **5530 parquet,停 2026-07-22 18:25**(sanguo-bs-akshare disabled 前最后一次);provider fundamentals 依赖,要续更
|
||
- **events/**: **全 MISSING**(龙虎榜/北向/两融/解禁/大宗/可转债/研报从未采集)
|
||
- **constituent_unified**: 7110 行 = baostock 2938(000016:195/000300:940/000905:1803) + akshare_cni 1172(深证 399001:702/399005:145/399006:175/399330:150) + akshare_csindex 3000(000852:1000/932000:2000);**静态,月度更新 schtask 无**
|
||
- **schtask 状态**: sanguo-bs-akshare / sanguo-index / sanguo-index-hist **全无**(方案A `stop_all_data_schtasks.ps1` 清了)
|
||
- **akshare_static_download.py 16 类分 5 组**:
|
||
- PER_STOCK(5500股循环,unit=`{symbol}_{type}`): valuation/northbound/share_capital/balance/income/cashflow/financial_abstract
|
||
- TOP_HOLDERS(per-stock×period): top_holders
|
||
- PER_DATE(每交易日,unit=`{date}_{type}`): dragon_tiger/block_trade/margin_sse/restricted
|
||
- PER_PERIOD(报告期,unit=`{period}_{type}`): forecast/express
|
||
- ONE_SHOT: index_const/industry
|
||
- marker 是 **symbol 级**(非 period 级)→ 三表/估值要更新新数据必须 `--force`(否则 marker 跳过永不更新)
|
||
- **merge_constituent.py 不可重跑**: `ALTER TABLE bs_index_constituent RENAME TO bs_index_constituent_old` 只能一次(_old 已存在);无 DROP/REPLACE constituent_unified
|
||
- **existing wrappers**(参考模式): `bs_eod_wrapper.ps1` / `xt_eod_wrapper.ps1`(unset proxy + log + 调 python);`register_schtasks.ps1`(schtasks /create 模板)
|
||
|
||
## schtask 清单(方案A §14.5 适配,按频率拆)
|
||
|
||
| schtask | 频率 | 时间 | 脚本 | 内容 |
|
||
|---|---|---|---|---|
|
||
| `sanguo-ak-eod` | daily | 19:00 | ak_eod_wrapper.ps1 | valuation + financial_abstract `--force`(日频,5500×2×0.8s≈2.2h 夜间) |
|
||
| `sanguo-ak-quarter` | monthly(财报季 5/9/11 月+年报4月) | 周末 02:00 | ak_quarter_wrapper.ps1 | balance + income + cashflow + forecast + express `--force`(季频,5500×3×0.8≈2.2h) |
|
||
| `sanguo-ak-events` | daily | 19:30 | ak_events_wrapper.ps1 | dragon_tiger + block_trade + margin_sse + restricted `--start today --end today`(per-date 日频,4 unit 快) |
|
||
| `sanguo-ak-stock` | weekly | 周六 03:00 | ak_stock_wrapper.ps1 | northbound + share_capital + top_holders(per-stock 慢,周频) |
|
||
| `sanguo-index` | monthly | 19:50 | index_monthly_wrapper.ps1 | 成份股 3 源 + merge(B) |
|
||
|
||
## File Structure
|
||
|
||
- **Modify:** `scripts/data_platform/merge_constituent.py`(可重跑: DROP/REPLACE 替代 RENAME)
|
||
- **Modify:** `scripts/data_platform/migrate_constituent.py`(可重跑: staging 隔离 + DROP staging 重建)
|
||
- **Create:** `scripts/data_platform/ak_eod_wrapper.ps1` / `ak_quarter_wrapper.ps1` / `ak_events_wrapper.ps1` / `ak_stock_wrapper.ps1`(4 个 akshare wrapper)
|
||
- **Create:** `scripts/data_platform/index_monthly_wrapper.ps1`(B 成份股 3 源编排)
|
||
- **Create:** `scripts/data_platform/register_akshare_schtasks.ps1`(注册 5 schtask)
|
||
- **Create:** `scripts/data_platform/verify_akshare_e2e.py`(验证全部)
|
||
- **Test:** `tests/portfolio/test_merge_constituent_rerun.py`(B 改造 TDD)
|
||
|
||
---
|
||
|
||
## Task A: 三表/估值增量(2 schtask)
|
||
|
||
**Files:** Create 4 akshare wrapper + register; 复用 `akshare_static_download.py`(不改)。
|
||
|
||
- [ ] **A1: ak_eod_wrapper.ps1**(日频估值/财务摘要)
|
||
```powershell
|
||
# unset proxy + 调 akshare_static_download.py --types valuation,financial_abstract --force
|
||
$env:http_proxy=""; $env:https_proxy=""; $env:all_proxy=""
|
||
cd C:\sanguo_vnpy_v2
|
||
C:\Python310\python.exe -X utf8 scripts\data_platform\akshare_static_download.py `
|
||
--types valuation,financial_abstract --force `
|
||
*>> C:\sanguo_vnpy_v2\data\ak_eod.log
|
||
```
|
||
- [ ] **A2: ak_quarter_wrapper.ps1**(季频三表+预告/快报,财报季)— 同上 `--types balance,income,cashflow,forecast,express --force`
|
||
- [ ] **A3: 验证 A** — scp wrapper + 手动跑 `--limit 5` 确认 valuation parquet 更新今日:
|
||
```bash
|
||
scp scripts/data_platform/ak_eod_wrapper.ps1 49.232.102.198:'C:/sanguo_vnpy_v2/scripts/data_platform/'
|
||
ssh 49.232.102.198 'cd /d C:\sanguo_vnpy_v2 && C:\Python310\python.exe -X utf8 scripts\data_platform\akshare_static_download.py --types valuation --force --limit 3'
|
||
# 验 static/valuation/<code>_valuation.parquet mtime = 今日
|
||
```
|
||
- [ ] **A4: 注册 schtask**(register_akshare_schtasks.ps1 含 sanguo-ak-eod daily 19:00 + sanguo-ak-quarter monthly)
|
||
|
||
---
|
||
|
||
## Task B: 成份股月度(sanguo-index)— 代码改造 TDD
|
||
|
||
**Files:** Modify `merge_constituent.py` + `migrate_constituent.py`; Create `index_monthly_wrapper.ps1`; Test `test_merge_constituent_rerun.py`。
|
||
|
||
- [ ] **B1: 写失败测试 — merge_constituent 可重跑**
|
||
```python
|
||
# tests/portfolio/test_merge_constituent_rerun.py
|
||
def test_merge_constituent_rerun_twice(tmp_path):
|
||
"""merge_constituent 跑两次不崩(第二次 DROP 重建,不 RENAME 已 _old 的表)。"""
|
||
db = tmp_path / "t.db"; c = sqlite3.connect(str(db))
|
||
# 造 constituent_unified + bs_index_constituent_old(已存在,_old 状态)
|
||
c.execute("CREATE TABLE constituent_unified(index_code,code,source,in_current,was_removed)")
|
||
c.execute("CREATE TABLE constituent_unified_staging(index_code,code,source,in_current,was_removed)")
|
||
c.execute("CREATE TABLE bs_index_constituent_old(code,date)") # _old 已存在
|
||
c.executemany("INSERT INTO constituent_unified_staging VALUES(?,?,?,?,?)",
|
||
[("000300","600519","baostock",1,0)])
|
||
c.commit(); c.close()
|
||
# 跑两次
|
||
import scripts.data_platform.merge_constituent as m # 或函数级 import
|
||
m.merge(str(db)) # 第一次: staging→unified(DROP 旧 unified 重建)
|
||
m.merge(str(db)) # 第二次: 不崩, unified 仍 1 行
|
||
c = sqlite3.connect(str(db))
|
||
assert c.execute("SELECT COUNT(*) FROM constituent_unified").fetchone()[0] == 1
|
||
```
|
||
|
||
- [ ] **B2: 改 merge_constituent.py 可重跑** — 把 `ALTER TABLE bs_index_constituent RENAME TO _old`(只能一次)改为:staging→`DROP TABLE IF EXISTS constituent_unified`→`CREATE constituent_unified AS SELECT FROM staging`。bs_index_constituent_old 已存在不碰。幂等。
|
||
|
||
- [ ] **B3: 改 migrate_constituent.py 可重跑** — staging 表 `DROP IF EXISTS constituent_unified_staging` 重建(每次重新聚合 baostock 988 时点 + akshare cni union + csindex),不依赖上次状态。
|
||
|
||
- [ ] **B4: index_monthly_wrapper.ps1**(3 源编排)
|
||
```powershell
|
||
$env:http_proxy=""; $env:https_proxy=""; $env:all_proxy=""
|
||
cd C:\sanguo_vnpy_v2
|
||
# 1. baostock 300/500/50 最新快照(单进程)
|
||
C:\Python310\python.exe -X utf8 scripts\data_platform\baostock_constituent_download.py --start 2026-01-01
|
||
# 2. akshare cni 深证 + csindex 中证(复用 P0 脚本 index_const_hist_download 或 akshare_static_download --types index_const)
|
||
C:\Python310\python.exe -X utf8 scripts\data_platform\akshare_constituent_download.py
|
||
# 3. migrate + merge(可重跑版)
|
||
C:\Python310\python.exe -X utf8 scripts\data_platform\migrate_constituent.py
|
||
C:\Python310\python.exe -X utf8 scripts\data_platform\merge_constituent.py
|
||
```
|
||
(注:akshare_constituent_download.py 若不存在,从 `akshare_static_download.py --types index_const` 或 P0 的 `index_const_hist_download.py` 复用;执行 agent 确认现有脚本)
|
||
|
||
- [ ] **B5: 验证 B** — 手动跑 wrapper,确认 `constituent_unified` 行数 ≥ 7110,source 含 baostock/akshare_cni/akshare_csindex,跑两次不崩。
|
||
- [ ] **B6: 注册 sanguo-index monthly 19:50**
|
||
|
||
---
|
||
|
||
## Task C: 事件类(per-date 日频 + per-stock 周频)
|
||
|
||
**Files:** Create `ak_events_wrapper.ps1` + `ak_stock_wrapper.ps1`(已在 A 的 register 注册)。
|
||
|
||
- [ ] **C1: ak_events_wrapper.ps1**(per-date 日频)— `--types dragon_tiger,block_trade,margin_sse,restricted --start {today} --end {today}`。每日 4 类×1 unit,快。落 `events/{type}/{date}_{type}.parquet`。
|
||
- [ ] **C2: ak_stock_wrapper.ps1**(per-stock 周频慢)— `--types northbound,share_capital,top_holders --force`。5500×3 慢,周六 03:00。
|
||
- [ ] **C3: 可转债/研报**(spec §14.2 列但 akshare_static_download.py 16 类无对应 fetcher) — **评估**:若 akshare 有 `bond_zh_hs_cov_min`/`stock_research_info_em` 接口,加 fetcher;否则 N/A 标注使用说明。执行 agent 确认 akshare 接口可用性,不可用则跳过并在 verify 标注。
|
||
- [ ] **C4: 验证 C** — 手动跑 events wrapper `--start today --end today`,确认 `events/dragon_tiger/{today}_dragon_tiger.parquet` 生成。
|
||
|
||
---
|
||
|
||
## Task D: 注册 + E2E
|
||
|
||
- [ ] **D1: register_akshare_schtasks.ps1** — 注册 5 schtask(sanguo-ak-eod/ak-quarter/ak-events/ak-stock/index),`/create /ru SYSTEM /rl HIGHEST`,verify 段 `schtasks /query` 确认 Status=Ready。
|
||
- [ ] **D2: verify_akshare_e2e.py** — 验证全部:
|
||
- static/valuation 最新 mtime = 近日
|
||
- events/dragon_tiger 有 parquet
|
||
- constituent_unified ≥ 7110 + 3 source
|
||
- [ ] **D3: 更新 memory + 使用说明** — `data-fusion-design-finalized` 的"剩余待办 akshare schtask"标完成;`docs/portfolio_local_unified_provider.md` 事件类从 N/A 更新。
|
||
|
||
---
|
||
|
||
## Self-Review
|
||
|
||
1. **spec §14.5 覆盖**: sanguo-bs-akshare(三表+事件)→ 拆 ak-eod/ak-quarter/ak-events/ak-stock(频率适配);sanguo-index 月度 → B。✓
|
||
2. **provider 依赖**: valuation/balance/income/cashflow/financial_abstract(LocalUnifiedProvider.get_fundamentals_df)→ A 覆盖。✓
|
||
3. **限流现实**: per-stock --force 全量慢,日频 valuation 2.2h / 季频三表 2.2h,夜间 + 周末 schtask。events per-date 日频快。✓
|
||
4. **B 可重跑**: merge DROP 重建幂等,migrate staging 隔离。TDD 跑两次不崩。✓
|
||
5. **风险**: akshare 东财封 IP(per-stock 全量)→ wrapper 内置断路器 + sleep;财报季触发 ak-quarter(`/sc monthly` 指定月或手动);可转债/研报接口待确认(C3)。
|
||
|
||
## Execution Handoff
|
||
|
||
Plan saved to `docs/superpowers/plans/2026-07-23-akshare-low-freq-schtask.md`。compact 后新 session 派 Sub Agent 执行(参考 LocalUnifiedProvider 模式:Task A→B→C→D,每 task 验证 + commit)。
|
||
|
||
## 关联文档/memory
|
||
|
||
- spec: `docs/superpowers/specs/2026-07-21-data-source-fusion-design.md` §14.5
|
||
- memory: `data-fusion-design-finalized`(方案A)/ `local-unified-provider-complete`(使用层)/ `vps-local-data-layout` / `baostock-concurrent-blacklist` / `schtasks-system-bat-gotchas`
|
||
- VPS 访问: ssh `49.232.102.198`,见 memory `windows-vps-access`
|