diff --git a/docs/static_data_gaps_design.md b/docs/static_data_gaps_design.md index bef75c1..ec11095 100644 --- a/docs/static_data_gaps_design.md +++ b/docs/static_data_gaps_design.md @@ -1,6 +1,11 @@ -# 静态数据 3 个真缺口 — 补充设计(2026-07-19 记录,待当前下载完再动手) +# 静态数据 3 个真缺口 — 补充设计(2026-07-19 记录) -> 当前下载(15min Mac补全 + akshare static + baostock static)跑完后再继续本设计。先记录存底。 +> **✅ 方案A 2026-07-22 落地后状态**(见 [[data-fusion-design-finalized]] memory §14): +> - **缺口1 日线换手/涨跌**:已实现——baostock `turn`/`pctChg` 拆 `valuation_baostock/.parquet` 按年宽表(2003-2026),非派生(原设计的"派生方案"已被 baostock 现成字段替代)。 +> - **缺口2 ETF 入 universe**:已完成——xtata `sanguo-xt-eod` 18:40 跑,universe 沪深A股∪ETF∪基金=7414,dividend_type='front',ETF 入 dbbardata 与个股共表。 +> - **缺口3 指数成分历史**:✅ **2026-07-23 全闭环**——`constituent_unified` 8466行/9指数(300/500/50 baostock 时点聚合全集 + 深证 4 指 akshare cni union 含被踢 + **中证1000/2000 csindex 公告回溯全集**)。**原"永久 gap"已推翻**:csindex 公告 JSON 接口(queryAnnouncementByVo + PDF/xlsx 附件)回溯调整公告治偏差,000852 1000→1672(was_removed=672)/ 932000 2000→2684(684)。详见 memory [[csindex-announce-backfill]]。残留 gap:932000 中间调整 csindex 无公告(launch∪current 近似)/ 000852 2007-2016 部分公告无附件(2016-12 起完整)。 +> +> 下文为 2026-07-19 原始设计,保留作历史参考。 ## 数据现状实测(VPS quant_trading.db + data/ 目录,非推理) - **DB dbbardata**: 15m(2025-07~2026-07,1年,xt_tacitdata源)/5m(1年)/**d日线(2010~2026,16年,5205 symbols,OHLCV+amount)**。dbbardata schema 有 turnover(=成交额amount),**无换手率/涨跌幅列**。 diff --git a/docs/superpowers/plans/2026-07-23-csi1000-constituent-backfill.md b/docs/superpowers/plans/2026-07-23-csi1000-constituent-backfill.md new file mode 100644 index 0000000..73c98b8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-csi1000-constituent-backfill.md @@ -0,0 +1,191 @@ +# 中证1000/2000 历史成份股补全 实施计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans。Steps 用 checkbox `- [ ]` 跟踪。 + +**Goal:** 把中证1000(000852)/中证2000(932000)从"纯当前快照"补成"治幸存者偏差的全集"(含被踢出的股票),接入 `constituent_unified`,并部署定期更新 schtask。 + +**Architecture:** csindex 官方公告 JSON 接口(`queryAnnouncementByVo` + `queryAnnouncementById`)抓调整公告 → 解析附件 PDF/xlsx 的调入/调出名单 → 聚合成"曾经入选集"(全集型,非时点型)→ 入 `constituent_unified`,`in_current`=当前快照、`was_removed`=曾经入选−当前。 + +**Tech Stack:** Python3 + pandas + openpyxl + pdfplumber + sqlite3 + PowerShell schtask + +## 诊断(已实证,2026-07-23) + +现状 `constituent_unified`(VPS quant_trading.db): +- `000852`: total=1000, in_current=1000, **was_removed=0**(纯快照,未治偏差) +- `932000`: total=2000, in_current=2000, **was_removed=0**(纯快照) +- 对比 `000300`: total=940, in_current=300, was_removed=640(已治偏差) + +**三处断点:** +1. **932000 launch xlsx 解析 bug**:`parse_csindex_announce.py:529` 取 `row[0]`(=指数代码 932000),应为 `row[3]`(证券代码)。→ 产出 distinct=1(2000 行全是 932000)。xlsx 实证 6 列:`指数代码/指数简称/指数英文简称/证券代码/证券中文简称/证券英文名称`。 +2. **000852 公告覆盖不全**:`filter_csi1000_notices`(162 行)用 `theme='指数调样'+title 含'中证1000'` 过滤,只拿 28 份(2018-07 起)。调查实证:列表 API payload 加 `indexCode:'000852'` 能拿 **96 条**(45 调样),可回溯到 **2014 发布期**(早期 HTML 表格,2018+ PDF/xlsx)。 +3. **migrate 没接 announce_union**:`migrate_constituent.py:118-131` 只读 `_snapshot.parquet`,没读 `_announce_union.parquet`。→ 1220 个治偏差集白产了。路径也对不上(parse 在 Mac 产 announce_union,migrate 读 VPS HIST,没同步)。 + +## 关键简化 + +`constituent_unified` 是**全集型**(300/500/50 = baostock 988 时点聚合成 in_current/was_removed),**不是时点型**。所以: +- **不需要**反向回溯引擎(生效日边界、逐时点 asof join) +- 只要"曾经入选集"= 所有公告 add 记录 ∪ initial ∪ current 的 distinct code +- `in_current` = akshare 当前快照(权威),`was_removed` = 曾经入选 − 当前 + +调查 agent 提的"生效日≠公告日"等坑是**时点型**需求才需要,本计划(全集型)不涉及。 + +## Global Constraints(spec 铁律) +- baostock 单进程单登录不并发(本计划不碰 baostock,无冲突) +- 直连不走代理:`unset http_proxy https_proxy all_proxy`(脚本已内置) +- 单线程限速:csindex 接口 sleep 1.0~1.5s +- staging→验证→合并,不直接写主库(migrate 走 staging→merge 两步,已幂等) +- provider 读 VPS 本地,不调 online(本计划是采集层,可调 csindex) +- commit message 无 Co-Authored-By + +--- + +### Task 1(#30):修 parse_csindex_announce.py 两处 + +**Files:** +- Modify: `scripts/data_platform/parse_csindex_announce.py:526-538`(932000 launch xlsx 列索引) +- Modify: `scripts/data_platform/parse_csindex_announce.py:120-182`(000852 列表搜索用 indexCode) + +**改动 1a — 932000 launch xlsx 列索引(:526-538):** +现:`code = _norm_code(row[0])`, `name = str(row[1])`。改为按 header 定位列(稳健),或直接 `code=row[3]`, `name=row[4]`。推荐 header 定位: +```python +header = rows[0] +# 找"证券代码"和"证券中文简称"列(中英文混合 header) +code_idx = next((i for i,h in enumerate(header) if h and "证券代码" in str(h)), 3) +name_idx = next((i for i,h in enumerate(header) if h and "证券中文简称" in str(h)), 4) +for row in rows[1:]: + code = _norm_code(row[code_idx] if len(row)>code_idx else None) + name = str(row[name_idx]).strip() if len(row)>name_idx and row[name_idx] else "" +``` + +**改动 1b — 000852 列表搜索用 indexCode(:120-182):** +现 `fetch_all_notices` 拉全量再 `filter_csi1000_notices` title 过滤。改为:对 000852 用 `indexCode` payload 直接搜: +```python +payload = {"lang":"cn","classlist":[],"indexlist":[], + "indexCode":"000852", # ← 新增,直接按指数搜 + "page":{"desc":"","key":"","page":page,"rows":100}, + "related_topics":[],"typelist":[]} +``` +保留旧 filter 作兜底(标题含中证1000+调整)。合并 indexCode 命中 ∪ 已知 REGULAR/TEMP_IDS 去重。932000 走全局 `related_topics:["index_rebalance"]` + PDF grep "中证2000" section(parse_pdf_adjustments 已支持 target_section)。 + +**验证探针:** +```bash +python3 scripts/data_platform/parse_csindex_announce.py --only 1000 +# 期望:filtered CSI 1000 公告 ≥ 40 条(原 28),date 范围早于 2018-07 +python3 scripts/data_platform/parse_csindex_announce.py --only 2000 +# 期望:932000_announce_union.parquet distinct codes ≈ 2000(原 bug=1) +``` + +- [ ] Step 1: 改 932000 launch xlsx 列索引(header 定位) +- [ ] Step 2: 改 000852 列表搜索(indexCode payload + 932000 related_topics) +- [ ] Step 3: Mac 重跑 `--only 1000` + `--only 2000`,验证探针 +- [ ] Step 4: commit + +--- + +### Task 2(#31):改 migrate_constituent.py 接 announce_union 聚合全集 + +**Files:** +- Modify: `scripts/data_platform/migrate_constituent.py:118-131`(加读 announce_union) +- Test: `tests/portfolio/test_migrate_announce_union.py`(新建,TDD) + +**聚合逻辑(全集型):** +```python +# 读 000852_announce_union.parquet + 932000_announce_union.parquet +# announce_union schema: updateDate/index_code/code/code_name/adjust_type(add|remove|current|initial|current)/notice_id/source +# 全集聚合: +for idx in ['000852','932000']: + ann = read(f"{idx}_announce_union.parquet") + snap = read(f"{idx}_snapshot.parquet") # akshare 当前快照,权威 in_current + current_codes = set(snap['code']) # 当前在册 + ever_codes = set(ann['code']) | current_codes # 曾经入选(所有 add/initial + current) + # 产出:ever_codes 每只一行 + # in_current = code in current_codes + # was_removed = code not in current_codes(曾入选已踢) + # source = 'csindex_announce' +``` +schema 对齐:`index_code/code/code_name/source/in_current/was_removed`。`code_name` 取 announce_union 或 snapshot 的(优先 snapshot 当前名)。 + +**合并进 staging:** 现有 `all_df = pd.concat([pool, df_deep, df_snap])`(:134)→ 把 000852/932000 的 announce_union 全集**替换** df_snap 里的 000852/932000 快照行(快照并入 announce 全集的 in_current),其他指数不动。 + +**TDD 测试(tests/portfolio/test_migrate_announce_union.py):** +- test announce_union 聚合:given announce(add A,B + remove C) + snapshot(current A,B,D),assert ever={A,B,C,D}, in_current={A,B,D}, was_removed={C} +- test 000852 distinct > 1000(治偏差证据) +- test 932000 distinct ≈ 2000(launch 修复) +- test 幂等(跑两次结果一致) + +- [ ] Step 1: 写聚合测试(RED) +- [ ] Step 2: 改 migrate 加 announce_union 聚合(GREEN) +- [ ] Step 3: 测试通过 +- [ ] Step 4: commit + +--- + +### Task 3(#32):重跑→同步VPS→migrate→merge→验证 + +**Files:** 无新文件(运行现有 pipeline) + +- [ ] Step 1: Mac 重跑 parse_csindex_announce.py --only both → 新 announce_union +- [ ] Step 2: scp 000852_announce_union.parquet + 932000_announce_union.parquet 到 VPS `C:\sanguo_vnpy_v2\data\index_const_hist\` +- [ ] Step 3: rsync 改后的 migrate_constituent.py 到 VPS +- [ ] Step 4: VPS 跑 migrate_constituent.py(SANGUO_DB 指向 quant_trading.db)→ merge_constituent.py +- [ ] Step 5: 验证(见下) + +**验证标准(VPS 查 constituent_unified):** +```sql +SELECT index_code, COUNT(*), SUM(in_current), SUM(was_removed) +FROM constituent_unified WHERE index_code IN ('000852','932000') GROUP BY index_code; +``` +- 000852: total > 1000(曾经入选 ~1200+), in_current=1000, **was_removed > 0**(治偏差) +- 932000: total ≈ 2000+, in_current=当前快照数, was_removed ≥ 0(launch ∪ current,中间调整无记录则 was_removed=0 可接受) +- 抽样:挑一只 known 被踢股(如 announce_union 里 remove 类型)→ constituent_unified 该 code was_removed=1 +- 回归:300/500/50/深证 行数不变(没误伤) + +--- + +### Task 4(#33):定期 schtask 方案+部署 + +**Files:** +- Create: `scripts/data_platform/csindex_constituent_wrapper.ps1` +- Create: `scripts/data_platform/register_csindex_schtasks.ps1` + +**schtask 设计:** +- 名:`sanguo-csindex-constituent` +- 频率:**每月 16 号 + 6月/12月定调后额外**(中证1000 定期调整 6月/12月,临时调整不定期 → 月度抓足够,缓存增量) +- 时间:**20:30**(避开 baostock 18:05/xt 18:40/akshare 19:00-19:50 窗口) +- 流程:parse_csindex_announce.py --refresh-list(抓新公告)→ 同步 announce_union 已在本机 → migrate → merge +- 幂等:migrate/merge 已 DROP+CREATE 可重跑;parse 有 notice cache 增量 + +**wrapper ps1(仿 bs_eod_wrapper.ps1 风格):** unset proxy → Set-Location → timestamped log → python parse + migrate + merge → exit code + +- [ ] Step 1: 写 wrapper ps1 + register ps1 +- [ ] Step 2: VPS 部署 + schtasks /create /ru SYSTEM /rl HIGHEST +- [ ] Step 3: 手动触发一次验证(schtasks /run) +- [ ] Step 4: commit + 同步安装目录 + +--- + +### Task 5(#34):更新 memory + +**Files:** +- Update: memory `data-fusion-design-finalized.md`(推翻 000852/932000 "永久 gap") +- Update: memory `static_data_gaps_design.md`(中证1000/2000 gap 关闭) +- Update: `MEMORY.md` 索引 + +**记:** csindex 公告 JSON 接口路推翻"永久 gap";000852 全集入库(曾经入选 1200+);932000 launch xlsx 列 bug 修复;全集型简化洞察(不需回溯引擎);定期 schtask;调查 agent 实证的 96 公告/45 调样/回溯到 2014。 + +- [ ] Step 1: 更新 3 个 memory 文件 +- [ ] Step 2: MEMORY.md 索引行 + +--- + +## Self-Review + +- spec 覆盖:① 调整补全→Task1-3 ② 定期抓取方案→Task4 ✓ +- 全集型简化避免过度设计(调查 agent 的回溯引擎是 future 时点型需求,现不做)✓ +- TDD:migrate 聚合逻辑先写测试 ✓ +- 不破坏:300/500/50/深证 migrate 路径不动,只加 000852/932000 announce 段 ✓ +- 约束:不走代理/单线程/staging→merge 幂等/不碰 baostock ✓ + +## 已知残留 gap(接受,不阻塞) +- 932000 中间调整(2023-08 launch 到 current 之间)csindex 无公告 → launch ∪ current 全集,中间被踢的不可补(2023 新指数,影响小) +- 000852 2014-2017 早期 HTML 表格解析格式松散,可能不全(扩 indexCode 搜索尽力补,实证 id=5/id=1585 等仍有表格) diff --git a/scripts/data_platform/index_monthly_wrapper.ps1 b/scripts/data_platform/index_monthly_wrapper.ps1 index c680573..3be22b4 100644 --- a/scripts/data_platform/index_monthly_wrapper.ps1 +++ b/scripts/data_platform/index_monthly_wrapper.ps1 @@ -1,13 +1,16 @@ # index_monthly_wrapper.ps1 — sanguo-index schtask wrapper (monthly 16号 19:50 成份股刷新) -# 1. index_const_hist_download 深证(A adjust_cni)+中证(B snapshot) akshare parquet 刷新(成份股随时间变) -# 2. migrate_constituent 重建 staging(从 bs_index_constituent_old baostock 全集 + 深证/中证 parquet) -# 3. merge_constituent staging -> constituent_unified (幂等 DROP+CREATE) -# 不碰 baostock online(全集已在 _old, migrate 读它); 不跑 C 新浪(baostock 权威) +# STEP0: parse_csindex_announce (csindex 调整公告 -> announce_union, 治幸存者偏差全集, 2026-07-23 新增) +# STEP1: index_const_hist_download --parts A B (深证 A adjust_cni + 中证 B snapshot) akshare parquet 刷新 +# STEP2: migrate_constituent 重建 staging (baostock _old 全集 + 深证 union + 中证 announce_union 全集) +# STEP3: merge_constituent staging -> constituent_unified (幂等 DROP+CREATE) +# 不碰 baostock online (全集已在 _old, migrate 读它); 不跑 C 新浪 (baostock 权威) +# STEP0 失败不阻塞 (csindex 网络抖动时 migrate 回退 snapshot, 治偏差降级但 pipeline 不死) $env:http_proxy = '' $env:https_proxy = '' $env:all_proxy = '' $env:PYTHONIOENCODING = 'utf-8' $env:INDEX_HIST_OUT_DIR = 'C:\sanguo_vnpy_v2\data\index_const_hist' +$env:CSINDEX_CACHE = 'C:\sanguo_vnpy_v2\data\csindex_raw' Set-Location C:\sanguo_vnpy_v2 $ts = Get-Date -Format 'yyyyMMdd_HHmmss' $logDir = 'C:\sanguo_vnpy_v2\data\migration_logs' @@ -16,11 +19,15 @@ $log = Join-Path $logDir "index_monthly_$ts.txt" $py = 'C:\Python310\python.exe' $dir = 'C:\sanguo_vnpy_v2\scripts\data_platform' -"[$ts] STEP1 index_const_hist_download --parts A B (深证 adjust + 中证 snapshot)" | Out-File $log -Encoding UTF8 +"[$ts] STEP0 parse_csindex_announce --only both (csindex 公告 -> announce_union, 治偏差全集)" | Out-File $log -Encoding UTF8 +& $py -X utf8 "$dir\parse_csindex_announce.py" --only both *>> $log +"[$ts] STEP0 done exit=$LASTEXITCODE (失败不阻塞, migrate 回退 snapshot)" | Out-File $log -Append -Encoding UTF8 + +"[$ts] STEP1 index_const_hist_download --parts A B (深证 adjust + 中证 snapshot)" | Out-File $log -Append -Encoding UTF8 & $py -X utf8 "$dir\index_const_hist_download.py" --parts A B *>> $log "[$ts] STEP1 done exit=$LASTEXITCODE" | Out-File $log -Append -Encoding UTF8 -"[$ts] STEP2 migrate_constituent (rebuild staging from _old + parquet)" | Out-File $log -Append -Encoding UTF8 +"[$ts] STEP2 migrate_constituent (rebuild staging from _old + parquet + announce_union)" | Out-File $log -Append -Encoding UTF8 & $py -X utf8 "$dir\migrate_constituent.py" *>> $log "[$ts] STEP2 done exit=$LASTEXITCODE" | Out-File $log -Append -Encoding UTF8 diff --git a/scripts/data_platform/register_akshare_schtasks.ps1 b/scripts/data_platform/register_akshare_schtasks.ps1 index 001d250..1cfd7de 100644 --- a/scripts/data_platform/register_akshare_schtasks.ps1 +++ b/scripts/data_platform/register_akshare_schtasks.ps1 @@ -1,15 +1,22 @@ # register_akshare_schtasks.ps1 — 注册 akshare 低频 5 schtask (spec §14.5 A+B+C) # akshare per-stock 全量慢+东财限流 → 夜间/周末/财报季; akshare_static_download 内置断路器(连30 failed exit)+0.8s sleep # 时间错开 bs-eod(18:05)/xt-eod(18:40) 之后, 无并发 -$ws = 'C:\sanguo_vnpy_v2\scripts\data_platform' -schtasks /create /tn sanguo-ak-eod /tr "powershell -ExecutionPolicy Bypass -File $ws\ak_eod_wrapper.ps1" /sc daily /st 19:00 /ru SYSTEM /rl HIGHEST /f -schtasks /create /tn sanguo-ak-events /tr "powershell -ExecutionPolicy Bypass -File $ws\ak_events_wrapper.ps1" /sc daily /st 19:30 /ru SYSTEM /rl HIGHEST /f -schtasks /create /tn sanguo-ak-stock /tr "powershell -ExecutionPolicy Bypass -File $ws\ak_stock_wrapper.ps1" /sc weekly /d SAT /st 03:00 /ru SYSTEM /rl HIGHEST /f -schtasks /create /tn sanguo-ak-quarter /tr "powershell -ExecutionPolicy Bypass -File $ws\ak_quarter_wrapper.ps1" /sc monthly /m APR,MAY,SEP,NOV /st 02:00 /ru SYSTEM /rl HIGHEST /f -schtasks /create /tn sanguo-index /tr "powershell -ExecutionPolicy Bypass -File $ws\index_monthly_wrapper.ps1" /sc monthly /d 16 /st 19:50 /ru SYSTEM /rl HIGHEST /f -Write-Output "=== VERIFY ===" +# +# 坑(2026-07-24 实证): PowerShell 调 `schtasks /tr "...$ws\xxx.ps1"` 时, 双引号内反斜杠路径被 +# 原生命令参数解析吃掉 → Arguments 丢前缀变 `\xxx.ps1` → schtask 触发成功但 powershell 找不到文件静默退出, 不写日志。 +# (Get-ScheduledTask 在中文 Win 报 0x80070057, 用 `schtasks /query /tn X /xml` 查 Arguments 才发现) +# 修复: 全路径硬编码 + `--%` stop-parsing 让 PowerShell 原样传参给 schtasks (反斜杠不丢)。 +schtasks --% /create /tn sanguo-ak-eod /tr "powershell -ExecutionPolicy Bypass -File C:\sanguo_vnpy_v2\scripts\data_platform\ak_eod_wrapper.ps1" /sc daily /st 19:00 /ru SYSTEM /rl HIGHEST /f +schtasks --% /create /tn sanguo-ak-events /tr "powershell -ExecutionPolicy Bypass -File C:\sanguo_vnpy_v2\scripts\data_platform\ak_events_wrapper.ps1" /sc daily /st 19:30 /ru SYSTEM /rl HIGHEST /f +schtasks --% /create /tn sanguo-ak-stock /tr "powershell -ExecutionPolicy Bypass -File C:\sanguo_vnpy_v2\scripts\data_platform\ak_stock_wrapper.ps1" /sc weekly /d SAT /st 03:00 /ru SYSTEM /rl HIGHEST /f +schtasks --% /create /tn sanguo-ak-quarter /tr "powershell -ExecutionPolicy Bypass -File C:\sanguo_vnpy_v2\scripts\data_platform\ak_quarter_wrapper.ps1" /sc monthly /m APR,MAY,SEP,NOV /st 02:00 /ru SYSTEM /rl HIGHEST /f +schtasks --% /create /tn sanguo-index /tr "powershell -ExecutionPolicy Bypass -File C:\sanguo_vnpy_v2\scripts\data_platform\index_monthly_wrapper.ps1" /sc monthly /d 16 /st 19:50 /ru SYSTEM /rl HIGHEST /f +Write-Output "=== VERIFY Arguments 路径(应全路径, 非 \xxx.ps1)===" foreach ($t in @('sanguo-ak-eod','sanguo-ak-events','sanguo-ak-stock','sanguo-ak-quarter','sanguo-index')) { - $q = schtasks /query /tn $t /fo list 2>&1 | Out-String - $status = ($q -split "`n" | Where-Object { $_ -match 'Status' } | Select-Object -First 1).Trim() - Write-Output (" {0,-22} {1}" -f $t, $status) + $xml = schtasks /query /tn $t /xml 2>&1 | Out-String + if ($xml -match 'File\s+([^<"]+)') { + Write-Output (" {0,-20} File {1}" -f $t, $matches[1]) + } else { + Write-Output (" {0,-20} (路径未找到)" -f $t) + } }