 claude_dev
|
21ebe7b4ef
|
feat(data_platform): static 三表事件驱动补洗上线(spec §20.11.7 采集层根治主件)——static_overlay_wash.py:dmsk NOTICE_DATE 雷达(近8分区×T-14窗,缺 static 文件=dmsk-only 覆盖段跳过)→命中股逐股 F10(emweb) 近12期 upsert 重写 per-stock 文件(按 REPORT_DATE 整期替换=重述天然胜,三日期列统一 YYYY-MM-DD,tmp+os.replace 原子写)→manifests/变更清单;pending 队列(F10 落库滞后 stale/网络失败次日重试,30天老化,成功出队);emweb 薄客户端=akshare stock_*_sheet_by_report_em 最小移植(ctype 正则解析缓存/5期分页/1s 节流/3 重试,NAS 容器无 akshare 零新依赖);rc 0/1/3=完赛/硬失败/墙钟停;run_dmsk.sh 增第三 lane(~07:2x 起跑 --until 15:00,rc 隔离不进聚合,免新 DSM 任务);push_vps_mirror_daily.sh 增独立 static manifest lane(文件级幂等,整单成功归档 pushed/,失败 WARN 不影响 corpus/dmsk);周日 sync_parquet 全量回冲=雷达窗内自愈+月度全量洗兜底;TDD 16 例(雷达三分支/upsert 重述胜+归一/原子写+manifest/stale/empty/pending 出队与老化/墙钟 rc3/客户端分页缓存重试),全套 442 绿;spec 调度拓扑落定+实施记录回填+runbook 镜像链补 static lane 行 [nas]
CI/CD / test (push) Successful in 26s
CI/CD / nas-deploy (push) Successful in 15s
CI/CD / nas-verify (push) Successful in 7s
|
2026-09-20 08:37:52 +08:00 |
|
|
claude_dev
|
367359d928
|
fix(data): akshare_static_download import 期副作用根治(基建 09-18 案)——模块体 L273 import 期调 setup_logging() 会 LOG_DIR.mkdir+开 FileHandler, POSIX 无 AK_STATIC_* env 时落到字面量 Windows 路径, 7 个测试文件 import 即中招, CI 工作区/dev 树长出 'C:\sanguo_vnpy_v2\data\static\' 目录(218 份垃圾日志,最新=今晨 run#1276 pytest 窗)+零字节 quant_trading.db;修=import 期只 getLogger(无 handler),setup_logging 延迟到 main() 首行,生产入口(VPS schtask 直跑)行为不变;回归测试 importlib.reload 断言 tmp cwd 零字面量残留(RED->GREEN);dev 树残留已清;420 绿(+1) [vps] [no-doc]
CI/CD / test (push) Successful in 29s
CI/CD / nas-deploy (push) Successful in 23s
CI/CD / nas-verify (push) Successful in 2s
|
2026-09-18 08:01:53 +08:00 |
|
|
claude_dev
|
3cce22a8b0
|
fix(data): gapsonly 七修——p101 服务墙三病(键集早停+日期下界+硬上限+无视state+段间独立,spec §20.10.5)——0911 半截补齐两次同点挂 [backfill TMC 069001002001 p101] 服务器繁忙,同点复现=确定性非限流:①早停「行数相等」判定在新块带翻页漂移dup(len虚高)/旧文件ps通道行数微差时恒False,连续5完整日永不触发一路裸翻;②EM单TMC段深翻页~100页/5万行确定性服务墙(TMC×DATE组合过滤又被服务端拒无法换锚,同墙宣告full模式对176万行大段永不可达,深史只能走per-stock);③gapsonly复用full模式state断点,挂死残留state(p100)毒化续跑立即再撞墙+单段早停跨段break跳过其余六段(各段夜间失败独立)。修复:键集判定(旧文件该段SCODE键集⊇新块去重键集=完整)+日期下界35自然日(≈24交易日≈深主板~55页远离墙,更深缺口归ps通道)+翻页硬上限90(到顶停段不raise,03:00无人值守不炸)+gapsonly无视state恒从p1(重翻由merge去重吸收,段末清state)+删跨段break七段全查。TDD+5例376绿;周六09-19 03:00首跑顺带merge洗净0915/0916/0917存量dup(直接洗文件被分类器拦,处置终态入spec) [vps]
CI/CD / test (push) Successful in 33s
CI/CD / nas-deploy (push) Successful in 21s
CI/CD / nas-verify (push) Successful in 2s
|
2026-09-17 22:04:00 +08:00 |
|
|
claude_dev
|
d94936cdd9
|
fix(data): margin TMC 块出口补翻页漂移 dedup(六修)——五修首跑验收发现 0915/0916 行数 6,890(预期~4,443),探针解剖实锤 EM sortColumns=DATE 同日内排序不稳定→翻页边界行漂移同一(DATE,SCODE)行进两页(069001002001 块内 883 行 dup 382,四段跨段 overlap=0 排除段重叠,0914 干净=概率性);by_date 通道 44d3ab9 起 merge dedup,五修①重构 recent_blocks 通道漏抄本防线;修=_fetch_tmc_block 出口 drop_duplicates(容错键列)+TDD 1 例 372 绿;存量 0915/0916 dup~2,445 行/日待部署后删文件+marker 自愈重拉(消费端两融余额会翻倍,修复前勿消费);spec §20.10.5 收录首跑验收四看全过+新病全案 [vps]
CI/CD / test (push) Successful in 32s
CI/CD / nas-deploy (push) Successful in 25s
CI/CD / nas-verify (push) Successful in 3s
|
2026-09-17 20:20:05 +08:00 |
|
|
claude_dev
|
75af7f2e42
|
fix(data): ak-events 五新域首两晚生产四病五修(用户全批,spec §20.9)——①margin 深段 TMC 代码级必然超时(原沪直查+4段整体包 30s,单段实测 26.3s→必挂,从未成功过):改逐层各自预算(沪 AK_TIMEOUT+每段 MARGIN_TMC_SEG_TIMEOUT=60s,seats 同款),任一层失败整体 None 不写残缺,09-11 自愈解堵 ②sw_industry 一级清单源乐咕乐股反爬 VPS(带UA=JS挑战页/裸403):31 行业改 SW_FIRST_LEVEL_STATIC 静态表(Mac 出口校准 2021-12 版),成份仍走申万宏源官网 API 不受影响 ③nb_hold 日增查当天必空(HKEX T+1 供应,同 margin_sse T日同构):基准日改 T-1+_hkex_hold_query strict 参数(日增空 skip 不抛/回补保持 raise),rc=1 主因拔除 ④financial_abstract 新浪源每晚 5.5k 连发撞限流(单发实测通=频率触发):fa 从 ak-eod 挪 ak-weekly 周六(量级-87%+错峰,schema 不变) ⑤margin_sse 窗口含 T 日每晚必 1 failed(EM 沪 DATE 视图 T+1):wrapper 改 T-2~T-1 全 settled;监控口径坑备案(类型级 failed 不抬出码,判健康须看各类型统计行);371 绿(+10 TDD) [vps]
CI/CD / test (push) Successful in 32s
CI/CD / nas-deploy (push) Successful in 22s
CI/CD / nas-verify (push) Successful in 4s
|
2026-09-16 20:00:23 +08:00 |
|
|
claude_dev
|
44d3ab9411
|
fix(ak): margin-ps 马拉松 OOM 根治——行dict分桶+每300股分段落盘+state只在flush后推进;根因实锤(Resource-Exhaustion-Detector 2004):by_date攒1行df到4272股/510万行≈510万pandas对象→32.6GB commit被强杀(09-13 16:17,页面文件太小+frps连带crashloop,两次8小时死=内存累积曲线非时间规律);同键DATE,SCODE merge+dedup幂等可重复吸收,crash最多重做一段;TDD 3例(段边界强杀/续跑落齐/keep-last)+spec §20.8 [vps]
CI/CD / test (push) Successful in 11s
CI/CD / nas-deploy (push) Successful in 35s
CI/CD / nas-verify (push) Successful in 14s
|
2026-09-14 07:48:06 +08:00 |
|
|
claude_dev
|
446fe4053c
|
fix(ak): margin 日采窗口改 T-3~T-1 仅工作日——数据实体抽查实锤 DATE 视图非沪锁而是结算时序(已结算日 count=4443 全市场 vs 未结算当晚沪-only 2000),原 T-2~T 当晚拉 T 写沪半截即标 marker=深段永久洞(09-11 实例沪2000/深0);spec §20.7 收录抽查实录+半场事故与验收锚点 [vps]
CI/CD / test (push) Successful in 34s
CI/CD / nas-deploy (push) Successful in 9s
CI/CD / nas-verify (push) Successful in 8s
|
2026-09-13 22:14:13 +08:00 |
|
|
claude_dev
|
844609825f
|
fix(ak): margin-ps 马拉松容错双修——①EM无数据双形态容忍(result=None+「返回数据为空」=合法空df,per-stock视图无两融标的股实测;繁忙仍fatal走重试);②ps循环逐股包call_ak_with_retry+重试耗尽跳过不杀马拉松(断点仍推进,周六gaps可补);TDD 3新测试全量绿 [vps]
CI/CD / test (push) Successful in 36s
CI/CD / nas-deploy (push) Successful in 33s
CI/CD / nas-verify (push) Successful in 13s
|
2026-09-12 10:31:48 +08:00 |
|
|
claude_dev
|
9585d01db7
|
fix(ak): margin-ps 通道 universe 本地化——_margin_ps_universe 读 valuation 域 marker 文件名(全A~5562码零网络),回落 akshare 列表;09-12 二次静默死亡教训(EM 抖动日 stock_info_a_code_name 在 VPS 挂起,马拉松启动不应依赖任何网络);TDD 2新测试全量绿 [vps]
CI/CD / test (push) Successful in 39s
CI/CD / nas-deploy (push) Failing after 12m34s
CI/CD / nas-verify (push) Has been skipped
|
2026-09-12 10:28:42 +08:00 |
|
|
claude_dev
|
6fd1fc6479
|
fix(ak): call_ak_with_timeout 超时后隐式 join 挂死线程根除——with 块 exit 隐式 shutdown(wait=True) 把瞬态挂起变永久卡死(09-12 margin-ps 启动卡死30分钟实锤,docstring声称不等join实际在等);改显式 try/finally+shutdown(wait=False);TDD 卡死复现测试(8.1s阻塞→1s内)全量353绿 [vps]
CI/CD / test (push) Successful in 6s
CI/CD / nas-deploy (push) Successful in 31s
CI/CD / nas-verify (push) Successful in 15s
|
2026-09-12 10:09:01 +08:00 |
|
|
claude_dev
|
8f48f43bc0
|
perf(ak): margin 全史回补双通道改判——TMC大查询实测5.7~20s/页全程70h不可行→新增--margin-backfill-ps per-stock快速通道(SCODE视图0.2s/页,全A≈10h,股级断点+19-24点避让ak-events防双进程合计破300/5min红线+内存按日分桶末尾merge);--margin-backfill降级退市史慢通道;周兜底改--margin-backfill-gaps gapsonly连续5完整日早停(分钟级);TDD 3新测试352全绿 [vps]
CI/CD / test (push) Successful in 33s
CI/CD / nas-deploy (push) Successful in 32s
CI/CD / nas-verify (push) Successful in 10s
|
2026-09-12 09:49:37 +08:00 |
|
|
claude_dev
|
d979c90fbe
|
feat(ak): 北向双域 nb_daily+nb_hold——HKEX官方渠道落地(spec §20.4二次探针终审);nb_daily=日统计js长表化(4市场×[汇总4指标+top10活跃],404写空+done);nb_hold=季度快照检查器(两步POST沪+?t=sz深,周六无效归一周五,td内嵌标题冒号取值,双日查询自证快照日,hash不变不写,文件名=季度末)+--nb-hold-backfill 12个月窗4季回补;实测0630快照4308行19s;EM参与者级孤段列为可选backlog;TDD 9新测试349全绿 [vps]
CI/CD / test (push) Successful in 35s
CI/CD / nas-deploy (push) Successful in 35s
CI/CD / nas-verify (push) Successful in 14s
|
2026-09-12 09:36:45 +08:00 |
|
|
claude_dev
|
55339fed09
|
feat(ak): 龙虎榜席位明细域 dragon_tiger_seats——当日榜单列表(RPT_DAILYBILLBOARD_DETAILSNEW)→逐股买卖top5席位(RPT_BILLBOARD_DAILYDETAILSBUY/SELL),SIDE列标注;实测680行/58股/190s(榜列表68行=同股多上榜原因去重58);逐股逐侧各自30s预算(整日期3分钟超AK_TIMEOUT,sw_industry同款分层);filter日期必须单引号;严格整体语义;增量域不回补;TDD 4新测试340全绿 [vps]
CI/CD / test (push) Successful in 35s
CI/CD / nas-deploy (push) Successful in 4s
CI/CD / nas-verify (push) Successful in 19s
|
2026-09-12 09:26:33 +08:00 |
|
|
claude_dev
|
b24f72da1f
|
feat(ak): 申万一级行业成份快照域 sw_industry——sw_index_first_info(31)+index_component_sw逐行业(成份+最新权重+计入日期,~5214行/夜37s实测);快照语义对齐ths_industry(§15.2);严格整体语义=任一行业失败整体failed次夜重跑;单unit 32连发内部0.8s逐发间隔防封;锚点600519∈801120食品饮料(权重39.8866/计入2021-12-13)实测入测试;TDD 6新测试336全绿 [vps]
CI/CD / test (push) Successful in 33s
CI/CD / nas-deploy (push) Successful in 0s
CI/CD / nas-verify (push) Successful in 17s
|
2026-09-12 09:08:41 +08:00 |
|
|
claude_dev
|
8f3ce48d1e
|
feat(ak): ak-weekly 周六档追加 margin 全史回补兜底+退役僵尸 northbound——深北段TMC最近两日块窗的旧缺口每周六整段翻页幂等补齐;northbound(EM个股北向持股2024-08已下线)每周2883空跑单位移除,北向转HKEX官方渠道(spec §20.4) [vps]
CI/CD / test (push) Successful in 32s
CI/CD / nas-deploy (push) Successful in 31s
CI/CD / nas-verify (push) Successful in 17s
|
2026-09-12 08:59:55 +08:00 |
|
|
claude_dev
|
f80ca2f09d
|
feat(ak): ak_events wrapper 挂载 margin 域 T-2~T 三天窗——深市TMC段T+1供应+最近两日块窗,三天窗留自愈;更旧缺口每周六--margin-backfill幂等兜底 [vps]
CI/CD / test (push) Successful in 30s
CI/CD / nas-deploy (push) Successful in 31s
CI/CD / nas-verify (push) Successful in 14s
|
2026-09-12 08:57:35 +08:00 |
|
|
claude_dev
|
f7fb7b7530
|
feat(ak): 两融个股明细域 margin(EM RPTA_WEB_RZRQ_GGMX)+--margin-backfill 全史回补通道——融合spec §20四域扩展首域;实测口径:DATE视图服务端universe锁沪市2000只,深/北走TMC段码(精确匹配,股票12位/ETF 9位,七段全集~570万行);日增量=沪DATE直查+深北TMC倒序最近两日块(深段T+1供应);回补=七段倒序整段翻页流式按日切块+断点state+与已有文件合并去重幂等;TDD 14新测试330全绿 [vps]
CI/CD / test (push) Successful in 33s
CI/CD / nas-deploy (push) Successful in 34s
CI/CD / nas-verify (push) Successful in 19s
|
2026-09-12 08:56:56 +08:00 |
|
|
claude_dev
|
e478aba80f
|
perf(xt): get_market_data_ex 显式传六列 field_list——桥fallback快路径(FormulaServer 0.025s vs 空列RPC 0.1-5.8s/次,今晚21:00全量夜班从~3.7h回分钟级);五样本含2北交空列vs六列等值实测EQUIV,TDD断言调用一等参 [vps]
CI/CD / test (push) Successful in 27s
CI/CD / nas-deploy (push) Successful in 2s
CI/CD / nas-verify (push) Successful in 21s
|
2026-09-11 13:40:01 +08:00 |
|
|
claude_dev
|
e756fcc0b0
|
fix(bs): 名单规模sanity守卫MIN_STOCK_LIST=4000——09-10实录query_stock_basic截断返2296静默半写exit=0,守卫转exit=1次夜LOOKBACK自愈 [vps]
Co-Authored-By: Claude Code <noreply@anthropic.com>
|
2026-09-10 22:52:35 +08:00 |
|
|
claude_dev
|
4bd856ab8d
|
fix(data): xt_eod数据腿切大QMT桥——wrapper注入xtquant_bridge PYTHONPATH+redis凭证env,替代已死的mini栈58610 [vps]
- 背景: miniQMT 58610 IPythonApiServer 自 09-09 误杀后不再自启(isAutoBind实验无效),
xt_eod连续断粮;桥shim(xtquant_compat)全量覆盖xt_eod所需函数
- 坑: PS5.1对无BOM文件按ANSI/GBK解析——wrapper含中文注释时多字节字符错位可吃掉
后续语句行(PYTHONPATH行被注释->仍走site-packages原版xtquant),故wrapper强制ASCII
- 实测: universe 2264+39/下载ok/入库ok=400 empty=0 fail=0 rows=9200进行中
|
2026-09-10 19:06:35 +08:00 |
|
|
claude_dev
|
29e3ecbe2d
|
feat(corpus): 巨潮504当日尾重试——主pass未done单元隔10min重走一轮,墙钟无余量跳过次日自愈兜底;spec§19.11.1 backlog执行记录 [nas]
CI/CD / test (push) Failing after 14m37s
CI/CD / nas-deploy (push) Has been skipped
CI/CD / nas-verify (push) Has been skipped
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-09 12:11:15 +08:00 |
|
|
claude_dev
|
f78e8f1349
|
feat(dmsk): DOMAINS扩forecast业绩预告/express业绩快报两域——探针钉边界2003/2005年报起,per-domain起始年+4键/3键(ID含NOTICE_DATE披露日),verify五域键唯一 [nas]
CI/CD / test (push) Successful in 26s
CI/CD / nas-deploy (push) Successful in 7s
CI/CD / nas-verify (push) Successful in 17s
|
2026-09-08 21:16:34 +08:00 |
|
|
claude_dev
|
150465a90e
|
feat(bs): --backfill-valuation估值历史窗回补入口——只拉9列不动dbbardata,断点buffer先落盘done后写,终局merge原子写回 [vps]
CI/CD / test (push) Failing after 13m26s
CI/CD / nas-deploy (push) Has been skipped
CI/CD / nas-verify (push) Has been skipped
|
2026-09-08 20:57:03 +08:00 |
|
|
claude_dev
|
040a39e9c1
|
feat(ak): gdhs一次性补史入口--gdhs-start——滚动3年窗外扩,源头边界2013Q1实测钉死 [vps]
|
2026-09-08 13:45:29 +08:00 |
|
|
claude_dev
|
f4404c17b5
|
docs(data): dmsk W4'完成终态入spec§19.8+verify修NOTICE显示(dropna防NaN串max) [nas]
CI/CD / test (push) Failing after 15m0s
CI/CD / nas-deploy (push) Has been skipped
CI/CD / nas-verify (push) Has been skipped
backfill 210unit/1771发/零失败,verify键唯一PASS,BJ 1万+行/域,净新增383只镜像外代码;边界=2009前退市双源皆无(源头缺失)
|
2026-09-08 07:25:49 +08:00 |
|
|
claude_dev
|
d71ee55e6f
|
fix(data): dmsk 翻页参数 pageNumber——东财静默忽略 pageNum=12页全回第一页(冒烟+500/6000实锤) [nas]
CI/CD / test (push) Successful in 30s
CI/CD / nas-deploy (push) Successful in 3s
CI/CD / nas-verify (push) Successful in 9s
|
2026-09-08 00:35:48 +08:00 |
|
|
claude_dev
|
236bd30eb8
|
feat(data): 财务136 W3' dmsk双lane落地+spec§19.8实施记录 [nas]
CI/CD / test (push) Successful in 32s
CI/CD / nas-deploy (push) Successful in 0s
CI/CD / nas-verify (push) Successful in 11s
- fundamentals_dmsk_download.py 单脚本(2009→今backfill+近3期daily NOTIC_DATE>=T-7),corpus骨架六修正全带入
- marker修正: daily按日折叠(§19.4原文「一次永逸」=day-2同款坑,52756e3教训);backfill键=域+期(域缺席连带跳过,首测抓出)
- 次新期金丝雀防「filter失效vs空期」不分导致的整段静默空跑
- run_dmsk_standalone.sh(DSM 07:10 sanguo-fundamentals-daily,一次性容器同corpus款)+verify_dmsk.py(键唯一硬门)
- 21测全mock绿,全量295绿;W1'镜像对账全绿+vb2026 H1缺口入档
|
2026-09-08 00:30:35 +08:00 |
|
|
claude_dev
|
52756e35ab
|
fix(data): daily增量marker按日折叠——首日全量done后次日整段跳过增量的day-2致命坑 [nas]
CI/CD / test (push) Failing after 12m23s
CI/CD / nas-deploy (push) Has been skipped
CI/CD / nas-verify (push) Has been skipped
09-07 首跑夜复查发现: daily ann/news marker 无日期维度,首跑日已写满
6168+6168 → 次日 06:00 首班会整段跳过增量采集,且 backfill 仍在跑会掩盖
症状。修=unit 名前挂 dt=YYYY-MM-DD/ 子目录(_dunit)+日初清旧日目录与
历史平铺 marker(_prune_daily_increment_markers);fulltext/pdf 段键=内容
id(不可变一次永逸)保持平铺不走折叠。契约测试: 次日重新拉窗承接新公告/
当日断点续传不变/旧日目录日初清。
|
2026-09-07 18:53:41 +08:00 |
|
|
claude_dev
|
a889b2ff0a
|
fix(data): corpus _filter_new 批内去重——巨潮单响应同id两条全放行成重键行
CI/CD / test (push) Failing after 5s
CI/CD / nas-deploy (push) Has been skipped
CI/CD / nas-verify (push) Has been skipped
09-07 首跑实锤 ann_meta 3行重键(verify_corpus FAIL): 1225400726/1225487003
整行完全相同×2 + 1225009396 ann_time差1秒(目录挂两条)。根因=has_any只对账
已有键,同一批内重复在统一add前全数放行。修=批内seen截留(首个出现者胜),
news_meta复合键同路径受益。事故形态钉进回归测试。 [nas]
|
2026-09-07 18:37:56 +08:00 |
|
|
claude_dev
|
d426619a96
|
fix(data): stats 改域内首末请求间隔(lane 总时长跨段失真) [nas]
CI/CD / test (push) Successful in 6s
CI/CD / nas-deploy (push) Successful in 7s
CI/CD / nas-verify (push) Successful in 9s
09-07 首跑 daily 收官 stats 乌龙: cninfo n=3 avg=3621s(被全文段时长摊薄)/
eastmoney 2.26s(真实1.25s); 改 DomainClient 首末请求时间戳 avg=(t_last-t_first)/(n-1),
后续 run 起直读真节奏
|
2026-09-07 12:27:46 +08:00 |
|
|
claude_dev
|
3b9b1e6634
|
fix(data): 账本启动对账自愈——分区有/账本无缺口永久闭合 [nas]
CI/CD / test (push) Successful in 3s
CI/CD / nas-deploy (push) Successful in 8s
CI/CD / nas-verify (push) Successful in 10s
09-07 巡检 verify FAIL 实锤: 旧代码部分段(news/fulltext/pdf)无 flush 计数器,
docker stop 的腿把内存账本丢掉(ann 差4.5k/news 差9.9w;分区数据完好 dups=0,
近窗索引兜底着去重但账本缺口30天后成理论重拉窗)。修=两 lane 启动时
_reconcile_ledgers: 近30日分区 key 对账补进账本并落盘,幂等;
新代码批量提交已让缺口收敛到单commit周期,对账再兜历史+未来任何崩溃窗
测试 55(新增对账自愈幂等钉);全套 272 绿
|
2026-09-07 07:30:55 +08:00 |
|
|
claude_dev
|
efcdc8d8e0
|
fix(data): 分区改多 part 追加写+批量提交,根治 day-1 O(n²) 劣化 [nas]
CI/CD / test (push) Successful in 25s
CI/CD / nas-deploy (push) Successful in 10s
CI/CD / nas-verify (push) Successful in 15s
09-07 首跑实锤: append_parquet 每 unit 整读整写当日唯一 part-0, news 段
13万+行后 5.3s/股(应1.15s)且随行数恶化,10:30 墙钟必跑不完且逐日更糟。
修(两层):
- DayBuffer: 当日新行内存缓冲→flush 落新 part-N.parquet(只写新行,
旧数据永不重写);id 去重职责收敛到调用方(账本+近窗双层,原就存在)
- Ctx 批量提交: unit_done 延后 marker 入批,每 FLUSH_EVERY unit 一次
commit=数据→账本→marker 依次落盘(崩溃=整批重拉,绝不产生
marker-done-但数据缺的洞);run_lane 段首清缓冲/finally 兜底 commit
- ③全文回补 known 检查补近窗兜底(账本 flush 崩溃窗防重)
- readers 全改 part-* glob(RecentIndex/_iter_partitions/verify_corpus)
- 测试 54:新增 多part语义/flush崩溃一致性/批量提交次序 三钉
|
2026-09-07 07:12:48 +08:00 |
|
|
claude_dev
|
308d4931e3
|
feat(data): T6 验收器 verify_corpus.py+每域请求统计日志 [nas]
CI/CD / test (push) Failing after 26s
CI/CD / nas-deploy (push) Has been skipped
CI/CD / nas-verify (push) Has been skipped
- corpus_download 收尾补 stats 行(每域 n_requests/avg_interval/[429]/[cooled])
=§18.5「日志限速合规」验收依据(平均间隔≥设定值可直读)
- verify_corpus.py 只读验收器(零写入零网络,容器内跑): 三域分区/行数/键重复/
账本一致性(ledger<uniq=FAIL)/PDF 索引↔文件/marker/depth/容量/最近日志尾部;
NAS 真数据首验 PASS(三域 dups=0, ledger=uniq, 索引=文件, 644G)
- 晨检一条命令复用(首周人工盯跑工具)
|
2026-09-07 01:31:00 +08:00 |
|
|
claude_dev
|
594e2efbf5
|
fix(data): fulltext 段补 --limit 预算门(三跑冒烟7分钟马拉松) [nas]
CI/CD / test (push) Successful in 23s
CI/CD / nas-deploy (push) Successful in 8s
CI/CD / nas-verify (push) Successful in 8s
_run_fulltext 漏门:new_articles 跨股去重后仍可远超 N(冒烟 2 股×100条
→~150 art @2.3s);真实运行有墙钟兜底故非事故,冒烟语义补齐+回归钉死
|
2026-09-06 23:56:05 +08:00 |
|
|
claude_dev
|
2fa678671c
|
feat(data): backfill ①年序翻转新→旧(近年优先供factor) [nas]
CI/CD / test (push) Successful in 28s
CI/CD / nas-deploy (push) Successful in 2s
CI/CD / nas-verify (push) Successful in 6s
spec§18.3 字面「2000→今」与其自设验收「回补~1周后即可供factor事件回测」
矛盾(升序前8天只有2000-2002老数据,近年最后才到);翻转为年DESC同PDF④
哲学: day1 即产当年数据,~1周完成近3-4年=事件因子可用,全量完结不受影响
(§18.5「600519 年报2001起逐年可查」照过)。空年照旧合法done零成本
含 cosmetic: reset_stage 归位 docstring 之后(前次批量插入错位)
|
2026-09-06 23:47:57 +08:00 |
|
|
claude_dev
|
ae6e2d92d0
|
fix(data): corpus 全 fetcher 补 UA+jsonp 解析防御 [nas]
CI/CD / test (push) Successful in 20s
CI/CD / nas-deploy (push) Successful in 5s
CI/CD / nas-verify (push) Successful in 7s
NAS 冒烟二跑实锤: search-api 对无 UA 请求返 200+空 body(金丝雀当时带 UA 故
未暴露)→s.index('(') ValueError 裸崩。修: 统一 UA 常量六处 fetcher 全带;
jsonp 非 cb(...) 形态 → TransportError(带响应片段,计入域名连续错走冷却
而非裸崩);容器内 A/B 实测 no-UA='' vs withUA=正常610条
|
2026-09-06 23:44:48 +08:00 |
|
|
claude_dev
|
89da5f71e0
|
fix(data): corpus --limit 改每段独立预算+pdf五类循环补门控 [nas]
CI/CD / test (push) Successful in 24s
CI/CD / nas-deploy (push) Successful in 0s
CI/CD / nas-verify (push) Successful in 8s
NAS 真实冒烟(--limit 2)假死 12min 事故根因(x2):
①预算共享: ann 段吃光 limit 后 news/fulltext 即刻 break(假死其一);
②pdf 五类查询循环完全无门控无计数 → 全池 6168 股 @1.3s 扫描 ~2.2h,
周日窗口 0 公告 0 日志=假死其二(docker stop 收尸, log 停在 lane start)
修: Ctx.stage_units+reset_stage()+budget_exhausted() 八段入口重置;
五类循环计数即耗预算;回归测试钉死(fivecat 调用数=预算数,非全池)
连带修正 smoke 语义=文档承诺的「每段最多 N unit」;52+269 全绿
|
2026-09-06 23:40:53 +08:00 |
|
|
claude_dev
|
1c3a61eb12
|
feat(data): T5 backfill ③全文 ④PDF+容量闸门 [nas]
CI/CD / test (push) Successful in 28s
CI/CD / nas-deploy (push) Successful in 2s
CI/CD / nas-verify (push) Successful in 6s
spec §18.7 T5(backfill lane 补完,四段齐):
- ③新闻全文回补: 扫 news_meta 库存→未入账 art 补正文;账本即进度(免
260万 marker 文件);404=墓碑行 content_text=None 入账不重试;文章域独立
限速 2s(RATE['article'],spec @2/s 慢车道)
- ④PDF六类回补(新→旧): ann_meta 库存标题谓词筛六类→ann_time desc;
文件在=跳过(文件即进度);404 墓碑 state/pdf_missing.json;索引复用
_append_pdf_index(daily/backfill 同管道)
- 容量闸门: 段首+每50GB 自查 _disk_free_gb,<200G 暂停 PDF 段+落
capacity_paused.json 告警(其余段不受影响,空间回升自动续)
- 测试51个(root fixture 默认 _disk_free_gb=645:曾吃 Mac 真实磁盘
8G 余量误触发闸门);全套 268 绿
|
2026-09-06 23:24:29 +08:00 |
|
|
claude_dev
|
10c486c64d
|
feat(data): T4 backfill lane ①ann_meta ②news_meta 落地 [nas]
CI/CD / test (push) Successful in 23s
CI/CD / nas-deploy (push) Successful in 26s
CI/CD / nas-verify (push) Successful in 6s
spec §18.7 T4(unit marker 断点+--limit 冒烟不落marker+墙钟rc3 全链复用):
- ①ann_meta 回补: unit=(stock,year) 2000→今年逐年按池序, 空年=合法done
(探测优先);今年窗口右端=today 相对;~15万 unit @1s≈3-6 跑日
- ②news_meta 回补: np-listapi unit=(stock,page) 翻到空页止;空页+page≥200
双止条件都落 news_end 哨兵(到头股零重扫);mkt前缀 6→1/其余→0
- 共用 _absorb_new(daily/backfill 同一条 id 去重+追加管道)
- 测试 46 个;修 wiring 网络泄漏:_wire_backfill_ann 曾漏锁
fetch_news_backfill_page 致 ann 测试真打 np-listapi(40s/真出网,
零网络铁律违反)——补锁后套件 159s→1.3s;全套 263 绿
|
2026-09-06 23:19:50 +08:00 |
|
|
claude_dev
|
e22f0e0aa8
|
feat(data): T2 corpus_download.py daily lane+断点+锁+36测全绿 [nas]
CI/CD / test (push) Successful in 26s
CI/CD / nas-deploy (push) Failing after 14m50s
CI/CD / nas-verify (push) Has been skipped
spec §18.7 T2(corpus 栈唯一新脚本,单脚本红线):
- daily 四段: 公告增量(巨潮 T-1~T 两天窗自愈)/新闻增量(search-api 带摘要,
(art_code,stock_code) 复合键=同稿多股 meta 多行)/新闻全文(昨日新 art_code,
art_code 全局一条)/当日 PDF(五类服务端码 category 码+业绩快报标题谓词——
实测 yjkb 码不存在,无效码被服务端静默忽略返回全量,故走客户端标题过滤)
- 断点: unit marker tmp+rename;--until 墙钟=完成当前 unit 后 rc=3;--limit 冒烟
绝不落 marker(5m 09-17 教训)
- 单实例锁: fcntl.flock(state/corpus.lock)——跨容器语义(bind mount 同 inode,
内核持有/进程死自动释放);5m 的 pid 活性检测在 docker run --rm 各自独立
PID namespace 下失效,corpus 换 flock(让路契约同款,锁忙 rc=3)
- id 去重双层: IdLedger 全局账本(state/ids_*.parquet int64 hash,满足 §18.5
announcement_id 全局无重)+RecentIndex 近30日分区索引(兜账本flush周期崩溃窗)
- 失败语义: 传输错 unit 不标done;真空=合法done;4xx确定性跳过不烧重试;
域名连续错×10或429冷却它域继续;rc 0完成/1致命/2限流/3墙钟或锁
- 股票池: 巨潮 szse_stock.json(实测含退市000003+北交920001,6247条)A前缀过滤,
非NAS副本库——orgId 为按股查询硬前提且退市覆盖更全(spec 偏差T6回写)
- 测试 36 个全 mock 零网络零写死日期;全套 253 绿
|
2026-09-06 23:00:42 +08:00 |
|
|
claude_dev
|
694a0d2238
|
fix(data): ak-weekly 断路器饿死修复=_safe_northbound_em+类型调序 [vps]——09-05 03:00 首跑 rc=2 根因:akshare stock_hsgt_individual_em 对无北向数据标的(~1117/2883,301xxx 等)在 data_json[result][pages] 抛 TypeError NoneType not subscriptable→failed→30 连败断路器→跳过后续类型,share_capital/top_holders/gdhs 全饿死(gdhs 目录至今不存在);端点健康(600519 返 1683 行,09-06 晨复测);修=①_safe_northbound_em 子串匹配 not subscriptable 返空 df 带 9 列 schema→走无北向持仓 empty 写空+done 既定语义不耗重试/断路器(同 _safe_top_10_em 手法)②weekly wrapper 调序 gdhs,share_capital,top_holders,northbound 最大最险者殿后防再连累③顺手 BOM;TDD 2 新测先红后绿+217 全绿;双脚本已 scp 生效位+Parse/py_compile 预检过;gdhs 手动补跑另行 [vps]
CI/CD / test (push) Successful in 23s
CI/CD / nas-deploy (push) Successful in 7s
CI/CD / nas-verify (push) Successful in 13s
|
2026-09-06 08:29:58 +08:00 |
|
|
claude_dev
|
c728f2368c
|
feat(data): hot_rank 挂载+wrapper 时间守卫 [vps]——墙测量5轮收官用户拍板方案A:emappdata 夜间开窗(22:28 实测 OK,晨午4轮全 FAIL),19:30 档失败零连带(单 unit 失败不触断路器 rc 仍 0,不写不标次夜自愈),首夜起 gap_check 即免费持续测量;三处同步=wrapper --types+PANEL_TYPES+EVENT_PANEL_TYPES(15类),零文件=零告警首日成功即成基线;时间守卫=param -Force+非 19:00-23:59 拒绝执行留痕 guard.log(治盘前手动误跑两犯:09-02 08:08/09-04 08:00,盘前拉当日=昨日态错标+marker 锁死当晚真数据);TDD 2 测翻新(白名单拒→放行+注册断言)先红后绿+806 全绿;VPS Parse 预检过+双脚本已 scp 生效位+wall_probe.py 已删 [vps]
CI/CD / test (push) Successful in 0s
CI/CD / nas-deploy (push) Successful in 5m3s
CI/CD / nas-verify (push) Successful in 11s
|
2026-09-04 18:46:46 +08:00 |
|
|
claude_dev
|
e2902cdf1f
|
fix(data): ak_events_wrapper 修启动级语法错+加BOM——09-03 19:30首跑即死(rc=1零日志=解析期死亡):真根因=line14 (Get-Date).AddDays(-1) -Format yyyyMMdd 为非法PS语法(表达式后不能接命令参数),无BOM时GBK误读把报错行号搅到13误导视线;修=.ToString('yyyyMMdd')+前置UTF-8 BOM根治PS5.1对无BOM中文文件的GBK误读隐患;三向hash已证本地=git=VPS无部署损坏;教训=ps1改动部署前必须Parser预检(VPS已验PARSE_OK);今晚手动补跑13类+margin_sse 0902(T-1窗首验) [vps]
CI/CD / test (push) Successful in 14s
CI/CD / nas-deploy (push) Successful in 3m1s
CI/CD / nas-verify (push) Successful in 19s
|
2026-09-04 07:59:19 +08:00 |
|
|
claude_dev
|
564160db11
|
fix(data): margin_sse 拆 T-1~T 两天窗——巡查实锤 SSE 融资融券 T+1 供应,当晚拉当天结构性永败 43 天(用户批准结构修) [vps]
CI/CD / test (push) Successful in 19s
CI/CD / nas-deploy (push) Successful in 5s
CI/CD / nas-verify (push) Successful in 6s
07-22 起慢性失血实锤: margin_sse 最后非空文件 20260721,之后 30 个交易日
全被旧码标成 空文件+done 锁死(每晚 19:30 拉当天,SSE 当日数据未发布→
空响应→akshare Length mismatch 崩→旧码转 empty 落盘+marker)。
已处置(同批): --force 回补 20260722~20260901 全区间 30/30 ok 59865 行;
本 commit=结构根治: wrapper 把 margin_sse 从 13 类当日批拆出,单独
--start T-1 --end T 两天窗——每晚稳定写昨日(数据已发布),当天失败不写
不标(新语义),次晚同窗自动重试补上,单晚漏跑自愈;两段 exit code 合并
任一非零即非零。0902 今晨已过 19:30,明晚首验(写 0902+0903)。
生效位=VPS scripts(手动 scp,CI 不覆盖 VPS)。
|
2026-09-02 22:47:21 +08:00 |
|
|
claude_dev
|
46a0569b9a
|
feat(data): A档全量落地——热度三件+股东户数收尾(用户批「按此方式实施,nas验证后推vps」);①xueqiu_hot雪球5633行+sina_sector新浪行业49行对账源进ak-events 19:30(同款快照语义)②hot_rank东财人气榜fetcher/注册先行,wrapper挂载留待恢复窗测量拍板(墙未定论不带不确定单元上线防缺日告警噪音)③gdhs股东户数进ak-weekly周六窗=per-period新形状:build_gdhs_units只枚举距今>90天季度末(披露守卫,防未披露期拉空走真空+done语义被marker锁死永不重拉的09-02设计坑),单次全市场5342行,marker去重无新期零网络;vintage缺日名单+2快照族(hot_rank留位);+6测试(注册/None/df契约/gdhs守卫两形态/缺日名单)215全绿 [vps]
CI/CD / test (push) Successful in 2s
CI/CD / nas-deploy (push) Successful in 9s
CI/CD / nas-verify (push) Successful in 16s
|
2026-09-02 09:16:14 +08:00 |
|
|
claude_dev
|
00cfd45d1a
|
feat(data): A档第二批接入——新浪资金流×2+同花顺行业/概念榜进ak-events 19:30(用户批方案A);东财push2墙死后免费无墙替代源双机实测秒级零限流(新浪行业资金流90行带净额/领涨股+概念387行,同花顺行业90行×12列+概念375行);快照族语义显式化:端点无date参数无历史回补,19:30采集=收盘终态,节假日拉到上交易日快照接受重复由消费方按交易日历过滤;vintage缺日检查扩全panel族并分档——可回补族(zt_pool×3)近窗洞给回补指引,快照族洞即永久告警查当晚日志无回补指引,旧洞不刷告警防疲劳;+5测试(注册/None/df契约×2+快照族永久洞语义)29绿 [vps]
CI/CD / test (push) Successful in 2s
CI/CD / nas-deploy (push) Successful in 8s
CI/CD / nas-verify (push) Successful in 12s
|
2026-09-02 08:57:39 +08:00 |
|
|
claude_dev
|
020d7cda4b
|
feat(data): vintage自检扩涨停池缺日防线——策略session共识「断档=不可逆损失须尽快」;per-date家族语义=每工作日(含节假日,空数据也写空文件)应有1个parquet,文件缺=当晚schtask漏跑=洞;洞≤42日历日(≈30交易日回补窗)→告警点名缺日+手动--start/--end回补指引,滑出窗=永久缺只入json不刷告警(防永久exit3疲劳);检查窗=首文件日→昨天(当日19:30才落盘不算洞);panel块进vintage_status.json契约(additive零破坏);+4测试(近窗洞告警/连续无洞/永久洞静默入json/目录缺不崩)13绿 [vps]
CI/CD / test (push) Successful in 22s
CI/CD / nas-deploy (push) Successful in 8s
CI/CD / nas-verify (push) Successful in 15s
|
2026-09-02 08:17:02 +08:00 |
|
|
claude_dev
|
db789d6e9c
|
feat(data): akshare事件类扩涨停池三件套——daily_stock_analysis调研落地第一步;双机实测(Mac+VPS生产IP)分流:push2ex集群(zt_pool/zbgc/dtgc)连打多轮全稳可用,push2 clist集群(行业/概念榜/板块资金流/主力资金)与push2his(cyq筹码/个股资金流)每IP长窗暗配额(首发200第二发起全reset,30s/镜像轮换/换头均不过)判敌对弃采;zt_pool 83只/zbgc 6只/dtgc 2只真实行数实证,dtgc空数据=当日真无跌停走既有写空+done语义;三类型进PER_DATE_TYPES家族(ak-events 19:30同窗+3单次调用<1min),零新schtask零新目录零新依赖;+4测试(注册/None契约/df契约/真空df落盘) [vps]
CI/CD / test (push) Successful in 4s
CI/CD / nas-deploy (push) Successful in 13s
CI/CD / nas-verify (push) Successful in 14s
|
2026-09-02 08:07:31 +08:00 |
|
|
claude_dev
|
e012342d0d
|
fix(data): register脚本对齐VPS现场命名+ak-monthly改XML注册——09-01晚实测暴露脚本漂移:08-20 schtask重组已改名ak-daily/ak-monthly/ak-weekly(wrapper不变)但本脚本没跟上,重跑旧脚本=拿过期地图改现场:①复活与ak-weekly完全重复的ak-stock(同wrapper同周六03:00会双开并发,已现场删除)②ak-quarter行「/m月份列表+/d SAT」在zh-CN schtasks报「/D选项无效」根本没注册上;修=①任务名对齐现场六件套(daily19:00/events19:30/weekly周六03:00/monthly首周六02:00/vintage09:00/index16号19:50)②monthly改嵌XML(ScheduleByMonthDayOfWeek Weeks=1 Saturday,Out-File Unicode+Schtasks /create /xml,绕开zh-CN星期名限制)③头注释记命名史+事故链防重演;现场已同款修完(ak-monthly next=9/5首周六,顺带周末全规模预演新代码) [vps]
CI/CD / test (push) Successful in 21s
CI/CD / nas-deploy (push) Successful in 8s
CI/CD / nas-verify (push) Successful in 20s
|
2026-09-01 20:49:35 +08:00 |
|
|
claude_dev
|
8bcca3146b
|
fix(data): vintage自检补覆盖率撕裂告警——VPS真数实测暴露v1盲区:cashflow洗库中段8%vs两表100%但max_period同为Q2'26,仅看max报了OK漏掉撕裂;补=三报表coverage最大-最小>50pp告警(三报表同批披露稳态应同步,中段/某表大面积失败即触发);+1真形态测试 [vps]
CI/CD / test (push) Successful in 7s
CI/CD / nas-deploy (push) Successful in 6s
CI/CD / nas-verify (push) Successful in 10s
|
2026-09-01 19:17:03 +08:00 |
|