feat(data_platform): corpus flash_meta 全市场快讯域上线(spec §20.12.3 补充类采纳)——金十 flash_newest.js 快照+东财 kuaixun 翻页(T-1 边界停界,20页硬顶)两源,与 news_meta 个股新闻语义并列不 join;flash_id=源前缀+源站稳定id跨源命名空间隔离,当日marker幂等,源级隔离单源失败不炸另一源;run_corpus.sh 第三lane rc隔离不进聚合(两主lane全让路也不漏当天快讯);镜像车 corpus 四子域扩五(新域指纹全新首跑自动全量推);corpus_reader.get_flash 空df带schema;TDD +13例全套465绿;Mac真网络冒烟=金十50+东财884新,21s完赛,dt跨度T-1边界实证,幂等复跑零请求;首班锚点=次日06:0x flash lane exit=0行 [nas]
This commit is contained in:
@@ -9,6 +9,9 @@ schema 契约(2026-09-17 NAS 实测落档,§20.11.3 数据契约③):
|
||||
sec_code(裸 6 位)/sec_name/title/short_title/ann_time/ann_type(|| 分隔多码)/
|
||||
ann_type_name/column_id/adjunct_url/adjunct_size/first_seen_date
|
||||
- ann_pdf: <corpus_root>/ann_pdf/<year>/<announcement_id>.pdf
|
||||
- flash_meta(§20.12.3, 2026-09-22): flash_id(jin10:/em_kuaixun: 前缀+源站稳定 id)/
|
||||
source/publish_time/title/content/important/channel/first_seen_date——
|
||||
全市场宏观快讯流(金十+东财kuaixun), 与 news_meta 个股新闻语义并列不 join
|
||||
|
||||
稳定 id(数据契约②配方):art_code/announcement_id 为源站稳定 id;爬虫层
|
||||
跨天重复行会在数据里原样出现——核内零去重,消费方按 id 自行幂等。
|
||||
@@ -48,6 +51,21 @@ def ann_pdf_dir(year):
|
||||
return os.path.join(pr.corpus_root(), "ann_pdf", str(year))
|
||||
|
||||
|
||||
FLASH_COLUMNS = ["flash_id", "source", "publish_time", "title", "content",
|
||||
"important", "channel", "first_seen_date"]
|
||||
|
||||
|
||||
def get_flash(start=None, end=None, sources=None):
|
||||
"""全市场快讯流(§20.12.3,dt 分区直读)。sources={"jin10","em_kuaixun"}
|
||||
过滤;空树返带 schema 的空 df(容错分级契约)。"""
|
||||
df = pr.read_partitions(pr.corpus_root(), "flash_meta", start, end)
|
||||
if df.empty:
|
||||
return df.reindex(columns=FLASH_COLUMNS)
|
||||
if sources:
|
||||
df = df[df["source"].isin(set(sources))]
|
||||
return df
|
||||
|
||||
|
||||
def get_watermark():
|
||||
"""镜像水位线(None=尚无成功推送记录)。"""
|
||||
return pr.read_watermark(pr.corpus_root())
|
||||
|
||||
Reference in New Issue
Block a user