claude_dev
421b3ada3b
- 连字符扩展：multi-agent → multi agent 提升搜索召回率 - 中文同义词支持：添加中英文混合查询扩展 - Snippet 高亮预留：为 FTS5 snippet 功能预留接口 - YAML 解析增强：支持 HTML 实体解码 - 标签关联维护：自动维护 wiki_page_tags 关联表 - 缓存失效优化：支持智能前缀匹配 - 设计文档更新：汇总近期改动 (v1.3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1091 lines
32 KiB
Markdown
1091 lines
32 KiB
Markdown
# sanguo_llmwiki 设计文档
|
||
|
||
## 1. 架构设计
|
||
|
||
### 1.1 系统架构
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Claude Code │
|
||
│ │
|
||
│ ┌─────────────────────┐ ┌─────────────────────────┐ │
|
||
│ │ Wiki Skills (11) │ │ Wiki MCP Server │ │
|
||
│ │ (一次性操作) │ │ (Python 服务) │ │
|
||
│ └─────────────────────┘ └─────────────────────────┘ │
|
||
│ │ │
|
||
└───────────────────────────────────────┼─────────────────────┘
|
||
│ MCP 协议
|
||
│ (stdio/SSE)
|
||
┌───────────────────────────────────────▼─────────────────────┐
|
||
│ Wiki MCP Server │
|
||
│ │
|
||
│ ┌─────────────────────────────────────────────────────┐ │
|
||
│ │ MCP Protocol Layer │ │
|
||
│ └─────────────────────────────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌───────────────────────▼───────────────────────────────┐ │
|
||
│ │ Tool Layer │ │
|
||
│ │ query │ memory │ status │ lint │ linker │ taxonomy │ synthesize│ │
|
||
│ └───────────────────────┬───────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌───────────────────────▼───────────────────────────────┐ │
|
||
│ │ Service Layer │ │
|
||
│ │ indexer │ query │ cache │ graph │ parser │ │
|
||
│ └───────────────────────┬───────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌───────────────────────▼───────────────────────────────┐ │
|
||
│ │ Storage Layer │ │
|
||
│ │ SQLite (WAL 模式) │ │
|
||
│ └─────────────────────────────────────────────────────┘ │
|
||
└───────────────────────────────────────┼─────────────────────┘
|
||
│
|
||
┌───────────────────────────────────────▼─────────────────────┐
|
||
│ Obsidian Wiki Vault │
|
||
│ /Volumes/KnowledgeBase/wiki-vault │
|
||
│ │
|
||
│ practices/ │ concepts/ │ entities/ │ projects/ │ skills/ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 1.2 部署架构
|
||
|
||
**开发阶段：**
|
||
```
|
||
Claude Code --stdio--> Wiki MCP Server (手动启动)
|
||
```
|
||
|
||
**生产阶段：**
|
||
```
|
||
PM2 --> Wiki MCP Server (SSE 模式)
|
||
│
|
||
└──> Claude Code --SSE--> Wiki MCP Server
|
||
```
|
||
|
||
---
|
||
|
||
## 2. 模块设计
|
||
|
||
### 2.1 MCP Protocol Layer
|
||
|
||
**职责：**
|
||
- MCP 协议解析和封装
|
||
- 工具注册和路由
|
||
- 错误处理和日志
|
||
|
||
**接口：**
|
||
```python
|
||
class MCPServer:
|
||
def register_tool(self, name: str, handler: Callable)
|
||
def handle_call(self, name: str, params: dict) -> dict
|
||
def log(self, level: str, message: str)
|
||
```
|
||
|
||
### 2.2 Tool Layer
|
||
|
||
**工具列表：**
|
||
|
||
| 工具 | 输入 | 输出 | 实现模块 |
|
||
|------|------|------|----------|
|
||
| wiki_query | query, tags, limit | results, citations | QueryTool |
|
||
| memory_bridge | tool_name, date_range | entries | MemoryTool |
|
||
| wiki_status | - | stats, pending | StatusTool |
|
||
| wiki_lint | path, level | issues, fixes | LintTool |
|
||
| cross_linker | path, dry_run | missing_links | LinkerTool |
|
||
| tag_taxonomy | path, enforce | conflicts | TaxonomyTool |
|
||
| wiki_synthesize | concepts, threshold | synthesis | SynthesizeTool |
|
||
| daily_update | - | updated, new | DailyTool |
|
||
|
||
**MemoryBridgeTool 详细设计：**
|
||
|
||
```python
|
||
class MemoryBridgeTool:
|
||
"""memory_bridge 工具实现 - 按 AI 工具来源浏览和对比 wiki 知识"""
|
||
|
||
async def handle(self, tool_name: str, date_range: str) -> dict:
|
||
"""
|
||
按 AI 工具来源浏览和对比 wiki 知识
|
||
|
||
Args:
|
||
tool_name: AI 工具名称（如 "claude", "web_reader", "gitea"）
|
||
date_range: 日期范围（如 "2024-01-01:2024-12-31"）
|
||
|
||
Returns:
|
||
{
|
||
"entries": [
|
||
{
|
||
"path": "practices/moziplus-orchestration.md",
|
||
"title": "moziplus 编排实践",
|
||
"summary": "...",
|
||
"updated_at": "2024-06-15",
|
||
"relevance_score": 0.85
|
||
}
|
||
],
|
||
"total": 12,
|
||
"tool_name": "claude",
|
||
"date_range": "2024-01-01:2024-12-31"
|
||
}
|
||
"""
|
||
# 1. 从索引中查询 source_tool = tool_name 的页面
|
||
# 2. 按 updated_at 在 date_range 内过滤
|
||
# 3. 计算相关性分数（基于摘要匹配）
|
||
# 4. 返回结果列表
|
||
```
|
||
|
||
**数据来源：** WikiPage.source_tool 字段（新增），记录页面来源的 AI 工具名称
|
||
|
||
### 2.3 Service Layer
|
||
|
||
**IndexerService（索引服务）：**
|
||
```python
|
||
class IndexerService:
|
||
async def index_page(self, path: str) -> None
|
||
async def index_batch(self, paths: List[str]) -> None
|
||
async def rebuild_index(self) -> None
|
||
async def get_dirty_pages(self) -> List[str] # 增量更新
|
||
```
|
||
|
||
**QueryService（查询服务）：**
|
||
```python
|
||
class QueryService:
|
||
"""查询服务 - 负责所有查询逻辑"""
|
||
|
||
def __init__(self, db: Database, cache: CacheService):
|
||
self.db = db
|
||
self.cache = cache
|
||
|
||
async def search(self, query: str, limit: int) -> List[WikiPage]:
|
||
"""FTS5 全文搜索"""
|
||
# 1. 检查缓存
|
||
cache_key = f"search:{query}:{limit}"
|
||
cached = await self.cache.get(cache_key)
|
||
if cached:
|
||
return cached
|
||
|
||
# 2. FTS5 搜索
|
||
results = await self.db.fts_search(query, limit)
|
||
|
||
# 3. 缓存结果
|
||
await self.cache.set(cache_key, results, ttl=3600)
|
||
|
||
return results
|
||
|
||
async def search_by_tags(self, tags: List[str]) -> List[WikiPage]:
|
||
"""按标签搜索"""
|
||
return await self.db.search_by_tags(tags)
|
||
|
||
async def get_page(self, path: str) -> Optional[WikiPage]:
|
||
"""获取单个页面"""
|
||
return await self.db.get_page(path)
|
||
|
||
async def get_links(self, path: str) -> Set[str]:
|
||
"""获取页面的出链"""
|
||
return await self.db.get_links(path)
|
||
|
||
async def get_backlinks(self, path: str) -> Set[str]:
|
||
"""获取页面的反向链接"""
|
||
return await self.db.get_backlinks(path)
|
||
|
||
async def find_orphans(self) -> Set[str]:
|
||
"""查找孤立页面（无反向链接）"""
|
||
all_pages = await self.db.get_all_pages()
|
||
orphans = set()
|
||
for page in all_pages:
|
||
backlinks = await self.db.get_backlinks(page.path)
|
||
if not backlinks and page.path != "index.md":
|
||
orphans.add(page.path)
|
||
return orphans
|
||
```
|
||
|
||
**CacheService（缓存服务）：**
|
||
```python
|
||
from functools import lru_cache
|
||
from collections import OrderedDict
|
||
import asyncio
|
||
|
||
class CacheService:
|
||
"""缓存服务 - LRU 缓存 + TTL 过期"""
|
||
|
||
def __init__(self, max_size: int = 1000):
|
||
self.cache: OrderedDict[str, tuple] = OrderedDict() # key -> (value, expire_time)
|
||
self.max_size = max_size
|
||
self.lock = asyncio.Lock()
|
||
|
||
async def get(self, key: str) -> Optional[Any]:
|
||
"""获取缓存值（异步，带锁）"""
|
||
async with self.lock:
|
||
if key not in self.cache:
|
||
return None
|
||
|
||
value, expire_time = self.cache[key]
|
||
|
||
# 检查是否过期
|
||
if expire_time and time.time() > expire_time:
|
||
del self.cache[key]
|
||
return None
|
||
|
||
# LRU: 移到末尾
|
||
self.cache.move_to_end(key)
|
||
return value
|
||
|
||
async def set(self, key: str, value: Any, ttl: int = 3600) -> None:
|
||
"""设置缓存值（异步，带锁）"""
|
||
async with self.lock:
|
||
expire_time = time.time() + ttl if ttl else None
|
||
|
||
# 如果缓存已满，删除最旧的条目
|
||
if len(self.cache) >= self.max_size and key not in self.cache:
|
||
self.cache.popitem(last=False) # FIFO 删除
|
||
|
||
self.cache[key] = (value, expire_time)
|
||
self.cache.move_to_end(key)
|
||
|
||
async def invalidate(self, pattern: str) -> int:
|
||
"""按模式清除缓存（支持 * 通配符）"""
|
||
async with self.lock:
|
||
if pattern == "*":
|
||
count = len(self.cache)
|
||
self.cache.clear()
|
||
return count
|
||
|
||
keys_to_delete = [k for k in self.cache.keys() if fnmatch.fnmatch(k, pattern)]
|
||
for key in keys_to_delete:
|
||
del self.cache[key]
|
||
return len(keys_to_delete)
|
||
```
|
||
|
||
**缓存策略：**
|
||
- **存储方式**：内存 LRU 缓存（OrderedDict）
|
||
- **最大容量**：1000 条（可配置）
|
||
- **淘汰策略**：FIFO 淘汰最旧条目
|
||
- **TTL**：默认 3600 秒（1 小时）
|
||
- **线程安全**：asyncio.Lock 保护
|
||
|
||
**GraphService（图服务）：**
|
||
```python
|
||
class GraphService:
|
||
async def get_links(self, path: str) -> Set[str]
|
||
async def get_backlinks(self, path: str) -> Set[str]
|
||
async def find_orphans(self) -> Set[str]
|
||
async def find_missing_links(self) -> List[Tuple[str, str]]
|
||
```
|
||
|
||
**ParserService（解析服务）：**
|
||
```python
|
||
class ParserService:
|
||
async def parse_frontmatter(self, content: str) -> dict
|
||
async def extract_links(self, content: str) -> List[str]
|
||
async def validate_page(self, path: str) -> List[str] # 返回问题列表
|
||
```
|
||
|
||
### 2.4 Storage Layer
|
||
|
||
**数据库连接（使用 aiosqlite 实现异步）：**
|
||
|
||
```python
|
||
import aiosqlite
|
||
import asyncio
|
||
|
||
class Database:
|
||
"""SQLite 数据库封装，使用 aiosqlite 实现真正的异步支持"""
|
||
|
||
def __init__(self, path: str):
|
||
self.path = path
|
||
self._conn = None
|
||
self._lock = asyncio.Lock()
|
||
|
||
async def connect(self):
|
||
"""建立连接，启用 WAL 模式和并发保护"""
|
||
self._conn = await aiosqlite.connect(self.path)
|
||
await self._conn.execute("PRAGMA journal_mode=WAL")
|
||
await self._conn.execute("PRAGMA busy_timeout=10000") # 10s
|
||
await self._conn.execute("PRAGMA synchronous=NORMAL")
|
||
await self._conn.commit()
|
||
|
||
async def execute(self, sql: str, params: tuple = ()):
|
||
"""执行 SQL（带写入锁）"""
|
||
async with self._lock:
|
||
cursor = await self._conn.execute(sql, params)
|
||
await self._conn.commit()
|
||
return cursor
|
||
|
||
async def fetch_all(self, sql: str, params: tuple = ()):
|
||
"""查询所有结果（读操作无需锁，WAL 自动处理）"""
|
||
cursor = await self._conn.execute(sql, params)
|
||
return await cursor.fetchall()
|
||
|
||
async def fts_search(self, query: str, limit: int) -> List[WikiPage]:
|
||
"""FTS5 全文搜索"""
|
||
sql = """
|
||
SELECT path, title, category, tags, summary,
|
||
lifecycle, created_at, updated_at, indexed_at
|
||
FROM wiki_fts
|
||
WHERE wiki_fts MATCH ?
|
||
ORDER BY rank
|
||
LIMIT ?
|
||
"""
|
||
rows = await self.fetch_all(sql, (query, limit))
|
||
return [self._row_to_page(row) for row in rows]
|
||
|
||
async def close(self):
|
||
"""关闭连接"""
|
||
if self._conn:
|
||
await self._conn.close()
|
||
```
|
||
|
||
**fix_dirty_states 恢复机制：**
|
||
|
||
```python
|
||
async def fix_dirty_states(db: Database):
|
||
"""启动时清理可能的脏状态"""
|
||
try:
|
||
# 1. 检查 WAL 文件是否损坏
|
||
await db.execute("PRAGMA wal_checkpoint(PASSIVE)")
|
||
|
||
# 2. 检查数据库完整性
|
||
result = await db.fetch_all("PRAGMA integrity_check")
|
||
if result and result[0][0] != "ok":
|
||
raise Exception(f"数据库损坏: {result}")
|
||
|
||
# 3. 清理可能的锁文件
|
||
# （WAL 模式下通常不需要）
|
||
|
||
except Exception as e:
|
||
logger.warning(f"检测到索引问题，尝试重建: {e}")
|
||
await rebuild_index(db)
|
||
```
|
||
|
||
---
|
||
|
||
## 3. 数据模型
|
||
|
||
### 3.1 WikiPage（页面模型）
|
||
|
||
```python
|
||
@dataclass
|
||
class WikiPage:
|
||
path: str # wiki 相对路径
|
||
title: str # 标题
|
||
category: str # 分类（practices/concepts/...）
|
||
tags: List[str] # 标签列表
|
||
summary: str # 摘要（≤200 字符）
|
||
content_hash: str # MD5 哈希
|
||
lifecycle: str # draft/verified/archived/disputed
|
||
source_tool: str # 来源工具（claude/web_reader/gitea/other）
|
||
created_at: datetime
|
||
updated_at: datetime
|
||
indexed_at: datetime
|
||
|
||
def is_stale(self, days: int = 90) -> bool:
|
||
return (datetime.now() - self.updated_at).days > days
|
||
```
|
||
|
||
> **注：** source_tool 字段用于 memory_bridge 功能，记录页面来源的 AI 工具
|
||
|
||
### 3.2 WikiIndex（索引模型）
|
||
|
||
```python
|
||
@dataclass
|
||
class WikiIndex:
|
||
pages: Dict[str, WikiPage]
|
||
links: Dict[str, Set[str]] # source -> {targets}
|
||
backlinks: Dict[str, Set[str]] # target -> {sources}
|
||
tags: Dict[str, Set[str]] # tag -> {pages}
|
||
orphans: Set[str] # 无反向链接的页面
|
||
stats: IndexStats
|
||
|
||
@dataclass
|
||
class IndexStats:
|
||
total_pages: int
|
||
total_links: int
|
||
total_tags: int
|
||
last_indexed: datetime
|
||
dirty_pages: int
|
||
```
|
||
|
||
### 3.3 SQLite 表结构（完整设计）
|
||
|
||
**页面索引表：**
|
||
```sql
|
||
CREATE TABLE wiki_pages (
|
||
path TEXT PRIMARY KEY,
|
||
title TEXT NOT NULL,
|
||
category TEXT,
|
||
tags TEXT, -- JSON 数组: ["tag1", "tag2"]
|
||
summary TEXT,
|
||
content_hash TEXT NOT NULL,
|
||
lifecycle TEXT DEFAULT 'draft', -- draft|verified|archived|disputed
|
||
source_tool TEXT DEFAULT 'other', -- claude/web_reader/gitea/other
|
||
created_at TIMESTAMP,
|
||
updated_at TIMESTAMP,
|
||
indexed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
-- 索引优化
|
||
CREATE INDEX idx_pages_category ON wiki_pages(category);
|
||
CREATE INDEX idx_pages_lifecycle ON wiki_pages(lifecycle);
|
||
CREATE INDEX idx_pages_updated ON wiki_pages(updated_at);
|
||
CREATE INDEX idx_pages_source_tool ON wiki_pages(source_tool); -- memory_bridge 查询优化
|
||
```
|
||
|
||
**FTS5 全文搜索表：**
|
||
```sql
|
||
-- 内容表（FTS5 外部内容表）- 必须先创建
|
||
CREATE TABLE wiki_content (
|
||
path TEXT PRIMARY KEY,
|
||
content TEXT NOT NULL
|
||
);
|
||
|
||
-- 将 FTS5 关联到内容表
|
||
CREATE VIRTUAL TABLE wiki_fts USING fts5(
|
||
path UNINDEXED,
|
||
title,
|
||
content,
|
||
summary,
|
||
content=wiki_content,
|
||
content_rowid=rowid,
|
||
tokenize = 'porter unicode61' -- 英文词干 + Unicode 分词
|
||
);
|
||
```
|
||
|
||
**链接关系表：**
|
||
```sql
|
||
CREATE TABLE wiki_links (
|
||
source TEXT NOT NULL,
|
||
target TEXT NOT NULL,
|
||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||
PRIMARY KEY (source, target)
|
||
);
|
||
|
||
-- 反向链接查询优化
|
||
CREATE INDEX idx_links_target ON wiki_links(target);
|
||
```
|
||
|
||
**标签索引表：**
|
||
```sql
|
||
CREATE TABLE wiki_tags (
|
||
tag TEXT PRIMARY KEY,
|
||
count INTEGER DEFAULT 0,
|
||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
-- 页面-标签关联表（多对多）
|
||
CREATE TABLE wiki_page_tags (
|
||
path TEXT NOT NULL,
|
||
tag TEXT NOT NULL,
|
||
PRIMARY KEY (path, tag),
|
||
FOREIGN KEY (path) REFERENCES wiki_pages(path) ON DELETE CASCADE,
|
||
FOREIGN KEY (tag) REFERENCES wiki_tags(tag) ON DELETE CASCADE
|
||
);
|
||
|
||
CREATE INDEX idx_page_tags_tag ON wiki_page_tags(tag);
|
||
```
|
||
|
||
**索引元数据表：**
|
||
```sql
|
||
CREATE TABLE wiki_meta (
|
||
key TEXT PRIMARY KEY,
|
||
value TEXT,
|
||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
-- 存储索引状态
|
||
INSERT INTO wiki_meta (key, value) VALUES
|
||
('index_version', '1'),
|
||
('last_full_reindex', ''),
|
||
('page_count', '0');
|
||
```
|
||
|
||
---
|
||
|
||
## 4. 接口设计
|
||
|
||
### 4.1 MCP Tool 接口
|
||
|
||
**wiki_query:**
|
||
```json
|
||
{
|
||
"name": "wiki_query",
|
||
"inputSchema": {
|
||
"type": "object",
|
||
"properties": {
|
||
"query": {"type": "string"},
|
||
"tags": {"type": "array", "items": {"type": "string"}},
|
||
"limit": {"type": "integer", "default": 10}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**memory_bridge:**
|
||
```json
|
||
{
|
||
"name": "memory_bridge",
|
||
"inputSchema": {
|
||
"type": "object",
|
||
"properties": {
|
||
"tool_name": {"type": "string"},
|
||
"date_range": {"type": "string"}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**wiki_status:**
|
||
```json
|
||
{
|
||
"name": "wiki_status",
|
||
"inputSchema": {
|
||
"type": "object",
|
||
"properties": {}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 4.2 内部服务接口
|
||
|
||
```python
|
||
# IndexerService
|
||
async def index_page(path: str) -> IndexResult
|
||
async def get_dirty_pages() -> List[str]
|
||
|
||
# QueryService
|
||
async def search(query: str, limit: int) -> List[WikiPage]
|
||
async def search_by_tags(tags: List[str]) -> List[WikiPage]
|
||
async def get_page(path: str) -> Optional[WikiPage]
|
||
|
||
# GraphService
|
||
async def get_links(path: str) -> Set[str]
|
||
async def get_backlinks(path: str) -> Set[str]
|
||
async def find_orphans() -> Set[str]
|
||
```
|
||
|
||
---
|
||
|
||
## 5. 核心算法
|
||
|
||
### 5.1 增量更新算法
|
||
|
||
```python
|
||
async def incremental_update():
|
||
"""增量更新索引 - 只处理变化的页面"""
|
||
# 1. 获取所有 wiki 页面
|
||
all_pages = scan_wiki_vault()
|
||
|
||
# 2. 检查每个页面的哈希
|
||
for page in all_pages:
|
||
current_hash = md5(page.content)
|
||
stored = await db.get_page_hash(page.path)
|
||
|
||
if stored != current_hash:
|
||
# 3. 只重索引变化的页面
|
||
await index_page(page)
|
||
|
||
# 4. 处理删除的页面
|
||
indexed_paths = await db.get_all_indexed_paths()
|
||
for path in indexed_paths:
|
||
if path not in all_pages:
|
||
await db.delete_page(path)
|
||
```
|
||
|
||
### 5.2 查询算法
|
||
|
||
```python
|
||
async def wiki_query(query: str, tags: List[str], limit: int):
|
||
"""组合查询算法"""
|
||
# 1. FTS5 全文搜索
|
||
if query:
|
||
results = await query_service.search(query, limit)
|
||
|
||
# 2. 标签过滤
|
||
if tags:
|
||
results = await query_service.search_by_tags(tags)
|
||
if query:
|
||
# 交叉引用：全文搜索结果中也需匹配标签
|
||
results = [r for r in results if any(t in r.tags for t in tags)]
|
||
|
||
# 3. 按相关性排序
|
||
sorted_results = rank_by_relevance(results, query)
|
||
|
||
# 4. 返回带 [[wikilink]] 的结果
|
||
return format_results(sorted_results[:limit])
|
||
```
|
||
|
||
### 5.3 cross_linker 算法
|
||
|
||
```python
|
||
async def find_missing_links():
|
||
"""查找缺失的交叉引用"""
|
||
# 1. 获取所有页面内容
|
||
pages = await load_all_pages()
|
||
|
||
# 2. 提取所有 [[wikilinks]]
|
||
all_links = extract_all_links(pages)
|
||
|
||
# 3. 找出缺失的链接
|
||
missing = []
|
||
for source, targets in all_links.items():
|
||
for target in targets:
|
||
if not await page_exists(target):
|
||
missing.append((source, target))
|
||
|
||
return missing
|
||
```
|
||
|
||
### 5.4 hot.md 生成算法
|
||
|
||
```python
|
||
async def generate_hot_md():
|
||
"""生成热点文件 - 记录最近活动和关键发现"""
|
||
# 1. 获取最近更新的页面（7 天内）
|
||
recent_pages = await db.get_recent_pages(days=7)
|
||
|
||
# 2. 获取新增标签
|
||
new_tags = await db.get_new_tags(days=7)
|
||
|
||
# 3. 检测新的孤立页面（可能需要链接）
|
||
orphans = await query_service.find_orphans()
|
||
|
||
# 4. 生成 markdown
|
||
hot_content = f"""# Wiki Hot - {datetime.now().strftime('%Y-%m-%d')}
|
||
|
||
## 最近更新
|
||
{format_page_list(recent_pages)}
|
||
|
||
## 新增标签
|
||
{format_tag_list(new_tags)}
|
||
|
||
## 待链接页面
|
||
{format_orphan_list(orphans)}
|
||
"""
|
||
|
||
# 5. 写入 hot.md
|
||
await write_file("hot.md", hot_content)
|
||
```
|
||
|
||
---
|
||
|
||
## 6. 配置设计
|
||
|
||
### 6.1 配置文件结构
|
||
|
||
```yaml
|
||
# ~/.sanguo-llmwiki/config.yaml
|
||
wiki:
|
||
vault_path: "/Volumes/KnowledgeBase/wiki-vault"
|
||
index_path: "~/.sanguo-llmwiki/index.db"
|
||
max_page_size_kb: 500
|
||
|
||
mcp:
|
||
mode: "stdio" # stdio 或 SSE
|
||
host: "localhost"
|
||
port: 8080
|
||
|
||
performance:
|
||
query_timeout_ms: 5000
|
||
cache_ttl_seconds: 3600
|
||
fts_cache_size_mb: 100
|
||
max_concurrent_indexing: 5
|
||
|
||
logging:
|
||
level: "INFO"
|
||
file: "~/.sanguo-llmwiki/wiki-mcp.log"
|
||
```
|
||
|
||
### 6.2 环境变量
|
||
|
||
```bash
|
||
# 环境变量优先级高于配置文件
|
||
WIKI_VAULT_PATH=/custom/path
|
||
WIKI_INDEX_PATH=/custom/index.db
|
||
MCP_MODE=sse
|
||
LOG_LEVEL=DEBUG
|
||
MAX_CONCURRENT_INDEXING=10
|
||
```
|
||
|
||
---
|
||
|
||
## 7. 错误处理
|
||
|
||
### 7.1 错误分类
|
||
|
||
| 错误类型 | 处理方式 |
|
||
|----------|----------|
|
||
| Wiki 路径不存在 | 启动失败，返回友好错误 |
|
||
| 索引文件损坏 | 自动重建 + WARN 日志 |
|
||
| SQLite 写入失败 | 回滚事务 + 重试 1 次 |
|
||
| 页面解析失败 | 记录日志 + 跳过该页面 |
|
||
| MCP 协议错误 | 返回标准错误格式 |
|
||
| 查询超时 | 返回部分结果 + WARN |
|
||
| 重试失败 | 降级为文件扫描（性能降低） |
|
||
|
||
### 7.2 错误响应格式
|
||
|
||
```json
|
||
{
|
||
"success": false,
|
||
"error": {
|
||
"code": "INDEX_CORRUPTED",
|
||
"message": "索引文件损坏，正在自动重建",
|
||
"details": {"rebuilding": true}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 7.3 重试策略
|
||
|
||
```python
|
||
async def execute_with_retry(db: Database, sql: str, params: tuple, max_retries: int = 2):
|
||
"""带重试的数据库操作"""
|
||
for attempt in range(max_retries):
|
||
try:
|
||
return await db.execute(sql, params)
|
||
except aiosqlite.OperationalError as e:
|
||
if "database is locked" in str(e) and attempt < max_retries - 1:
|
||
await asyncio.sleep(0.1 * (2 ** attempt)) # 指数退避
|
||
continue
|
||
raise
|
||
```
|
||
|
||
---
|
||
|
||
## 8. 安全考虑
|
||
|
||
虽然是本地系统，但仍需考虑：
|
||
|
||
1. **路径安全**：验证路径在 wiki vault 范围内（防止路径遍历）
|
||
2. **资源限制**：限制查询返回数量、内存使用
|
||
3. **日志脱敏**：日志中不记录敏感内容
|
||
|
||
---
|
||
|
||
## 9. 性能优化
|
||
|
||
### 9.1 索引优化
|
||
|
||
- 使用 FTS5 全文搜索索引
|
||
- 标签单独建立索引
|
||
- 定期 VACUUM（每周）
|
||
|
||
### 9.2 查询优化
|
||
|
||
- 查询结果缓存（TTL 1 小时）
|
||
- 限制返回数量（默认 10）
|
||
- 使用 prepared statements
|
||
|
||
### 9.3 并发优化
|
||
|
||
- SQLite WAL 模式
|
||
- aiosqlite 真正的异步支持
|
||
- asyncio.Lock 写入串行化
|
||
- busy_timeout=10s
|
||
|
||
---
|
||
|
||
## 10. Wiki Skills 设计
|
||
|
||
### 10.1 Skill 模板
|
||
|
||
每个 Skill 遵循统一结构：
|
||
|
||
```markdown
|
||
---
|
||
name: wiki-xxx
|
||
description: >
|
||
简短描述（1-2 句）
|
||
触发条件
|
||
---
|
||
|
||
# Wiki XXX
|
||
|
||
## 使用场景
|
||
用户何时触发这个 Skill
|
||
|
||
## 操作步骤
|
||
1. ...
|
||
2. ...
|
||
|
||
## 输出格式
|
||
...
|
||
```
|
||
|
||
### 10.2 Skills 列表（与需求对齐）
|
||
|
||
**优先级 P0（核心）：**
|
||
1. wiki-setup - 初始化 wiki vault
|
||
2. wiki-ingest - 蒸馏文档
|
||
3. wiki-capture - 保存对话
|
||
|
||
**优先级 P1（重要）：**
|
||
4. wiki-rebuild - 重建 wiki
|
||
5. data-ingest - 录入非结构化数据
|
||
6. ingest-url - 抓取 URL
|
||
7. wiki-export - 导出知识图谱
|
||
|
||
**优先级 P2（可选）：**
|
||
8. wiki-research - 多轮搜索研究
|
||
9. impl-validator - 验证实现
|
||
10. graph-colorize - 着色
|
||
11. wiki-agent - 录入历史
|
||
|
||
---
|
||
|
||
## 11. 性能基准测试
|
||
|
||
### 11.1 benchmark.py 设计
|
||
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""
|
||
Wiki MCP Server 性能基准测试
|
||
参考 BitNet 实践 - 可验证的性能指标
|
||
"""
|
||
|
||
import asyncio
|
||
import time
|
||
import statistics
|
||
from typing import List
|
||
|
||
class Benchmark:
|
||
"""基准测试类"""
|
||
|
||
def __init__(self, query_service: QueryService):
|
||
self.query_service = query_service
|
||
self.results = []
|
||
|
||
async def benchmark_query(self, query: str, iterations: int = 100) -> dict:
|
||
"""测试查询性能"""
|
||
latencies = []
|
||
|
||
for _ in range(iterations):
|
||
start = time.perf_counter()
|
||
await self.query_service.search(query, limit=10)
|
||
end = time.perf_counter()
|
||
latencies.append((end - start) * 1000) # ms
|
||
|
||
return {
|
||
"query": query,
|
||
"iterations": iterations,
|
||
"avg_ms": statistics.mean(latencies),
|
||
"p50_ms": statistics.median(latencies),
|
||
"p99_ms": statistics.quantiles(latencies, n=100)[98],
|
||
"min_ms": min(latencies),
|
||
"max_ms": max(latencies),
|
||
"qps": iterations / sum(latencies) * 1000
|
||
}
|
||
|
||
async def benchmark_indexing(self, page_count: int = 1000) -> dict:
|
||
"""测试索引性能"""
|
||
start = time.perf_counter()
|
||
|
||
# 模拟索引 N 个页面
|
||
pages = generate_mock_pages(page_count)
|
||
await indexer_service.index_batch(pages)
|
||
|
||
end = time.perf_counter()
|
||
total_ms = (end - start) * 1000
|
||
|
||
return {
|
||
"page_count": page_count,
|
||
"total_ms": total_ms,
|
||
"avg_ms_per_page": total_ms / page_count
|
||
}
|
||
|
||
async def run_all(self) -> dict:
|
||
"""运行所有基准测试"""
|
||
results = {}
|
||
|
||
# 1. 查询性能测试
|
||
queries = [
|
||
"SQLite 并发",
|
||
"性能优化",
|
||
"架构设计"
|
||
]
|
||
for query in queries:
|
||
results[f"query_{query}"] = await self.benchmark_query(query)
|
||
|
||
# 2. 索引性能测试
|
||
results["indexing"] = await self.benchmark_indexing()
|
||
|
||
# 3. 内存占用测试
|
||
results["memory"] = measure_memory_usage()
|
||
|
||
return results
|
||
|
||
# 基准测试目标（参考需求 4.1）
|
||
TARGETS = {
|
||
"query_p99_ms": 100, # P99 延迟 < 100ms
|
||
"fts_search_p99_ms": 200, # 全文搜索 < 200ms
|
||
"index_avg_ms_per_page": 10, # 索引 < 10ms/页
|
||
"memory_mb": 500 # 内存 < 500MB
|
||
}
|
||
|
||
if __name__ == "__main__":
|
||
# 运行基准测试
|
||
benchmark = Benchmark(query_service)
|
||
results = asyncio.run(benchmark.run_all())
|
||
|
||
# 输出结果
|
||
print("=== Wiki MCP Server Benchmark ===")
|
||
print(json.dumps(results, indent=2))
|
||
|
||
# 检查是否达标
|
||
for key, target in TARGETS.items():
|
||
actual = results.get(key)
|
||
if actual and actual > target:
|
||
print(f"WARNING: {key} ({actual}) exceeds target ({target})")
|
||
```
|
||
|
||
---
|
||
|
||
## 12. 测试设计
|
||
|
||
### 12.1 单元测试
|
||
|
||
覆盖所有 Service 层的核心逻辑：
|
||
- IndexerService 测试
|
||
- QueryService 测试
|
||
- GraphService 测试
|
||
- ParserService 测试
|
||
- Database 并发测试（模拟并发写入）
|
||
|
||
### 12.2 集成测试
|
||
|
||
- MCP 协议层测试（使用 MCP SDK mock）
|
||
- SQLite 操作测试
|
||
- Wiki 解析测试
|
||
- FTS5 搜索测试
|
||
|
||
### 12.3 E2E 测试
|
||
|
||
使用真实 wiki 数据集测试：
|
||
- 查询场景
|
||
- 搜索场景
|
||
- 索引更新场景
|
||
- 并发查询场景
|
||
|
||
---
|
||
|
||
## 13. 部署设计
|
||
|
||
### 13.1 开发部署
|
||
|
||
```bash
|
||
# 手动启动
|
||
cd ~/.openclaw/sanguo_projects/sanguo_llmwiki
|
||
python -m mcp_server.main
|
||
```
|
||
|
||
### 13.2 生产部署
|
||
|
||
```bash
|
||
# PM2 配置
|
||
cat > ecosystem.config.cjs << 'EOF'
|
||
module.exports = {
|
||
apps: [{
|
||
name: 'wiki-mcp',
|
||
script: 'python',
|
||
args: '-m mcp_server.main',
|
||
cwd: '/Users/chufeng/.openclaw/sanguo_projects/sanguo_llmwiki',
|
||
instances: 1,
|
||
autorestart: true,
|
||
watch: false,
|
||
max_memory_restart: '500M',
|
||
env: {
|
||
PYTHONUNBUFFERED: '1',
|
||
LOG_LEVEL: 'INFO'
|
||
}
|
||
}]
|
||
}
|
||
EOF
|
||
|
||
pm2 start ecosystem.config.cjs
|
||
pm2 save
|
||
```
|
||
|
||
### 13.3 MCP 配置
|
||
|
||
**stdio 模式（开发）：**
|
||
```json
|
||
{
|
||
"mcpServers": {
|
||
"wiki": {
|
||
"command": "python",
|
||
"args": ["-m", "mcp_server.main"],
|
||
"cwd": "/Users/chufeng/.openclaw/sanguo_projects/sanguo_llmwiki",
|
||
"env": {
|
||
"WIKI_VAULT_PATH": "/Volumes/KnowledgeBase/wiki-vault"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
**SSE 模式（生产）：**
|
||
```json
|
||
{
|
||
"mcpServers": {
|
||
"wiki": {
|
||
"type": "sse",
|
||
"url": "http://localhost:8080/sse"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
> **注：** SSE 模式需要 MCP Server 实现 HTTP 端点，v1.0 暂不实现，v1.1 预留接口
|
||
|
||
---
|
||
|
||
## 14. 修订记录
|
||
|
||
### v1.3（2026-06-28）- 搜索改进与代码修复
|
||
|
||
**搜索改进（P0-P1）：**
|
||
- ✅ **连字符扩展**：在 `Database.update_fts_content()` 中添加 `_expand_hyphenated_text()` 方法，将 `multi-agent` 扩展为 `multi agent` 提升搜索召回率
|
||
- ✅ **中文同义词支持**：在 `QueryService` 中添加 `CHINESE_SYNONYMS` 映射表和 `_expand_query_with_synonyms()` 方法，支持中英文混合查询（如"多智能体"自动扩展为包含 "multi-agent" 的查询）
|
||
- ✅ **Snippet 高亮预留**：在 `Database.fts_search()` 中添加 `with_snippets` 参数，为 FTS5 snippet 功能预留接口
|
||
|
||
**代码修复：**
|
||
- ✅ **YAML 解析增强**：添加 `html.unescape()` 支持解码 HTML 实体（如 `>` → `>`）
|
||
- ✅ **标签关联维护**：添加 `Database.update_page_tags()` 方法，自动维护 `wiki_page_tags` 关联表
|
||
- ✅ **缓存失效优化**：在 `QueryService.invalidate_cache()` 中添加智能前缀处理，支持 `page:/links:/search:` 前缀匹配
|
||
|
||
**数据模型更新：**
|
||
- ✅ **FTS5 表结构优化**：移除外部内容表，直接存储所有字段在 FTS 表中
|
||
- ✅ **索引策略调整**：title 和 summary 字段存储扩展版本（原版 + 连字符扩展版本）
|
||
|
||
### v1.2（2026-06-26）- 第二轮评审修复
|
||
|
||
**Major（已修复）：**
|
||
- ✅ **M1**: 修复 FTS5 表结构语法错误（第 3.3 节）- 删除重复定义，调整表创建顺序
|
||
- ✅ **M2**: 解决 memory_bridge 的 tool_name 数据来源问题（第 2.2 节 / 3.1 节）- 在 WikiPage 中添加 source_tool 字段
|
||
- ✅ **M3**: 明确 QueryService 缓存策略（第 2.3 节）- 补充 CacheService 详细实现，包括 LRU 缓存和大小限制
|
||
|
||
### v1.1（2026-06-26）
|
||
|
||
**修复的问题：**
|
||
|
||
**Critical（必须修复）：**
|
||
- ✅ **C1**: 补充 QueryService 模块设计（第 2.3 节）
|
||
- ✅ **C2**: 补充完整的 SQLite 表结构设计（第 3.3 节）
|
||
|
||
**Major（建议修复）：**
|
||
- ✅ **M1**: Database 类改用 aiosqlite 实现真正的异步（第 2.4 节）
|
||
- ✅ **M2**: 补充 MemoryBridgeTool 详细设计（第 2.2 节）
|
||
- ✅ **M3**: 配置项与需求对齐（第 6.1 节）
|
||
- ✅ **M4**: Wiki Skills 优先级与需求对齐（第 10.2 节）
|
||
- ✅ **M5**: 新增 benchmark.py 设计（第 11 节）
|
||
|
||
**Minor（可选改进）：**
|
||
- ✅ **m1**: 删除 WikiPage.sources 字段并添加说明
|
||
- ✅ **m2**: 补充 hot.md 生成算法（第 5.4 节）
|
||
- ✅ **m3**: 补充重试策略和降级方案（第 7.3 节）
|
||
- ✅ **m4**: 明确 SSE 模式为 v1.1 预留（第 13.3 节）
|
||
- ✅ **m5**: 补充 Python 3.11+ 版本要求（需求文档）
|
||
|
||
---
|
||
|
||
*文档版本：v1.3*
|
||
*创建时间：2026-06-26*
|
||
*更新时间：2026-06-28*
|