基于 ai-dynamo/dynamo 代码库探索整理 (
lib/kv-router、lib/llm/src/kv_router、lib/llm/src/block_manager)。
整理日期:2026-06-12
Router(KV-aware routing)与 Offload(KVBM 分层 KV cache)之间没有直接函数调用, 而是通过 KV cache 事件总线异步耦合:Offload 状态变化以 Store/Remove 事件经 Consolidator 汇聚后发布到 event plane,Router indexer 订阅并按 storage tier 更新索引。
关键结论:
StorageTier: Device / HostPinned / Disk / External),通过 RouterEvent.storage_tier 和 medium 字段路由到不同 indexer。tier,lower-tier indexer 可单独记录 host/disk 命中,配合 host_cache_hit_weight / disk_cache_hit_weight 参与选 worker。A. 请求进入 → Router 决策
KvPushRouter::generate → select_workerKvRouter::find_best_match_details:算 token block hashes → Indexer::find_matches_by_tier → 加权 cache hit → KvScheduler::schedule → DefaultWorkerSelector::select_workerB. Worker 执行 → Offload 触发
ManagedBlockPool::register_blocks 注册 immutable blockoffload_priority 存在 → enqueue_offload → OffloadManager worker 线程执行 D2H/H2Diskregister_blocksC. 状态更新 → Router 感知
DynamoEventManager → KvEventConsolidatorHandle::handle_store/removeDedupCacheStatusTracker:首 store 发 STORE;末 source remove 发 REMOVEKvEventConsolidatorPublisher → ZMQ(medium 字段)KvEventPublisher 的 zmq_listener 收 batch → event_processor 批处理sinks::emit 发布 RouterEvent 到 kv-events subjectstart_subscriber 收事件 → handle_live_event → Indexer::apply_eventD. Router 后续决策
find_matches_by_tier 使用更新后的 radix tree / lower-tier continuationoverlap_credit = overlap_score_credit × device
+ host_cache_hit_weight × host_pinned
+ disk_cache_hit_weight × disk
+ shared
prefill_blocks = max(raw_prefill − overlap_credit, 0)
logit = prefill_cost + decode_cost // 越小越优
见 lib/kv-router/src/scheduling/selector.rs 的 DefaultWorkerSelector::worker_logit。
| 结构 | 定义位置 | 方向 | 含义 |
|---|---|---|---|
StoreEventInput / RemoveEventInput | kv_consolidator/tracker.rs | KVBM → Consolidator | block_hash + source=Kvbm + tier |
ConsolidatedEvent::{Store,Remove,ClearAll} | kv_consolidator/tracker.rs | Consolidator 内部队列 | 去重后的对外事件 |
Event::BlockStored { medium } | kv_consolidator/publisher.rs | Consolidator → ZMQ | medium = GPU / CPU_TIER1 / CPU_TIER2 |
PlacementEvent | kv-router/src/protocols.rs | Worker publisher 内部 | placement.tier + KvCacheEvent |
RouterEvent | kv-router/src/protocols.rs | Worker → Event plane → Router | worker_id + storage_tier + event |
TierOverlapBlocks | kv-router/src/scheduling/types.rs | Indexer → Scheduler | device / host_pinned / disk 每 worker 命中数 |
SchedulingRequest | kv-router/src/scheduling/types.rs | Router → Selector | tier_overlap + effective_overlap |
| 配置 | 位置 | 作用 |
|---|---|---|
--router-mode kv | frontend | 启用 KV-aware routing |
--router-kv-events / use_kv_events | KvRouterConfig | 是否消费 worker KV 事件 |
--router-kv-overlap-score-credit | KvRouterConfig | GPU prefix overlap 权重 |
host_cache_hit_weight / disk_cache_hit_weight | kv-router/src/scheduling/config.rs | CPU / disk tier 命中权重 |
DYN_KVBM_KV_EVENTS_ENABLE_CONSOLIDATOR | consolidator_config.py | 是否启用 Consolidator |
DYN_KVBM_CPU_CACHE_GB / DYN_KVBM_DISK_CACHE_GB | block_manager/config.rs | G2 / G3 容量,影响 bypass CPU |
DYN_KVBM_LEADER_ZMQ_PUB_PORT | consolidator endpoints | Consolidator 输出 ZMQ 端口 |
| 文件 / 模块 | 关键内容 |
|---|---|
lib/kv-router/src/protocols.rs | StorageTier, KvCacheEvent, RouterEvent, KV_EVENT_SUBJECT |
lib/kv-router/src/indexer/ | KvIndexer, LowerTierIndexers, TieredMatchDetails |
lib/kv-router/src/scheduling/ | LocalScheduler, DefaultWorkerSelector, TierOverlapBlocks, KvRouterConfig |
lib/kv-router/src/zmq_wire/convert.rs | convert_event:medium → StorageTier → PlacementEvent |
lib/llm/src/kv_router.rs | KvRouter, find_best_match_details, cache_hit_weight_for_tier |
lib/llm/src/kv_router/push_router.rs | KvPushRouter, select_worker, AsyncEngine::generate |
lib/llm/src/kv_router/indexer/ | find_matches_by_tier, apply_event, start_subscriber, handle_live_event |
lib/llm/src/kv_router/publisher/ | KvEventPublisher:zmq_listener → event_processor → sinks::emit |
lib/bindings/python/rust/llm/kv.rs | KvRouter / KvPushRouter / KvEventPublisher PyO3 绑定 |
| 文件 / 模块 | 关键内容 |
|---|---|
lib/llm/src/block_manager/offload.rs | OffloadManager:D2H / H2D / H2Disk / D2D workers |
lib/llm/src/block_manager/pool/managed/state.rs | register_blocks → enqueue_offload(offload 触发点) |
lib/llm/src/block_manager/events.rs | DynamoEventManager::publish_store_events → handle_store |
lib/llm/src/block_manager/kv_consolidator/ | KvEventConsolidator:tracker(Dedup/Passthrough)+ ZMQ publisher |
lib/llm/src/block_manager/config.rs | should_bypass_cpu_cache():G1 → G3 直传 |
lib/bindings/kvbm/.../connector/leader/slot.rs | process_offload_request(G1→G2 或 G1→G3) |
lib/bindings/kvbm/python/kvbm/vllm_integration/consolidator_config.py | should_enable_consolidator, get_consolidator_endpoints |
| 路径 | 用途 |
|---|---|
tests/kvbm_integration/test_consolidator_router_e2e.py | Consolidator + Router E2E;STORE/REMOVE dedup 跨 G1/G2/G3 |
tests/router/test_router_e2e_with_vllm.py | Router E2E |
examples/backends/vllm/launch/agg_kvbm_router.sh | Aggregated + KV routing + KVBM |
docs/design-docs/router-design.md | Router 设计 |
docs/design-docs/kvbm-design.md | KVBM 分层与数据流 |