capability

package
v0.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 22, 2026 License: None detected not legal advice Imports: 0 Imported by: 0

Documentation

Overview

Package capability 是引擎模型能力探测的可复用核心.

它把原 cmd/capability-probe 的纯逻辑 (向 provider/model 发最小化探测 请求, 实测 streaming / thinking / tool_use / structured_output / caching / schema_ref / max_tools, 给每条能力打来源戳, 输出 ModelCapabilities 矩阵) 抽成包, 供 server / registry / pricing 等 引擎内消费者复用. cmd/capability-probe 退成薄 CLI 壳.

Package capability is the reusable core of the engine's model capability probing. It extracts the pure logic from the original cmd/capability-probe (sending minimal probe requests to a provider/model to empirically test streaming / thinking / tool_use / structured_output / caching / schema_ref / max_tools, stamping each capability with its Source, and emitting a ModelCapabilities matrix) into a package reusable by in-engine consumers (server / registry / pricing). cmd/capability-probe becomes a thin CLI wrapper.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsFullyProbed

func IsFullyProbed(mc *ModelCapabilities) bool

IsFullyProbed 是 isFullyProbed 的公共导出版 (缓存 / skip 消费者用的完整性检查).

IsFullyProbed is the exported wrapper of isFullyProbed (a completeness check for cache / skip consumers).

Types

type Capability

type Capability struct {
	Value  any    `json:"value"`
	Source Source `json:"source"`
	// Exhaustive 表示 Value 是否是穷尽测试得到的真上限.
	// 升华改进(ELEVATED): 原先 max_tools 这种字段写 value=128 时,下游消费者无法区分
	// "测了 128 都过(其实可能更多)"和"在 129 处确认拒绝"这两种语义.
	// nil = 概念不适用(如 streaming/thinking 这种 bool 字段);
	// false = 测试未触顶,Value 是已知下界,真值可能更大;
	// true = 测试穷尽,Value 是确认上限.
	// 替代方案:<把 value 写成 null + 文本 note 描述> - 否决:文本不可结构化解析,
	// 未来 loader/registry/UI 等程序消费者要正则提取数字才能用,违背"数据驱动行为"原则.
	Exhaustive *bool          `json:"exhaustive,omitempty"`
	Evidence   map[string]any `json:"evidence,omitempty"`
	Note       string         `json:"note,omitempty"`
}

Capability 是一个带来源标签的能力字段.

精妙之处(CLEVER): Value 用 any 而非泛型-- 一个能力可能是 bool(streaming),int(context_window),float64(price), string(note),统一成 any 让 JSON 序列化自然处理类型差异. Evidence 是可选的原始证据(如 token 计数,错误信息,API 响应摘要), 让下游能重现 probe 的判断过程.

type CapabilityReport

type CapabilityReport struct {
	SchemaVersion string                        `json:"schema_version"`
	GeneratedAt   string                        `json:"generated_at"`
	Models        map[string]*ModelCapabilities `json:"models"`
}

CapabilityReport 是整个 probe 运行输出的聚合报告.

type CapabilityResult

type CapabilityResult struct {
	Provider                  string
	Model                     string
	Streaming                 tristate
	Thinking                  tristate
	ToolUse                   tristate
	StructuredOut             tristate
	Caching                   tristate
	SchemaRef                 tristate
	ToolCount                 int             // 实际可处理工具数,0 表示未探测或探测失败
	ToolCountExhaustive       bool            // true=Value 是确认上限;false=测试未触顶,Value 是已知下界
	ToolCountNote             string          // 探测诊断信息
	MaxOutputTokens           int             // 单次响应实测输出 token 数 (含 thinking + text + tool_use), 0 表示未探测或探测失败
	MaxOutputTokensExhaustive bool            // true=stop_reason=max_tokens/length 触底; false=stop_reason=end_turn/stop_sequence model 自停, 实际上限可能 ≥ Value
	MaxOutputTokensNote       string          // 探测诊断信息
	SchemaFeatures            map[string]bool // L1174: schema 特性 → 是否被 API 接受
	// ADR-0007 capability tracking 实测三件套 (TD-20).
	//
	// ToolNameRegex: 模型实际接受的工具名 regex 实证. 当前 prober 只测一个
	// 触发字符 (`.`), 输出 "strict" (拒含 `.`) / "permissive" (接受) / 空 (未测).
	// 完整 regex inference 留 follow-up — 单点探测足以暴露 r22 类业务 bug.
	//
	// ReasoningPassbackMode: 模型在多轮 tool calling 中是否要求 prior assistant
	// 的 reasoning_content 在下一轮 messages 中 passback. "string" / "none" / 空.
	// 实证方法: 2 round-trip, round 2 不回传 reasoning_content, 看是否 4xx.
	//
	// ProviderKind: 静态标 — 注册时按 target.providerKind 直接落到 result,
	// 不实测 (direct vs aggregator 是注册期人类知识, 不需 API 调用).
	//
	// ADR-0007 capability tracking probed triplet (TD-20).
	//   ToolNameRegex: empirical evidence of the model's accepted tool-name
	//     pattern. Current prober tests one trigger char (`.`) and reports
	//     "strict" (rejects `.`) / "permissive" (accepts) / empty (untested).
	//     Full regex inference deferred — single-point probe already covers
	//     the r22-class business bug surface.
	//   ReasoningPassbackMode: whether the model requires the prior
	//     assistant's reasoning_content to be echoed in the next request
	//     during multi-turn tool calling. "string" / "none" / empty.
	//   ProviderKind: registration-time static label — never probed (direct
	//     vs aggregator is human knowledge, no API call needed).
	ToolNameRegex         string
	ToolNameRegexNote     string
	ReasoningPassbackMode string
	ReasoningPassbackNote string
	ProviderKind          string // 静态来自 target.providerKind, 无 prober
	Notes                 []string
}

CapabilityResult 是单个 provider+model 的能力探测结果.

func Probe

func Probe(ctx context.Context, opts ProbeOpts, model string) CapabilityResult

Probe 是想要原始 tristate 结果的低层入口 (包装 probe()).

Probe is a lower-level entry for callers wanting the raw tristate result (wraps probe()).

type ModelCapabilities

type ModelCapabilities struct {
	Provider string `json:"provider"`
	Model    string `json:"model"`

	ContextWindow        Capability `json:"context_window"`
	MaxOutputTokens      Capability `json:"max_output_tokens"`
	InputPricePer1M      Capability `json:"input_price_per_1m"`
	OutputPricePer1M     Capability `json:"output_price_per_1m"`
	CacheReadPricePer1M  Capability `json:"cache_read_price_per_1m,omitempty"`
	CacheWritePricePer1M Capability `json:"cache_write_price_per_1m,omitempty"`

	Streaming     Capability `json:"streaming"`
	Thinking      Capability `json:"thinking"`
	ToolUse       Capability `json:"tool_use"`
	StructuredOut Capability `json:"structured_output"`
	Caching       Capability `json:"caching"`
	SchemaRef     Capability `json:"schema_ref"`
	MaxTools      Capability `json:"max_tools"`

	Vision            Capability `json:"vision"`
	PDF               Capability `json:"pdf"`
	Batch             Capability `json:"batch"`
	ParallelToolCalls Capability `json:"parallel_tool_calls"`
	StrictJSON        Capability `json:"strict_json"`

	// ADR-0007 capability tracking 三件套 (TD-20).
	// 不纳入 isFullyProbed (避免 ADR-0007 之前的缓存全部失效, 让新字段
	// 渐进 backfill 而非 hard-cutover).
	//
	// ADR-0007 capability tracking triplet (TD-20). Not included in
	// isFullyProbed to keep pre-ADR-0007 caches valid; new fields
	// backfill incrementally rather than hard-cutover.
	ToolNameRegex         Capability `json:"tool_name_regex,omitempty"`
	ReasoningPassbackMode Capability `json:"reasoning_passback_mode,omitempty"`
	ProviderKind          Capability `json:"provider_kind,omitempty"`

	// SchemaFeatures 是 JSON Schema 特性的详细支持数据 (L1174).
	// 不纳入 isFullyProbed 7 字段 -- 补充探测数据, 已缓存 target 需 --force 重跑.
	SchemaFeatures map[string]Capability `json:"schema_features,omitempty"`

	ProbedAt    string   `json:"probed_at"`
	ProbeErrors []string `json:"probe_errors,omitempty"`
}

ModelCapabilities 是一个 provider+model 组合的完整能力画像.

字段分三组:

  1. 基础规格(Context/Tokens/Price):从 provider.Models() 查,标 SourceDocumented
  2. 可实测能力(Streaming/Thinking/Tool/...):probe 结果,标 SourceProbed
  3. 文档能力(Vision/PDF/Batch/Strict/Parallel):暂不实测,标 SourceDocumented/Untested

func ProbeModel

func ProbeModel(ctx context.Context, model string, opts ProbeOpts) (*ModelCapabilities, error)

ProbeModel 跑完整 probe + merge 流水线, 返回一个 model 的 probed + Models() + documented 合并能力画像.

它从 opts 构造内部 target, 运行 probe() + buildModelCapabilities, 返回 *ModelCapabilities. probe() 本身从不 hard-fail (诊断记进 Note 后继续), 故 error 仅保留给 nil-Provider 校验 / ctx 取消.

ProbeModel runs the full probe + merge pipeline for one model and returns the merged probed + Models() + documented capability picture. It builds the internal target from opts, runs probe() + buildModelCapabilities, and returns *ModelCapabilities. probe() itself never hard-fails (it records diagnostics in Notes and continues), so error is reserved for nil-Provider validation / ctx cancellation.

type ProbeOpts

type ProbeOpts struct {
	// ProviderName 是结果标注用的 provider 名 (如 "anthropic").
	// ProviderName is the provider label for results (e.g. "anthropic").
	ProviderName string
	// Provider 是必需的基础 provider 句柄.
	// Provider is the REQUIRED base provider handle.
	Provider flyto.ModelProvider
	// ThinkingProvider 是配置了 ThinkingBudget 的实例; nil -> 跳过 thinking.
	// ThinkingProvider is a ThinkingBudget-configured instance; nil -> skip thinking.
	ThinkingProvider flyto.ModelProvider
	// CachingClient 是直连 internal/transport 的 client (Anthropic / MiniMax
	// cache_control 直路); nil -> 跳过该路径.
	// CachingClient is a direct internal/transport client for the
	// Anthropic / MiniMax cache_control path; nil -> skip that path.
	CachingClient *api.Client
	// CachingProvider 是 EnableCaching=true 实例 (OpenRouter) 或同一实例
	// (DeepSeek long-prefix 阶梯); nil -> 走 generic 路径.
	// CachingProvider is an EnableCaching=true instance (OpenRouter) or the
	// same instance (DeepSeek long-prefix ladder); nil -> generic path.
	CachingProvider flyto.ModelProvider
	// ProviderKind 是静态 "direct" / "aggregator" 标签.
	// ProviderKind is the static "direct" / "aggregator" label.
	ProviderKind string

	// --- 成本控制: 贵探针可跳过 (见 target 注释) ---
	// Cost controls: the expensive probes can be skipped (see target docs).
	MaxProbeTools         int // 0 -> 默认 128 (替换原写死值). 0 -> default 128 (replaces hardcode).
	SkipCaching           bool
	SkipMaxOutputTokens   bool
	SkipToolCount         bool
	SkipReasoningPassback bool
	SkipSchemaFeatures    bool
}

ProbeOpts 携带 probe 需要的四个 provider 句柄 + 静态 providerKind 标签 + per-probe skip 控制.

升华改进(ELEVATED): 单 provider 签名不足 — caching 与 thinking 需各自 单独配置的实例 (ThinkingBudget / EnableCaching / cache_control 直路). 把四个句柄折进 ProbeOpts, 既保留 cmd 原有的 key-dependent 构造灵活性, 又给 server 等消费者一个干净的公共入口.

ProbeOpts carries the four provider-ish handles the probes need plus the static providerKind label and per-probe skip controls. A single provider signature is insufficient -- caching and thinking each need separately configured instances (ThinkingBudget / EnableCaching / the cache_control direct path). Folding the four handles into ProbeOpts keeps cmd's key-dependent construction flexibility while giving server-side consumers a clean public entry.

type Source

type Source string

Source 标识能力数据的来源.

const (
	SourceProbed     Source = "probed"     // 实测过
	SourceDocumented Source = "documented" // 官方文档/AI 分析
	SourceManual     Source = "manual"     // 人工标注(专家知识)
	SourceUntested   Source = "untested"   // 计划测但未实现
	SourceUntestable Source = "untestable" // 无法程序化测试
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL