// vision.go - the cross-provider vision (image-to-text) capability // contract. PROMOTED here from the minimax package (2026-06-04): vision // was originally minimax-only (the vlm /v1/coding_plan/vlm endpoint) and // the types lived in providers/minimax under the rule-of-two ("keep the // contract local until a second provider needs the same shape"). The // OpenAI provider is now the second implementer (an OpenAI-compatible // vision model -- e.g. a local oMLX gemma4 reached via OPENAI_BASE_URL -- // extracts images through /v1/chat/completions image_url content), so the // contract graduates to flyto as a shared capability. // // VisionProvider is OPTIONAL: not every ModelProvider implements it. A // provider exposes vision by implementing ExtractVision; consumers type- // assert (provider, ok := p.(flyto.VisionProvider)) and degrade when // absent. This is deliberately SEPARATE from the streaming Provider.Stream // path: vision here is a single-shot vendor-endpoint RPC (minimax) or a // single non-streamed chat call (openai), NOT the engine's multi-turn // orchestration. Wiring image BlockImage into Provider.Stream is a larger, // future change; this contract keeps "extract one image" as its own seam. // // vision.go - 跨 provider 的视觉 (图转文) 能力契约. 2026-06-04 从 minimax // 包**提上来**: 视觉原本只 minimax 有 (vlm /v1/coding_plan/vlm 端点), 按 // rule-of-two 类型留在 providers/minimax ("够两家再 promote"). 现在 OpenAI // provider 是第二个实现者 (OpenAI 兼容视觉模型 -- 例如经 OPENAI_BASE_URL // 接的本地 oMLX gemma4 -- 走 /v1/chat/completions image_url content 抽图), // 故契约升到 flyto 作共享能力. // // VisionProvider 是**可选**的: 不是每个 ModelProvider 都实现. provider 实现 // ExtractVision 即暴露视觉; 消费者 type-assert (provider, ok := p.(flyto. // VisionProvider)) 并在缺失时降级. 它刻意与 streaming Provider.Stream 路径 // **分开**: 这里的视觉是单次 vendor 端点 RPC (minimax) 或单次非流式 chat // (openai), 不是引擎的多轮编排. 把图片 BlockImage 接进 Provider.Stream 是 // 更大的将来改动; 本契约把 "抽一张图" 留作独立接缝. package flyto import "context" // VisionRequest is a single-shot image extraction request. Image is RAW // bytes (not base64); each provider encodes for its own wire (minimax: a // data URI for /v1/coding_plan/vlm; openai: a data URI in an image_url // content block). MediaType defaults to image/png when blank. Model is the // provider model id -- REQUIRED by openai (per-request selection, e.g. // gemma4-moe-26b-a4b-q6), IGNORED by minimax (its vlm endpoint is model- // locked server-side). MaxTokens caps generation -- 0 means provider // default; matters for reasoning models (oMLX gemma4 spends thousands of // tokens in reasoning_content before emitting the answer in content, so a // low cap clips the JSON). // // VisionRequest 是单次图像抽取请求. Image 是**原始**字节 (非 base64); 各 // provider 自行编码上线 (minimax: data URI 发 /v1/coding_plan/vlm; openai: // data URI 放 image_url content 块). MediaType 留空默认 image/png. Model 是 // provider 模型 id -- openai **必填** (per-request 选, 如 gemma4-moe-26b-a4b-q6), // minimax **忽略** (其 vlm 端点服务端锁模型). MaxTokens 限生成 -- 0 表示 // provider 默认; 对推理模型要紧 (oMLX gemma4 在 reasoning_content 里花数千 // token 推理才在 content 出答案, cap 太低会切掉 JSON). type VisionRequest struct { Prompt string // instructions for the model Image []byte // raw image bytes (not base64) MediaType string // e.g. "image/png", "image/jpeg"; empty -> "image/png" Model string // provider model id; required by openai, ignored by minimax MaxTokens int // 0 = provider default } // VisionResponse carries the model's extracted content -- the raw string // the model returned (typically JSON when the prompt asked for structured // output). The consumer parses it; reasoning-model providers strip their // own thinking and return only the answer here. // // VisionResponse 携带模型抽取内容 -- 模型返的原始字符串 (prompt 要结构化 // 输出时通常 JSON). 消费者解析; 推理模型 provider 剥掉自己的思考, 这里只 // 返答案. type VisionResponse struct { Content string } // VisionProvider is the optional vision capability. *minimax.Provider and // *openai.Provider satisfy it. Consumers depend on this interface (not a // concrete provider) so the VLM backend is swappable / injectable / // mockable. Single-shot + stateless: safe to share across goroutines. // // VisionProvider 是可选视觉能力. *minimax.Provider 与 *openai.Provider 满足 // 它. 消费者依赖此接口 (非具体 provider), 让 VLM 后端可换 / 可注入 / 可 mock. // 单次 + 无状态: 可跨 goroutine 共享. type VisionProvider interface { ExtractVision(ctx context.Context, req *VisionRequest) (*VisionResponse, error) }