[{"data":1,"prerenderedAt":900},["ShallowReactive",2],{"article:llm-eval-loop":3},{"meta":4,"raw":18,"references":19,"prev":47,"next":58,"body":69,"toc":887},{"slug":5,"title":6,"date":7,"updated":8,"tags":9,"summary":14,"draft":15,"lang":16,"minutes":17},"llm-eval-loop","评测闭环：让每次 Prompt 变更都可回归","2026-07-14","2026-10-11",[10,11,12,13],"LLM","评测","Evals","质量","从小规模回归集起步，区分确定性校验、模型裁判与人工评审；给出技术问答 rubric、统计边界、版本记录和上线门槛，避免用平均分掩盖退化。",false,"zh-CN",7,"改一句 prompt 后，多数问题更流畅，少数问题却开始漏掉单位。没有逐用例对照，这类退化很容易被“整体感觉更好”掩盖。评测闭环的目标是**把业务要求变成可重复检查的证据**，而非追求一个漂亮总分。\n\n## 先定义成功，再选裁判\n\n技术问答任务的成功可拆成：答案正确、步骤有效、覆盖所有子问题、表达清楚、引用真实、无越权或隐私泄露。结构合法、用户采纳和最终正确是不同指标。\n\n能用程序确认的先用程序：JSON schema、数值与单位、引用 ID、工具执行状态。开放解释用 rubric 加模型裁判，领域争议和高风险样本保留人工评审。LLM 裁判不是唯一能规模化的方式，也不能代替可计算的标准。\n\n![评测闭环：样本与标准、分层评分、版本对照、灰度观测和坏案例回流](\u002Fdiagrams\u002Feval-loop.svg)\n\n*图 1 · 每次变更保留逐用例结果，线上反馈用于修正评测的代表性。*\n\n## 20 条可以起步，不能证明“提高 3%”\n\n先建一个小回归集，比如 20 条覆盖主要能力与已知失败；它能发现严重回归，不能精确代表生产正确率。冷启动可用人工设计、合成样本和授权真实样本组合，并逐条验证。\n\n数据集分三份：开发集用于调试，锁定回归集用于发布检查，保留集用于阶段性评估泛化。长期只盯同一批样本调 prompt，会对评测集过拟合。线上失败样本脱敏后加入回归；过期或不允许保留的样本需要更新、替换或删除。\n\n```json\n{\n  \"id\": \"batch-count-unit-01\",\n  \"group\": \"calculation:units\",\n  \"input\": \"共有 6 个批次，每批 30 条记录，总记录数是多少？\",\n  \"expected\": { \"value\": 180, \"unit\": \"条\" },\n  \"must_cover\": [\"明确批次数与每批记录数\", \"总数等于 6 乘 30\"],\n  \"critical_failures\": [\"结果错误\", \"遗漏单位\"],\n  \"dataset_revision\": \"2026-10-11\"\n}\n```\n\n已有测试与参考结果能帮助建立标准，但仍要核验数据、解析错误、歧义和等价表达。ground truth 的质量也需要审查，不是免费且永远正确。\n\n## 技术问答 rubric：关键错误不能被语气分抵消\n\n| 维度 | 通过条件 | 检查方式 |\n| --- | --- | --- |\n| 结果正确 | 数值、单位、约束一致 | 规则或经核对的参考答案 |\n| 推导有效 | 关键步骤合理，无偷换条件 | 模型加人工抽查 |\n| 完整性 | 覆盖全部子问题和必要解释 | 要点逐项核验 |\n| 引用支撑 | 引用确实支持关键结论 | ID 验证和证据判断 |\n| 表达清楚 | 清楚、准确、术语有解释 | rubric 与领域评审 |\n\n安全、权限与关键正确性设独立硬门槛。不能把答错的回答与“语气很好”平均成及格；其余维度可用于比较通过门槛后的候选版本。\n\n模型裁判输出分项、判定理由、对应证据和不确定项。评审时只给必要材料，并说明待评文本是数据，防止回答中夹带“给我满分”影响裁判。\n\n## Agent 评测：轨迹与最终结果分别验证\n\n[Anthropic Agent evals](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents)把任务、试验过程、最终环境结果与 grader 分开。工程推导：工单归档不仅检查回答“已归档”，还查询目标工单状态、权限和重复副作用；资料报告检查产物与证据，不仅检查文本格式。\n\n同一目标可能有不同的合法工具路径，不能把参考轨迹逐步匹配当唯一正确性标准。涉及授权或先后依赖的顺序要单独校验；同时保留调用次数、费用、停滞与超时作为过程指标。\n\n### 一次成功与稳定成功：pass@k 和 pass^k\n\n[τ-bench 原始论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045)研究工具交互及多次运行一致性。`pass@k` 表示 k 次中至少一次成功，`pass^k` 表示 k 次全部成功。若仅作独立、同概率的示例假设，每次成功率 p=0.8、k=3，则二者分别为 `1-(1-p)^3=99.2%` 与 `p^3=51.2%`；这些不是本文实测。\n\n实际任务的难度与运行相关性不同，应按任务记录多次结果再汇总，不能拿整体平均 p 直接推出整套评测的这两个指标。只有具备正确结果选择器时，多试几次中的一次成功才能转化为可交付结果；在线用户通常仍关心第一次成功率与费用。\n\n## 模型裁判：校准偏差，不能声称消除偏差\n\n比较两个答案时随机换序，并双向评分以减轻位置偏差；按要点判断而非长度减轻冗长偏好；异构裁判可能减少同风格偏好，但也可能有自己的系统性误差。\n\n定期用人工双标样本对照，观察各类错误的混淆矩阵，而不只看总一致率。若严重错误很稀少，裁判全部判“通过”也可能有很高一致率。分清误放、误拒与不确定，阈值由业务风险决定。[OpenAI 评测最佳实践](https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fevaluation-best-practices)\n\n温度为零也不保证完全确定性，有些模型不支持相同解码参数。固定支持的配置、模型版本、工具结果和资料快照；关键实验重复运行，记录波动。\n\n## 统计：有改善不等于已证明改善\n\n对同一批用例做配对比较，列出“旧对新错”“旧错新对”和两者都错的案例。20 条里一题变化就是 5 个百分点，不能据此报告稳定的 1% 增益。\n\n均值之外看任务分组、置信区间与重复运行分布。正确率可用相应二项区间，配对差值可用配对 bootstrap 等方法；不要把逐次相关的工具轮次当作独立样本。连续试很多 prompt 再挑最高分，会产生选择偏差，应使用保留集确认。\n\n线上 A\u002FB 随机化单位要与业务一致，同一用户跨会话的影响不能当完全独立。采纳率、重问率和放弃率是体验代理，不等于答案正确；曝光顺序、难度和延迟都可能影响它们。\n\n## 接入发布流程\n\n```text\n固定数据集、模型配置、工具和资料版本\n运行基线与候选，保存原始输出和逐项评分\n检查硬门槛、关键分组退化、延迟与费用\n人工看新增失败及裁判不确定项\n灰度上线，观察真实业务指标\n发现问题则回滚，并将有效样本加入回归\n```\n\n这些步骤可接入任意支持版本化配置的工具。模型、prompt、检索索引、工具 schema、裁判 rubric 和数据集都要记录版本；仅保存“这次 0.86 分”无法复现。调用失败也记录，不能悄悄从分母移除。\n\n具体 CI 门槛如“关键集合全部通过、开放题无明显退化、费用不超预算”是业务决策，不能照抄一个通用裁判分数阈值。评测调用也有费用、并发与数据访问要求，应设独立预算。\n\n## 错误分类帮助定位，但不是唯一映射\n\n检索缺证据、生成推理错、引用错、格式损坏、过度拒绝、越权分别维护案例。一个回答可能同时有多种错误；通过人工证据替换、禁用记忆、确定性工具替换等对照实验缩小原因，避免见到“幻觉”就只改 prompt。\n\n## 落地检查\n\n1. 定义业务成功和硬失败，程序检查优先。\n2. 小集合起步，分开发、回归与保留集。\n3. 裁判与人工校准，严重错误单独统计。\n4. 固定版本，重复关键实验，报告样本量与波动。\n5. CI 保留逐用例差异，灰度和回滚一起设计。\n6. 坏案例依法留存和更新，线上指标验证代表性。\n\n## 参考资料\n\n- [OpenAI — Evaluation best practices](https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fevaluation-best-practices)\n- [Anthropic — Multi-agent research system](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fmulti-agent-research-system)\n- [Ragas — Metrics](https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002F)\n- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685)\n- [Anthropic — Demystifying evals for AI agents](https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents)\n- [τ-bench — Tool-Agent-User Interaction in Real-World Domains](https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045)",[20,25,30,34,39,43],{"title":21,"url":22,"type":23,"note":24,"checked":8},"OpenAI — Evaluation best practices","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fevaluation-best-practices","官方文档","数据集、grader、持续评测与偏差校准建议；人工与程序检查仍不可省略。",{"title":26,"url":27,"type":28,"note":29,"checked":8},"Anthropic — Multi-agent research system","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fmulti-agent-research-system","工程报告","研究型多 Agent 的设计、评测与成本经验；报告收益不是所有多 Agent 的定律。",{"title":31,"url":32,"type":23,"note":33,"checked":8},"Ragas — Metrics","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Fmetrics\u002Favailable_metrics\u002F","RAG 评测指标的定义与适用问题；指标不能当一对一修复指令。",{"title":35,"url":36,"type":37,"note":38,"checked":8},"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena","https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.05685","研究论文","LLM-as-a-judge 的位置、冗长等偏差与评测研究；一致率不代表裁判无偏。",{"title":40,"url":41,"type":28,"note":42,"checked":8},"Anthropic — Demystifying evals for AI agents","https:\u002F\u002Fwww.anthropic.com\u002Fengineering\u002Fdemystifying-evals-for-ai-agents","任务、试验轨迹、结果与 grader 的评测方法；不代替本业务验收标准。",{"title":44,"url":45,"type":37,"note":46,"checked":8},"τ-bench — Tool-Agent-User Interaction in Real-World Domains","https:\u002F\u002Farxiv.org\u002Fabs\u002F2406.12045","工具对话后的数据库结果验证与多次一致性评测；论文分数有模型和环境边界。",{"slug":48,"title":49,"date":50,"updated":8,"tags":51,"summary":56,"draft":15,"lang":16,"minutes":57},"device-cloud-api-sync","端云协同 API 设计：离线同步、弱网重试与版本兼容","2026-08-18",[52,53,54,55],"架构","API 设计","离线优先","端云协同","用本地事务与 outbox 保住离线记录，以业务操作 ID 去重；设计逐项确认、游标失效、删除传播、冲突策略和内容包原子切换，并验证断电与重连。",5,{"slug":59,"title":60,"date":61,"updated":8,"tags":62,"summary":67,"draft":15,"lang":16,"minutes":68},"multi-agent-communication","多 Agent 如何通信：任务书、结构化回传与 A2A 协议","2026-06-16",[63,64,52,65,66],"Agent","多智能体","A2A","MCP","从可并行任务出发，设计输入、状态、成果与失败契约；说明共享文件和跨机引用的区别、MCP 与 A2A 的职责，以及协调成本与验收方法。",6,{"type":70,"children":71},"root",[72,88,95,100,105,114,123,129,134,139,535,540,546,669,674,679,685,698,703,710,753,758,764,769,780,785,791,796,801,806,811,821,826,831,837,842,847,882],{"type":73,"tag":74,"props":75,"children":76},"element","p",{},[77,80,86],{"type":78,"value":79},"text","改一句 prompt 后，多数问题更流畅，少数问题却开始漏掉单位。没有逐用例对照，这类退化很容易被“整体感觉更好”掩盖。评测闭环的目标是",{"type":73,"tag":81,"props":82,"children":83},"strong",{},[84],{"type":78,"value":85},"把业务要求变成可重复检查的证据",{"type":78,"value":87},"，而非追求一个漂亮总分。",{"type":73,"tag":89,"props":90,"children":92},"h2",{"id":91},"先定义成功再选裁判",[93],{"type":78,"value":94},"先定义成功，再选裁判",{"type":73,"tag":74,"props":96,"children":97},{},[98],{"type":78,"value":99},"技术问答任务的成功可拆成：答案正确、步骤有效、覆盖所有子问题、表达清楚、引用真实、无越权或隐私泄露。结构合法、用户采纳和最终正确是不同指标。",{"type":73,"tag":74,"props":101,"children":102},{},[103],{"type":78,"value":104},"能用程序确认的先用程序：JSON schema、数值与单位、引用 ID、工具执行状态。开放解释用 rubric 加模型裁判，领域争议和高风险样本保留人工评审。LLM 裁判不是唯一能规模化的方式，也不能代替可计算的标准。",{"type":73,"tag":74,"props":106,"children":107},{},[108],{"type":73,"tag":109,"props":110,"children":113},"img",{"alt":111,"src":112},"评测闭环：样本与标准、分层评分、版本对照、灰度观测和坏案例回流","\u002Fdiagrams\u002Feval-loop.svg",[],{"type":73,"tag":74,"props":115,"children":116},{},[117],{"type":73,"tag":118,"props":119,"children":120},"em",{},[121],{"type":78,"value":122},"图 1 · 每次变更保留逐用例结果，线上反馈用于修正评测的代表性。",{"type":73,"tag":89,"props":124,"children":126},{"id":125},"_20-条可以起步不能证明提高-3",[127],{"type":78,"value":128},"20 条可以起步，不能证明“提高 3%”",{"type":73,"tag":74,"props":130,"children":131},{},[132],{"type":78,"value":133},"先建一个小回归集，比如 20 条覆盖主要能力与已知失败；它能发现严重回归，不能精确代表生产正确率。冷启动可用人工设计、合成样本和授权真实样本组合，并逐条验证。",{"type":73,"tag":74,"props":135,"children":136},{},[137],{"type":78,"value":138},"数据集分三份：开发集用于调试，锁定回归集用于发布检查，保留集用于阶段性评估泛化。长期只盯同一批样本调 prompt，会对评测集过拟合。线上失败样本脱敏后加入回归；过期或不允许保留的样本需要更新、替换或删除。",{"type":73,"tag":140,"props":141,"children":146},"pre",{"className":142,"code":143,"language":144,"meta":145,"style":145},"language-json shiki shiki-themes vitesse-light vitesse-light","{\n  \"id\": \"batch-count-unit-01\",\n  \"group\": \"calculation:units\",\n  \"input\": \"共有 6 个批次，每批 30 条记录，总记录数是多少？\",\n  \"expected\": { \"value\": 180, \"unit\": \"条\" },\n  \"must_cover\": [\"明确批次数与每批记录数\", \"总数等于 6 乘 30\"],\n  \"critical_failures\": [\"结果错误\", \"遗漏单位\"],\n  \"dataset_revision\": \"2026-10-11\"\n}\n","json","",[147],{"type":73,"tag":148,"props":149,"children":150},"code",{"__ignoreMap":145},[151,163,210,248,286,374,434,492,526],{"type":73,"tag":152,"props":153,"children":156},"span",{"class":154,"line":155},"line",1,[157],{"type":73,"tag":152,"props":158,"children":160},{"style":159},"--shiki-default:#999999;--shiki-dark:#999999",[161],{"type":78,"value":162},"{\n",{"type":73,"tag":152,"props":164,"children":166},{"class":154,"line":165},2,[167,173,179,184,189,195,201,205],{"type":73,"tag":152,"props":168,"children":170},{"style":169},"--shiki-default:#99841877;--shiki-dark:#99841877",[171],{"type":78,"value":172},"  \"",{"type":73,"tag":152,"props":174,"children":176},{"style":175},"--shiki-default:#998418;--shiki-dark:#998418",[177],{"type":78,"value":178},"id",{"type":73,"tag":152,"props":180,"children":181},{"style":169},[182],{"type":78,"value":183},"\"",{"type":73,"tag":152,"props":185,"children":186},{"style":159},[187],{"type":78,"value":188},":",{"type":73,"tag":152,"props":190,"children":192},{"style":191},"--shiki-default:#B5695977;--shiki-dark:#B5695977",[193],{"type":78,"value":194}," \"",{"type":73,"tag":152,"props":196,"children":198},{"style":197},"--shiki-default:#B56959;--shiki-dark:#B56959",[199],{"type":78,"value":200},"batch-count-unit-01",{"type":73,"tag":152,"props":202,"children":203},{"style":191},[204],{"type":78,"value":183},{"type":73,"tag":152,"props":206,"children":207},{"style":159},[208],{"type":78,"value":209},",\n",{"type":73,"tag":152,"props":211,"children":213},{"class":154,"line":212},3,[214,218,223,227,231,235,240,244],{"type":73,"tag":152,"props":215,"children":216},{"style":169},[217],{"type":78,"value":172},{"type":73,"tag":152,"props":219,"children":220},{"style":175},[221],{"type":78,"value":222},"group",{"type":73,"tag":152,"props":224,"children":225},{"style":169},[226],{"type":78,"value":183},{"type":73,"tag":152,"props":228,"children":229},{"style":159},[230],{"type":78,"value":188},{"type":73,"tag":152,"props":232,"children":233},{"style":191},[234],{"type":78,"value":194},{"type":73,"tag":152,"props":236,"children":237},{"style":197},[238],{"type":78,"value":239},"calculation:units",{"type":73,"tag":152,"props":241,"children":242},{"style":191},[243],{"type":78,"value":183},{"type":73,"tag":152,"props":245,"children":246},{"style":159},[247],{"type":78,"value":209},{"type":73,"tag":152,"props":249,"children":251},{"class":154,"line":250},4,[252,256,261,265,269,273,278,282],{"type":73,"tag":152,"props":253,"children":254},{"style":169},[255],{"type":78,"value":172},{"type":73,"tag":152,"props":257,"children":258},{"style":175},[259],{"type":78,"value":260},"input",{"type":73,"tag":152,"props":262,"children":263},{"style":169},[264],{"type":78,"value":183},{"type":73,"tag":152,"props":266,"children":267},{"style":159},[268],{"type":78,"value":188},{"type":73,"tag":152,"props":270,"children":271},{"style":191},[272],{"type":78,"value":194},{"type":73,"tag":152,"props":274,"children":275},{"style":197},[276],{"type":78,"value":277},"共有 6 个批次，每批 30 条记录，总记录数是多少？",{"type":73,"tag":152,"props":279,"children":280},{"style":191},[281],{"type":78,"value":183},{"type":73,"tag":152,"props":283,"children":284},{"style":159},[285],{"type":78,"value":209},{"type":73,"tag":152,"props":287,"children":288},{"class":154,"line":57},[289,293,298,302,306,311,315,320,324,328,334,339,343,348,352,356,360,365,369],{"type":73,"tag":152,"props":290,"children":291},{"style":169},[292],{"type":78,"value":172},{"type":73,"tag":152,"props":294,"children":295},{"style":175},[296],{"type":78,"value":297},"expected",{"type":73,"tag":152,"props":299,"children":300},{"style":169},[301],{"type":78,"value":183},{"type":73,"tag":152,"props":303,"children":304},{"style":159},[305],{"type":78,"value":188},{"type":73,"tag":152,"props":307,"children":308},{"style":159},[309],{"type":78,"value":310}," {",{"type":73,"tag":152,"props":312,"children":313},{"style":169},[314],{"type":78,"value":194},{"type":73,"tag":152,"props":316,"children":317},{"style":175},[318],{"type":78,"value":319},"value",{"type":73,"tag":152,"props":321,"children":322},{"style":169},[323],{"type":78,"value":183},{"type":73,"tag":152,"props":325,"children":326},{"style":159},[327],{"type":78,"value":188},{"type":73,"tag":152,"props":329,"children":331},{"style":330},"--shiki-default:#2F798A;--shiki-dark:#2F798A",[332],{"type":78,"value":333}," 180",{"type":73,"tag":152,"props":335,"children":336},{"style":159},[337],{"type":78,"value":338},",",{"type":73,"tag":152,"props":340,"children":341},{"style":169},[342],{"type":78,"value":194},{"type":73,"tag":152,"props":344,"children":345},{"style":175},[346],{"type":78,"value":347},"unit",{"type":73,"tag":152,"props":349,"children":350},{"style":169},[351],{"type":78,"value":183},{"type":73,"tag":152,"props":353,"children":354},{"style":159},[355],{"type":78,"value":188},{"type":73,"tag":152,"props":357,"children":358},{"style":191},[359],{"type":78,"value":194},{"type":73,"tag":152,"props":361,"children":362},{"style":197},[363],{"type":78,"value":364},"条",{"type":73,"tag":152,"props":366,"children":367},{"style":191},[368],{"type":78,"value":183},{"type":73,"tag":152,"props":370,"children":371},{"style":159},[372],{"type":78,"value":373}," },\n",{"type":73,"tag":152,"props":375,"children":376},{"class":154,"line":68},[377,381,386,390,394,399,403,408,412,416,420,425,429],{"type":73,"tag":152,"props":378,"children":379},{"style":169},[380],{"type":78,"value":172},{"type":73,"tag":152,"props":382,"children":383},{"style":175},[384],{"type":78,"value":385},"must_cover",{"type":73,"tag":152,"props":387,"children":388},{"style":169},[389],{"type":78,"value":183},{"type":73,"tag":152,"props":391,"children":392},{"style":159},[393],{"type":78,"value":188},{"type":73,"tag":152,"props":395,"children":396},{"style":159},[397],{"type":78,"value":398}," [",{"type":73,"tag":152,"props":400,"children":401},{"style":191},[402],{"type":78,"value":183},{"type":73,"tag":152,"props":404,"children":405},{"style":197},[406],{"type":78,"value":407},"明确批次数与每批记录数",{"type":73,"tag":152,"props":409,"children":410},{"style":191},[411],{"type":78,"value":183},{"type":73,"tag":152,"props":413,"children":414},{"style":159},[415],{"type":78,"value":338},{"type":73,"tag":152,"props":417,"children":418},{"style":191},[419],{"type":78,"value":194},{"type":73,"tag":152,"props":421,"children":422},{"style":197},[423],{"type":78,"value":424},"总数等于 6 乘 30",{"type":73,"tag":152,"props":426,"children":427},{"style":191},[428],{"type":78,"value":183},{"type":73,"tag":152,"props":430,"children":431},{"style":159},[432],{"type":78,"value":433},"],\n",{"type":73,"tag":152,"props":435,"children":436},{"class":154,"line":17},[437,441,446,450,454,458,462,467,471,475,479,484,488],{"type":73,"tag":152,"props":438,"children":439},{"style":169},[440],{"type":78,"value":172},{"type":73,"tag":152,"props":442,"children":443},{"style":175},[444],{"type":78,"value":445},"critical_failures",{"type":73,"tag":152,"props":447,"children":448},{"style":169},[449],{"type":78,"value":183},{"type":73,"tag":152,"props":451,"children":452},{"style":159},[453],{"type":78,"value":188},{"type":73,"tag":152,"props":455,"children":456},{"style":159},[457],{"type":78,"value":398},{"type":73,"tag":152,"props":459,"children":460},{"style":191},[461],{"type":78,"value":183},{"type":73,"tag":152,"props":463,"children":464},{"style":197},[465],{"type":78,"value":466},"结果错误",{"type":73,"tag":152,"props":468,"children":469},{"style":191},[470],{"type":78,"value":183},{"type":73,"tag":152,"props":472,"children":473},{"style":159},[474],{"type":78,"value":338},{"type":73,"tag":152,"props":476,"children":477},{"style":191},[478],{"type":78,"value":194},{"type":73,"tag":152,"props":480,"children":481},{"style":197},[482],{"type":78,"value":483},"遗漏单位",{"type":73,"tag":152,"props":485,"children":486},{"style":191},[487],{"type":78,"value":183},{"type":73,"tag":152,"props":489,"children":490},{"style":159},[491],{"type":78,"value":433},{"type":73,"tag":152,"props":493,"children":495},{"class":154,"line":494},8,[496,500,505,509,513,517,521],{"type":73,"tag":152,"props":497,"children":498},{"style":169},[499],{"type":78,"value":172},{"type":73,"tag":152,"props":501,"children":502},{"style":175},[503],{"type":78,"value":504},"dataset_revision",{"type":73,"tag":152,"props":506,"children":507},{"style":169},[508],{"type":78,"value":183},{"type":73,"tag":152,"props":510,"children":511},{"style":159},[512],{"type":78,"value":188},{"type":73,"tag":152,"props":514,"children":515},{"style":191},[516],{"type":78,"value":194},{"type":73,"tag":152,"props":518,"children":519},{"style":197},[520],{"type":78,"value":8},{"type":73,"tag":152,"props":522,"children":523},{"style":191},[524],{"type":78,"value":525},"\"\n",{"type":73,"tag":152,"props":527,"children":529},{"class":154,"line":528},9,[530],{"type":73,"tag":152,"props":531,"children":532},{"style":159},[533],{"type":78,"value":534},"}\n",{"type":73,"tag":74,"props":536,"children":537},{},[538],{"type":78,"value":539},"已有测试与参考结果能帮助建立标准，但仍要核验数据、解析错误、歧义和等价表达。ground truth 的质量也需要审查，不是免费且永远正确。",{"type":73,"tag":89,"props":541,"children":543},{"id":542},"技术问答-rubric关键错误不能被语气分抵消",[544],{"type":78,"value":545},"技术问答 rubric：关键错误不能被语气分抵消",{"type":73,"tag":547,"props":548,"children":549},"table",{},[550,574],{"type":73,"tag":551,"props":552,"children":553},"thead",{},[554],{"type":73,"tag":555,"props":556,"children":557},"tr",{},[558,564,569],{"type":73,"tag":559,"props":560,"children":561},"th",{},[562],{"type":78,"value":563},"维度",{"type":73,"tag":559,"props":565,"children":566},{},[567],{"type":78,"value":568},"通过条件",{"type":73,"tag":559,"props":570,"children":571},{},[572],{"type":78,"value":573},"检查方式",{"type":73,"tag":575,"props":576,"children":577},"tbody",{},[578,597,615,633,651],{"type":73,"tag":555,"props":579,"children":580},{},[581,587,592],{"type":73,"tag":582,"props":583,"children":584},"td",{},[585],{"type":78,"value":586},"结果正确",{"type":73,"tag":582,"props":588,"children":589},{},[590],{"type":78,"value":591},"数值、单位、约束一致",{"type":73,"tag":582,"props":593,"children":594},{},[595],{"type":78,"value":596},"规则或经核对的参考答案",{"type":73,"tag":555,"props":598,"children":599},{},[600,605,610],{"type":73,"tag":582,"props":601,"children":602},{},[603],{"type":78,"value":604},"推导有效",{"type":73,"tag":582,"props":606,"children":607},{},[608],{"type":78,"value":609},"关键步骤合理，无偷换条件",{"type":73,"tag":582,"props":611,"children":612},{},[613],{"type":78,"value":614},"模型加人工抽查",{"type":73,"tag":555,"props":616,"children":617},{},[618,623,628],{"type":73,"tag":582,"props":619,"children":620},{},[621],{"type":78,"value":622},"完整性",{"type":73,"tag":582,"props":624,"children":625},{},[626],{"type":78,"value":627},"覆盖全部子问题和必要解释",{"type":73,"tag":582,"props":629,"children":630},{},[631],{"type":78,"value":632},"要点逐项核验",{"type":73,"tag":555,"props":634,"children":635},{},[636,641,646],{"type":73,"tag":582,"props":637,"children":638},{},[639],{"type":78,"value":640},"引用支撑",{"type":73,"tag":582,"props":642,"children":643},{},[644],{"type":78,"value":645},"引用确实支持关键结论",{"type":73,"tag":582,"props":647,"children":648},{},[649],{"type":78,"value":650},"ID 验证和证据判断",{"type":73,"tag":555,"props":652,"children":653},{},[654,659,664],{"type":73,"tag":582,"props":655,"children":656},{},[657],{"type":78,"value":658},"表达清楚",{"type":73,"tag":582,"props":660,"children":661},{},[662],{"type":78,"value":663},"清楚、准确、术语有解释",{"type":73,"tag":582,"props":665,"children":666},{},[667],{"type":78,"value":668},"rubric 与领域评审",{"type":73,"tag":74,"props":670,"children":671},{},[672],{"type":78,"value":673},"安全、权限与关键正确性设独立硬门槛。不能把答错的回答与“语气很好”平均成及格；其余维度可用于比较通过门槛后的候选版本。",{"type":73,"tag":74,"props":675,"children":676},{},[677],{"type":78,"value":678},"模型裁判输出分项、判定理由、对应证据和不确定项。评审时只给必要材料，并说明待评文本是数据，防止回答中夹带“给我满分”影响裁判。",{"type":73,"tag":89,"props":680,"children":682},{"id":681},"agent-评测轨迹与最终结果分别验证",[683],{"type":78,"value":684},"Agent 评测：轨迹与最终结果分别验证",{"type":73,"tag":74,"props":686,"children":687},{},[688,696],{"type":73,"tag":689,"props":690,"children":693},"a",{"href":41,"rel":691},[692],"nofollow",[694],{"type":78,"value":695},"Anthropic Agent evals",{"type":78,"value":697},"把任务、试验过程、最终环境结果与 grader 分开。工程推导：工单归档不仅检查回答“已归档”，还查询目标工单状态、权限和重复副作用；资料报告检查产物与证据，不仅检查文本格式。",{"type":73,"tag":74,"props":699,"children":700},{},[701],{"type":78,"value":702},"同一目标可能有不同的合法工具路径，不能把参考轨迹逐步匹配当唯一正确性标准。涉及授权或先后依赖的顺序要单独校验；同时保留调用次数、费用、停滞与超时作为过程指标。",{"type":73,"tag":704,"props":705,"children":707},"h3",{"id":706},"一次成功与稳定成功passk-和-passk",[708],{"type":78,"value":709},"一次成功与稳定成功：pass@k 和 pass^k",{"type":73,"tag":74,"props":711,"children":712},{},[713,719,721,727,729,735,737,743,745,751],{"type":73,"tag":689,"props":714,"children":716},{"href":45,"rel":715},[692],[717],{"type":78,"value":718},"τ-bench 原始论文",{"type":78,"value":720},"研究工具交互及多次运行一致性。",{"type":73,"tag":148,"props":722,"children":724},{"className":723},[],[725],{"type":78,"value":726},"pass@k",{"type":78,"value":728}," 表示 k 次中至少一次成功，",{"type":73,"tag":148,"props":730,"children":732},{"className":731},[],[733],{"type":78,"value":734},"pass^k",{"type":78,"value":736}," 表示 k 次全部成功。若仅作独立、同概率的示例假设，每次成功率 p=0.8、k=3，则二者分别为 ",{"type":73,"tag":148,"props":738,"children":740},{"className":739},[],[741],{"type":78,"value":742},"1-(1-p)^3=99.2%",{"type":78,"value":744}," 与 ",{"type":73,"tag":148,"props":746,"children":748},{"className":747},[],[749],{"type":78,"value":750},"p^3=51.2%",{"type":78,"value":752},"；这些不是本文实测。",{"type":73,"tag":74,"props":754,"children":755},{},[756],{"type":78,"value":757},"实际任务的难度与运行相关性不同，应按任务记录多次结果再汇总，不能拿整体平均 p 直接推出整套评测的这两个指标。只有具备正确结果选择器时，多试几次中的一次成功才能转化为可交付结果；在线用户通常仍关心第一次成功率与费用。",{"type":73,"tag":89,"props":759,"children":761},{"id":760},"模型裁判校准偏差不能声称消除偏差",[762],{"type":78,"value":763},"模型裁判：校准偏差，不能声称消除偏差",{"type":73,"tag":74,"props":765,"children":766},{},[767],{"type":78,"value":768},"比较两个答案时随机换序，并双向评分以减轻位置偏差；按要点判断而非长度减轻冗长偏好；异构裁判可能减少同风格偏好，但也可能有自己的系统性误差。",{"type":73,"tag":74,"props":770,"children":771},{},[772,774],{"type":78,"value":773},"定期用人工双标样本对照，观察各类错误的混淆矩阵，而不只看总一致率。若严重错误很稀少，裁判全部判“通过”也可能有很高一致率。分清误放、误拒与不确定，阈值由业务风险决定。",{"type":73,"tag":689,"props":775,"children":777},{"href":22,"rel":776},[692],[778],{"type":78,"value":779},"OpenAI 评测最佳实践",{"type":73,"tag":74,"props":781,"children":782},{},[783],{"type":78,"value":784},"温度为零也不保证完全确定性，有些模型不支持相同解码参数。固定支持的配置、模型版本、工具结果和资料快照；关键实验重复运行，记录波动。",{"type":73,"tag":89,"props":786,"children":788},{"id":787},"统计有改善不等于已证明改善",[789],{"type":78,"value":790},"统计：有改善不等于已证明改善",{"type":73,"tag":74,"props":792,"children":793},{},[794],{"type":78,"value":795},"对同一批用例做配对比较，列出“旧对新错”“旧错新对”和两者都错的案例。20 条里一题变化就是 5 个百分点，不能据此报告稳定的 1% 增益。",{"type":73,"tag":74,"props":797,"children":798},{},[799],{"type":78,"value":800},"均值之外看任务分组、置信区间与重复运行分布。正确率可用相应二项区间，配对差值可用配对 bootstrap 等方法；不要把逐次相关的工具轮次当作独立样本。连续试很多 prompt 再挑最高分，会产生选择偏差，应使用保留集确认。",{"type":73,"tag":74,"props":802,"children":803},{},[804],{"type":78,"value":805},"线上 A\u002FB 随机化单位要与业务一致，同一用户跨会话的影响不能当完全独立。采纳率、重问率和放弃率是体验代理，不等于答案正确；曝光顺序、难度和延迟都可能影响它们。",{"type":73,"tag":89,"props":807,"children":809},{"id":808},"接入发布流程",[810],{"type":78,"value":808},{"type":73,"tag":140,"props":812,"children":816},{"className":813,"code":815,"language":78,"meta":145},[814],"language-text","固定数据集、模型配置、工具和资料版本\n运行基线与候选，保存原始输出和逐项评分\n检查硬门槛、关键分组退化、延迟与费用\n人工看新增失败及裁判不确定项\n灰度上线，观察真实业务指标\n发现问题则回滚，并将有效样本加入回归\n",[817],{"type":73,"tag":148,"props":818,"children":819},{"__ignoreMap":145},[820],{"type":78,"value":815},{"type":73,"tag":74,"props":822,"children":823},{},[824],{"type":78,"value":825},"这些步骤可接入任意支持版本化配置的工具。模型、prompt、检索索引、工具 schema、裁判 rubric 和数据集都要记录版本；仅保存“这次 0.86 分”无法复现。调用失败也记录，不能悄悄从分母移除。",{"type":73,"tag":74,"props":827,"children":828},{},[829],{"type":78,"value":830},"具体 CI 门槛如“关键集合全部通过、开放题无明显退化、费用不超预算”是业务决策，不能照抄一个通用裁判分数阈值。评测调用也有费用、并发与数据访问要求，应设独立预算。",{"type":73,"tag":89,"props":832,"children":834},{"id":833},"错误分类帮助定位但不是唯一映射",[835],{"type":78,"value":836},"错误分类帮助定位，但不是唯一映射",{"type":73,"tag":74,"props":838,"children":839},{},[840],{"type":78,"value":841},"检索缺证据、生成推理错、引用错、格式损坏、过度拒绝、越权分别维护案例。一个回答可能同时有多种错误；通过人工证据替换、禁用记忆、确定性工具替换等对照实验缩小原因，避免见到“幻觉”就只改 prompt。",{"type":73,"tag":89,"props":843,"children":845},{"id":844},"落地检查",[846],{"type":78,"value":844},{"type":73,"tag":848,"props":849,"children":850},"ol",{},[851,857,862,867,872,877],{"type":73,"tag":852,"props":853,"children":854},"li",{},[855],{"type":78,"value":856},"定义业务成功和硬失败，程序检查优先。",{"type":73,"tag":852,"props":858,"children":859},{},[860],{"type":78,"value":861},"小集合起步，分开发、回归与保留集。",{"type":73,"tag":852,"props":863,"children":864},{},[865],{"type":78,"value":866},"裁判与人工校准，严重错误单独统计。",{"type":73,"tag":852,"props":868,"children":869},{},[870],{"type":78,"value":871},"固定版本，重复关键实验，报告样本量与波动。",{"type":73,"tag":852,"props":873,"children":874},{},[875],{"type":78,"value":876},"CI 保留逐用例差异，灰度和回滚一起设计。",{"type":73,"tag":852,"props":878,"children":879},{},[880],{"type":78,"value":881},"坏案例依法留存和更新，线上指标验证代表性。",{"type":73,"tag":89,"props":883,"children":885},{"id":884},"参考资料",[886],{"type":78,"value":884},[888,889,890,891,894,895,896,897,898,899],{"id":91,"depth":165,"text":94},{"id":125,"depth":165,"text":128},{"id":542,"depth":165,"text":545},{"id":681,"depth":165,"text":684,"children":892},[893],{"id":706,"depth":212,"text":709},{"id":760,"depth":165,"text":763},{"id":787,"depth":165,"text":790},{"id":808,"depth":165,"text":808},{"id":833,"depth":165,"text":836},{"id":844,"depth":165,"text":844},{"id":884,"depth":165,"text":884},1792400815607]