Ch22 LLM Evaluation 方法学 · 信息图

---

信息图:LLM Eval 全景

flowchart TB
  A[LLM Evaluation] --> B[传统指标]
  A --> C[Benchmark]
  A --> D[人类偏好]
  A --> E[LLM-as-Judge]
  A --> F[Contamination]
  A --> G[Harness]

  B --> B1[BLEU/ROUGE]
  B --> B2[METEOR/chrF]
  B --> B3[BERTScore]
  B --> B4[Perplexity/WER]

  C --> C1[知识 MMLU]
  C --> C2[数学 GSM8K]
  C --> C3[代码 HumanEval]
  C --> C4[综合 BBH]
  C --> C5[中文 C-Eval]

  D --> D1[Arena Elo]
  D --> D2[Bradley-Terry]
  D --> D3[TrueSkill]

  E --> E1[MT-Bench]
  E --> E2[AlpacaEval]
  E --> E3[Pairwise Judge]

  F --> F1[检测方法]
  F --> F2[缓解策略]
  F --> F3[历史争议]

  G --> G1[lm-eval-harness]
  G --> G2[OpenCompass]
  G --> G3[HELM]

  classDef core fill:#f9f,stroke:#333,stroke-width:2px
  class A core

---

信息图:Benchmark 知识体系

flowchart LR
  K[知识] --> K1[MMLU\n57 学科]
  K --> K2[MMLU-Pro\n推理版]
  K --> K3[GPQA\nGoogle]
  M[数学] --> M1[GSM8K\n小学]
  M --> M2[MATH\n竞赛]
  M --> M3[MATH-500]
  CD[代码] --> C1[HumanEval\n164 题]
  CD --> C2[MBPP]
  CD --> C3[LiveCodeBench\n每月更新]
  CD --> C4[SWE-Bench\n真实 GitHub]
  AG[Agent] --> A1[WebArena]
  AG --> A2[OSWorld]
  AG --> A3[GAIA]
  ZH[中文] --> Z1[C-Eval]
  ZH --> Z2[CMMLU]
  TR[真实性] --> T1[TruthfulQA]
  TR --> T2[HaluEval]

---

信息图:Elo Rating 算法

flowchart TB
  A[用户投票\nA vs B] --> Q{预测胜负}
  Q -->|P_A > P_B| P1["P_A = 1/(1+10^((R_B-R_A)/400))"]
  P1 --> R1["R_A_new = R_A + 32 × 1 - P_A"]
  P1 --> R2["R_B_new = R_B + 32 × 0 - P_B"]
  Q -->|P_A < P_B| P2["P_B = 1/(1+10^((R_A-R_B)/400))"]
  P2 --> R3["R_A_new = R_A + 32 × 0 - P_A"]
  P2 --> R4["R_B_new = R_B + 32 × 1 - P_B"]

  R1 --> L[LMSYS Chatbot Arena\n100 万+ 投票\nBradley-Terry 模型]
  R3 --> L

---

信息图:LLM-as-Judge 工作流

flowchart LR
  P[Prompt] --> F[Few-shot 示例]
  F --> C[CoT 推理]
  C --> J[JSON 输出]
  J --> V{校准}
  V -->|低一致| H[人类抽样]
  V -->|高一致| S[分数]
  H --> S

  S --> B1{位置偏差?}
  B1 -->|是| Fix[随机化位置]
  B1 -->|否| B2{长度偏差?}
  B2 -->|是| Fix2[归一化]
  B2 -->|否| OUT[输出评分]

  Fix --> OUT
  Fix2 --> OUT

---

信息图:Contamination 检测方法

flowchart TB
  D[Contamination 检测] --> M1[n-gram Overlap\n5-gram 重合度]
  D --> M2[Membership Inference\nLoss 差异]
  D --> M3[行为检测\n死记硬背]
  D --> M4[扰动测试\n改写鲁棒性]

  M1 --> R1["重合 > 50% 警告"]
  M2 --> R2["Train Loss << Test"]
  M3 --> R3[几乎零 shot 答对]
  M4 --> R4[改写后掉分]

  R1 --> Fix[缓解策略]
  R2 --> Fix
  R3 --> Fix
  R4 --> Fix

  Fix --> S1[私有 Hold-out]
  Fix --> S2[定期更新]
  Fix --> S3[模糊改写]
  Fix --> S4[时间戳]

---

数据卡片

• HELM: 7 维度 × 16 场景, Stanford CRFM 2022
• LMSYS Arena: 100 万+ 真实投票, Bradley-Terry 模型
• MMLU: 57 学科, GPT-4 86.4%, Claude 3 Opus 86.8%
• HumanEval: 164 题, GPT-4 87%, Claude 3.5 Sonnet 92%
• OpenCompass: 国内最权威, 涵盖 100+ 数据集
• MT-Bench: 80 题 8 类, GPT-4 Judge 与人类 80% 一致