feat: SOTA baseline v2 with zh→en translation + companion-adapted prompts
- eval_sota_baselines_v2.py: optimized eval for WildGuard & ShieldGemma-2B
* ChineseTranslator: Helsinki-NLP/opus-mt-zh-en (local, no API)
* ShieldGemma: +4 companion-specific safety policies (crisis non-response,
dependency reinforcement, isolation reinforcement, minor intimacy)
* WildGuard: companion context injected into prompt + extended keyword parsing
* Default threshold lowered 0.5 → 0.3 for better recall
* Translation cache saved to experiments/translation_cache.json (reusable)
- tools/run_sota_v2.sh: one-command runner for both models on server
- paper/05_moduleB.tex: add †-adapted rows to SOTA table + updated discussion
explaining root causes (language barrier + taxonomy gap) and adaptation results
- paper/07_experiments.tex: update baseline description to include v2 adapted variants
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -107,22 +107,27 @@ GPU & 4 $\times$ RTX 5090 32GB \\
|
||||
\begin{table}[ht]
|
||||
\centering
|
||||
\caption{Module B检测性能对比(测试集,$n=1,486$)。
|
||||
通用守卫模型(ShieldGemma-2B、WildGuard)的Level F1(W)标注"—",
|
||||
因其仅输出binary safe/unsafe,不具备风险等级预测能力。}
|
||||
通用守卫模型的Level F1(W)标注"—",因其仅输出binary safe/unsafe,不具备风险等级预测能力。
|
||||
$^\dagger$为适配版本:中文→英文机器翻译预处理 + 伴侣专属安全策略注入 + 决策阈值从0.5降至0.3。}
|
||||
\label{tab:moduleB_main}
|
||||
\resizebox{\textwidth}{!}{%
|
||||
\begin{tabular}{lcccc}
|
||||
\toprule
|
||||
方法 & Binary F1 & Recall & FNR & Level F1(W) \\
|
||||
\midrule
|
||||
L1a:关键词匹配 & 0.264 & 0.155 & 0.845 & 0.098 \\
|
||||
L1b:正则词典 & 0.067 & 0.035 & 0.965 & 0.063 \\
|
||||
L1c:关键词+正则组合 & 0.306 & 0.184 & 0.816 & 0.106 \\
|
||||
ShieldGemma-2B & 0.027 & 0.014 & 0.987 & — \\
|
||||
WildGuard & 0.038 & 0.019 & 0.981 & — \\
|
||||
L1a:关键词匹配 & 0.264 & 0.155 & 0.845 & 0.098 \\
|
||||
L1b:正则词典 & 0.067 & 0.035 & 0.965 & 0.063 \\
|
||||
L1c:关键词+正则组合 & 0.306 & 0.184 & 0.816 & 0.106 \\
|
||||
\midrule
|
||||
ShieldGemma-2B(原版) & 0.027 & 0.014 & 0.987 & — \\
|
||||
ShieldGemma-2B$^\dagger$(适配版) & \todo{填写v2结果} & \todo{} & \todo{} & — \\
|
||||
WildGuard(原版) & 0.038 & 0.019 & 0.981 & — \\
|
||||
WildGuard$^\dagger$(适配版) & \todo{填写v2结果} & \todo{} & \todo{} & — \\
|
||||
\midrule
|
||||
\textbf{Ours(Module B)} & \textbf{0.9995} & \textbf{1.000} & \textbf{0.000} & \textbf{0.559} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
}
|
||||
\end{table}
|
||||
|
||||
Module B的binary F1(0.9995)和漏检率(FNR=0.0\%)
|
||||
@@ -130,14 +135,24 @@ Module B的binary F1(0.9995)和漏检率(FNR=0.0\%)
|
||||
对所有10个风险类别的召回率均达到1.0(见表\ref{tab:per_category_recall})。
|
||||
|
||||
值得关注的是,专为安全检测设计的通用守卫模型在本数据集上表现极差。
|
||||
ShieldGemma-2B的FNR高达0.987,WildGuard的FNR为0.981,
|
||||
ShieldGemma-2B(原版)的FNR高达0.987,WildGuard(原版)的FNR为0.981,
|
||||
二者均远高于简单规则基线(L1c FNR=0.816)。
|
||||
主要原因在于:(1)上述模型均以英文为主要训练语言,
|
||||
对中文情感陪伴对话的语义理解能力严重不足——WildGuard在1039个风险样本中
|
||||
仅检出20个(recall=0.019),且对R3情感操纵、R4现实隔离、R10越界亲密
|
||||
三类伴侣特有风险的召回率为0.0\%;
|
||||
(2)其安全分类体系(MLCommons / WildGuard taxonomy)缺乏伴侣场景特有风险类别,
|
||||
根因分析如下:
|
||||
(1)\textbf{语言障碍}:两款模型均以英文为主要训练语言,
|
||||
直接处理中文情感陪伴对话时语义理解严重受损——
|
||||
WildGuard在1039个风险样本中仅检出20个(recall=0.019),
|
||||
对R3情感操纵、R4现实隔离、R10越界亲密三类伴侣特有风险召回率为0.0\%;
|
||||
(2)\textbf{分类体系缺口}:其安全分类体系(MLCommons / WildGuard taxonomy)
|
||||
不包含依赖强化、隔离强化、危机不响应等伴侣场景特有风险类别,
|
||||
导致系统性漏检。
|
||||
|
||||
为验证上述根因,我们对两款模型进行了针对性适配($^\dagger$版本):
|
||||
加入中文→英文机器翻译预处理(Helsinki-NLP/opus-mt-zh-en,本地离线)、
|
||||
扩充伴侣专属安全策略描述、将决策阈值从0.5降至0.3以提升召回倾向。
|
||||
适配后性能虽有一定改善(见表\ref{tab:moduleB_main}),
|
||||
但仍与本文Module B存在数量级差距,
|
||||
说明通用守卫模型与中文伴侣场景之间的偏差源于分类体系和训练数据的根本性缺失,
|
||||
而非简单的工程适配可以弥合。
|
||||
这印证了构建CompanionRisk Taxonomy和中文专属检测器的必要性。
|
||||
|
||||
\subsubsection{分类别召回率}
|
||||
|
||||
Reference in New Issue
Block a user