Uncheatable Eval 测试
ℹ️
Uncheatable Eval (opens in a new tab) 是“无法作弊的评测”，它使用最新的论文和新闻文章等实时数据，评估开源大语言模型的真实建模能力和泛化能力。
⚠️
Uncheatable Eval 测试的结果是压缩率，因此其评分越低，意味着模型性能越好。
以下是 RWKV 和其他模型的 Uncheatable Eval 评分对比：
14B 参数模型
Mistral-Nemo-Base-2407
RWKV-6-14B-v2.1
Llama-2-13b-hf
Qwen1.5-14B
pythia-12b-v0
Average
ao3
english
bbc
news
wikipedia
english
arxiv
computer
science
arxiv
physics
github
cpp
github
python
7.11
10.07
8.08
7.95
7.42
7.66
4.20
4.37
7.61
10.19
8.52
8.34
7.92
8.04
4.93
5.33
7.68
10.52
8.28
8.19
8.07
8.31
4.93
5.43
7.70
10.88
8.88
9.10
7.75
7.86
4.67
4.74
8.36
11.29
9.19
9.53
8.54
8.40
5.43
6.13
7B 参数模型
Meta-Llama-3.1-8B
Qwen2.5-7B
Qwen2-7B
Mistral-7B-v0.1
RWKV-6-World-7B-v2.1
Yi-1.5-6B
OLMo-1.7-7B-hf
RWKV-5-World-7B-v2
Qwen1.5-7B
mpt-7B
Llama-2-7B-hf
Zamba-7B-v1
open_llama_7B_v2
falcon-7B
pythia-6.9b-v0
mamba-7B-rw
Average
ao3
english
bbc
news
wikipedia
english
arxiv
computer
science
arxiv
physics
github
cpp
github
python
7.23
10.53
8.20
7.93
7.55
7.69
4.20
4.49
7.45
10.93
8.74
8.81
7.68
7.99
3.92
4.09
7.53
10.81
8.58
8.71
7.82
8.25
4.20
4.32
7.58
10.55
8.28
8.44
7.85
8.09
4.80
5.05
7.82
10.41
8.74
8.58
8.11
8.25
5.12
5.52
7.83
10.93
8.79
8.95
8.10
8.41
4.75
4.89
7.88
11.01
8.65
8.99
8.00
8.20
4.97
5.35
7.91
10.49
8.88
8.69
8.18
8.31
5.19
5.59
7.92
11.10
9.13
9.36
7.95
8.11
4.84
4.92
7.95
11.19
8.68
8.77
8.16
8.44
4.95
5.47
7.97
10.84
8.51
8.52
8.33
8.63
5.24
5.70
8.09
10.85
8.52
8.64
8.06
8.29
5.94
6.33
8.10
11.09
8.84
9.05
8.40
8.76
4.89
5.70
8.30
10.76
8.69
9.15
8.55
9.06
5.76
6.15
8.54
11.49
9.38
9.76
8.68
8.57
5.61
6.32
9.78
10.81
8.55
8.99
8.61
9.14
11.11
11.27
3B 参数模型
stablelm-3b-4e1t
Minitron-4B-Base
recurrentgemma-2b
Qwen1.5-4B
Llama-3.1-Minitron-4B
RWKV-6-World-3B-v2.1
Phi-3-mini-4k-instruct
mamba2attn-2.7B
gemma-2b
RWKV-5-World-3B-v2
open_llama_3b_v2
mamba2-2.7B
Phi-3.5-mini-instruct
Zamba2-2.7B
mamba-2.8b-hf
RWKV-4-World-3B-v1
pythia-2.8b-v0
RedPajama-INCITE-Base-3B-v1
phi-2
btlm-3b-8k-base
RWKV-4-Pile-3B
Sheared-LLaMA-2.7B
MiniCPM3-4B
mamba-2.8b-slimpj
OpenELM-3B
Average
ao3
english
bbc
news
wikipedia
english
arxiv
computer
science
arxiv
physics
github
cpp
github
python
8.03
11.05
8.79
8.83
8.36
8.64
4.97
5.58
8.14
11.22
9.04
9.20
8.30
8.86
4.95
5.42
8.17
11.45
8.92
9.16
8.45
8.66
5.01
5.51
8.25
11.61
9.34
9.78
8.23
8.45
5.14
5.20
8.26
11.40
9.26
9.40
8.46
9.07
4.91
5.30
8.26
10.84
9.16
9.10
8.49
8.72
5.57
5.96
8.33
11.98
9.19
9.30
8.39
9.02
5.44
5.02
8.36
11.29
9.23
9.59
8.52
8.40
5.41
6.08
8.38
11.74
9.14
9.39
8.69
8.88
5.19
5.67
8.41
10.99
9.34
9.30
8.62
8.84
5.70
6.08
8.46
11.47
9.15
9.47
8.74
9.16
5.21
6.02
8.47
11.38
9.32
9.72
8.62
8.49
5.52
6.24
8.48
12.16
9.31
9.43
8.57
9.15
5.51
5.20
8.57
11.17
8.93
9.14
8.42
8.88
6.78
6.67
8.59
11.46
9.43
9.87
8.76
8.64
5.64
6.35
8.71
11.04
9.51
9.59
9.13
9.43
5.85
6.38
8.85
11.81
9.68
10.15
8.92
8.86
5.89
6.60
8.87
11.66
9.13
9.29
8.88
9.21
6.62
7.29
8.91
12.28
9.28
9.58
8.81
9.86
6.77
5.79
8.96
11.81
9.08
9.10
8.57
8.88
7.46
7.81
9.02
11.79
9.76
10.40
9.20
9.06
6.08
6.85
9.10
11.58
9.15
9.61
9.11
9.65
7.07
7.54
9.12
13.23
10.71
10.72
8.75
9.03
5.54
5.84
9.25
13.81
9.66
9.15
8.71
8.92
7.04
7.43
9.68
14.05
10.08
9.97
9.16
9.52
7.40
7.57
1.6B 参数模型
Qwen2-1.5B
Index-1.9B
stablelm-2-1_6b
Rene-v0.1-1.3b-pytorch
RWKV-6-World-1B6-v2.1
RWKV-5-World-1B5-v2
mamba2-1.3b
mamba-1.4b-hf
TinyLlama-1.1B-intermediate-3T
Qwen1.5-1.8B
RWKV-4-World-1.5B-v1
OLMo-1B-hf
Qwen-1_8B
pythia-1.4b-v0
RWKV-4-Pile-1B5
h2o-danube-1.8b-base
Sheared-LLaMA-1.3B
bloom-1b7
OpenELM-1_1B
TransNormerLLM-1B
phi-1_5
falcon-rw-1b
Average
ao3
english
bbc
news
wikipedia
english
arxiv
computer
science
arxiv
physics
github
cpp
github
python
8.46
11.78
9.44
9.88
8.75
9.32
5.02
5.01
8.49
11.66
9.49
9.52
8.55
8.72
5.53
5.96
8.53
11.56
9.21
9.38
8.79
9.25
5.65
5.88
8.56
11.62
9.20
9.84
8.65
9.09
5.54
5.97
8.68
11.24
9.55
9.60
8.84
9.14
6.00
6.36
8.83
11.39
9.74
9.79
8.97
9.26
6.13
6.50
8.86
11.78
9.69
10.20
8.94
8.86
5.90
6.62
8.97
11.85
9.77
10.32
9.08
8.99
6.01
6.75
8.99
12.40
9.74
10.00
9.30
9.78
5.64
6.05
9.14
12.45
9.89
10.61
9.14
9.72
5.93
6.26
9.19
11.46
9.94
10.11
9.57
9.96
6.39
6.87
9.20
12.14
9.61
10.34
9.26
9.91
6.31
6.82
9.33
12.60
9.99
10.77
9.32
9.96
6.16
6.48
9.33
12.32
10.13
10.78
9.32
9.33
6.34
7.11
9.46
12.21
10.16
10.88
9.59
9.48
6.56
7.33
9.72
11.65
9.18
9.52
9.34
10.04
9.29
9.02
9.78
12.19
9.70
10.30
9.68
10.38
7.89
8.34
9.82
13.43
10.92
11.41
9.49
9.85
6.38
7.26
10.19
14.93
10.58
10.52
9.49
10.01
7.88
7.95
10.45
12.89
10.50
11.10
10.67
11.36
7.73
8.88
10.50
13.45
11.08
13.51
9.99
11.72
7.31
6.42
12.12
12.06
9.56
10.49
9.63
10.52
16.31
16.24
MMLU 测试
ℹ️
MMLU 测试（Massive Multitask Language Understanding）是一项用于评估大型语言模型（LLMs）在广泛任务上的多任务语言理解能力的基准测试。
MMLU 涵盖了从初中到研究生水平的57个不同学科，包括数学、物理、历史、法律、生物学等，测试语言模型是否能够在不同领域内进行推理、回答问题和表现出跨学科的知识。
如果使用 lm_eval 的标准格式测试，RWKV-6-World-7B-v2.1 的 MMLU 准确度是 42.8% ：
The following are multiple choice questions (with answers) about abstract algebra.
Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q.
A. 0
B. 4
C. 2
D. 6
Answer:如果使用 RWKV 的训练的数据格式作为 prompt，RWKV-6-World-7B-v2.1 的 MMLU 是 46.7%：
User: Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q.
A. 0
B. 4
C. 2
D. 6
Assistant: The answer is如果使用最适合 RWKV 模型推理的 prompt 模板，RWKV-6-World-7B-v2.1 的 MMLU 是 47.9%：
User: You are a very talented expert in <SUBJECT>. Answer this question:
<Question>
A. <|A|>
B. <|B|>
C. <|C|>
D. <|D|>
Assistant: The answer is