Reasoning
LMCA Leaderboard
LMCA (Language Model Conceptual Argumentation) scores models on expert-rated arguments across philosophy, decision theory, and AI risk — domains where there is no empirical feedback loop to check an answer against. It is part of the Conceptual Reasoning Index, built with Anthropic by researchers Emery Cooper and Caspar Oesterheld.
Source: epoch56 open models ranked+141 proprietaryData through Oct 2026
Open models ranked on LMCA
# shows rank among open models / rank overall (including proprietary).
| # | Model | Score | VRAM @ Q4 |
|---|---|---|---|
| 1 / 45 | GLM 5.3 · 753.3B | 55.5% | ~456 GB |
| 2 / 54 | Kimi K3 · 2779.9B | 52.7% | ~1674 GB |
| 3 / 82 | DeepSeek V4.1 Flash · 763.2B | 47.0% | ~458 GB |
| 4 / 86 | GLM 5.2 · 753.3B | 45.8% | ~456 GB |
| 5 / 88 | DeepSeek V4 Pro 0813 · 1650.5B | 45.5% | ~991 GB |
| 6 / 102 | DeepSeek V4 Flash 0731 · 304.2B | 41.7% | ~183 GB |
| 7 / 104 | Qwen3.8 27B · 27.8B | 41.4% | ~17 GB |
| 8 / 105 | DeepSeek V4 Pro · 1598.8B | 41.2% | ~960 GB |
| 9 / 110 | Gemma 4 31B IT · 31.3B | 39.3% | ~20 GB |
| 10 / 115 | Qwen3.5 397B A17B · 403.4B | 37.9% | ~242 GB |
| 11 / 117 | Inkling · 952.4B | 37.6% | ~572 GB |
| 12 / 120 | Kimi K2.6 · 1026.9B | 37.3% | ~620 GB |
| 13 / 122 | NVIDIA Nemotron 3 Ultra 550B A55B BF16 · 560.5B | 36.9% | ~370 GB |
| 14 / 125 | DeepSeek V4 Flash · 290.9B | 35.9% | ~175 GB |
| 15 / 126 | Mistral Large 4 1T A52B · 1000B | 35.8% | ~604 GB |
| 16 / 129 | Qwen3.6 27B · 27.8B | 34.5% | ~17 GB |
| 17 / 131 | Qwen3.5 27B · 27.8B | 34.0% | ~17 GB |
| 18 / 132 | MiniMax M3 · 427.0B | 33.7% | ~257 GB |
| 19 / 134 | Qwen3.5 122B A10B · 125.1B | 32.2% | ~75 GB |
| 20 / 137 | Qwen3.6 35B A3B · 36.0B | 29.7% | ~22 GB |
| 21 / 138 | Gemma 4 26B A4B IT · 25.8B | 29.7% | ~16 GB |
| 22 / 139 | MiMo V2.5 Pro · 1023.2B | 29.5% | ~614 GB |
| 23 / 140 | Qwen3.5 35B A3B · 36.0B | 29.5% | ~22 GB |
| 24 / 141 | Qwen3 235B A22B Thinking 2507 · 235.1B | 29.3% | ~142 GB |
| 25 / 143 | DeepSeek V3.2 Exp · 685.4B | 29.1% | ~415 GB |
| 26 / 145 | DeepSeek V3.2 · 685.4B | 28.8% | ~415 GB |
| 27 / 146 | DeepSeek V3.1 Terminus · 684.5B | 28.6% | ~415 GB |
| 28 / 150 | Mistral Medium 3.5 128B · 127.7B | 26.1% | ~78 GB |
| 29 / 152 | Qwen3 235B A22B · 235.1B | 25.0% | ~142 GB |
| 30 / 153 | Qwen3.5 9B · 9.7B | 24.5% | ~6.4 GB |
| 31 / 154 | DeepSeek V3.1 · 684.5B | 24.3% | ~415 GB |
| 32 / 156 | Qwen3 235B A22B Instruct 2507 · 235.1B | 23.5% | ~142 GB |
| 33 / 157 | Qwen3 30B A3B Instruct 2507 · 30.5B | 22.4% | ~19 GB |
| 34 / 159 | GPT OSS 120B · 116.8B | 22.1% | ~71 GB |
| 35 / 161 | Mistral Small 4 119B 2603 · 119.4B | 20.6% | ~73 GB |
| 36 / 162 | Qwen3 30B A3B Thinking 2507 · 30.5B | 19.5% | ~19 GB |
| 37 / 165 | Qwen3 14B · 14.8B | 18.2% | ~9.5 GB |
| 38 / 167 | Llama 3.3 70B Instruct · 70.6B | 17.5% | ~47 GB |
| 39 / 168 | Qwen3 32B · 32.8B | 17.3% | ~20 GB |
| 40 / 171 | Mistral Large 3 675B Instruct 2512 · 675B | 16.7% | ~446 GB |
| 41 / 173 | Llama 4 Maverick 17B 128E Instruct · 401.6B | 15.9% | ~265 GB |
| 42 / 174 | Qwen3 30B A3B · 30.5B | 15.8% | ~19 GB |
| 43 / 175 | DeepSeek v3 0324 · 684.5B | 15.5% | ~415 GB |
| 44 / 177 | Llama 3.1 70B Instruct · 70.6B | 14.8% | ~47 GB |
| 45 / 178 | GPT OSS 20B · 20.9B | 14.5% | ~13 GB |
| 46 / 179 | Qwen2.5 72B Instruct · 72.7B | 13.4% | ~45 GB |
| 47 / 180 | Gemma 3 27B IT · 27.4B | 12.3% | ~18 GB |
| 48 / 181 | Llama 4 Scout 17B 16E Instruct · 108.6B | 12.0% | ~72 GB |
| 49 / 183 | C4ai Command A 03 2025 · 111.1B | 10.3% | ~73 GB |
| 50 / 186 | C4ai Command R 08 2024 · 32.3B | 9.2% | ~21 GB |
| 51 / 188 | Qwen3 8B · 8.2B | 8.8% | ~5.5 GB |
| 52 / 191 | Qwen2.5 7B Instruct · 7.6B | 6.4% | ~5.0 GB |
| 53 / 194 | Llama 3.1 8B Instruct · 8.0B | 5.4% | ~5.3 GB |
| 54 / 195 | C4ai Command R Plus 08 2024 · 103.8B | 5.0% | ~69 GB |
| 55 / 196 | Gemma 3 12B IT · 12.2B | 4.5% | ~8.0 GB |
| 56 / 197 | Gemma 3 4B IT · 4.3B | 2.8% | ~2.8 GB |
Score vs model size
Which models give the most quality for their size — the ones worth running locally.
- Gemma 3 4B IT, 4B, score 2.8% — on the efficiency frontier (best score at its size or smaller).
- Qwen2.5 7B Instruct, 8B, score 6.4% — on the efficiency frontier (best score at its size or smaller).
- Qwen3 8B, 8B, score 8.8% — on the efficiency frontier (best score at its size or smaller).
- Qwen3.5 9B, 10B, score 24.5% — on the efficiency frontier (best score at its size or smaller).
- Gemma 4 26B A4B IT, 26B, score 29.7% — on the efficiency frontier (best score at its size or smaller).
- Qwen3.8 27B, 28B, score 41.4% — on the efficiency frontier (best score at its size or smaller).
- DeepSeek V4 Flash 0731, 304B, score 41.7% — on the efficiency frontier (best score at its size or smaller).
- GLM 5.3, 753B, score 55.5% — on the efficiency frontier (best score at its size or smaller).
LMCA: frequently asked questions
- What is the best open LLM on LMCA?
- GLM 5.3 is the top open model on LMCA, scoring 55.5%. Among all models tested — including proprietary ones — it ranks #45. The top model overall is Claude Sonnet 5.5 Max (Anthropic) at 68.3%.
- What's the best LMCA model you can run on a 24 GB GPU?
- Qwen3.8 27B is the highest-scoring open model that fits in 24 GB at 4-bit quantization (~17 GB), scoring 41.4% on LMCA.
- What's the best LMCA model you can run on a 12 GB GPU?
- Qwen3.5 9B is the highest-scoring open model that fits in 12 GB at 4-bit quantization (~6.4 GB), scoring 24.5% on LMCA.
- Can open models match proprietary models on LMCA?
- Not quite on LMCA: the strongest proprietary model (Claude Sonnet 5.5 Max) scores 68.3%, ahead of the best open model (GLM 5.3) at 55.5% — but you can run the open one yourself.
Scores aggregated from epoch. llmrun does not run this benchmark — see the source for methodology, or the about benchmarks for what it measures.