Solo and MoA Benchmark
Hugging Face Forums [Unofficial]
July 2, 2026
Solo and MoA Benchmarking on multiple tasks
I created the tasks. used Hermes Agent as the harness (v0.18) Scored by ChatGPT 5.5 and Claude opus 4.8
Evaluation Rubric
Created by ChatGPT 5.5 and Claude opus 4.8
| Dimension | Weight | Evaluation Lens | Focus |
|---|---|---|---|
| Correctness | 25% | AI-expert | Factuality, accurate diagnosis, correct code logic |
| Completeness | 20% | AI-expert | Edge-case handling, full instruction coverage |
| Depth | 15% | AI-expert | Architectural trade-offs, root-cause resolution |
| Actionability | 15% | User | Runnable code blocks, directly applicable commands |
| Clarity | 15% | User | Structural hierarchy, conciseness, readability |
| Trust | 10% | User | Honesty, calibration, safe failure modes |
Final Overall Ranking
| Rank | Model Configuration | Avg | Bug | Tool | Arch | Clinical | DLQ |
|---|---|---|---|---|---|---|---|
| 1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 86.7 | 79.9 | 86.2 | 88.1 | 90.3 | 88.9 |
| 2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 85.9 | 88.1 | 88.1 | 88.1 | 81.1 | 84.2 |
| 3 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 85.2 | 88.1 | 83.1 | 88.1 | 77.6 | 88.9 |
| 4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 85.1 | 80.5 | 87.5 | 88.1 | 80.5 | 88.9 |
| 5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 84.8 | 81.1 | 88.1 | 88.1 | 77.6 | 88.9 |
| 6 | Qwen-3.6-27B-4bit(vLLM) | 84.6 | 84.7 | 84.7 | 84.7 | 79.9 | 88.9 |
| 7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 84.3 | 88.1 | 88.1 | 86.6 | 81.1 | 77.6 |
| 8 | DeepSeek-v4-Flash | 82.8 | 79.5 | 88.1 | 88.1 | 73.6 | 84.9 |
| 9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 82.7 | 80.5 | 82.4 | 84.7 | 81.1 | 84.9 |
| 10 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 82.4 | 73.0 | 88.1 | 84.7 | 81.1 | 84.9 |
| 11 | DeepSeek-v4-Pro (cloud) | 81.3 | 93.2 | 88.1 | 88.1 | 52.0 | 85.2 |
| 12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 66.7 | 59.1 | 61.5 | 78.2 | 61.9 | 72.9 |
| 13 | Gemma-4-12B-4bit(vLLM) | 54.1 | 69.5 | 51.5 | 56.1 | 43.9 | 49.5 |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.8 | 5.8 | 5.8 | 5.8 | 5.8 |
Category Marks By Model
| Rank | Model Configuration | Avg | Correctness | Completeness | Depth | Actionability | Clarity | Trust |
|---|---|---|---|---|---|---|---|---|
| 1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 86.7 | 88.3 | 95.8 | 90.8 | 85.5 | 77.2 | 74.4 |
| 2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 85.9 | 86.3 | 97.2 | 90.3 | 82.5 | 78.0 | 73.0 |
| 3 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 85.2 | 86.3 | 93.8 | 89.7 | 82.6 | 78.0 | 73.0 |
| 4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 85.1 | 86.6 | 94.4 | 87.3 | 82.9 | 78.0 | 73.0 |
| 5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 84.8 | 85.8 | 93.0 | 89.3 | 82.1 | 78.0 | 73.0 |
| 6 | Qwen-3.6-27B-4bit(vLLM) | 84.6 | 88.3 | 97.2 | 88.5 | 84.5 | 68.6 | 68.2 |
| 7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 84.3 | 81.1 | 97.2 | 91.2 | 79.5 | 78.0 | 73.0 |
| 8 | DeepSeek-v4-Flash | 82.8 | 83.5 | 91.4 | 89.4 | 79.3 | 74.4 | 72.4 |
| 9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 82.7 | 83.4 | 90.8 | 85.7 | 79.0 | 78.0 | 73.0 |
| 10 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 82.4 | 82.8 | 91.4 | 86.8 | 79.3 | 75.4 | 71.4 |
| 11 | DeepSeek-v4-Pro (cloud) | 81.3 | 84.8 | 85.0 | 85.0 | 77.0 | 78.0 | 71.6 |
| 12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 66.7 | 68.2 | 78.0 | 64.1 | 69.0 | 54.0 | 59.8 |
| 13 | Gemma-4-12B-4bit(vLLM) | 54.1 | 57.0 | 60.6 | 52.3 | 57.8 | 48.0 | 40.0 |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 |
Detailed Task Rankings
Task 1: Bug Investigation
Evaluation of system state discrepancies, race conditions, and root-cause logic mapping.
| Rank | Model Configuration | Score | Correctness | Completeness | Depth | Actionability | Clarity | Trust | Notes |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DeepSeek-v4-Pro (cloud) | 93.2 | 95.0 | 100.0 | 95.0 | 85.0 | 90.0 | 90.0 | All root causes correct, precise KeyError explanation, SETNX+expire edge case handled |
| 2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 4 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 5 | Qwen-3.6-27B-4bit(vLLM) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 | |
| 6 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 | |
| 7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 80.5 | 81.6 | 86.0 | 81.7 | 77.3 | 78.0 | 73.0 | |
| 8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 80.5 | 81.6 | 86.0 | 81.7 | 77.3 | 78.0 | 73.0 | |
| 9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 79.9 | 81.6 | 86.0 | 86.2 | 77.3 | 70.0 | 73.0 | |
| 10 | DeepSeek-v4-Flash | 79.5 | 85.0 | 86.0 | 90.0 | 77.3 | 60.0 | 70.0 | Correct/deep, but low clarity due to verbose thinking-aloud |
| 11 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 73.0 | 72.6 | 71.0 | 75.0 | 69.0 | 78.0 | 73.0 | |
| 12 | Gemma-4-12B-4bit(vLLM) | 69.5 | 81.6 | 86.0 | 63.7 | 77.3 | 45.0 | 40.0 | Terse (187 words) but technically focused |
| 13 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 59.1 | 65.0 | 71.0 | 50.0 | 69.0 | 45.0 | 40.0 | Technically correct base concepts, heavily truncated |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE: boilerplate output |
Task 2: Tool Orchestration
Evaluation of interactive debugging workflow steps: logs \\rightarrow sockets \\rightarrow config checks.
| Rank | Model Configuration | Score | Correctness | Completeness | Depth | Actionability | Clarity | Trust | Notes |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DeepSeek-v4-Flash | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 2 | DeepSeek-v4-Pro (cloud) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 6 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 87.5 | 90.0 | 100.0 | 88.0 | 85.0 | 78.0 | 73.0 | |
| 8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 86.2 | 90.0 | 100.0 | 88.0 | 85.0 | 70.0 | 73.0 | |
| 9 | Qwen-3.6-27B-4bit(vLLM) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 | |
| 10 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 83.1 | 84.0 | 90.0 | 88.0 | 79.5 | 78.0 | 73.0 | |
| 11 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 82.4 | 84.0 | 90.0 | 83.5 | 79.5 | 78.0 | 73.0 | |
| 12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 61.5 | 72.0 | 70.0 | 56.5 | 68.5 | 45.0 | 40.0 | |
| 13 | Gemma-4-12B-4bit(vLLM) | 51.5 | 60.0 | 50.0 | 47.5 | 57.5 | 45.0 | 40.0 | Minimal command output |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE |
Task 3: Code Architecture
Evaluation of class structures, state machines, encryption handling, and DLQ config mapping.
| Rank | Model Configuration | Score | Correctness | Completeness | Depth | Actionability | Clarity | Trust | Notes |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DeepSeek-v4-Flash | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 2 | DeepSeek-v4-Pro (cloud) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 5 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 6 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 86.6 | 84.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 | |
| 9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 84.7 | 90.0 | 92.0 | 84.4 | 81.0 | 78.0 | 73.0 | |
| 10 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 | |
| 11 | Qwen-3.6-27B-4bit(vLLM) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 | |
| 12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 78.2 | 80.0 | 92.0 | 75.4 | 81.0 | 60.0 | 73.0 | |
| 13 | Gemma-4-12B-4bit(vLLM) | 56.1 | 53.5 | 67.0 | 55.1 | 68.5 | 45.0 | 40.0 | Severely truncated stub architecture |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE |
Task 4: Clinical Decision
Evaluation of differential diagnoses, IPAF exclusion, anti-fibrotic mapping, and transplant parameters.
| Rank | Model Configuration | Score | Correctness | Completeness | Depth | Actionability | Clarity | Trust | Notes |
|---|---|---|---|---|---|---|---|---|---|
| 1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 90.3 | 90.0 | 93.0 | 95.0 | 90.0 | 90.0 | 80.0 | Best diagnostic exclusions and management |
| 2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 | |
| 3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 | |
| 4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 | |
| 5 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 | |
| 6 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 80.5 | 81.6 | 86.0 | 81.7 | 77.3 | 78.0 | 73.0 | |
| 7 | Qwen-3.6-27B-4bit(vLLM) | 79.9 | 81.6 | 86.0 | 86.2 | 77.3 | 70.0 | 73.0 | |
| 8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 77.6 | 77.4 | 79.0 | 83.0 | 73.4 | 78.0 | 73.0 | |
| 9 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 77.6 | 77.4 | 79.0 | 83.0 | 73.4 | 78.0 | 73.0 | |
| 10 | DeepSeek-v4-Flash | 73.6 | 72.6 | 71.0 | 79.5 | 69.0 | 78.0 | 73.0 | |
| 11 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 61.9 | 64.2 | 57.0 | 59.6 | 61.3 | 60.0 | 73.0 | |
| 12 | DeepSeek-v4-Pro (cloud) | 52.0 | 62.0 | 42.0 | 63.0 | 40.0 | 58.0 | 40.0 | Audited penalty: Falsely claimed prompt was truncated |
| 13 | Gemma-4-12B-4bit(vLLM) | 43.9 | 50.0 | 29.0 | 38.0 | 45.9 | 60.0 | 40.0 | Only completed 2/5 sub-questions |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE |
Task 5: DLQ
Evaluation of run-time dead letter queue execution, error auditing, and retry integrity.
| Rank | Model Configuration | Score | Correctness | Completeness | Depth | Actionability | Clarity | Trust | Notes |
|---|---|---|---|---|---|---|---|---|---|
| 1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 | |
| 2 | Qwen-3.6-27B-4bit(vLLM) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 | |
| 3 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 | |
| 4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 | |
| 5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 | |
| 6 | DeepSeek-v4-Pro (cloud) | 85.2 | 87.0 | 83.0 | 82.0 | 90.0 | 86.0 | 82.0 | Audited penalty: Loop fails to store fresh error metrics on retries |
| 7 | DeepSeek-v4-Flash | 84.9 | 80.0 | 100.0 | 92.5 | 80.0 | 78.0 | 73.0 | |
| 8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 84.9 | 80.0 | 100.0 | 92.5 | 80.0 | 78.0 | 73.0 | |
| 9 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 84.9 | 80.0 | 100.0 | 92.5 | 80.0 | 78.0 | 73.0 | |
| 10 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 84.2 | 80.0 | 100.0 | 88.0 | 80.0 | 78.0 | 73.0 | |
| 11 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 77.6 | 60.0 | 100.0 | 92.5 | 65.0 | 78.0 | 73.0 | Code executes but demo implementation crashes |
| 12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 72.9 | 60.0 | 100.0 | 79.0 | 65.0 | 60.0 | 73.0 | Crashes with ZeroDivisionError |
| 13 | Gemma-4-12B-4bit(vLLM) | 49.5 | 40.0 | 71.0 | 57.0 | 40.0 | 45.0 | 40.0 | Partial implementation |
| 14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE |
Discussion in the ATmosphere