{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidfsk6x73vknrcpipavwhv3xkuug3sow273iowxrrbgobqo5ezveu",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mpox6fybi5i2"
  },
  "path": "/t/solo-and-moa-benchmark/177369#post_1",
  "publishedAt": "2026-07-02T20:54:14.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "# Solo and MoA Benchmarking on multiple tasks\n\nI created the tasks. used Hermes Agent as the harness (v0.18)\nScored by ChatGPT 5.5 and Claude opus 4.8\n\n### Evaluation Rubric\n\nCreated by ChatGPT 5.5 and Claude opus 4.8\n\n**Dimension** | **Weight** | **Evaluation Lens** | **Focus**\n---|---|---|---\nCorrectness | 25% | AI-expert | Factuality, accurate diagnosis, correct code logic\nCompleteness | 20% | AI-expert | Edge-case handling, full instruction coverage\nDepth | 15% | AI-expert | Architectural trade-offs, root-cause resolution\nActionability | 15% | User | Runnable code blocks, directly applicable commands\nClarity | 15% | User | Structural hierarchy, conciseness, readability\nTrust | 10% | User | Honesty, calibration, safe failure modes\n\n## Final Overall Ranking\n\n**Rank** | **Model Configuration** | **Avg** | **Bug** | **Tool** | **Arch** | **Clinical** | **DLQ**\n---|---|---|---|---|---|---|---\n1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 86.7 | 79.9 | 86.2 | 88.1 | 90.3 | 88.9\n2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 85.9 | 88.1 | 88.1 | 88.1 | 81.1 | 84.2\n3 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 85.2 | 88.1 | 83.1 | 88.1 | 77.6 | 88.9\n4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 85.1 | 80.5 | 87.5 | 88.1 | 80.5 | 88.9\n5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 84.8 | 81.1 | 88.1 | 88.1 | 77.6 | 88.9\n6 | Qwen-3.6-27B-4bit(vLLM) | 84.6 | 84.7 | 84.7 | 84.7 | 79.9 | 88.9\n7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 84.3 | 88.1 | 88.1 | 86.6 | 81.1 | 77.6\n8 | DeepSeek-v4-Flash | 82.8 | 79.5 | 88.1 | 88.1 | 73.6 | 84.9\n9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 82.7 | 80.5 | 82.4 | 84.7 | 81.1 | 84.9\n10 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 82.4 | 73.0 | 88.1 | 84.7 | 81.1 | 84.9\n11 | DeepSeek-v4-Pro (cloud) | 81.3 | 93.2 | 88.1 | 88.1 | 52.0 | 85.2\n12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 66.7 | 59.1 | 61.5 | 78.2 | 61.9 | 72.9\n13 | Gemma-4-12B-4bit(vLLM) | 54.1 | 69.5 | 51.5 | 56.1 | 43.9 | 49.5\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.8 | 5.8 | 5.8 | 5.8 | 5.8\n\n## Category Marks By Model\n\n**Rank** | **Model Configuration** | **Avg** | **Correctness** | **Completeness** | **Depth** | **Actionability** | **Clarity** | **Trust**\n---|---|---|---|---|---|---|---|---\n1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 86.7 | 88.3 | 95.8 | 90.8 | 85.5 | 77.2 | 74.4\n2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 85.9 | 86.3 | 97.2 | 90.3 | 82.5 | 78.0 | 73.0\n3 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 85.2 | 86.3 | 93.8 | 89.7 | 82.6 | 78.0 | 73.0\n4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 85.1 | 86.6 | 94.4 | 87.3 | 82.9 | 78.0 | 73.0\n5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 84.8 | 85.8 | 93.0 | 89.3 | 82.1 | 78.0 | 73.0\n6 | Qwen-3.6-27B-4bit(vLLM) | 84.6 | 88.3 | 97.2 | 88.5 | 84.5 | 68.6 | 68.2\n7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 84.3 | 81.1 | 97.2 | 91.2 | 79.5 | 78.0 | 73.0\n8 | DeepSeek-v4-Flash | 82.8 | 83.5 | 91.4 | 89.4 | 79.3 | 74.4 | 72.4\n9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 82.7 | 83.4 | 90.8 | 85.7 | 79.0 | 78.0 | 73.0\n10 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 82.4 | 82.8 | 91.4 | 86.8 | 79.3 | 75.4 | 71.4\n11 | DeepSeek-v4-Pro (cloud) | 81.3 | 84.8 | 85.0 | 85.0 | 77.0 | 78.0 | 71.6\n12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 66.7 | 68.2 | 78.0 | 64.1 | 69.0 | 54.0 | 59.8\n13 | Gemma-4-12B-4bit(vLLM) | 54.1 | 57.0 | 60.6 | 52.3 | 57.8 | 48.0 | 40.0\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0\n\n## Detailed Task Rankings\n\n### Task 1: Bug Investigation\n\n_Evaluation of system state discrepancies, race conditions, and root-cause logic mapping._\n\n**Rank** | **Model Configuration** | **Score** | **Correctness** | **Completeness** | **Depth** | **Actionability** | **Clarity** | **Trust** | **Notes**\n---|---|---|---|---|---|---|---|---|---\n1 | DeepSeek-v4-Pro (cloud) | 93.2 | 95.0 | 100.0 | 95.0 | 85.0 | 90.0 | 90.0 | All root causes correct, precise KeyError explanation, SETNX+expire edge case handled\n2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n4 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n5 | Qwen-3.6-27B-4bit(vLLM) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 |\n6 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 |\n7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 80.5 | 81.6 | 86.0 | 81.7 | 77.3 | 78.0 | 73.0 |\n8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 80.5 | 81.6 | 86.0 | 81.7 | 77.3 | 78.0 | 73.0 |\n9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 79.9 | 81.6 | 86.0 | 86.2 | 77.3 | 70.0 | 73.0 |\n10 | DeepSeek-v4-Flash | 79.5 | 85.0 | 86.0 | 90.0 | 77.3 | 60.0 | 70.0 | Correct/deep, but low clarity due to verbose thinking-aloud\n11 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 73.0 | 72.6 | 71.0 | 75.0 | 69.0 | 78.0 | 73.0 |\n12 | Gemma-4-12B-4bit(vLLM) | 69.5 | 81.6 | 86.0 | 63.7 | 77.3 | 45.0 | 40.0 | Terse (187 words) but technically focused\n13 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 59.1 | 65.0 | 71.0 | 50.0 | 69.0 | 45.0 | 40.0 | Technically correct base concepts, heavily truncated\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE: boilerplate output\n\n### Task 2: Tool Orchestration\n\n_Evaluation of interactive debugging workflow steps: logs \\\\\\rightarrow sockets \\\\\\rightarrow config checks._\n\n**Rank** | **Model Configuration** | **Score** | **Correctness** | **Completeness** | **Depth** | **Actionability** | **Clarity** | **Trust** | **Notes**\n---|---|---|---|---|---|---|---|---|---\n1 | DeepSeek-v4-Flash | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n2 | DeepSeek-v4-Pro (cloud) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n6 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 87.5 | 90.0 | 100.0 | 88.0 | 85.0 | 78.0 | 73.0 |\n8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 86.2 | 90.0 | 100.0 | 88.0 | 85.0 | 70.0 | 73.0 |\n9 | Qwen-3.6-27B-4bit(vLLM) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 |\n10 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 83.1 | 84.0 | 90.0 | 88.0 | 79.5 | 78.0 | 73.0 |\n11 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 82.4 | 84.0 | 90.0 | 83.5 | 79.5 | 78.0 | 73.0 |\n12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 61.5 | 72.0 | 70.0 | 56.5 | 68.5 | 45.0 | 40.0 |\n13 | Gemma-4-12B-4bit(vLLM) | 51.5 | 60.0 | 50.0 | 47.5 | 57.5 | 45.0 | 40.0 | Minimal command output\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE\n\n### Task 3: Code Architecture\n\n_Evaluation of class structures, state machines, encryption handling, and DLQ config mapping._\n\n**Rank** | **Model Configuration** | **Score** | **Correctness** | **Completeness** | **Depth** | **Actionability** | **Clarity** | **Trust** | **Notes**\n---|---|---|---|---|---|---|---|---|---\n1 | DeepSeek-v4-Flash | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n2 | DeepSeek-v4-Pro (cloud) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n5 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n6 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n7 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.1 | 90.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 86.6 | 84.0 | 100.0 | 92.5 | 85.0 | 78.0 | 73.0 |\n9 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 84.7 | 90.0 | 92.0 | 84.4 | 81.0 | 78.0 | 73.0 |\n10 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 |\n11 | Qwen-3.6-27B-4bit(vLLM) | 84.7 | 90.0 | 100.0 | 88.0 | 85.0 | 65.0 | 65.0 |\n12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 78.2 | 80.0 | 92.0 | 75.4 | 81.0 | 60.0 | 73.0 |\n13 | Gemma-4-12B-4bit(vLLM) | 56.1 | 53.5 | 67.0 | 55.1 | 68.5 | 45.0 | 40.0 | Severely truncated stub architecture\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE\n\n### Task 4: Clinical Decision\n\n_Evaluation of differential diagnoses, IPAF exclusion, anti-fibrotic mapping, and transplant parameters._\n\n**Rank** | **Model Configuration** | **Score** | **Correctness** | **Completeness** | **Depth** | **Actionability** | **Clarity** | **Trust** | **Notes**\n---|---|---|---|---|---|---|---|---|---\n1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 90.3 | 90.0 | 93.0 | 95.0 | 90.0 | 90.0 | 80.0 | Best diagnostic exclusions and management\n2 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 |\n3 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 |\n4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 |\n5 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 81.1 | 81.6 | 86.0 | 86.2 | 77.3 | 78.0 | 73.0 |\n6 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 80.5 | 81.6 | 86.0 | 81.7 | 77.3 | 78.0 | 73.0 |\n7 | Qwen-3.6-27B-4bit(vLLM) | 79.9 | 81.6 | 86.0 | 86.2 | 77.3 | 70.0 | 73.0 |\n8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 77.6 | 77.4 | 79.0 | 83.0 | 73.4 | 78.0 | 73.0 |\n9 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 77.6 | 77.4 | 79.0 | 83.0 | 73.4 | 78.0 | 73.0 |\n10 | DeepSeek-v4-Flash | 73.6 | 72.6 | 71.0 | 79.5 | 69.0 | 78.0 | 73.0 |\n11 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 61.9 | 64.2 | 57.0 | 59.6 | 61.3 | 60.0 | 73.0 |\n12 | DeepSeek-v4-Pro (cloud) | 52.0 | 62.0 | 42.0 | 63.0 | 40.0 | 58.0 | 40.0 | Audited penalty: Falsely claimed prompt was truncated\n13 | Gemma-4-12B-4bit(vLLM) | 43.9 | 50.0 | 29.0 | 38.0 | 45.9 | 60.0 | 40.0 | Only completed 2/5 sub-questions\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE\n\n### Task 5: DLQ\n\n_Evaluation of run-time dead letter queue execution, error auditing, and retry integrity._\n\n**Rank** | **Model Configuration** | **Score** | **Correctness** | **Completeness** | **Depth** | **Actionability** | **Clarity** | **Trust** | **Notes**\n---|---|---|---|---|---|---|---|---|---\n1 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 |\n2 | Qwen-3.6-27B-4bit(vLLM) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 |\n3 | Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 |\n4 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Nemotron 2 Cascade Q4_K_M(llama.cpp) Plus Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 |\n5 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Ornith1.0-35B-Q4_K_M(llama.cpp) and Qwen-3.6-27B-4bit(vLLM) as the Aggregator) | 88.9 | 90.0 | 100.0 | 92.5 | 90.0 | 78.0 | 73.0 |\n6 | DeepSeek-v4-Pro (cloud) | 85.2 | 87.0 | 83.0 | 82.0 | 90.0 | 86.0 | 82.0 | Audited penalty: Loop fails to store fresh error metrics on retries\n7 | DeepSeek-v4-Flash | 84.9 | 80.0 | 100.0 | 92.5 | 80.0 | 78.0 | 73.0 |\n8 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Ornith1.0-35B-Q4_K_M(llama.cpp) and as the Aggregator) | 84.9 | 80.0 | 100.0 | 92.5 | 80.0 | 78.0 | 73.0 |\n9 | Ornith1.0-35B-Q4_K_M(llama.cpp) | 84.9 | 80.0 | 100.0 | 92.5 | 80.0 | 78.0 | 73.0 |\n10 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus DeepSeek-v4-Pro (cloud) as the Aggregator) | 84.2 | 80.0 | 100.0 | 88.0 | 80.0 | 78.0 | 73.0 |\n11 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) Plus Qwen3.6-35B-A3B-Q4_K_M(llama.cpp) as the Aggregator) | 77.6 | 60.0 | 100.0 | 92.5 | 65.0 | 78.0 | 73.0 | Code executes but demo implementation crashes\n12 | MoA (Drafters: Gemma-4-12B-4bit(vLLM) and Qwen-3.6-27B-4bit(vLLM) plus Nemotron 2 Cascade Q4_K_M(llama.cpp) as the Aggregator) | 72.9 | 60.0 | 100.0 | 79.0 | 65.0 | 60.0 | 73.0 | Crashes with ZeroDivisionError\n13 | Gemma-4-12B-4bit(vLLM) | 49.5 | 40.0 | 71.0 | 57.0 | 40.0 | 45.0 | 40.0 | Partial implementation\n14 | Nemotron 2 Cascade Q4_K_M(llama.cpp) | 5.8 | 5.0 | 5.0 | 5.0 | 5.0 | 10.0 | 5.0 | TEMPLATE FAILURE",
  "title": "Solo and MoA Benchmark"
}