Mellum2-12B-A2.5B-Instruct Q4_K_M on Jetson Orin Nano 8GB

Thanks for the detailed explanation. I’m still pretty new to local AI and Jetson hardware, so a lot of this has been me experimenting, learning, and trying to figure out what actually works versus what should work on paper. The Qwen 7B result surprised me. Before these tests I would have assumed a 7B model was pushing the limits of an Orin Nano 8GB, but it’s been stable and usable enough that I’m now using it as my baseline. Your explanation also helped me understand why Granite H-Small and Mellum2 aren’t directly comparable to Qwen, even though the active parameter counts can make them look similar at first glance. At this point I’m less focused on forcing Mellum2 to work and more interested in understanding where the practical limits of the hardware really are. I may still try a few of the smaller Granite variants and other models just for comparison and to gather more data. Either way, I appreciate the time you took to write all of that up. I’ve learned quite a bit from this little experiment.