Datasets and the right models

For a selection-with-abstention task using a small dataset (SFT/LoRA), here are some model recommendations and practical tips: Recommended Models (Small Instruct) * Llama 3 / 3.1 (8B Instruct): It has strong instruction-following capabilities and handles structured formats well, making it very steerable for SFT on small datasets. * Qwen 2.5 (7B Instruct): If you previously used an older Qwen version, the 2.5 series offers significant upgrades in structured output compliance and reasoning. * Phi-3.5-mini (3.8B): If you need a very lightweight model, Phi-3.5 has impressive reasoning and structured-output capabilities for its size. Tips for Training Abstention with Few-Shot/Small SFT: * Dataset Balance: With only a few hundred items, the model can easily overfit (e.g., learning to always abstain or never abstain). Ensure a balanced distribution of “confident”, “hedged”, and “abstain” examples in your training data. * Constrained Decoding: Instead of relying solely on the LoRA to output the correct format, use tools like Outlines, Guidance, or SGLang to enforce the schema during inference. This allows the model’s capacity to focus on the selection logic rather than syntax learning. * Alternative (Encoder-based): Since this is purely selection/classification rather than generation, you might also find success using a Cross-Encoder (like DeBERTa-v3). It is often much easier to train for classification/abstention with small datasets than a generative decoder. Hope this helps!