Could collections hypothetically store keys and values inline?

Speed aside, you can't create a &Vec or &mut Vec without literally having a valid Vec in memory and if you have a valid Vec in memory then which frees the allocation when dropped and you need to do CompactVec::into() and then CompactVec::from()—which requires ownership and so requires ugly swapping if you're, for example, working with an array of CompactVecs.

For example, what would have been array_of_arrays[i].push(foo) becomes

{
    let mut temp: Vec<_> = std::mem::take(&mut array_of_arrays[i]).into();
    temp.push(foo);
    array_of_arrays[i] = temp.into();
}

and similarly at every other place of usage. The ergonomics of that are sufficiently bad that I don't think anyone is going to do this outside of situations with very strict memory constraints.