Rearrange mode structure to avoid wasting memory from padding.