Removing out_mem from the encoder state.