flac/encode.c: Fix an integer overflow
authorErik de Castro Lopo <erikd@mega-nerd.com>
Tue, 14 Feb 2017 10:21:54 +0000 (21:21 +1100)
committerErik de Castro Lopo <erikd@mega-nerd.com>
Tue, 14 Feb 2017 10:26:43 +0000 (21:26 +1100)
When encoding a 24 bit multichannel W64 file on a 32 bit system, a
calculation could overflow resulting in a "got partial sample"
error and early termination of the encoding process.

The fix was to replace the cast to `size_t` (which is only 32 bits
on a 32 bit system) with a cast to `uin64_t`.

Thanks to Janne Hyv√§rinen for helping to track down the problem.

Closes: https://sourceforge.net/p/flac/bugs/453

src/flac/encode.c

index 73c1576..be3869f 100644 (file)
@@ -1387,9 +1387,9 @@ int flac__encode_file(FILE *infile, FLAC__off_t infilesize, const char *infilena
                        case FORMAT_AIFF_C:
                                while(encoder_session.fmt.iff.data_bytes > 0) {
                                        const size_t bytes_to_read =
-                                               min (sizeof (ubuffer.u8),
-                                                       min ((size_t)encoder_session.fmt.iff.data_bytes,
-                                                               CHUNK_OF_SAMPLES * (size_t)encoder_session.info.bytes_per_wide_sample));
+                                               (size_t) min (sizeof (ubuffer.u8),
+                                                       min (encoder_session.fmt.iff.data_bytes,
+                                                               CHUNK_OF_SAMPLES * (uint64_t) encoder_session.info.bytes_per_wide_sample));
                                        size_t bytes_read = fread(ubuffer.u8, sizeof(uint8_t), bytes_to_read, infile);
                                        if(bytes_read == 0) {
                                                if(ferror(infile)) {