Audacity Forum

Converting from float to int by truncating sounds wrong on many levels I can't properly understand what truncate means in this context...

bgravato wrote:I can't properly understand what truncate means in this context...

rounding down.

I understand the "idea", my failure to understand is in terms of programming/computing... what could it mean to truncate a float and get an int without float->int conversion?
But never mind I'm just being picky about languistics (Ed influences? hehe)

bgravato wrote:what could it mean to truncate a float and get an int without float->int conversion?

It's something like:

Multiply the original 32 bit float value by 2^23 then round down to get the signed 24 bit integer.
In 32 bit float notation, 0 dB is +/- 1
In 24 bit integer notation, 0 dB is +/- 2^23

steve wrote:

bgravato wrote:what could it mean to truncate a float and get an int without float->int conversion?

It's something like:

Multiply the original 32 bit float value by 2^23 then round down to get the signed 24 bit integer.
In 32 bit float notation, 0 dB is +/- 1
In 24 bit integer notation, 0 dB is +/- 2^23

Multiplying and rounding is not truncating... it's multiplying and rounding

steve wrote:

Gale Andrews wrote:From ImportPCM.cpp:

Code: Select all

// In general, go with the user's preferences.  However, if
// the file is higher-quality, go with a format which preserves
// the quality of the original file.

   if (mFormat != floatSample &&
       sf_subtype_more_than_16_bits(mInfo.format))
      mFormat = floatSample;

It looks like that was written at a time when there was no 24 bit option. Has there ever been such a time?

Those lines were last modified in 2002 (around the time of 1.0.0) when there was no sample format preference. 1.2.0 (2004) had the 24-bit format preference.

steve wrote:It looks to me like 24 bit files are converted to 32 bit float (surprising perhaps, but perhaps not a "bad" thing), but then on Export to 24 bit, the sample values are truncated to 24 bit. Export should properly convert to 24 bit (with dither if enabled) and not just truncated to 24 bit.

Is it possible to be a bit more definitive e.g. by testing this in 1.2.6 and HEAD and comparing sample values like for like?

steve wrote:@Gale - I vaguely recall some fuss a long time ago about something in the manual that said that converting from 32 bit float to 24 bit integer was done by truncating. Do you remember that? Does it relate to this issue?

I think all that was about the Manual stating that Audacity "by default" processed internally at 32-bit, when in truth it always does that irrespective of the sample format of the audio - there is no option.


Gale

Gale Andrews wrote:Is it possible to be a bit more definitive

Yes, certainly.

Gale Andrews wrote: e.g. by testing this in 1.2.6 and HEAD and comparing sample values like for like?

Sadly no, because there was a different bug in early versions of Audacity that caused 24 bit sample values to be calculated incorrectly and that didn't get fixed until well into 1.3.x versions.


Nyquist code to generate a sequence of every 24 bit sample value (about 6 min 20 seconds duration):
Nyquist code to read sample values in 24 bit format. All values should be integers. The sample values are displayed with one decimal place just to show that they are integers. Note: Nyquist runs in 32 bit float. If dither is enabled and the track that it is generating into is less than 32 bit float, the output will be dithered.

Test results:

1) Generate samples into a 32 bit float track.
The samples test as sequential integer values. Example output from 4min into the track : 2) Generate samples into a 24 bit track, dither = none.
Same results as 1 (as expected). Example output from 1 min into the track: 3) Generate samples into a 24 bit track, dither = triangle.
Sample values are still integers (as they must be for 24 bit) but there is a degree of random variation in the values due to the triangle dither.
Example output from 4min into the track : 4) Generate samples into a 32 bit float track. (the sample values are in sequence).
Export as 24 bit with dither = none.
Import the track and test the sample values.
The sample values are still integers in sequence as expected.
Example output from 4min into the track : 5) Generate samples into a 32 bit float track. (the sample values are in sequence).
Export as 24 bit with dither = triangle.
Import the track and test the sample values.
The sample values are still integers in sequence. Dither has not been applied.
Example output from 4min into the track (Note that the sample values are identical to test number 4) : 6) Modify the generator code so that we get some fractional values: Modify the sample reading code so that we get 2 decimal places by changing (setq *float-format* "%1.1f") to (setq *float-format* "%1.2f")
Generate samples into a 32 bit float track.
The samples test as sequential half-integer values. Example output from 4min into the track : 7) Export the track from test 6 as 24 bit with dither = none.
I would have expected integer results rounded to the nearest value but we actually get rounded down values.
Example output from 4min into the track : 8) Export the track from test 6 as 24 bit with dither = Triangle.
We would expect integer results with a degree of randomness to the values due to the dither.
We actually get results identical to test 7 - the values are rounded down.
9) Same as test 8 but with dither = shaped.
Again we get identical results as test 7.


10) Same as test 8 but with dither = rectangle.
Again we get identical results as test 7.


Conclusion:
Exporting 32 bit float audio to "signed 24 bit Integer PCM", using "Other uncompressed files" always rounds down the sample values and dither is never applied.

Results in Audacity 1.3.12 are the same as Audacity 2.0

Results in Audacity 1.3.10 are slightly different - the values appear to be rounded rather than rounded down. Dither is not applied even when selected in Preferences.
Audacity 1.3.4 also appears to round to the nearest. Dither is not applied even when selected in Preferences.

steve wrote:The samples test as sequential half-integer values. Example output from 4min into the track :

Code: Select all

    548848.00
    548848.25
    548848.50
    548848.75
    548849.00

7) Export the track from test 6 as 24 bit with dither = none.
I would have expected integer results rounded to the nearest value but we actually get rounded down values.
Example output from 4min into the track :

Code: Select all

    548848.00
    548848.00
    548848.00
    548848.00
    548849.00

Thanks for the tests.

So I'm clear, what values would you expect in 7 (what direction should the rounding go in)? And 1.3.10 and 1.3.4 do this?

I tested 32-bit float silence exported to 24-bit WAV with shaped dither set in Preferences in 1.3.2, 1.3.0 and 1.2.6. All WAV's had no dither applied.

So looks like two separate bugs in addition to #22 (or three if you regard 24-bit quality setting importing 16- or 24-bit files as 32-bit as a bug).


Gale

Gale Andrews wrote:...what direction should the rounding go in?...

Mathematically speaking "rounding" rounds to the nearest integer. Therefore ".00" through ".49" should round down to ".00" and ".50" through ".99" should round up to next ".00"