Audacity Forum

PGA wrote:

Gale Andrews wrote:...what direction should the rounding go in?...

Mathematically speaking "rounding" rounds to the nearest integer. Therefore ".00" through ".49" should round down to ".00" and ".50" through ".99" should round up to next ".00"

Actually Peter, Gale is quite "mathematically" correct. There are several mathematical methods for rounding - see: http://en.wikipedia.org/wiki/Rounding_numbers

WC

waxcylinder wrote:Actually Peter, Gale is quite "mathematically" correct. There are several mathematical methods for rounding - see: http://en.wikipedia.org/wiki/Rounding_numbers

Which just goes to show that there really is a world-wide conspiracy to confuse the hell out of everybody!

Gale Andrews wrote:So I'm clear, what values would you expect in 7 (what direction should the rounding go in)? And 1.3.10 and 1.3.4 do this?

I would have expected rounding to the nearest 24 bit integer value, but I don't think that it matters which kind of rounding is used. I think that you get the same amount of quantize error whether you round up, down, toward zero or away from zero. If I was programming it I'd go for whichever is fastest. If there's no difference in speed, I'd go for whichever is simplest to code. If there's no difference in code complexity I'd go for rounding to the nearest.

Audacity 1.3.10 and lower almost round to the nearest but there's something wrong because some sample values are rounded the wrong way. I'm not sure what's going on there, but it's not quite right.

PGA wrote:Which just goes to show that there really is a world-wide conspiracy to confuse the hell out of everybody!

steve wrote:I don't think that it matters which kind of rounding is used. I think that you get the same amount of quantize error whether you round up, down, toward zero or away from zero.

OK, but if so what is wrong with the rounding down, or for all we know so far, rounding towards zero, that happens now?

Or another question, is there something constituting a bug in this image in one of your earlier posts ?






Gale

Gale Andrews wrote:OK, but if so what is wrong with the rounding down, or for all we know so far, rounding towards zero, that happens now?

Having thought a bit more about the issue of rounding, rounding down (or up) will introduce a small (extremely small) DC offset. Although insignificant for practical purposes, that should not really happen. Rounding toward zero will cause a minute reduction in amplitude and rounding away from zero will create an equally small increase in amplitude. Rounding to the nearest value will produce the closest match with the original (which I think is what happens when converting to 16 bit without dither).

Converting from a higher bit format to a lower one must involve some form of rounding - the problem is those cases when dither is not being applied on export when converting from 32 bit float down to 24 bit when dither is enabled in Preferences.


In the tests below a 440 Hz sine wave is generated in 32-bit float, then a duplicate made. The original is inverted so that it can be used to "cancel out" the sine wave from the duplicate and so reveal any changes to the duplicate. For the 16 bit tests the mix of the two tracks is amplified by 50 dB so that the difference shows up clearly on the spectrum plot. For the 24 bit tests the mix has been amplified by 100 dB.

Test 1
The duplicate is converted to 16 bit without dither. The noise we see here is characteristic of quantize errors. Note that in the mid-range there are peaks up to about -75 dB.
Test 2
This image shows shaped dither noise when converting from 32 bit float down to 16 bit. Note that for frequencies below 4 kHz the noise floor is around -90 dB, so the noise in the region that hearing is most sensitive, the noise level is actually a lot lower than without dither.
Test 3
The duplicate is converted to 24 bit without dither. The noise we see here is characteristic of quantize errors, but we have an extra peak at 440 Hz (because rounding is not going to closest value the inverted sine does not quite match the sine component of track 2). The extra peak at 440 Hz shows up more clearly with a small FFT size (second image). Looking at the first image we see a noise level in the mid range of about -75 dB.


Test 4
Here we see the 24 bit sine wave correctly dithered (shaped dither). The noise level in the mid range is below -85 dB. This is what we should get when exporting from a 32 bit float track to a 24 bit file but we don't - we get the same results as test 3.