DC offset trickery

[steve]

Doesn't the contribution by edgar-rft to the thread referenced above expain what is going on? Surely the asymmetry of peaks above and below the axis is irrelevant to whether there is DC offset? What matters is whether the integral of the waveform over a reasonable length of time is nonzero

You're absolutely correct. When you understand what is happening, then the behaviour is not surprising.

The second graph, over a longer time period, shows patches of a waveform with DC offset mixed with some zero-offset silence, a combination I would be surprised to see in reality

but they do occasionally occur (though not as badly as my illustration), and cause a lot of confusion when they do.

I've been searching for a good "real world" example, and I've found one. It's quite a nice trumpet sample that has no obvious clipping.
Try Normalizing this sample to 0 dB with "Remove any DC offset" enabled.

Also try the Nyquist DC offset tool from here (default settings): http://forum.audacityteam.org/viewtopic ... 13#p195713

[Gale Andrews]

non-linear gain

Thanks, Steve.

This will have to be all explained in course of time to mere mortals (non-audio-engineers).

Can you define "non-linear gain" concisely as part of that? Googling doesn't produce anything very usable.

So, are we saying


has negative DC offset because in any given length there is a greater number of samples below zero which outweighs the samples above zero having greater amplitude?

And so when Audacity tries to remove the offset, it moves the waveform upwards, but the result is "wrong" (obviously not because it still "looks" asymmetrical) because the sum is still non-zero?

And:

Code: Select all

(setq step (round (/ *sound-srate* 4))) ; 1/4 second in samples

(force-srate *sound-srate*
  (snd-avg s step  step op-average))

shows this "non-zero-ness" for quarter second steps, so shows very clearly where the Audacity DC remove is still wrong? (And this code only works on mono tracks, right)?

This visual result isn't very "scientific". Is it possible to get some debug output that says which 1/4 second regions were offset?

Your Nyquist "DC offset" has multiple removal options.

Is the "absolute" one what Audacity uses?

Would your default "dynamic" method (clearly better visually than Audacity removal) show no 1/4 second sections with offset?

Is the dynamic always the best method? Or high pass if you can be sure to avoid clicks? Or have you got to use different methods each time? Can the result be "verified" statistically?


Gale

[steve]

Can you define "non-linear gain" concisely as part of that?

Probably easiest by example.

Let's say that we have a series of sample values from an audio signal:

-10 dB
-15 dB
-20 dB
-25 dB

If we amplify the signal by +1 dB we would expect the sample values to become:

-9 dB
-14 dB
-19 dB
-24 dB

If we amplify by a further +1 dB (+2 dB total) we would expect the sample values to become:

-8 dB
-13 dB
-18 dB
-23 dB

This is "linear amplification" (linear gain) and is what an "ideal amplifier" will do.

In the real world of analogue electronics, amplifiers are never quite "ideal", though usually they are very close with modern electronics.
Let's say that we have a very bad amplifier and after attempting to amplify our original signal by +2 dB we get the following sample values:

-8.3 dB
-13.1 dB
-15.9 dB
-22.7 dB

We can see that the highest sample value (was -10 dB) has been amplified by +1.7 dB, whereas the lowest sample value (was -25 dB) has been amplified by +2.3 dB.
The relationship between the input signal (the original sample values) and the output signal (the amplified sample values) is not linear - the "gain" changes according to the amplitude. This is an example of "non-linear gain".

This image shows the typical current gain of a single transistor circuit. It shows that at very low current, the gain is non-linear, but then there is a (fairly) linear region as the input signal increases, until the transistor becomes "saturated", at which point the gain begins to decrease (non-linear):

[steve]

has negative DC offset because in any given length there is a greater number of samples below zero which outweighs the samples above zero having greater amplitude?

It has negative DC offset because the overall sum of samples below the 0.0 centre line outweighs the overall sum of samples above the line.
For "DC" offset we are interested in the net value of all the samples. By definition, "DC" is a "constant" value.

For a signal that has no DC component, if we add up the values of all samples, the total is zero. This is how the Audacity DC offset removal works - it adds up all of the sample values and calculates the average sample value (which is zero if there is no DC offset). If the average sample value is not zero then there is a "DC offset".

Quick definition: "DC offset" is how much the average sample value is offset from zero.

Coming back to your definition:
"has negative DC offset because in any given length there is a greater number of samples below zero which outweighs the samples above zero"
there is a potential problem:
What if the "average sample value" in the first half of the audio is different to the average sample value in the final half of the audio?
Overall, the samples may balance out (no DC offset), but if we look only at the first half we would see DC offset, and if we look at the second half we would see a different DC offset. This is where the Audacity DC offset falls down.

As an illustration, consider:

firsttrack000.png (9.93 KiB) Viewed 2580 times

If we apply DC offset correction to the first half of the track we get:

firsttrack001.png (10.23 KiB) Viewed 2580 times

and if we then apply DC offset correction to the second half of the track we get:

firsttrack002.png (10.09 KiB) Viewed 2580 times

but if we applied DC offset correction to the entire track, then we would have (no change):

firsttrack003.png (9.93 KiB) Viewed 2580 times

The problem here is that there is a gradually drifting offset, but overall the offset is zero.
The DC component (overall) is zero, but we can clearly see that there is a negative "bias" at the start of the track which gradually becomes a positive bias at the end of the track.

What my "Dynamic bias adjustment" does is to look at a rolling average rather than an overall average. The period for each subset of samples must be great enough to not be influenced by low audio frequencies. The result of "dynamic bias adjustment looks like this:

firsttrack005.png (9.25 KiB) Viewed 2580 times

To complete this series of illustrations, this is the effect of the high-pass filter method:

firsttrack006.png (9.13 KiB) Viewed 2580 times

[steve]

shows this "non-zero-ness" for quarter second steps, so shows very clearly where the Audacity DC remove is still wrong? (And this code only works on mono tracks, right)?

This visual result isn't very "scientific". Is it possible to get some debug output that says which 1/4 second regions were offset?

For a "scientific" measure, apply this code to a track:

Code: Select all

(defun analyze (sig)
  (let ((step (round (/ *sound-srate* 4)))) ; 1/4 second in samples
    (snd-avg sig step step op-average)))
(multichan-expand #'analyze s)

Then use "Sample Data Export" (Analyze menu) to export a table of 1/4 second bias measurements.

Is the "absolute" one what Audacity uses?

Yes.

Would your default "dynamic" method (clearly better visually than Audacity removal) show no 1/4 second sections with offset?

Here is the Sample Data Export (linear scale) for the waveform in my previous post: http://forum.audacityteam.org/viewtopic ... 46#p202646

Audacity DC Offset correction:

Code: Select all

1	-0.15000
2	-0.05000
3	0.05000
4	0.15000

High-pass filter (LADSPA DC Removal):

Code: Select all

1	-0.01694
2	0.00907
3	0.00907
4	0.00907

Dynamic Bias Adjustment:

Code: Select all

1	-0.00520
2	-0.00011
3	0.00011
4	0.00053

[steve]

Is the dynamic always the best method? Or high pass if you can be sure to avoid clicks? Or have you got to use different methods each time?

Based on my Nyquist implementation of "dynamic bias adjustment":

"Dynamic" is best if there is a changing bias.
"Absolute" is marginally more accurate than "Dynamic" if there is constant DC offset (a "bias" is present that does not vary over time).
"Absolute" is better that "High-pass" if there is constant DC offset.
"High-pass filter" is better than "Absolute" if there is a non-constant bias, but not as good as "Dynamic".

The results can be verified with the method described in the post above this one.

A C++ implementation of "Dynamic" could improve the accuracy for "pure" DC offset (a "bias" is present that does not vary over time).

I think that ideally, offset correction should test if non-constant bias is present and use "dynamic" if it is, and "absolute" if it isn't. With careful implementation this could probably be done with very little performance overhead as Audacity already loops through the samples to calculate the overall offset.

[Gale Andrews]

Thanks for describing "non-linear gain".

Here is the Sample Data Export (linear scale) for the waveform in my previous post: http://forum.audacityteam.org/viewtopic ... 46#p202646

Can you post a WAV file of that - it might be easier.

Audacity DC Offset correction:

Code: Select all

1	-0.15000
2	-0.05000
3	0.05000
4	0.15000

High-pass filter (LADSPA DC Removal):

Code: Select all

1	-0.01694
2	0.00907
3	0.00907
4	0.00907

Dynamic Bias Adjustment:

Code: Select all

1	-0.00520
2	-0.00011
3	0.00011
4	0.00053

Sorry I am not following what these "bias measurement" values are showing.

I was assuming that the less you could "see" in the track after

Code: Select all

(setq step (round (/ *sound-srate* 4))) ; 1/4 second in samples

(force-srate *sound-srate*
  (snd-avg s step  step op-average))

the better.

In terms of your sets of four values, which is "most accurate" and why?

And what values would the "perfect" result show?


Gale

[Gale Andrews]

"Dynamic" is best if there is a changing bias.
"Absolute" is marginally more accurate than "Dynamic" if there is constant DC offset (a "bias" is present that does not vary over time).
"Absolute" is better that "High-pass" if there is constant DC offset.
"High-pass filter" is better than "Absolute" if there is a non-constant bias, but not as good as "Dynamic".

In http://forum.audacityteam.org/viewtopic ... 93#p195694 you were recommending absolute + filter as best for non-constant offset.


Gale

[Gale Andrews]

Let's consider the "bassoon" "dc-offset-sample.flac" ( http://forum.audacityteam.org/viewtopic ... 48#p195648 ) and "trumpet-offset-example.flac" ( http://forum.audacityteam.org/viewtopic ... 69#p196369 ) .

99 users out of 100 would probably expect offset removal to move the bassoon waveform down and the trumpet waveform up, but that's another issue.

So is Audacity offset removal getting both these wrong because of non-constant bias, or because parts of the selection have no bias? I assume the former? I notice if you just select the first trumpet note and remove offset with Normalize, Audacity seems to do a respectable job.

Are we now saying that non-constant offset is a feature of brass recordings, or a feature of brass recordings made with non-linear equipment, or a feature of any type of recordings made with non-linear equipment?

Are we now saying that asymmetrical recordings made with good equipment should be fine - they are asymmetrical (so look to a novice like they are in need of offset removal) but won't have offset?


Gale

[steve]

Can you post a WAV file of that - it might be easier.

It was just a temporary test tone. I don't remember the frequency that I used, but this file has the same offset:

bias.wav

(86.24 KiB) Downloaded 79 times

Sorry I am not following what these "bias measurement" values are showing.

They are showing the amount that the audio is offset in each 1/4 second period. The audio sample is 1 second duration, so there are 4 offset readings.

I was assuming that the less you could "see" in the track after

Code: Select all

(setq step (round (/ *sound-srate* 4))) ; 1/4 second in samples
(force-srate *sound-srate*
  (snd-avg s step  step op-average))

the better.

The closer the samples are to zero amplitude the better.

In terms of your sets of four values, which is "most accurate" and why?

The "Audacity DC Offset correction" is absolutely accurate for "DC" (overall bias).
-0.15000 + -0.05000 + 0.05000 + 0.15000 = 0.00000
However we can also see that there is an offset in the first 1/4 second of -0.15, and an offset in the second 1/4 second of -0.05, and so on.

The LADSPA effect shows a significant offset in the first 1/4 second of -0.01694. After the first 1/4 second it settles down to an offset of 0.00907

The "Dynamic bias adjustment" shows an offset in the first 1/4 second of -0.0052, which is much less than either of the other methods. After the first 1/4 second it settles down to a tiny offset of 0.00011, with an increase in the offset at the end, but still a lot better (closer to zero) than the LADSPA effect.

And what values would the "perfect" result show?

For the first and final 1/4 seconds a "perfect" result is debatable. It's a bit like the situation with repairing clipped audio, where it is impossible to know what "should" replace the truncated audio. The best that can be done for the start and end of the selection is to make intelligent guesses.

Other than the initial and final 1/4 seconds, the samples should ideally be zero.
An intelligent guess for the initial and final 1/4 seconds is that they should be close to zero. The "unknown" factor is that there is no way of knowing whether the very first and very last samples in the audio selection should be at zero or a non-zero value. Probably the best guess for the first sample would come from extrapolating the offset from the third 1.4 second and the second 1/4 second to the first 1/4 second, then taking an average of the extrapolated value and the actual offset value for the first 1/4 second. The Nyquist implementation is not that sophisticated as it would be too slow.