Normalize and DC offset correction

Programming and Development Nyquist
1

This is not really a practical plug-in which is why I’ve not posted it to the plug-ins page, but it includes some features that may be of interest for pug-in authors.

I wrote this plug-in to help me work out why the Audacity Normalize effect doesn’t work properly in all cases (a bug that is currently being fixed). As such it is closely based on the Normalize effect in Audacity 2.0. On Linux it is quite a lot slower than the standard Audacity effect but on Windows it is almost as fast.

The first thing required is a function to calculate maximum and minimum peaks.
The obvious method is to use the “peak” function:

(peak (s-max 0 s) ny:all) ; positive peak level
(peak (s-min 0 s) ny:all) ; negative peak level

The problem here is also one of the bugs (now fixed) in the Audacity Normalize. If the audio signal (s) is all above or all below the audio track centre line (DC offset is greater than the peak amplitude) then this does not work because it only looks at positive values and negative values, not at the “highest peak” and “lowest peak”.
A simple way to get this to work properly is to add a large offset to the sound, for example:

;;; get absolute peak values
(defun MaxPeak (sig)
  (- 10 (peak (sum 10 sig) ny:all)))  
(defun MinPeak (sig)
  (- 10 (peak (sum 10 (mult -1 sig)) ny:all)))

The next thing required is a method to calculate DC offset.
The way that the Audacity effect does it is too loop through the sound sample by sample, much like this:

(setq offset 0)                   ; initialise offset
(dotimes (i (truncate len))       ; step though samples
  (setq offset 
    (sum offset (snd-fetch s))))  ; sum all sample values
(setq offset (- (/ offset len)))  ; divide by number of samples

This works fine but looping through samples in Nyquist is really slow.
A much faster way is to use the function “integrate” which steps through the samples (in highly optimised C+ and sums them.
We can then look at the value of the final sample using “sref” (note that we can’t use “peak” because we need to know the signed value).

(sref 
  (sound (mult (integrate s)(/ (get-duration 1))))
  (/ (1- len) *sound-srate*))

“sref” interprets its time argument with reference to local time, so we need to apply “sound” so that “S” is evaluated correctly.


Finally we get down to the business of calculating the gain and offset and applying them. This is where it gets complicated because we have need to consider every combination of “with/without DC correction”, “with/without normalizing” and “linked/separate channels”.
I’ll not go through all of the detail, other than to say that if DC offset correction is to be used then it needs to be calculated before calculating the max/min amplitude of the audio so that the calculated max/min take into account the offset for that channel.
Normalize.ny (3.21 KB)

2

Steve is this still the vn. I tested yesterday, or have you modified the GUI based on my commenys?

Peter.

3

The GUI is the same (I forgot about the typo, I’ll fix that now).
About your GUI comments (and thanks again for testing).

_" When the “Normalize Amplitude?” is set to “No” the “Normalize to” and its attendant slider should both be greyed out and in-operable."_*
Unfortunately that is not possible in Nyquist plug-ins. The GUI is not “interactive” so it is not possible for one GUI element to affect another. I live in hope that one day a developer may take an interest and improve the GUI for Nyquist plug-ins but for now we have what we have, which is 4 “widget” types and (limited) text formatting.

*_" The slider will only max out at 0dB whereas one can type a positive number in the “Normalize to” input box

  • When you do type a positive number in the “Normalize to” input box and implement effect this generates clipping (so 4 above should not be allowed)"_**
    If this was a plug-in for “release” then I’d limit the maximum “used” value to 0 dB to avoid clipping.
    In fact I’ll add this - it’s just one line that ensures that the “used” value is not above than zero:
(setq amp (min amp 0))

Again, because the GUI is not interactive there is no way to “grey out” the OK button if an invalid value is entered. The “slider widget” only allows numerical characters, but there’s no way to stop someone from typing “-19.+099,123–++,.456” (which would be read as “-19”)


Minor typo: “Stereo Channels Indepedet” shoul read: “Stereo Channels Indepedent” "
Oops :blush: Fixed (but actually the spelling should be “Independent”) :wink:


The code is a bit different.
This version is a lot faster than the one that you tested.
As described in the text in my previous post, the DC offset is now calculated using “integrate” rather than stepping through the samples in a Nyquist/LISP loop.

I’ve also fixed a bug (that you missed :wink:). I think that if you invert your test sample and then try the old version you will find that the Normalize level is incorrect. It’s fixed in this version.

As the attachment in the start of this topic has not yet been downloaded I’ll just replace it with the new version.

4

Done.

5

Using INTEGRATE for computing the DC component is a great speed improvement, and usually it’s even sufficient to compute the DC component of a few seconds of a sound, because the DC component never changes, otherwise it’s no DC. This gives room for further speed improvements.

But again reading variable names like MinPeak and MaxPeak makes me go nuts:

A signal is not a graph. A signal consists of an AC component (Alternating Current, in audio processing this component contains the audio information) and a DC component (Direct Current, generally unwanted in audio processing, but sometimes useful in envelopes).

  • The Minimum value of a signal is always zero, there is no “Minimum Peak”.


  • The Peak value of a signal is the higher absolute value of the most-positive sample and the most-negative sample, computed from the AC component, with the DC component already removed.

This means that the PEAK value of the AC-component is the same as the PEAK value of the inverted AC-component, so there is no need for two peak variables, but it’s important first to remove the DC-component from the signal before computing the PEAK value.

The Nyquist PEAK function already takes care of computing the absolute value of the most-positive and most-negative sample, if the sound, given to the PEAK function as argument, is the AC-component without the DC-component.

Never confuse “most-negative” with “min” in signal processing, this leads to very hard to find bugs. Nyquist itself is a bad example in this regard, because S-MIN and S-MAX both treat signals like graphs and therefore return wrong values in the DSP sense.

Simplified version (I hope it’s easy to understand, but not really good coding style):

(defun normalize-mono (input-sound)
  (let* ((DC (sref (sound (mult (integrate input-sound)
                                (/ (get-duration 1))))
                   (/ (1- len) *sound-srate*)))
         (AC (sum input-sound (- DC)))
         (MaxPeak (peak AC ny:all)))
    (if (zerop MaxPeak) ; avoids divison-by-zero error
        (if (zerop DC) input-sound AC) 
        (scale (/ 1.0 MaxPeak) AC))))

In case of “normalizing without DC removal” the last SCALE line becomes:

(sum DC (scale (/ (- 1.0 (abs DC)) MaxPeak) AC))

Question: What is a practical use-case for “normalizing without DC removal”? Do I have overlooked something?

Sorry for the rant, the code includes all-in-all some very good improvement ideas. :slight_smile:

  • edgar
6

That’s nice. Yes there are some excellent improvement ideas. I’ll need to contemplate a while :slight_smile:

Speed.

DC is not “sound”, so a “sound” card has no right to pass DC, so the DC component of a sound recording should always be zero, so removing a DC component should be unnecessary.

Are you suggesting that the Normalize effect should “always” remove any DC component and that it should not be a user selectable option?

7

Now (six o’clock in the morning) a new version, rewritten according to the “AC and DC components” theory. This is probably still not the final version because I have e.g. still not found a good abstraction how the code could be MULTICHAN-EXPANDed to an arbitrary number of channels.

…but only in theory, I’m often enough forced to work with bullsh**t equipment, where removing unwanted DC is the most important function of all.

The point is that I never have found a single use-case where normalizing without removing unwanted DC ever made sense. But it’s also not a good attitude to bigheadedly say that nobody will ever need it. It often happens that one day I need some particular settings and the next day in a similar situation I need exactly the opposite. It’s difficult to say what is “unnecessary” and what not.

But currently there are some nonsensical settings in the plugin like “neither remove DC, nor normalize” what in end effect does nothing… ???
Normalize-2.ny (4.42 KB)

8

Hi Edgar,

the points that I made when I wrote the proposal were:

  1. DC offset removal (if present) should ideally be the first thing done after capture and before all other editing
  2. Normalizing/amplitude adjustment is normally the final editing function prior to exporting usefaul audio files and after all other editing has been done.
    Hence the need /desire to separate the two effects - am wrong in this?

I do agree that if a user has DC on the signal then if they’ve not removed it prior to other editing then it is useful to use the opportunity of normalizing to correct the DC (but rememeber that until 1.3.13 and 2.0 we were advising users to make amplitude adjustments with Amplify rather than Normalize, as until then Normalize insisted on working on each channel of a stereo pair independently).

The other reason I don’t like “hiding” DC removal in the Normalize function is that it tends to blind the user to the fact that they have DC on the signal, which is normally the sign of a poor soundcard and should really be dealt with at the hardware level by replacement.

My other gripe is that if you run Normalize just to remove DC offset ,with amplitude adjustment turned off, the progress dialog still ays "Normalizing … " when it is clearly doing no such thing.

Peter.

9

Another problem with bullsh**t equipment is that the offset may not be constant DC but rather a slowly drifting offset. I think this is a case for offering more than one type of offset correction (probably not as part of the Normalize effect but rather as a separate “Offset Correction” effect). A drifting offset is subsonic AC rather than DC, so a high-pass filter is required to remove it rather than (only) DC offset correction, but this is drifting off into a different topic.


I’ve been thinking hard for an example of when Normalizing without removing DC makes sense. The only example that I can think of is if Audacity is being used for processing/editing non-audio signals (which some users do use it for). So even though it is very much a fringe case I think that DC correction should be retained as an option.

There is also the issue of speed.
In the Audacity effect, calculating the DC offset requires looping through the samples in an analysis stage and this definitely slows down the processing (by about 25% on my computer). There’s a comment in the Audacity Normalize code:
“// we don’t need an analysis pass if not doing dc removal”
So it looks the intention is to minimise processing time.
In the Audacity Normalize implementation calculating minimum and maximum sample values is (so I’ve been told) a lot faster than calculating the offset.
In Nyquist, using the “integrate” function there appears to be little if any speed penalty in calculating the DC offset.


It does seem nonsensical to include an option to disable normalizing in the “Normalize” effect :confused:
I think that this possibility exists because there is no separate “Offset Correction” effect. It would imho make a lot more sense for the “Normalize” effect to always normalize (with DC correction as an option) and if offset correction is required without normalizing then an “Offset Correction” effect should be used.

We see quite a few posts from users that have DC offset but have not found the correction tool. If offset correction was a separate tool then it would be a lot more visible as well as resolving the nonsense of the “do nothing” option in Normalize.


Thanks for the Normalize-2.ny code. It’s very instructive.
It also very clearly indicates the bizarreness of normalizing without DC offset correction.
(if normalize without DC correction → normalize the AC signal and add the DC offset)

10

The bad news is that the Nyquist speed improvement is caused by the fact that Nyquist loads all samples from Audacity section into memory, and analyzing a sound in memory is of course much faster than reading the samples from disc.

But unfortunately this means that the Nyquist speed improvement is based on the same effect that causes the well-known Nyquist memory problems, or in other words, with a proper Audacity Nyquist implementation, Nyquist would not be faster than the Audacity “Normalize” code, it would more likely be slower.

This also means that the current plugin code will crash Audacity if the selection does not fit into the computer’s memory. But it’s an experimental plugin, and I think that we still work in the right direction.

More detailed answer will come later …

  • edgar
11

Unfortunately my code is wrong, because “normalize without DC correction” must be:

  • normalize the AC signal + add the DC offset, amplified by the same amount as the AC signal

and this is indeeed the same as:

  • MaxPeak = difference of the most-positive sample to 1.0
  • MinPeak = difference of the most-negative sample to -1.0
  • amplify the entire signal AC+DC by: (min (abs MinPeak) (abs MaxPeak))

This is the reason for the “without DC removal we need no analysis pass” comment in the Audacity C/C++ code, and this is indeed faster than my AC/DC code. I still try to understand the code in “audacity/src/effect/Normalize.cpp” … :frowning:

  • edgar
12 



            ;; no dc-removal, but normalizing
            (if (zerop pk)
                input-sound  ; silence
                (mult (/ amp (+ pk (abs dc)))(sum dc ac)))))))
13

Another minor problem is that the DC offset calculation is not always exact.

(defun compute-dc (input-sound)
  (sref (sound (mult (integrate input-sound)
                     (/ (get-duration 1))))
        (/ (1- len) *sound-srate*)))

If the computed value is not exactly the same as actual DC offset then for a constant, non-zero signal

(peak (sum input-sound (- dc)) ny:all)

will be non-zero so the “silent” but ever so slightly offset sound will be normalized to the target value rather than simply having the offset removed.
A solution is to allow for a little inaccuracy by changing

(if (zerop pk)

to something like:

(if (< pk (db-to-linear -96))
14

I think this version could be adapted for “multichan-expand’ing” if dc-list and peak-list were arrays rather than lists but I’ve just used a “Do” loop instead.

I’ve removed the nonsensical “don’t normalize” option.
“Remove any DC offset: (default Yes)” has been reversed to “Retain DC offset: (default No)”.
“Normalize Stereo Channels Independently: (default No)” has been reversed to “Link Stereo Channels: (default Yes)”.

This version supports multi-channel sound (although Audacity currently only supports a maximum of 2 channels).
Normalize-3.ny (2.21 KB)

15

Re-thinking all that stuff over and over again and taking into account that PEAK already includes an ABS function then this means that “normalize without DC correction” is nothing else but:

(mult s (/ amp (peak s ny:all)))

and in this case there is indeed no need for a DC analysis.

Is this really true? … I think yes.

  • edgar
16

This is not nonsensical (is not nonsensical = sensical?), I need “DC-removal without normalizing” really often. What I meant was:

  • Normalize with DC-removal - makes sense
  • Normalize without DC-removal - makes sense?
  • Do not normalize with DC-removal - makes sense
  • Do not normalize without DC-removal - is complete nonsense

Also the trick using peak-lists with MAPC for multichannel sounds is a great idea.

  • edgar
17

Quotes from waxcylinder in post 178468 on the first page of this thread:

DC-removal (if DC is present) must be the very first step after importing. This must be hammered into every user’s brain.

Normalizing is the first step after importing, while the last step before exporting is usually called “mastering” and goes far beyond normalizing, it usually includes dynamics compression and intentional peak clipping a.k.a. limiting. Unfortunately Audacity has no real-time audio effects, so it’s nearly impossible with Audacity to do dynamics compression or mastering in a meaningful way, so with Audacity there is no other way than using normalizing instead of mastering, but this is nothing but a limitation imposed by Audacity. In real life normalizing and mastering are two pretty much different things.

I fully agree, there should appear a “Your soundcard is sh*t” window to make the user aware that s/he has paid money for a piece of crap.

  • edgar
18

ROFLMAO

Seriously though Edgar, does your response mean that you would support the separation of the DC-Removal and Normalize functions? My proposal on the Wiki could do with a bit more support. :sunglasses:

Peter.

19

The practical situation looks like this:

  • Normalize is a two-pass process: (1) find the maximum peak, (2) amplify to the given level
  • DC-Removal is a two-pass process: (1) find the DC component, (2) subtract the DC component


  • Normalize (1) and DC-Removal (1) both can be done in the one and the same pass
  • Normalize (2) and DC-Removal (2) both can also be done in the one and the same pass

If you want to perform “Normalize with DC removal” with two separate Normalize and DC-Removal functions then you need 4 passes instead of 2 passes, and if “Normalize with DC removal” is needed with long audio recordings it makes a difference if you need to wait ten mintues or twenty minutes until the work is done. You can save a lot of time if both functions are unified in the same effect. This is the only reason.

From the “logical” workflow viewpoint it seems to make sense to separate the functions, but in practice it is no good idea.

  • edgar
20

Ok so I can see the need for retaining DC removal in the Normalize effect for:

  1. those who feel the need for speed
  2. those who overlooked it after capture

But I would still like to have an additional effect which just does DC-removal.

  1. It makes the concept of DC on the signal more obviouus to the user and more discoverable in the documentation
  2. If James ever gets around to enhancing the effects hot-key bindings (in 2.0.1 alpha now) to become really HOT-keys with immediate action and no dialog then I for one would definitely want separate DC Removal and Normalize effects (but happy to have the DC removal still in the Normailize - I just wouldn’t use it there).

Peter