Noise removal "tinklebells" and how to fix them

[Paul L]

I think you just answered a different question from what I asked.

When doing noise removal, not accumulating the profile, I believe it is hazardous to take just one window and compare it to the profile. Look closely at the spectrogram of sound with rectangular window of 2048. Many bands appear to fluctuate rapidly in magnitude from sample to sample, even within a note or a vowel. Now remember that the algorithm only sees every 1024th value in any one of those bands. If you get an unlucky choice, one of these might dip below the noise threshold for that band, causing the naive algorithm to drop out that band for a whole window of 2048.

But take just one other window, and take maximum, and you reduce the probability of these artifacts. The algorithm becomes "insensitive" where it should be. But then it also becomes insensitive where it shouldn't, in the pauses, and so we need other compensations like sensitivity.

Now it needn't be maximum of two, it might be average of two, or midmost of three... those others might work to prevent artifacts in foreground sound, but something must be done that considers one or more neighboring windows, not just considering the center window alone.

Minimum of two -- what existed before -- makes the algorithm TOO "sensitive" and perhaps spreads the artifact over TWO windows of foreground sound (overlapping for 3/2 times 2048 samples).

If the comments in the Wiki about the noise removal algorithm were by Dominic -- and part of it replicates his comments from code -- then it seems he was anxious about the tinkles in the quiet parts, which is well, but his solution introduced these other artifacts in the foreground.

[Paul L]

Some history:

Version 5979 by Dominic in 2007 made major changes introducing the algorithm we now have. Something rather different existed before which did appear to do averaging within each band in GetProfile. Some other algorithm with the comment

Code: Select all

   /* try to eliminate tinkle bells */

was doing frequency smoothing too by some other means in RemoveNoise.

There was no attack/decay, but a sort of time smoothing that was an exponential decay only forward in time. There was no queue of buffers. One window, only, was compared with the profile values.

There was no attenuation factor -- instead gain of 1 was assigned to non-noise, 0 to noise, for true removal, not reduction (though with smoothings). There was one slider controlling "level" which was more like a "sensitivity" affecting the criterion for deciding that a bin of a window is noise or not-noise.

The big change in 5979 introduced attack/decay time, the reduction factor, and the frequency smoothing, but eliminated level.

So it remained until version 10512 in 2010 by Marco Diego Aurelio reintroduced sensitivity.

It looks like the smoothings and the sensitivity all exist to combat the tinkles, but nothing completely satisfactory has been found. I don't thnk Dominic or Marco could be pressed for comment now?

The treatment of the profile has gone from averaging over the whole profile (of the LOGARITHM of power, just ignoring the zeroes -- huh? That's not RMS?) to the present thing that takes minima in each of many 50 ms windows, then takes maximum of the minima (got that?).

[Paul L]

My first trial using the max-of-min algorithm with a finer step size did NOT produce desirable results, for a reason that I should have anticipated. Just as unlucky values make the minimum too small when doing the removal, they can also do that when determining the noise thresholds, and the finer step made that more likely to happen, so my thresholds were too low for many bands and I ended up removing too little noise.

A simple (arithmetic) mean power computation over the whole profile seems NEVER to have been the algorithm, simple as it sounds! Instead the most ancient versions used this unexplained GEOMETRIC mean of power... with the infinities from log of 0 just thrown out of the sum!

[steve]

The algorithm change in Audacity 1.3.3 (r5979) was, for most material, a considerable improvement over the older algorithm. In particular, it was/is much better for music that has very low level noise as there is insignificant effect on the music while giving a good amount of reduction to the noise during "silences". However, for some types of material, the old algorithm was demonstrably better. This was particularly noticeable with bass notes over a high level of hiss: with the new algorithm, the hiss could be heard through the notes. The Sensitivity slider was introduced specifically to reintroduce the benefit of the old algorithm into the new version.

Interesting about the different ways that the profile has been calculated over the years. I suspect that the "best" way depends on the type of noise being removed. I'd guess that if the noise is very constant (such as generated white noise), it probably does not matter very much which method is used, other than how much "Removal" and how much "Sensitivity" to use.

Perhaps the profile (or corrective gain) should take into account the equal loudness contour for an assumed listening level?

[Paul L]

There is still dissatisfaction with noise removal and a sense that it is inferior to other products.

Do you think I am onto something with using min instead of max to avoid unwanted artifacts in foreground sounds?

I am convinced from these spectrograms that I have shown.

I am not yet convinced that I have a better idea for analyzing the profile.

[steve]

The level of satisfaction seems to be very dependent on how much the user is expecting of the effect (the main reason why raising expectations with the name Noise "Removal" is a bad idea imo). For reducing gentle hiss from a reasonably good "home studio" recording, I find the current effect very acceptable. For slightly higher levels of noise, or particularly sensitive audio, tweaking the settings for optimal results can be quite fiddly. This seems to be one of the main problems for less experienced users who would welcome a simpler effect (not easy to do without sacrificing quality). For more difficult material (such as more than a little noise), I think the Audacity effect starts to fall behind some of the alternatives.

Do you think I am onto something with using min instead of max to avoid unwanted artifacts in foreground sounds?

For the type of material that I generally deal with, tinkly artefacts in foreground sounds are rarely the major problem, so I've not done much testing for that. Consequently I don't feel that I can give an informed opinion at this stage.

There are certainly some bugs that you have identified that are worth fixing, such as the incorrect attack/release times (yes I think it should probably be "release" rather than "decay" as it seems to be much like the "release" of a dynamic processor). If you can make separate patches for each bug I expect that they can be applied quiet soon after the 2.0.6 release.

I am convinced from these spectrograms that I have shown.

The most important test - does it sound better?

I am not yet convinced that I have a better idea for analyzing the profile.

and I'm not sure that is the major problem.
There is certainly a problem with how low frequencies are handled.

Here's an example - I don't know why this happens, but perhaps you can fathom it:

1. Generate a 10 second "Chirp" from 20 Hz to 200 Hz, 0.8 to 0.1 amplitude.
2. Generate 10 seconds of silence.
3. Take the noise profile from the silence and apply it with the default settings.

Ideally, nothing should happen (because we are removing "silence"), but that is not what we see.

Now repeat but this time make the noise profile from 10 seconds of pink noise, amplitude -30 dB (not unreasonable considering some of the audio samples sent in). Why does this happen?

tracks006.png (22.46 KiB) Viewed 3853 times

[Paul L]

For fun, try mixing your test cases with the inversion of the original chirp. With the pink noise example, there are some curious "chattery" artifacts. For both, there is a slow pulsing. Now look at the original chirp in spectrogram with 2048 rectangular windows, notice what happens in the lowest band, and notice how that relates to the difference waveform as you view it in Waveform dB.

I guessed even before trying these out for myself that you are seeing the effects of that bug that blanks out the DC component before inverse fft.

Try the pink noise with 48 and no attack and no smoothing. Subtract the original and listen to all the weird whispering. You see, I thought something like this is what the messed up Isolate feature ought to be doing, and hearing so much junk in the foreground noise convinced me that something had to be done about it.

I tried all this in 2.0.5 and then in my build. The first example, with silence for profile, does nothing to the sound as expected. The difference of tracks is not exactly 0 but had peak amplitude of -127.6 dB.

For the pink noise example, the differences between the tracks are mostly inaudible, with a little squeak near the end. I don't know yet how much of that is due to the DC offset fix, and how much else to that replacing of min with max.

[Paul L]

For slightly higher levels of noise, or particularly sensitive audio, tweaking the settings for optimal results can be quite fiddly.

Shouldn't we make it less so then? I think time smoothing, frequency smoothing, and sensitivity exist to you can compensate for threshold setting by profiling that isn't good enough. For my proposed change to fix foreground sounds, there is then too little sensitivity to background sounds. I would rather seek the smarter profiling algorithm so results are often good with no smoothings or sensitivity.

For more difficult material (such as more than a little noise), I think the Audacity effect starts to fall behind some of the alternatives.

How so? Results or convenience or both?

Do you think I am onto something with using min instead of max to avoid unwanted artifacts in foreground sounds?

For the type of material that I generally deal with, tinkly artefacts in foreground sounds are rarely the major problem, so I've not done much testing for that. Consequently I don't feel that I can give an informed opinion at this stage.

I was using extreme settings on some voice: 48 dB reduction and no smoothing. I was calculating the difference from the original. I was in short trying to do what Isolate is supposed to do on one interpretation -- just pass the pure "noise" part as the program detects noise. Sensitivity affects "what is noise" but smoothings do not. I think that test should do what is expected, and I made a great improvement that takes the foreground parts and makes SILENCE of them. If we fix Isolate to do something like this, silence for the foreground is what I want to hear, or else I won't trust the effect. At least that was my own experience with Noise Removal in the beginning: I came to mistrust it because Isolate was lying to me. Then I learned to ignore Isolate, then to subtract sound from sound for myself, and still I was not satisfied.

There are certainly some bugs that you have identified that are worth fixing, such as the incorrect attack/release times (yes I think it should probably be "release" rather than "decay" as it seems to be much like the "release" of a dynamic processor). If you can make separate patches for each bug I expect that they can be applied quiet soon after the 2.0.6 release.

What is the procedure for making and using .patch files? I don't know it. If I make a sequence of changes to one file that may each get accepted or not, should I make each change based on head and have merge problems, or make my patches cumulative according to some prioritization of these fixes?

I am convinced from these spectrograms that I have shown.

The most important test - does it sound better?

As I said, extreme settings to do what Isolate ought to do, not what I would use to treat speech for real with only slight reduction, but yes. There are noticeable distortions in the words with 48 dB reduction, and I know that lesser attenuation or smoothing merely mitigate that, but I am happier knowing I can eliminate them instead.

I am not yet convinced that I have a better idea for analyzing the profile.

and I'm not sure that is the major problem.

I have had another idea, that for each band, we find some high quantile of the power for each bin... Of course max is very easy to find but might be too sensitive and would be corrupted by just one click somewhere. So not the 100th percentile. Outlying lows make the present max-of-min procedure work badly when the noise is examined more closely with less step between windows.

There is certainly a problem with how low frequencies are handled.

Well, didn't I tell you that in the earlier thread? The DC components always got zeroed and that explains the oscillating envelope for the pink noise profile and the nonzero effects for the silence.

Now repeat but this time make the noise profile from 10 seconds of pink noise, amplitude -30 dB (not unreasonable considering some of the audio samples sent in). Why does this happen?

What do you think should happen with pink noise? I think the chirp should be likewise unaffected by removal of pink noise. But instead there are whispery artifacts, which are obvious in 2.0.5 and less so with the change from max to min. I think they could be made zero or close to it by ANOTHER variation in procedure we haven't discussed yet.

The noise sample, and the profile, are transformed by FFT with rectangular window. A Hann window is used only to blend the overlapping windows together after inverse FFT.

The alternative would use a Hann or other window BEFORE FFT.

This was what Dominic did, until version 2200. The comment for the revision is

Applied Craig DeForest's fix for the Noise Removal (window after
FFT) and fixed 47.5-second problem (improper overlapping between
blocks)

I have no other explanation why the approach was changed. After this change, Hann windows were NOT applied to the profile. But the noise was forward - fft'd TWICE -- the frequency data, to be back-transformed, without the Hann window. But another forward fft, yielding the spectrum used for noise discrimination, WITH the window.

This remained the state of the program until the major revision, and this was the algorithm that you say some thought better for some uses. You said it handled something like a hiss under a cello sound better. I think I can guess why now.

This post is getting too long, I want to think and experiment some more.

[Paul L]

I tried all this in 2.0.5 and then in my build. The first example, with silence for profile, does nothing to the sound as expected. The difference of tracks is not exactly 0 but had peak amplitude of -127.6 dB.

In other words I get my original samples back with errors only in the sixth decimal place. Maybe that is just the roundoff error of the FFT inversion and nothing more.

[Paul L]

Look at this, Steve!

First picture: your chirp, pink noise, the mix of those two, and the result of noise removal (in my build with fixes) with the pink noise as profile, and extreme 48/0/0/0 settings, viewed with 2048 Blackman-Harris windows (and default gain and range settings for the palette). Which are good windows for very low side lobes but a wide central lobe, so frequency bins are not contaminated at "long distance." Looks like noise removal didn't work very well!

tracks011.jpg (487.09 KiB) Viewed 3821 times

Second picture: all of the same, but using Rectangular windows.

tracks012.jpg (503.82 KiB) Viewed 3821 times

Do you see how the second picture explains the bad noise removal seen in the first picture?

This is something like the hiss under the cello maybe.

Once more with Hann windows.

tracks013.jpg (446.1 KiB) Viewed 3821 times