Noise Removal Tests

Reading this thread: https://forum.audacityteam.org/t/1-2-6-noise-removal-superior-to-1-3-6/6213/1 led me to investigate the NR effect in 1.3.13.

My method was as follows:

  • New Project
    Generate noise, white, level 0.01, 10 seconds
    New track
    Generate tone, sine, 400 Hz, 9 seconds
    Move tone track so the two tracks end at the same time
    Select All then Mix and Render
    Export to WAV or AIF
    Select noise only portion, Effect > Noise Removal and Get Noise Sample
    Select entire track
    Effect > Noise Removal with Noise Reduction 12, Sensitivity 0, Smoothing 130 and Attack/decay 0.10

Result:

  • Noise is removed from the first second of the track
    Once the tone starts, noise is not removed, and instead “wobbles” in volume

In my understanding, this is not correct behaviour. If the NR algorithm is using FFT to sample the noise profile, then applying downward expansion to each frequency band according to the level of audio in that band, the noise should be removed even in the presence of the tone. Noise would remain the the band containing the 400 Hz tone, but would be removed in all other bands. Thus the “masking” effect would make it appear that the noise has been completely removed.

To confirm this:

  • Open the exported noise+tone file in Brian Davies’ DeNoise program
    Set “Limit reduction to 12 dB”
    Check the “Automatic” checkbox
    Click the “Start” button

Import the DeNoised file in the same project and compare the two results.

The DeNoised file has the noise removed in the presence of the tone.

Virtually identical results can be achieved in DeNoise by setting the “White” checkbox, or sampling the noise profile at the start of the file per instructions on page 14 in the DeNoise manual.

Possible conclusions:

  1. DeNoise is superior to Audacity’s Noise Removal effect.
  2. The Noise Removal effect is broken.

Conclusion #1 seems obvious from this experiment.

Conclusion #2 is speculative. It all depends of how the Noise Removal effect is intended to operate. If the NR effect is intended to operate as described above, then I submit that it is broken.

– Bill

Interesting Bill - this may have some bearing on some recent posts I can vaguely recall where folks had problems with Audacity’s NR.

Are you planning to potentially report this as a bug?

WC

Potentially, yes. I’m hoping to get some indication from those in the know of how the effect is meant to operate, and if my test is valid.

– Bill

Further tests … Gale and Steve suggested I try increasing the Sensitivity slider to get more aggressive noise reduction. The results were not much different.

Plot Spectrum of the unprocessed tone+noise:
SpectrumNRbefore.png
Plot Spectrum after Audacity NR with Sensitivity = 0:
SpectrumNRafter.png
As you can see, the noise that is spread out along the -78 dB line is virtually untouched. There is some minimal reduction above 6000 Hz. The increase below 40 Hz is the “wobble”.

Plot Spectrum after Audacity NR with Sensitivity = 12
SpectrumNRplusSensAfter.png
There is now some clear NR above 3000 Hz. The wobble remains.

Plot Spectrum after processing by DeNoise:
SpectrumNRafterDavies.png
There is now clearly 12 dB of NR up to about 10 kHz. No wobble.

More off-forum discussion has occurred while I was composing this. Might as well post it, then respond on -quality to the comments. I’m posting here instead of -quality in order to include the attachments. Hope that’s OK.

– Bill

There is a pretty clear bug/unintended behaviour showing up in this test.
Irrespective of what settings are used, the Noise Removal effect generates low frequencies (as shown by the peaks at around 12 Hz and 30 Hz in the screen shots).
This low frequency noise is audible.

Apart from that problem, pretty good results can be obtained using these settings:
Noise Reduction = 24
Sensitivity = 18
Frequency smoothing = 100
Attack/Decay = 0.0
screenshot.png
With the following settings, the plot looks almost identical, but when listening the noise has been almost completely eliminated.
NR = 48
S = 16
FS = 100
A/D = 0.0
screenshot2.png
Listening to the “isolated” noise reveals that virtually nothing has been removed from the original sine tone, but the low frequency noise is also clearly evident.

BTW, the current version of Audacity NR (with “Sensitivity” adjustment) is a considerable improvement over the previous version (currently in Audacity 1.3.12).

There is also another bug in Noise Removal (1.3.13 alpha):
The Sensitivity and Attack/Decay sliders do not respond correctly to keyboard control. (tested on Linux only)

Now try those settings on a real-world example. For instance, import a CD track, add white noise at -40 dB and apply those settings. The result is horrible.

– Bill

Of course the results are horrible :smiley: but I think that is half of the point - the optimum settings are dependent on the type of audio and the type of noise. If there were settings that worked regardless of the material, then the effect could be preprogrammed with the optimum values and no sliders. To compare the performance of different Noise Removal effects you need to specify the test material, then compare the optimum settings for that test material (which is much harder to do with real world samples). I don’t think that Audacity NR is particularly good at removing white noise, but does it do better with other types of noise? Many specialist audio restoration programs will have several different effects for dealing with different types of noise (hum, buzz, hiss, crackle, pops…). Other than “Click Removal” Audacity has just one NR.

Artificial tests can indicate possible issues (such as the “wobble” as you called it/“low frequency noise” as I called it) but are quite limited when it comes to assessing what real world performance will be like. As an example, the two plots that I posted look extremely similar, and I think that you would agree that you would expect them to sound virtually the same, but if you repeat the test you should notice that they sound remarkably different.

I’m just wondering what sort of tests we can manage -
Do you have a licensed version of DeNoise? (I don’t, but I could run the 21 day trial)
What are your C++ skills like? (mine are non-existent)
Do you only run Macs or other platforms as well? (I’m mostly on Linux, but have XP in VirtualBox)
Do you have any other Noise Removal programs for comparison? (I’ve got Gnome Wave Cleaner)
Do you have a host program that fully supports VST effects? (I don’t, but I can get limited access to one).

Yes I have a licensed version of DeNoise.

I have zero C++ skills :frowning:

Mac only, I’m afraid, so can’t try GWC.

No programs other than DeNoise. I could download the 14-day trial of SoundSoap SE, but it apparently mangles the output during the trial period.

And no programs other than Audacity that support VST effects (and my Mac is prone to the “one VST effect or I’ll hang” bug).

I’ve been thinking of real-world sources that would expose what I see as the shortcomings of the effect. A cappella vocals, solo guitar, and just about any classical piece (lots of dynamics)? Perhaps adding noise at -40 dB is too severe. Perhaps -50 or -60 would be more realistic, but at -40 it’s easy to hear.

There’s a parallel discussion on the -quality list (as you know), but I’ll repeat something here for those who are not subscribed to that list. It appears that the Audacity NR effect is not doing multiband noise gating [I’ll call it “gating” although it is really downward expansion] with the gate thresholds set by the noise profile. Instead it seems to be using the overall full-spectrum signal level to decide whether or not to open or close the gates. The upshot is that as soon as the signal starts, the noise removal is negligible to non-existent. Raising the Sensitivity slider just causes the effect to treat more of the signal as noise.

– Bill

Here are some more comparisons in 1.3.13 HEAD with Bill’s tone/noise sample showing the spectrum of the tone after processing (Noise reduction 34 dB, FS 140 Hz, A/D 0.00):

First with Sensitivity at 0.00 dB:
1.3.13_hiss_not_removed.png
Little hiss suppression evident or audible.

Second with Sensitivity at 20.00 dB:
1.3.13_hiss_removed.png
This seems to show the hiss reduced by about 30 dB above 3000 Hz. Subjectively I can’t hear the original hiss at all.

Then a comparison with 1.2.6 with the slider half-way (using the 1.3.13 Plot Spectrum of course):
1.2.6_processed_noise.png
This seems to show little hiss reduction on the face of it. There are no low frequency spikes comparable to 1.3.13 but there is a significant spike at 300 Hz. Here is a wav from that processed 1.2.6 audio:

To me I am not audibly sure if much hiss has been removed in 1.2.6 or not. There is less wobble than in 1.3.13 but the main thing I hear is the 300 Hz which now makes it sound like a dual tone. The 300 Hz does seem to smother the hiss but I still sense there actually is less hiss and that the -78 dB audio sounds like a mix of hiss and quiet tone artefacts.

My conclusion, none of the results are perfect in view of the fairly loud artefacts to the tone. 1.2.6 and 1.3.13 both produce tone artefacts at about -42 dB, but at different frequencies. The 1.3.13 “hiss removed” audio sounds much the best to me.

How does the DeNoise tone sound after processing? Is it much less artefacted? I agree we should be worrying about real life samples more than tone artefacts.

Note that Marco’s extra code (in the Windows Unicode Nightly) gives a better result again in my view (tone artefacts less audible, although the plot looks similar). However you have to sample the noise and the signal with that or the results are often worse than in SVN HEAD (more artefacts, and large amplitude reductions in the result).


Gale

There’s a free tool that may help in comparing the performance of different Noise Removal methods
http://libinst.com/Audio%20DiffMaker.htm

It’s a Windows program, but works perfectly in Wine.

If you look at the flow diagram at the bottom of that page the software is merely inverting one recording then summing the two recording. It is apparently smart enough to deal with sample rate drift. But otherwise, for the purposes of our tests, why not just make a copy of the original noisy recording, invert it then mix that inverted copy with the noise-reduced samples. I just tried this and it is very revealing. Don’t know why I didn’t think of it before. :astonished:

– Bill

Gale:
The DeNoised tone is free of artefacts.

Tone tests are useful since they can exposes problems in the effect through both listening and spectrum plots. Spectrum plots are less useful with real-world samples, so we are left with listening tests. The “invert and sum” method can help expose artefacts in real-world samples. Spectrum plots of the result of “invert and sum” can be quite revealing.

For my first real world test I chose “Foxglove” by Bruce Cockburn - a fingerpicking solo guitar piece. I summed the stereo recording to mono, amplified to -2 dB then added noise at -40 dB. I processed it with Audacity NR with NR 20, Sens 0, Smoothing 100 and A/D 0.25. Then processed it with Denoise with NR 20, Noise Floor -40 (equivalent to Sens = 0) and noise profile = “white”. Listening to the results, with the Audacity NR I can hear some slight degradation of the guitar sound, and I can hear the noise “pumping”; with DeNoise there is no apparent degradation of the guitar, and there is no pumping audible. Particularly revealing is the decay of the last guitar notes at the end. With the Audacity NR I can hear a “seashell over the ear” kind of effect; with DeNoise there is just constant hiss at -60 dB under the decaying notes.

– Bill

<<>>

Pretty much why I stopped using it for critical work – or anything, really. It always does something to the sound. I wonder how that other package is doing it’s work.

Koz

And all of which confirms my personal view that Brian Davies has built some finel-honed tools to do some particularly narrow audio tasks extremely well - just as Chris has with his famous Compressor.

And Audacity still provides an excellent audio “workbench” to glue all these specificic external software tasks together.

WC

From the DeNoise manual:

The DeNoise application works in the frequency domain, dividing the incoming audio stream into
80 overlapping frames per second, and transforming each such frame into 1024 equally spaced fre-
quency bands (2048 if the sample rate exceeds 48 kHz).

  • For sound sampled at the CD rate – 44.1 kHz – this means that each band has a width of 22Hz. From
    about 200Hz this enables fine-grained processing; however it is not satisfactory for the treatment of
    rumble, hum, and the like. This is the reason for the development of a separate application (De-
    NoiseLF) for the treatment of low-frequency noise.

Most readers will be familiar with the operation of graphic equalizers. Normally they are restricted
to no more than 30 bands, and they are adjusted infrequently. DeNoise operates as a type of “intel-
ligent” graphic equalizer. There are 1024 (or even 2048) frequency bands, and the incoming audio is
processed in overlapping frames at the rate of 80 frames per second. The gain in each of the bands
is adjusted continuously from frame to frame, > using algorithms which attempt to judge what would
be perceived as noise, and what would be perceived as desirable audio
> . The objective is to suppress
the former without doing perceptible damage to the latter.

The emphasis is mine. Up to that point he’s describing a multi-band noise gate. This corresponds to the description of the Audacity NR effect on the wiki Missing features - Audacity Support . Nowhere does Davies describe the algorithms in detail, but later in the manual refers to “psycho-acoustic algorithms”. The implication is that he’s doing something more sophisticated than simple noise gating.

Note: I know that the Audacity NR effect and DeNoise both use downward expansion rather than simple gating. It’s just easier to write “multi-band noise gate” than “multi-band noise downward expander”. :wink:

– Bill

The part that bothers me most about the Audacity NR effect is the low frequency “beats”. I wonder if that’s caused by “overlapping frames” that are at a lower frequency than 80 Hz and don’t overlap very well. I’ve not come across this type of artefact in any other Noise Removal application.

Davies…DeNoise.

…using algorithms which attempt to judge what would
be perceived as noise, and what would be perceived as desirable audio…

Exactly. That would be McDonalds “Secret Sauce,” the Coca Cola ingredient locked in a vault in Atlanta, etc.

And yes, without that, you have Tesco/7-Eleven microwave burgers and off-brand tap cola.

Ewwww.

Koz

Confirmed on Windows so I added a bug:
http://bugzilla.audacityteam.org/show_bug.cgi?id=227

I have assumed the same problem on Mac too but if not, please change the OS information for the bug.

I guess we can agree then that at a given Noise Reduction setting, Bill’s tone/noise sample has little/no hiss removed at Sens = 0 dB and very mild artifacts (based on mixing the processed audio with an inverted copy of the original). At high Sens settings much of the noise is removed (based on plots and running NR in Isolate mode), but tone artifacts are much worse, despite the plot of the low frequency spikes looking very similar to that for Sens = 0 dB.

I’d still argue that artifacting with high sensitivities and real world audio is nothing like as bad as with tones, based on testing some classical piano and low amplitude string samples, and that a lot of real world noise encountered will be narrow band (Audacity does fairly well removing hum from my USB turntable, I think).

I tend to use Goldwave for NR. If no-one else does, I might try some tests with it, though I think they will confirm my long-held subjective impression that Goldwave is to be preferred for NR over Audacity (this is partly because it deals better with intermittent vinyl artifact noise, not only that it produces fewer NR artifacts than Audacity).

I guess in the absence of getting Marco or a mainstream developer interested before 2.0, one possible kludge might be to start “Sensitivity” at + 5 or + 10 dB to make NR a bit more aggressive, then use some vague scale as in Click Removal. I still don’t personally see further changes as critical for 2.0. Click Removal is also a relatively weak feature, but tends to get less discussion.



Gale

Davies is (IIRC) a retired mathematician. I don’t think he’s “invented” anything, but instead probably read the literature and implemented the best algorithms described there.

For example, if you look at the third plot in my original post you can see that full noise reduction (12 dB) does not occur until approximately one octave below and above the 440 Hz tone. A simple noise gate would not do that.

– Bill