Noise Removal Tests

[billw58]

Reading this thread: viewtopic.php?f=28&t=7010 led me to investigate the NR effect in 1.3.13.

My method was as follows:

New Project
Generate noise, white, level 0.01, 10 seconds
New track
Generate tone, sine, 400 Hz, 9 seconds
Move tone track so the two tracks end at the same time
Select All then Mix and Render
Export to WAV or AIF
Select noise only portion, Effect > Noise Removal and Get Noise Sample
Select entire track
Effect > Noise Removal with Noise Reduction 12, Sensitivity 0, Smoothing 130 and Attack/decay 0.10

Result:

Noise is removed from the first second of the track
Once the tone starts, noise is not removed, and instead "wobbles" in volume

In my understanding, this is not correct behaviour. If the NR algorithm is using FFT to sample the noise profile, then applying downward expansion to each frequency band according to the level of audio in that band, the noise should be removed even in the presence of the tone. Noise would remain the the band containing the 400 Hz tone, but would be removed in all other bands. Thus the "masking" effect would make it appear that the noise has been completely removed.

To confirm this:

Open the exported noise+tone file in Brian Davies' DeNoise program
Set "Limit reduction to 12 dB"
Check the "Automatic" checkbox
Click the "Start" button

Import the DeNoised file in the same project and compare the two results.

The DeNoised file has the noise removed in the presence of the tone.

Virtually identical results can be achieved in DeNoise by setting the "White" checkbox, or sampling the noise profile at the start of the file per instructions on page 14 in the DeNoise manual.

Possible conclusions:
1) DeNoise is superior to Audacity's Noise Removal effect.
2) The Noise Removal effect is broken.

Conclusion #1 seems obvious from this experiment.

Conclusion #2 is speculative. It all depends of how the Noise Removal effect is intended to operate. If the NR effect is intended to operate as described above, then I submit that it is broken.

-- Bill

[waxcylinder]

Interesting Bill - this may have some bearing on some recent posts I can vaguely recall where folks had problems with Audacity's NR.

Are you planning to potentially report this as a bug?

WC

[billw58]

Are you planning to potentially report this as a bug?

Potentially, yes. I'm hoping to get some indication from those in the know of how the effect is meant to operate, and if my test is valid.

-- Bill

[billw58]

Further tests ... Gale and Steve suggested I try increasing the Sensitivity slider to get more aggressive noise reduction. The results were not much different.

Plot Spectrum of the unprocessed tone+noise:

SpectrumNRbefore.png (74.79 KiB) Viewed 5810 times

Plot Spectrum after Audacity NR with Sensitivity = 0:

SpectrumNRafter.png (77.22 KiB) Viewed 5811 times

As you can see, the noise that is spread out along the -78 dB line is virtually untouched. There is some minimal reduction above 6000 Hz. The increase below 40 Hz is the "wobble".

Plot Spectrum after Audacity NR with Sensitivity = 12

SpectrumNRplusSensAfter.png (78.64 KiB) Viewed 5810 times

There is now some clear NR above 3000 Hz. The wobble remains.

Plot Spectrum after processing by DeNoise:

SpectrumNRafterDavies.png (77.55 KiB) Viewed 5811 times

There is now clearly 12 dB of NR up to about 10 kHz. No wobble.

More off-forum discussion has occurred while I was composing this. Might as well post it, then respond on -quality to the comments. I'm posting here instead of -quality in order to include the attachments. Hope that's OK.

-- Bill

[steve]

There is a pretty clear bug/unintended behaviour showing up in this test.
Irrespective of what settings are used, the Noise Removal effect generates low frequencies (as shown by the peaks at around 12 Hz and 30 Hz in the screen shots).
This low frequency noise is audible.

Apart from that problem, pretty good results can be obtained using these settings:
Noise Reduction = 24
Sensitivity = 18
Frequency smoothing = 100
Attack/Decay = 0.0

screenshot.png (38.98 KiB) Viewed 5809 times

With the following settings, the plot looks almost identical, but when listening the noise has been almost completely eliminated.
NR = 48
S = 16
FS = 100
A/D = 0.0

screenshot2.png (39.05 KiB) Viewed 5807 times

Listening to the "isolated" noise reveals that virtually nothing has been removed from the original sine tone, but the low frequency noise is also clearly evident.

[steve]

BTW, the current version of Audacity NR (with "Sensitivity" adjustment) is a considerable improvement over the previous version (currently in Audacity 1.3.12).

There is also another bug in Noise Removal (1.3.13 alpha):
The Sensitivity and Attack/Decay sliders do not respond correctly to keyboard control. (tested on Linux only)

[billw58]

There is a pretty clear bug/unintended behaviour showing up in this test.
Irrespective of what settings are used, the Noise Removal effect generates low frequencies (as shown by the peaks at around 12 Hz and 30 Hz in the screen shots).
This low frequency noise is audible.

Apart from that problem, pretty good results can be obtained using these settings:
Noise Reduction = 24
Sensitivity = 18
Frequency smoothing = 100
Attack/Decay = 0.0

Now try those settings on a real-world example. For instance, import a CD track, add white noise at -40 dB and apply those settings. The result is horrible.

-- Bill

[steve]

Now try those settings on a real-world example. For instance, import a CD track, add white noise at -40 dB and apply those settings. The result is horrible.

Of course the results are horrible but I think that is half of the point - the optimum settings are dependent on the type of audio and the type of noise. If there were settings that worked regardless of the material, then the effect could be preprogrammed with the optimum values and no sliders. To compare the performance of different Noise Removal effects you need to specify the test material, then compare the optimum settings for that test material (which is much harder to do with real world samples). I don't think that Audacity NR is particularly good at removing white noise, but does it do better with other types of noise? Many specialist audio restoration programs will have several different effects for dealing with different types of noise (hum, buzz, hiss, crackle, pops...). Other than "Click Removal" Audacity has just one NR.

Artificial tests can indicate possible issues (such as the "wobble" as you called it/"low frequency noise" as I called it) but are quite limited when it comes to assessing what real world performance will be like. As an example, the two plots that I posted look extremely similar, and I think that you would agree that you would expect them to sound virtually the same, but if you repeat the test you should notice that they sound remarkably different.

I'm just wondering what sort of tests we can manage -
Do you have a licensed version of DeNoise? (I don't, but I could run the 21 day trial)
What are your C++ skills like? (mine are non-existent)
Do you only run Macs or other platforms as well? (I'm mostly on Linux, but have XP in VirtualBox)
Do you have any other Noise Removal programs for comparison? (I've got Gnome Wave Cleaner)
Do you have a host program that fully supports VST effects? (I don't, but I can get limited access to one).

[billw58]

I'm just wondering what sort of tests we can manage -
Do you have a licensed version of DeNoise? (I don't, but I could run the 21 day trial)
What are your C++ skills like? (mine are non-existent)
Do you only run Macs or other platforms as well? (I'm mostly on Linux, but have XP in VirtualBox)
Do you have any other Noise Removal programs for comparison? (I've got Gnome Wave Cleaner)
Do you have a host program that fully supports VST effects? (I don't, but I can get limited access to one).

Yes I have a licensed version of DeNoise.

I have zero C++ skills

Mac only, I'm afraid, so can't try GWC.

No programs other than DeNoise. I could download the 14-day trial of SoundSoap SE, but it apparently mangles the output during the trial period.

And no programs other than Audacity that support VST effects (and my Mac is prone to the "one VST effect or I'll hang" bug).

I've been thinking of real-world sources that would expose what I see as the shortcomings of the effect. A cappella vocals, solo guitar, and just about any classical piece (lots of dynamics)? Perhaps adding noise at -40 dB is too severe. Perhaps -50 or -60 would be more realistic, but at -40 it's easy to hear.

There's a parallel discussion on the -quality list (as you know), but I'll repeat something here for those who are not subscribed to that list. It appears that the Audacity NR effect is not doing multiband noise gating [I'll call it "gating" although it is really downward expansion] with the gate thresholds set by the noise profile. Instead it seems to be using the overall full-spectrum signal level to decide whether or not to open or close the gates. The upshot is that as soon as the signal starts, the noise removal is negligible to non-existent. Raising the Sensitivity slider just causes the effect to treat more of the signal as noise.

-- Bill

[Gale Andrews]

Here are some more comparisons in 1.3.13 HEAD with Bill's tone/noise sample showing the spectrum of the tone after processing (Noise reduction 34 dB, FS 140 Hz, A/D 0.00):

First with Sensitivity at 0.00 dB:

1.3.13_hiss_not_removed.png (877.54 KiB) Viewed 5794 times

Little hiss suppression evident or audible.

Second with Sensitivity at 20.00 dB:

1.3.13_hiss_removed.png (877.54 KiB) Viewed 5790 times

This seems to show the hiss reduced by about 30 dB above 3000 Hz. Subjectively I can't hear the original hiss at all.

Then a comparison with 1.2.6 with the slider half-way (using the 1.3.13 Plot Spectrum of course):

Bill's tone/noise sample processed by 1.2.6

1.2.6_processed_noise.png (877.54 KiB) Viewed 5790 times

This seems to show little hiss reduction on the face of it. There are no low frequency spikes comparable to 1.3.13 but there is a significant spike at 300 Hz. Here is a wav from that processed 1.2.6 audio:

126noise.wav

(430.72 KiB) Downloaded 107 times

To me I am not audibly sure if much hiss has been removed in 1.2.6 or not. There is less wobble than in 1.3.13 but the main thing I hear is the 300 Hz which now makes it sound like a dual tone. The 300 Hz does seem to smother the hiss but I still sense there actually is less hiss and that the -78 dB audio sounds like a mix of hiss and quiet tone artefacts.

My conclusion, none of the results are perfect in view of the fairly loud artefacts to the tone. 1.2.6 and 1.3.13 both produce tone artefacts at about -42 dB, but at different frequencies. The 1.3.13 "hiss removed" audio sounds much the best to me.

How does the DeNoise tone sound after processing? Is it much less artefacted? I agree we should be worrying about real life samples more than tone artefacts.

Note that Marco's extra code (in the Windows Unicode Nightly) gives a better result again in my view (tone artefacts less audible, although the plot looks similar). However you have to sample the noise and the signal with that or the results are often worse than in SVN HEAD (more artefacts, and large amplitude reductions in the result).


Gale