The Notch filter effect in Audacity is a simple “Nyquist” effect. http://wiki.audacityteam.org/wiki/Nyquist_Plug-ins_Reference
Nyquist is a scripting language that has been included in Audacity and allows users to easily use custom effects, or customize existing Nyquist effects.
The thing that makes it simple is that Nyquist scripts are written in plain text, so can be written / edited in any plain text editor (such as NotePad, though the free NotePad++ is much better)
The core code for the notch filter is simply:
(notch2 sound frequency width
Where “sound” is a mono or stereo sound to be processed, “frequency” is the centre frequency of the notch, and “width” is the “q” value (narrowness) of the filter.
You can run this code in the Nyquist Prompt effect (http://manual.audacityteam.org/o/man/nyquist_prompt.html) by substituting actual values and typing or pasting the code into the Nyquist Prompt text box.
For example, for a 1000 Hz notch filter with a “q” of 2.0 you would use:
(notch2 s 1000 2.0)
Note that the letter “s” is a special character that represents the selected audio.
In order to invert a sound, all you need to do is to multiply it by -1 (minus one). The code to do this in Nyquist is:
(mult -1 sound)
where “sound” is the sound that you want to invert.
For example, to invert the audio in a wave track using the Nyquist prompt you could use:
(mult s -1)
Note again that “s” is used to represent the audio in the current track selection.
“Mixing” sounds is simply “addition”, and in Nyquist can be accomplished with the function “SUM”, so putting this all together we have:
(sum s (mult -1 (notch2 s 1000 2.0)))
Does that all make sense?
The Notch filter plug-in in your Audacity plug-ins folder can likewise be customised.
For an alternative approach, you may be interested in modifying this band stop filter: http://wiki.audacityteam.org/wiki/Nyquist_Effect_Plug-ins#Band_Stop_Filter