I was playing with ChannelMixer and decided to hit the debug button to look at the preset parameters. Fine. Then I tried it again with a different preset but hit the OK button - and the debug window appeared!
Upon investigation, the use of the Debug button appears to be stored on an effect by effect basis, and once you’ve used the debug button in any given Nyquist effect, clicking the OK button will always invoke the debug dialog in that effect.
It appears that the only way to clear this is the quit and restart Audacity.
Tested on Audacity 1.3.13-alpha-Sep 21 2010, Mac PPC OS X 10.5.8.