I pass the noise floor test until I start talking..

RMS value is performance plus noise-floor.

Amplifying so the RMS is higher inevitably increases the noise-floor by the same amount.

An expander plug-in (like couture) can selectively squash down the noise-floor when you’re not speaking,
but IMO only about -6db reduction is possible before it sounds conspicuously processed.