That’s good because if the noise isn’t bad you might be close to passing. 
They are (obviously) measuring the quiet-parts when there is no talking… They aren’t listening.
If you don’t have it already ACX Check, will check your peak, RMS, and noise levels.
Koz’ Audiobook Mastering Macro will set the peak and RMS levels (and won’t hurt anything if they are already good), plus it includes a high-pass filter to filter-out any low-frequency noise (below the voice frequency range). The worst measuring noise is often in the lower frequencies where it is harder to hear, but they’ll still measure it.
If it still doesn’t pass you can try Noise Reduction which requires you to feed-in a Noise Profile (A sample of noise only).