Even if you remove the silences, odds-are the waveform will then be discontinuous at the join: there will be a click at the join.
There is a DeClicker plugin for Audacity which automatically repairs clicks, here … Updated De-Clicker and new De-esser for speech
That plugin can be slow, (depending on the settings used), but it’s much faster than repairing individual clicks manually.