Let me give an analogy to shed some light on this question.
Consider the value “5”.
“5” is an integer, so has a precision of one whole number (0 decimal places).
“5.0000” has a precision of 4 decimal places.
“5” is exactly equal to “5.0000”.
How that applies to digital recordings:
16 bit, 44100 Hz PCM is a reasonably high quality audio format (CD quality). It is capable of faithfully reproducing the full fidelity of a vinyl recording. The frequency range of audio that is sampled at 44100 Hz is about 20 kHz (theoretically up to 22050 Hz, but is limited by the practicalities of technology to around 20000 Hz). People cannot hear above 20 kHz (20000 Hz) - not even so much as a hint. 16 bit audio has a dynamic range of up to about 100 dB, which far exceeds all but the very best, pristine vinyl recordings assuming perfect reproduction and ideal conditions. In reality the dynamic range of a really good vinyl record played on really good equipment in a really good listening room is unlikely to be much over 80 dB.
So, we have analogue audio data from the vinyl, which has a finite frequency range and a finite dynamic range, both of which can just be reproduced digitally in full at 16 bit 44.1 kHz.
Other than the limits of technology there is nothing to stop us from storing the digital data at 10000000000 Hz sample rate and 100000000000 bits per sample, but that would clearly be overkill. What would all of those extra samples and bit actually be doing? They are like the extra 0’s in “5.0000” - they don’t actually do anything other than make the data bigger.
So if 16 bit 44100 Hz is “just enough”, do we ever need more?
Yes we do, and “more is better” is true up to a point.
When processing audio, there will usually be errors in the order of 1 LSB (least significant bit). What that means in practice is that if we process 16 bit audio, the result will probably only be accurate to around 14 or 15 bits, which is not quite enough for “pristine quality”. Audacity uses “32 bit float” format internally, which provides extreme accuracy when processing. In effect, processing in 32 bit float format is virtually perfect in terms of accuracy - you can apply thousands of processes with no loss of sound quality. at all.
44100 Hz is “just enough” to capture the full audio frequency range, but there may be a measurable amount of “ringing artefacts” in the extreme high frequency range due to the steepness of the anti-alising filters. Increasing the sample rate to around 80 kHz allows much more gentle anti-aliasing filters to be used, so that frequencies above 20 kHz are rolled off less steeply and so removing the possibility of ringing in the extreme high frequency range. (Note: There is no evidence that reconstruction and anti-aliasing issues are audible).
So “more is better” up to a point, and that “point” is in the region of:
For recording: 24 bit 80 kHz.
For processing: 32 bit float 80 kHz.
80 kHz, is not a standard rate and is not an exact multiple of the clock rate used in audio hardware, so to avoid resampling errors professional quality recording more often uses the nearest standard rate, which is 96 kHz. For audio, there are no benefits to increasing the sample rate or bit depth beyond 32 bit float 96 kHz (though there are disadvantages to doing so).
There is an interesting article about this and related issues here: http://wiki.hydrogenaudio.org/index.php?title=Myths_(Vinyl)
