Audacity works internally in extreme high quality 32-bit float.
When you set the project’s bit format to 16 or 24 bit, then Audacity converts the audio after processing to 16 / 24 bit as you told it to. In the case that you describe, the round trip from 24-bit → 16-bit → 24-bit will have truncated the sample values from 24-bit to 16-bit and added a tiny amount of dither noise. Theoretically this would slightly reduce the sound quality, but in practice it will probably not be noticeable (or hardly noticeable).
I would highly recommend using the default 32-bit float format for all projects so as to achieve the best sound quality.