<<<I was told to hook up an RCA cable from the master booth in my mixer to the line input in my computer (my microphone jack).>>>
That would have worked out perfectly…on a Mac. Macs actually do have Stereo Line-In connections. Your computer is expecting a tiny, delicate, weak, mono microphone signal and you’re feeding it a massive, powerful, stereo, high volume, Line Level Signal. So right there, we’re in trouble.
Doesn’t the mixer have a Master Out or Line-Out? Generally, the Booth signal follows the little monitor knob on the mixer and not the Master Mix Faders.
You can get around some of these problems with one of the many fine USB audio adapters we reviewed…
https://forum.audacityteam.org/t/sound-card-reviews/8375/1
Pick one that has stereo Line-In connections on one end and USB on the other. Chuck it in between the mixer and the computer and follow the instructions to set it all up.
<<<if I were to plug headphones in, I would hear myself>>>
Most computers won’t do that, or they will, but the headphone signal will be late, sorta like reverb or echo. We strongly recommend listening to the mixer and not the computer.
Set up Audacity Preferences for 44100, 16-bit, Stereo, and not the settings that it woke up with. Restart Audacity with the new settings to make sure they stick. See if some of those error messages don’t go away.
Koz