I have plunked down at least twice what it would have cost to purchase new albums (CDs) to no avail.
My usual advice is to buy the CD (or MP3) if it’s available. 
Back to the adjustments - You shouldn’t need the force gauge as long as the tone arm has a calibrated adjustment and you zero it (float it) first. If there’s an anti-skating adjustment, adjust that to match your stylus force adjustment.
Although it’s counter-intuitive, it’s usually recommended that you use the maximum recommended tracking force for the particular cartridge. (1.5 grams for the Shure M97XE). The Shure brush/stabilizer weighs 1/2 gram (which subtracts from the actual stylus force) so you’d set the tone arm to read 2 grams.
Record can be damaged (and you can get tracking distortion) if the force is too low, and you’ll get more wear if the force is too high. But, since you are digitizing the records, and probably not planning on playing them 100s of times, a little extra wear is no big deal.
Many I have tried appear to be scratch free but still crackle and pop too much even after several cleanings.
Perhaps you’ve forgotten how BAD most records sounded! 
Cleaning sometimes helps a little, but it rarely makes the record sound like a CD or MP3. With older records there is usually wear & damage beyond dirt & dust, although playing it dirty can cause more permanent damage.
The different effect in Audacity work for some of the cracking and popping but not always. Loud songs cover corrections but more subtle music seems to lose too much of the quality with noise reduction etc. (at least at default setting). Perhaps additional learning time with Audacity will aid in my quest to remove the noise without losing to much of the original track.
For individual clicks & pops, the Click Removal effect is your best bet and you can select a short section of audio so you’re not altering the good audio.
Sometimes you can re-draw the waveform with the Draw Tool. But, it’s tricky to get good-sounding results, and it’s tricky to zoom-in and find the find the defect visually. (Switching back-and-forth between the normal Waveform View and the Spectrogram View can help when zooming-in on the defect. Do that with the drop-down arrow to the left of the waveform.)
For constant hiss, hum, or crackle, regular Noise Reduction is best, but there can be side effects so you sometimes have to decide if the cure is worse than the disease. Sometimes, you can apply noise reduction only during fade-in or fade-out so if there are nose reduction artifacts they will only exist during fade-in/fade-out and the bulk of the song won’t be affected/damaged. You can also try that during quiet passages, but “damaged music” during a quiet passage might be more objectionable than during the fade-in/fade-out.
I use [u]Wave Repair[/u] ($30) to remove clicks & pops. It does an audibly perfect job on most vinyl defects, and in the manual mode it only “touches” the audio where you identify a defect. But, it usually takes me a full weekend to digitize and clean-up an LP, so my plan is to try one of the more-automated applications whenever I do this again.
[u]This page[/u] (written by the developer of Wave Repair) lists several software options and has tons of advice for digitizing records.
It seems that the cartridge picks up every little flaw on my old records.
Well, yes… If it’s going to pick-up all of the music it’s also going to pick-up all of the defects. There was a “case” here recently where there someone was digitizing 78’s and there was excessive high-frequency noise that didn’t exist when the records were played on a Victrola. Rolling-off the higher frequencies to more closely match the Victrola’s frequency response was the solution, but since LPs generally have (nearly) full-audio frequency range, you probably don’t want to do that.
