Hi,
‘Subtract’ might be a misnomer here because when I looked this up, I did not get what I was looking for.
Here’s what I’m trying to do:
I have a song and the instrumental of the song. What I want to do is remove the frequencies of the instrumental from the complete song to get just the vocals.
I have tried the “Isolate Vocals” effect, but the results are fuzzy.
I have also tried the “Invert” function, but that hasn’t worked.
Please help!
I have a song and the instrumental of the song.
As separate songs? Are they MP3 downloads? It would seem that you should be able to put them on separate timelines one above the other and manipulate them so the instruments cancel. In practice however, the MP3 process pretty much kills that idea. MP3 gets its convenient, tiny files by shuffling music tones around and actually leaving some of them out. So the music behind the voice and the music on the separate song are different enough that they will not cancel.
Split, flip, and cancel rarely works either because that assumes no stereo effects have been added to the voice. Stereo effects are always added to the voice and the result, as you found, is a ratty, mushy cancellation.
Why do you want to do this?
Koz
AI software like Spleeter, (but the results are still “fuzzy”).
I have tried the “Isolate Vocals” effect, but the results are fuzzy.
I’ve never tried spleeter, but these are generally “novelty effects”, not suitable for high-quality audio production. 
I have also tried the “Invert” function, but that hasn’t worked.
That’s also how the basic vocal remover works but Vocal Reduction and Isolation has some additional features and processing.
And I’m not sure what you tried but it’s Invert plus a couple more steps. And the built-in vocal removal effect is better…
It is subtraction… If the vocals are in the “center” (identical and in-phase in the left & right channels) they can be perfectly subtracted-out, along with everything else in the center, including the bass. (The bass is almost always centered, but that can fairly-easily be left-out of the subtraction so it’s not a big deal.) But, there usually are other centered (or almost centered) sounds that do get subtracted-out (or reduced). And often when the vocals are centered, there is out-of-phase reverb that gets inverted back in-phase so it gets summed instead of subtracted (subtracting a negative is addition).