Not really. There is https://forum.audacityteam.org/t/need-pointers-on-adding-feature-to-search-pattern/33729/3 which isolates similar sounds. You could use that on a duplicate of the track, label the detected sounds with Analyze > Sound Finder…, then use the Edit > Labeled Audio commands to remove the labeled audio in the original track.
But that would remove the voices, shortening the track, not remove some voices from the background, leaving the background and leaving the length the same. And it doesn’t work so well if the voices you want to remove are mixed with other sounds.
You would have to wait for a proper Audio Diff feature.
Gale