There are pros and cons to each approach, but you need not use just one approach.
Forking Audacity is, as you say, the most ambitious. It is the most flexible, and probably the most difficult. The code for Audacity is very complex, with around 600,000 lines of code (mostly C++). Even if you are a highly experienced C++ developer, it is likely to require a substantial amount of time and effort just to gain familiarity with the code base. As we use GitHub to host the Audacity code, making a fork is easy, and so long as your custom version does not diverge too far, it should be relatively easy to merge updates from the Audacity code, and to push updates from your version back “upstream” to Audacity.
Nyquist is probably the easiest. It is based on a dialect of the LISP programming language (“XLISP”) with a strong focus on audio. Prototyping can be very quick as there is a large library of audio related functions. Although Nyquist runs as an interpreted language, many of the primitives are written in C, so with careful programming it is often possible to achieve performance similar to a pure C/C++ application.
To give an idea of how rapidly features can be developed in Nyquist, (how little code), here is a script that can be run from the Nyquist Prompt effect that will add a label each time the selected audio exceeds the specified threshold:
;type analyze
;version 4
;control thresh "Threshold" float "dB" -24 -72 -6
(defun label-sounds (sig)
(setf step (round (/ *sound-srate* 100)))
(let* ((sig (snd-avg sig step step OP-PEAK))
(srate (snd-srate sig))
(labels ())
(sound-flag nil))
(do ((val (snd-fetch sig)(snd-fetch sig))
(count 0 (+ 1 count)))
((not val) labels)
(cond
((> val thresh)
(unless sound-flag
(push (list (/ count srate) "Sound") labels))
(setf sound-flag t))
(t (setf sound-flag nil))))))
(setf thresh (db-to-linear thresh))
(if (soundp *track*)
(label-sounds *track*)
(print "Mono track required"))
Although Nyquist is quite an easy language, it may not be the best approach for everything. In particular, working with FFT is very slow and quite difficult.
“Mod-script-pipe” is the new kid on the block, and has recently received a major upgrade.
Basically, mod-script-pipe allows you to control Audacity from an external script (such as Python). Almost everything that you can do in Audacity manually (via the GUI) can be done via “scripting commands”.
Most of the scripting commands are also available to Audacity’s “Macro” feature (see: https://manual.audacityteam.org/man/macros.html), though with macros you can only run a linear sequence of commands.
A large portion of the scripting commands are also available to Nyquist via a special “AUD-DO” function (https://manual.audacityteam.org/man/nyquist_macros.html). The main limitation for running scripting commands from Nyquist is that you can’t run Nyquist scripts from an aud-do command (Nyquist currently does not support multiple simultaneous processes or threads).
I’d suggest not bothering with LADSPA as it is becoming obsolete.
VST “should” work on Windows and Linux, though we have to use 3rd party open source headers for VST, so there can be compatibility problems.
I don’t know much about VAMP, but I’ve seen some impressive VAMP plug-ins, and it is open source, and they are intended specifically for audio analysis. There’s some information here: https://www.vamp-plugins.org/
Regarding a starting point:
Audacity is an extremely versatile open source multi-track audio recorder and editor. We’ve had visitors to this forum from people that use Audacity in other fields of zoological research, including someone that was studying Orca, and someone that was studying nocturnal behaviour of apes. I think that spending some time becoming familiar with Audacity would be time well spent.
Although Nyquist may not be suitable for ‘all’ of your requirements, it is probably the easiest and quickest way in to making custom plug-ins. Because Nyquist plug-ins are just plain text files (compiling not required), installing and modifying Nyquist plug-ins is very easy. We also have a forum board specifically for questions about Nyquist: https://forum.audacityteam.org/viewforum.php?f=39
)