Audacity supports a number of different plug-in formats, including VST (mostly on Windows), LADSPA, AU (Mac OS X only) and Nyquist.
Of these, probably the easiest to write are “Nyquist” plug-ins.
Because Nyquist plug-ins are specifically for Audacity, we have quite a lot of information. See this post: https://forum.audacityteam.org/t/manuals-and-reference-material/32817/1