Append Import

[Update: The current version of this plug-in is fully functional - see the included help screens for usage and limitations ]

This feature gets requested from time to time, so I thought I’d look to see if it could be done with Nyquist.

Here’s the first (experimental) version as a proof of concept.
This version can import up to 10 files, end to end in one track.

Limitations:

  • Only mono files are supported in the first version (marked “Obsolete” below). For stereo support, download the “latest” version.
  • All files should have the same sample rate.
  • Probably only WAV files supported at present.
  • The computer must have enough RAM to hold all of the files at the same time before they are written to the track.
  • There is no error checking, so incorrect user input will fail, probably with no error message.

I think that Nyquist only supports reading uncompressed formats, so I’ll not be able to do anything about that particular constraint, but it should be possible to overcome the other limitations in future versions.

When installed, this plug-in will appear in the Generate menu. See Installing Nyquist plug-ins .

OBSOLETE VERSION:
AppendImport.ny (1.46 KB)
LATEST VERSION: AppendImport.ny

Here’s a new version.

  • Both stereo and mono files of any sample rate can now be imported.
  • To import files into a stereo track, a stereo track must be selected before running Append Import. (Generate plug-ins can only create new mono tracks, but can return stereo audio to a stereo track if the stereo track is selected before running the plug-in).
  • When “Mono” is selected as the output format, stereo files will be mixed down to mono.
  • Append Import only supports uncompressed audio formats.
  • The computer must have enough RAM for the entire import.

As with all Generate plug-ins, if the tracks are imported without selecting a track, then the imported track will be at the “Project Sample Rate” (as shown in the lower left corner of the main Audacity window). If a track is selected before running “Append Import” then the track sample rate will be used.

Unfortunately it does not appear to be possible to trap input errors. If an invalid path or file name is entered, then the import will fail without any meaningful message. If the “Debug” button is used rather than the “OK” button, then the error will be shown in the debug window, though not in a very user friendly way.

Nyquist only supports reading uncompressed file types (AIFF, WAV) so it is not possible to use this plug-in directly with MP3, WMA, FLAC or other compressed file types.

This plug-in could be adapted to concatenate longer files, but only by writing directly to disk rather than reading into a track, and there’s better ways of doing that (for example by using SoX). For a modern computer (with 2GB or more of RAM) it should be possible to import well over an hour of stereo audio.

One nice little bonus feature is that the specified directory is the “default” import folder. Files from other locations may be imported at the same time if they are typed with a fully qualified address. For example, if the default folder is /home/user/Desktop/ and file1 is test1.wav and file2 is /home/user/soundfiles/test2.wav then test1.wav will be found from the default directory and test2.wav will be found because it has a fully defined path. I’m not sure how this works on Mac and Windows, so perhaps someone can test it and let me know.
AppendImport.ny (2.49 KB)

A possible update on this could be for the plug-in to read a list of files from a text file. This would overcome the issue of limited space in the GUI so that more files could be imported at the same time. Would that be of interest to anyone?

A minor update.
This version is able to check that the default Import folder exists.
AppendImport.ny (2.63 KB)

Well I thought it was a good idea, but also to have the option to import all of the files in a specific folder (in alphanumeric order).

This version does just that.

"Append Import will look in the selected “Import Folder” for a text file called filelist.txt.
If found, it will attempt to read each audio file that is listed in filelist.txt.
filelist.txt must contain only a list of uncompressed audio files of type .WAV, .AIFF or .AU.
The file extension .AIF is also allowed.

Each file must be on a new line and must be either

  1. A file in the ‘Import Folder’
  2. A fully qualified file name.

If filelist.txt is not found, all supported audio files in the Import Folder will be loaded in alphanumeric order.
Clicking the Debug button rather than the OK button will show a list of the imported file names.

This information is also available in the built-in help screen.
AppendImport.ny (3.58 KB)

I’ve noticed a small bug.
When importing all files from the Import Folder, import may fail if there are files with very short file names.
I think this will be a trivial fix but I don’t have time to fix it right now - so this is a reminder that it needs fixing.

This version includes a fix for the bug mentioned in my previous post.
AppendImport.ny (3.63 KB)

This version adds a feature that may help to recover stereo recordings from a crashed Audacity 1.2 (though the plug-in requires Audacity 1.3.8 or later to run).

The extra feature is an “Import Type” option. The choices are:
Sequential: This is the (default) standard mode that imports files end to end.
Alternate Channels: This is the option that may help recover stereo recordings from a crashed Audacity 1.2.x project. The first file read will be placed in the left channel, the next file will be placed in the right channel, the next appended to the left, and so on.

Note:
Audacity 1.2.x will create sequentially numbered .AU files when recording, however, as soon as there is any editing the file numbers will probably be no longer in numerical sequence.
AppendImport_Mode.ny (4.69 KB)

Thanks for this plug-in Steve and the “import mode” modification.

This needs more explanation I think,Steve. Are you saying that you can leave “Import Folder” empty and it uses the last used Audacity Open/Import directory, or the working directory? On WIn 7, if the “Import Folder” box is empty you receive “Directory “” does not exist”. If you enter the fully qualified path to a file it tells you that directory does not exist.

If you can type multiple files or folder names, what are valid separators?

Is it expected that Unicode or extended ASCII characters in folder names will fail? Folders called “Söndag” and “Älska” are reported not to exist on Win 7 here, though they do.

Also for these and other Nyquist plug-ins that look in folders, do they/should they search recursively?



Gale

No, but this feature has changed since the first (prototype) version.

How it works now:

The “Import Folder”:

  • The “Import Folder” tells the plug-in where to look.
  • If the “Import Folder” is not specified, the plug-in will only look in Nyquist’s default-sf-dir (default sound file directory). The location of the default-sf-dir is platform dependent though I think it could be displayed in the Help screen. For clarity the user should always specify this folder by entering a valid, fully qualified path as the “Import Folder”.
  • The plug-in will look in the “Import Folder” to see if there is a file called filelist.txt.
  • If filelist.txt is found, then the plug-in will import each file that is listed in “filelist.txt” in the order that they are listed.
  • If filelist.txt is not found, then the plug-in will import all supported audio files that exist in that folder.

The filelist.txt file:

  • If this file is present in the “Import Folder” folder it will be used and only audio files listed in this file will be imported.
  • filelist.txt is a plain text file.
  • Each file on the list must be on a new line.
  • Blank lines are ignored.
  • Each audio file must be either (a) a file in the “Import Folder” or (b) a fully qualified file name.
  • All files listed must exist.
  • All files listed must be of supported file types.

If there is no filelist.txt file:

  • All supported files in the “Import Folder” will be imported.
  • Unsupported file types will be ignored.


That may be OS dependent. Söndag" and “Älska” both work here (Linux) as file names or folder names. There’s some (unresolved) discussion about this here: http://audacity.238276.n2.nabble.com/Unicode-Problems-td5681026.html

This plug-in does not search recursively. It looks only in the “Current directory”.
Searching recursively would be difficult and I think that it would probably be more confusing for users.

Fairly obviously, most users are not on Linux, so better help screens listing limitations will be needed. Where is the Nyquist default dir on Windows? I think you 'll need to find someone on Mac too to test before “release”, including testing support for non-ASCII characters.

If we can put together what the non-ASCII problems in Audacity-Nyquist are (and if indeed they are Windows only) then it would probably be an idea to put it on Bugzilla.

I would suggest the help screen information could be round the wrong way and that most people will not have a list of files prepared.

The filelist.txt file… Each audio file must be either (a) a file in the “Import Folder” or (b) a fully qualified file name.

Does this mean that if the AIFF/WAV/AU file is in the “Import Folder” (i.e. the Nyquist dir if left empty or the typed in folder name) that you can just type “filename.wav” in filelist.txt; if the file is not in the “Import Folder” then it can be in the list but the full path must be given? I don’t think this is ideally clear in the Help screen either.

When this plug-in is published I think it would be worth mentioning it as another “solution” on the Wiki Crash Recovery page for the case where the recovery utilities don’t work create the recovery WAV - this should certainly be easier for most users than the scripts listed there.

Thanks


Gale

Updated the Help screen and corrected the default “Import Folder”.
The default “Import Folder” is now set correctly (previous version incorrectly set it to the root folder).
I think that the default should be the users home folder on all platforms (feedback required from users with other OS’s).
The default “Import Folder” is now displayed in the Help screen.

It should be possible to set the “Import Folder” relative to the default (Users home folder?).
For example on Linux the “Import Folder” can be set to the Desktop by either entering:
Desktop
or
/home//Desktop
Again, it would be helpful if this could be checked on other OS’s.

It should not matter if a file separator (“” or “/” depending on the OS) is added to the end of the Import Folder path or not, though it IS important at the beginning of the path.

~/” (tilde slash) notation for the users home folder is not currently supported (Linux).
%UserName% notation for the users home folder is not currently supported (Windows).

Is “~/” also used on Mac?

I think that these shortcuts could be supported if it is thought necessary, though it’s much simpler to just have the option of “relative to home” (as now).
IF “relative to home” works on all OS’s I can add it to the Help screen. If not then I’ll leave it undocumented.

[Update] Just seen the reply fro Gale while I was typing.
The Help screen is full, but I agree that as there are several limitations that cannot be error checked there needs to be documentation of the limitations, so I’ll have to add a second Help screen.

Re. non-ASCII characters I’ll try to contact Edgar to see if there is any update on this (He knows a lot more about this than myself).

Already done in this version.
I’ve also changed the control name “Import Type” to “Import Mode” as I though that “Type” could be confused with “file type”.

This is the Help screen as now:

Append Import will look in the ‘Import Folder’ for
a text file called ‘filelist.txt’. If found, it will
attempt to read each audio file that is listed in
‘filelist.txt’. If not found, all supported audio
files in the ‘Import Folder’ will be loaded in alpha-
numeric order.

The default Import Folder is:
“/home/steve”.

If used, ‘filelist.txt’ must contain only a list of
uncompressed audio files of type .WAV .AIFF or .AU
Each file must be on a new line and must be either:

  1. A file in the ‘Import Folder’
  2. A fully qualified file name.

Import Mode:
‘Sequential’ will append the files in a single sequence.
‘Alternate Channels’ will assume that the first file
is intended for the left channel, the next file for
the right, and so on.

To see a list of imported file names, click the
Debug button.

If the AIFF/WAV/AU file IS in the “Import Folder”, then you can type:
Just the file name on a line in filelist.txt.
or
The fully qualified file name.

If the AIFF/WAV/AU file is NOT in the “Import Folder”, then you must use the fully qualified file name.

For example, if the “Import Folder” is /home/gale/ and you want to import the file /home/gale/test.wav then you could enter a line in filelist.txt as test.wav or as /home/gale/test.wav. But if you want to import the file /home/gale/Music/test.wav (not in the “Import Folder”) then you would need to enter it as /home/gale/Music/test.wav.

Did you get that meaning from the new help screen? If not, can you suggest alternative wording?

I’m still thinking that a cut-down version specifically for recovering unedited Audacity 1.2.x recordings could be useful. The Help screen(s) could be targeted specifically for the task and there would be no need for the filelist.txt or format conversion features. In fact all that it would need (in addition to the Help menu) would be “Mono/Stereo” and the input path.

Can the other scripts do anything that this plug-in can’t?
AppendImport.ny (4.9 KB)

Added a “Limitations” help screen.

This plug-in is not tolerant of user error. Invalid
input may cause the import to fail without warning.

To import files into a stereo track, a stereo track
must be selected before running Append Import.

Non-ascii characters are not supported for file or
folder names on Windows.

The computer must have enough RAM for the entire
import. Attempting to import more audio than can fit
in the available memory will cause Audacity to freeze
and/or crash.

Progress bar and Remaining Time are non-functional.

Errors in the ‘filelist.txt’ file (if used) may
return a ‘Nyquist did not return audio’ error.
Using the Debug button to run the plug-in may give
some clue of what went wrong.

Imported files will be resampled to the current
track sample rate.

AppendImport.ny (5.76 KB)

Edgar has already seen it and will give a more detailed answer how to test this later on…

*** :stuck_out_tongue: ***

Original question was:

Gale: > Folders called “Söndag” and “Älska” are reported not to exist on Win7…
Steve: > “Söndag” and “Älska” both work here (Linux)…

Edgar: Audacity, Nyquist, XLISP and Unicode

XLISP is “interpreted C with Lisp syntax” what has the strange consequence that in XLISP strings internally are stored as 8-bit integer arrays. Originally these string arrays were designed to hold one single 8-bit ASCII charater per array cell, but with multibyte Unicode characters (what in Unicode is every character that’s not contained in ISO-8859-1), every array cell holds one byte of the multibyte character.

Strangely this works, as long as no XLISP character function tries to access an array cell where a byte outside ASCII_0…127 is stored. In XLISP, only the character functions test for ASCII_0…127, but NOT the string functions. The string functions internally only see integer numbers in array cells and do not care about the values of the integers.

IMO this XLISP string-function vs. character-function behaviour is nothing but a big mess, but accidentally (forunately?) this way XLISP string functions (theoretically) can work with multibyte Unicode characters, as long as no XLISP character function is involved.

The problem with the Audacity Nyquist interface was that it seems to be OS dependent (or wx-widgets dependent) in what encoding the strings from the user interface are given from Audacity to Nyquist and vice-versa.

An Audacity Nyquist test-plugin could work like this:

  • Read a string containing non-ASCII_0…127 characters from a Audacity Nyquist text widget. The string then automatically gets stored in a Nyquist variable.


  • Write the string from the variable out into a text file, so with a text editor could be investigated in what encoding non-ASCII_0…127 characters are stored in the file. If there are nonsense characters in the text file then this might be the reason why under Win7 files and directories with non-ASCII_0…127 characters in their names are not found. This test must be done with every OS Audacity supports.


  • Return the string from Nyquist back to Audacity to see if non-ASCII_0…127 characters are correctly displayed in the Audacity pop-up text window. If there are nonsense characters in the text window then it’s an Audacity or wx-widgets problem. This test should also be done with every OS Audacity supports.

Anything overlooked or forgotten?

  • edgar

P.S.: if you’re looking for an attached plugin - the test-plugin still needs to be written…
P.P.S.: what happens if an Audacity Nyquist plugin writes text files with file names like “Söndag.txt” or “Älska.txt”?

Yes.

Certainly don’t object to a separate utility, and as I said before I think AppendImport.ny (or a customised version of it) should be listed on the Wiki Crash Recovery page. I have a few reservations.

  • The only real problem with the python-based GUI tool recommended for Mac and Windows is that it can throw up errors once you get to 1000 or more .au files. Since that’s 1 GB I suspect you wouldn’t want to recommend anything like 1000 files to import into RAM with AppendImport.ny, and that AppendImport.ny will thus “reach the limit” long before the recovery tool will.
  • The process of browsing to the folder location in the python tool raises an OS window which is likely to be much more reassuring for novice or stressed users than AppendImport.ny (to access non-ASCII paths on Windows with AppendImport.ny users will have to move the files).
  • User will have to stop and download latest 1.3 (while that may be a good thing it’s another step).

I think all the above are OK if the limitations are clearly mentioned on the Wiki, but if we have instructions the need for a custom plug-in is reduced anyway.

One point in favour of AppendImport.ny is that the python tool only understands continuous runs of .au files. If any .au files are missing it will start a new recovery.wav at the point where the sequence breaks.

Don’t know if it helps, but I tried Steve’s sampleprinter3.ny on Win 7. It can’t write to a non-ASCII folder path.

If you ask it to write to an “Älska.txt” file:

; write output as file
(setq fp (open "Älska.txt" :direction :output))

it disappears from the Analyze menu on account of the “Ä”.


Gale

I don’t yet know what “the limit” is for the number of files, but on a modern computer with plenty of RAM it should be able to handle a lot. I’ll need to do some stress testing.

I agree that the lack of a file browser is a major weakness - now that would be a nice enhancement for Audacity Nyquist - a browser widget. :sunglasses:

Yes, this helps a lot. If the following happens:

You install sampleprinter3.ny in the Audacity plug-ins folder and it works OK, but as soon as you take a text editor, change the filename in the code to “Älska.txt”, (like shown above) and if you try to apply the modified sampleprinter3.ny to an Audacity track, instead of writing an output file the sampleprinter3 just simply disappears from the Analyze menu, then Audacity itself can’t read the “Ä” on Win7.

If Audacity starts, or if you modify a Nyqusit plugin while Audacity is running, the plugin is re-parsed by Audacity as soon as you try to apply the modified plugin. The parsing of the plugin text happens outside of Nyquist, if I remember right by “/src/effects/nyquist/Nyquist.cpp:ParseFile()” in the Audacity Nyquist interface. If there’s any problem found in the plugin text during parsing, like obviously the “Ä” on Win7, then the plugin is removed from the Audacity GUI (the plugin entry disappears from the respective Audacity menu).

I do NOT think that this is a Nyquist bug, everything looks as if Audacity itself can’t read the “Ä” on Win7. Unfortunately I have no idea where exactly the “Ä” problem happens in the Audacity C/C++ code.

In case of doubt we must ask on the developers list.

  • edgar

P.S.: in another case of doubt we must open a new thread for this because it starts to become a bit confusing here…

Is there any reason not to publish this on http://wiki.audacityteam.org/wiki/Download_Nyquist_Plug-ins ? I thought we have most limitations covered in the Help screens?



Gale

There is a bug in the current version.
When using “Alternate Channels” input mode, if there are an odd number of files, Audacity will freeze (the plug-in is waiting for a non-existent file to import).

I’m working on a fix now along with adding a new feature:
With “Sequential” import, as the imported files are “merged” into a single track, there is no easy way see where each file starts and ends, so I’m adding a feature that allows the position of the imported files to be labelled.