I am facing some issues using chains on a large number of files, i.e. up to 20,000
Issue 1) When trying to select a large number of files, i.e. > 3,000 - audacity views the filenames as having strange non-unicode characters in them and says there is no track.
Issue 2) If I select a moderately large number of files, i.e. > 2,000 but < 3,000 - the processing begins and appears to be working fine, but will just stop about half way and doesn’t give an error.
The system I am running this one is a 3.6/3.8Ghz Core i7 3820, 16GB DDR3 RAM, 128GB SSD - so I don’t think that hardware resources is the issue.
Thanks for the report. That seems like an unreasonable number to me. Most computers would probably take half and hour just to select all the files in the file open dialogue.
Where exactly does it see strange characters?
If you are using 2.0.1 or earlier, turn off “Audio Cache” in the Directories Preferences.
As for the 20,000 being an unreasonable number, I can understand it would seem so! But it takes about 45-60 seconds to select all the files in the open dialogue.
As for the strange characters, I will be able to show you a screenshot when I get home from work, but basically it shows it in an error dialogue after I select all the files and press OK.
As for the Audio cache tip, thanks, I’ll give that a try, though I am using version 2.0.2
Is Audacity still responsive after processing stops after 1500 files or so? Could you restart batch processing with another 1500 files or so if you wanted to, or do you have to force quit?
The temporary space should be cleared by Audacity after processing each file, so disk space should not be a problem.
One thing you could try. Open Windows Task Manager, Processes tab, right-click over audacity.exe > Set Affinity. If Audacity is set to use all processors (it almost certainly doesn’t), just try selecting one processor.
I presume that this problem is independent from what functions are in the Chain?
For example if you use an empty chain, (containing only “- END -”) does it still crash out at > 1000 files?
What is the (input) file format?
In “Edit > Preferences > Import / Export”, are uncompressed files set to be copied or read directly?
There is a chance you are running out of space on the hard drive. Audacity can fail with no warning nor error message in this case. A 128 GB drive might seem large but if it is the C drive it is likely quite full ( 1000’s of MP3s, OS and apps, etc.) before you begin chain processing. If you have another drive try pointing Audacity’s Temp file to a folder there:
Also, if the MP3s are on your SSD C drive try putting them on a different drive for processing.
Audacity also has a Log which might give us some needed info; after it fails (but is still responsive to menu actions) choose menu item Help > Show Log…, the log will look something like:
This Log window had a menu; choose Log > Save… then, after saving it somewhere, examine the last few lines to see if there is anything interesting. If the text is not hundreds of lines you may paste it into a post here; if it is very large, zip it up and send it to Gale, Steve or myself (use the PM function of the board).
The chain works fine if I select 1,000 files or less.
But, if I were to select 2,000 files, it would only process about 60-70 and then stop
What size are the files you are processing? Since Chains cleans the temp folder after processing each file, you only need three times the disk space of that file available, but you also need the space for writing the new file.
That could be significant. There is no way (I know of) to make Chains overwrite an original file, so you are continually adding to disk space. Even if you apply the chain to the original file, it will export the modified files to a new “Cleaned” directory in the folder the files came from.
The log would be worth looking at, I agree. And do you have that screenshot of the strange characters after selecting 3000 files?
Thanks for your replies and apologies for my delay in response.
To clarify, disk space is not an issue here. The disk has 56GB free space (have added a 64GB SSD purely for the section of audio I am working on), the files are from roughly 10kb to 150kb in size.
The chain is:
Normalize
Truncate Silence
Export to MP3
Below is a screenshot of the error I have if I select over 2,000 sound files: http://imgur.com/07m3F
The issue here appears to be something to do with the temp files. Why it has this error, I do not know, as when I browse to the directory in the error dialog, the file exists.
Below is a screenshot of the error I have if I select a large number of files, in this case files 7000 → 15000 http://i.imgur.com/088wN.png
This issue here, I really am not sure… I think it must be something to do with the way it generates the list of files for processing before the job begins, as it should start a 7,000 but instead starts at 10,000 (7 > 1) as it hasn’t realised the extra ‘0’ at the end…? I’ve noticed when selecting files in the explorer window, it takes a second or two to re-sort the list of files correctly… Also, I have no idea why it is adding the strange character and space to the start of the filenames (and saying they are raw)… The logs also display this (for this particular scenario)
18:51:41: FFmpeg directory is in PATH.
18:51:41: Checking for monolithic avformat from ‘avformat-52.dll’.
18:51:41: Error: Failed to load shared library ‘avformat-52.dll’ (error 126: the specified module could not be found.)
18:51:41: Loading avutil from ‘avutil-50.dll’.
18:51:41: Error: Failed to load shared library ‘avutil-50.dll’ (error 126: the specified module could not be found.)
18:51:41: Loading avcodec from ‘avcodec-52.dll’.
18:51:41: Error: Failed to load shared library ‘avcodec-52.dll’ (error 126: the specified module could not be found.)
18:51:41: Loading avformat from ‘avformat-52.dll’.
18:51:41: Error: Failed to load shared library ‘avformat-52.dll’ (error 126: the specified module could not be found.)
18:51:41: Error: Failed to load FFmpeg libraries.
18:51:41: Error: Failed to find compatible FFmpeg libraries.
Could you try moving, say 2000 files to a location that has a significantly shorter path (for example F:/test/) and set your temp directory to a shorter path and see if that makes a difference.
I’m just guessing here, but what I’m thinking is that Audacity audacity needs to temporarily store a list of all of the file names that are to be processed, which I would guess is stored in RAM. If it is storing the fully qualified file names of over 1000 files, that may require more space in RAM than is being allocated. For example, if it allocates say half a MB of address space, you could easily be filling that and overflowing, so corrupting the file list.
