Audacity Chains with thousands of files

Hi there,

I am facing some issues using chains on a large number of files, i.e. up to 20,000

Issue 1) When trying to select a large number of files, i.e. > 3,000 - audacity views the filenames as having strange non-unicode characters in them and says there is no track.
Issue 2) If I select a moderately large number of files, i.e. > 2,000 but < 3,000 - the processing begins and appears to be working fine, but will just stop about half way and doesn’t give an error.

The system I am running this one is a 3.6/3.8Ghz Core i7 3820, 16GB DDR3 RAM, 128GB SSD - so I don’t think that hardware resources is the issue.

And tips/pointers will be greatly appreciated.

Thanks for the report. That seems like an unreasonable number to me. Most computers would probably take half and hour just to select all the files in the file open dialogue.

Where exactly does it see strange characters?

If you are using 2.0.1 or earlier, turn off “Audio Cache” in the Directories Preferences.



Gale

Hi Gale,

Thanks for your reply, much appreciated.

As for the 20,000 being an unreasonable number, I can understand it would seem so! But it takes about 45-60 seconds to select all the files in the open dialogue.

As for the strange characters, I will be able to show you a screenshot when I get home from work, but basically it shows it in an error dialogue after I select all the files and press OK.

As for the Audio cache tip, thanks, I’ll give that a try, though I am using version 2.0.2

Cheers,
Gerard

Audacity 2.0.2 does not have the audio cache feature (which is why Gale specified “2.0.1 or earlier”).

Is Audacity still responsive after processing stops after 1500 files or so? Could you restart batch processing with another 1500 files or so if you wanted to, or do you have to force quit?

The temporary space should be cleared by Audacity after processing each file, so disk space should not be a problem.


Gale

Hi Gale,

Yep - Audacity is still completely responsive after the processing stops mid-way, and there is no messages in event viewer.

I /can/ restart at where it stops, but as mentioned, I have thousands and thousands of files to process (Normalize, truncate silence, export to mp3).

Again, thanks for your input.

Cheers,
Gerard

Thanks.

One thing you could try. Open Windows Task Manager, Processes tab, right-click over audacity.exe > Set Affinity. If Audacity is set to use all processors (it almost certainly doesn’t), just try selecting one processor.


Gale

Hi Gale,

I’ll give that a try later on, thanks for your help.

Cheers,
Gerard

Hi Gale,

I’ve tried fiddling with the CPU affinity, alas to no avail.

Interestingly, the chain works fine if I select 1,000 files or less.

But, if I were to select 2,000 files, it would only process about 60-70 and then die…

Also, after I’ve been processing/exporting for a while, I get an error which requires me to restart Audacity before continuing:

“Error writing autosave file”
“Couldn’t write to file C:usersusernameAppDataroamingAudacityAutosavedate-time … etc.tmp: error opening file”

This is really starting to drive me nuts! I really want to avoid moving towards a proprietary solution such as soundforge.

Again, any ideas appreciated.

Cheers,
Gerard

I presume that this problem is independent from what functions are in the Chain?
For example if you use an empty chain, (containing only “- END -”) does it still crash out at > 1000 files?
What is the (input) file format?
In “Edit > Preferences > Import / Export”, are uncompressed files set to be copied or read directly?

This can sometimes happen anyway and isn’t related to Chains. Are .tmp files being left behind in the AutoSave folder when this happens?

You may be able to use dBPowerAmp or SoX (command-line) - it may depend exactly where you want to trim silences in the files.


Gale

There is a chance you are running out of space on the hard drive. Audacity can fail with no warning nor error message in this case. A 128 GB drive might seem large but if it is the C drive it is likely quite full ( 1000’s of MP3s, OS and apps, etc.) before you begin chain processing. If you have another drive try pointing Audacity’s Temp file to a folder there:
tempDir.png
Also, if the MP3s are on your SSD C drive try putting them on a different drive for processing.

Audacity also has a Log which might give us some needed info; after it fails (but is still responsive to menu actions) choose menu item Help > Show Log…, the log will look something like:
log.png
This Log window had a menu; choose Log > Save… then, after saving it somewhere, examine the last few lines to see if there is anything interesting. If the text is not hundreds of lines you may paste it into a post here; if it is very large, zip it up and send it to Gale, Steve or myself (use the PM function of the board).

I had assumed not, since Gerard had said

The chain works fine if I select 1,000 files or less.

But, if I were to select 2,000 files, it would only process about 60-70 and then stop

What size are the files you are processing? Since Chains cleans the temp folder after processing each file, you only need three times the disk space of that file available, but you also need the space for writing the new file.

That could be significant. There is no way (I know of) to make Chains overwrite an original file, so you are continually adding to disk space. Even if you apply the chain to the original file, it will export the modified files to a new “Cleaned” directory in the folder the files came from.

The log would be worth looking at, I agree. And do you have that screenshot of the strange characters after selecting 3000 files?


Gale

Hi All,

Thanks for your replies and apologies for my delay in response.

To clarify, disk space is not an issue here. The disk has 56GB free space (have added a 64GB SSD purely for the section of audio I am working on), the files are from roughly 10kb to 150kb in size.

The chain is:

  • Normalize
  • Truncate Silence
  • Export to MP3

Below is a screenshot of the error I have if I select over 2,000 sound files:
http://imgur.com/07m3F

The issue here appears to be something to do with the temp files. Why it has this error, I do not know, as when I browse to the directory in the error dialog, the file exists.


Below is a screenshot of the error I have if I select a large number of files, in this case files 7000 → 15000
http://i.imgur.com/088wN.png

This issue here, I really am not sure… I think it must be something to do with the way it generates the list of files for processing before the job begins, as it should start a 7,000 but instead starts at 10,000 (7 > 1) as it hasn’t realised the extra ‘0’ at the end…? I’ve noticed when selecting files in the explorer window, it takes a second or two to re-sort the list of files correctly… Also, I have no idea why it is adding the strange character and space to the start of the filenames (and saying they are raw)… The logs also display this (for this particular scenario)
18:51:41: FFmpeg directory is in PATH.
18:51:41: Checking for monolithic avformat from ‘avformat-52.dll’.
18:51:41: Error: Failed to load shared library ‘avformat-52.dll’ (error 126: the specified module could not be found.)
18:51:41: Loading avutil from ‘avutil-50.dll’.
18:51:41: Error: Failed to load shared library ‘avutil-50.dll’ (error 126: the specified module could not be found.)
18:51:41: Loading avcodec from ‘avcodec-52.dll’.
18:51:41: Error: Failed to load shared library ‘avcodec-52.dll’ (error 126: the specified module could not be found.)
18:51:41: Loading avformat from ‘avformat-52.dll’.
18:51:41: Error: Failed to load shared library ‘avformat-52.dll’ (error 126: the specified module could not be found.)
18:51:41: Error: Failed to load FFmpeg libraries.
18:51:41: Error: Failed to find compatible FFmpeg libraries.


Any further suggestions much appreciated!

Cheers,
Gerard

How is it formatted? NTFS, FAT32, something else?

Hi Steve,

The disk is formatted as NTFS using default allocation unit size.

Cheers,
Gerard

Could you try moving, say 2000 files to a location that has a significantly shorter path (for example F:/test/) and set your temp directory to a shorter path and see if that makes a difference.

I’m just guessing here, but what I’m thinking is that Audacity audacity needs to temporarily store a list of all of the file names that are to be processed, which I would guess is stored in RAM. If it is storing the fully qualified file names of over 1000 files, that may require more space in RAM than is being allocated. For example, if it allocates say half a MB of address space, you could easily be filling that and overflowing, so corrupting the file list.

Hi Steve,

Thanks for your reply, that is definitely an interesting theory.

I will give it a shot when I get home and post results.

Cheers,
Gerard