possible to batch edit 17,000 slightly different .mp3 files?

I have 17,000 .mp3 files from a dictionary project.

The first file starts off with a recording of someone saying, “One.” What follows after that is the part of the audio recording that I want to keep.
The second file starts off with someone saying, “Two,” and then the stuff I want keep.
Etc., etc., etc. through the 17,000th file.

Of course after each number is said, there is a brief silent pause.

It will not work to just delete the first second (for example) of each file, because that would be too long for some files (it doesn’t take long to say, “one”) and too short for others (it would take longer to say, “16,783,” for example).

Can Audacity somehow be set up to await that brief pause, and then split that file into two parts: (1) the part where the number is said, which I’d be glad to delete, (2) the part with the rest of the file that I want to keep?

Helllllllp me please! I’m an Audacity rookie. You will not offend me by being super specific and detailed in your replies and directions. :smiley:

And thank you!

I think we can do the silent sense, but I don’t think we can make the Chains/Batch system look for the marks, and then only use the first mark.

Someone better at Batch/Chains may post.

Also, there may be a limit to the number of files. Audacity makes copies of The Whole Job every time you take an action so it can perform UNDO. That process is going to get enormous after a while.

Koz

Thanks @Koz, any wisdom how I might engage someone who is "better at “Batch/Chains” with this question?

The file sizes are about 350kb each, relatively small, but we could do them in groups to overcome the file size challenge.

how I might engage someone

No, and that’s not the only problem you’re going to have. You violated a process recommendation—never do production in MP3. Nasty things can happen.

Audacity doesn’t edit MP3. It converts MP3s to its internal editing format and then makes a new MP3 when it’s done, so anything you do is going to have double the MP3 compression sound distortion. You can avoid the increase by exporting the finished works as WAV and never making another MP3, or you can get less damage by making very high quality MP3s. Both of those will give you much higher file sizes than you think.

You may be better off with one of the pure MP3 editors that don’t take the sound apart and put it back together.

Scroll down to Other MP3 Editing Tools.

Koz

I certainly don’t think you can do it with Chains.

But for the next release 2.3.0 we are changing Chains to Macros with added functionality (more commands that you can put in a Macro - and other stuff)

We recommend never processing more than 500 files at a time in a Chain or Macro (smaller numbers may be even better)

You can’t do it right now with Macros AFAICT, but I am in discussion with the developer who is handling Macros - we may need to add a command (or two) if he is prepared to do that - I’ll let you know …

The Export process in Macros can be a bit flaky right now - but we should have that fixed in the next month - so can you wait a bit


First let me ask you a few questions about your files:

  1. are there other 1-plus second silences in each file,

  2. is one second the minimum length of silence between the spoken number and the required audio,

  3. what output format are you expecting (audio files, Audacity Projects),

  4. are you expecting to write back, overwrite, the existing input file(s) probably not a good idea,

  5. can I assume that you would make a safety backup copy of all thes files vefore exposing tem to Macro processing,

  6. would you be prepared to take the (managed) risk of using Beta software for this ?

Peter

Audacity’s Chains provide a simple “list” processing feature. That is, you have a list of commands, and the Chain works down the list from top to bottom. There is no mechanism for logical operations (such as conditional branches).

Also, Chains should not generally be used for more than a couple of hundred files at a time. Better to split your collection of 17000 into smaller groups. Say, 34 groups of 500 files.

Possible options depend on budget. Is this a commercial job, is there a budget?
I’m also wondering why you have 17000 specially prepared files, before having a clear plan of how to process them in the way that you need. What’s the job? What’s the big picture?

Do you (or your team) have any programming experience?

Here’s the document on Chains.

http://manual.audacityteam.org/man/chains_for_batch_processing_and_effects_automation.html

Koz

Not possible to do with Chains …

However with 2.3.0 alpha Macros some of the added commands help us here

First the good news.
This little Macro does the job:
SelectAll:
SilenceFinder:labelbeforedur=“0.3” sil-dur=“1” sil-lev=“26”
SelectAll:
SplitLabels:
SelectAll:
CursProjectStart:
SelNextClip:
Delete:
ExportWav:

The silence finder command will find all silences of longer than 2.0 seconds abd above the thrfeshold - those are settable parameters
It will then place a label 0.3 second before the non-silence starts (settable parameter)
we then place split markers in the audio at all the label positions (that’s the SplitLabels command)
We then move the cursor back to home T=0
and then select all up to the next clip (i.e. from T=0 to 0.3 seconds before your required audio_
We than delete the section
final step is to export

Right now I have the export set to WAV - this is for a couple of reasons.

  1. As you input is MP3 re=exporting to MP3 will increase the compression damage
  2. as it stands tight now exporting to MP3 would just overwrite your existing source MP3 with no warning (this is one of the wrinkles we want to fix)

Now the bad news
The SplitLabels command has a bug which we only discovered when I was setting up and testing this Macro.
Basically it pops an error when you try to run it the first time (or with out a prexisting label track) with a selection apparently I’m told)

This of course makes it unusable as a batch-process.

I have managed to bench-test the Macro - but on the open project and not a set of files - I have to run it twice, and the second time it places the split marker(s) - I will see if I can find a better workaround, but I’m not hopeful.

I will shortly be logging the bug.

In the meantime if you want to be bold and test this on one of your files using a nightly alpha you would be welcome. You can get the latest Mac nightlies from here: https://www.fosshub.com/Audacity-Mac-Nightlies.html

I have attached the actual .txt file with the Macro - you would need to download 2.3.0 and add this to your Macros folder.

The draft documentation for the Macros can be found here: https://alphamanual.audacityteam.org/man/Macros

And thanks to James Crook (Macros developer) for helping me with what looks on the surface a very little bit of Macro coding - but it’s only easy when you know how :sunglasses:

Cheers,
Peter
silence remove.txt (165 Bytes)

Thanks!

And now that James, the developer, understands the underlying issue for the bug - he’s given mee a workaround.

The Macro now reads:
SelectAll:
SilenceFinder:labelbeforedur=“0.3” sil-dur=“1” sil-lev=“26”
SelectAll:
ShowExtraMenus:
ShowExtraMenus:
SplitLabels:
SelectAll:
CursProjectStart:
SelNextClip:
Delete:
ExportWav:

Note that we now have two additional extraneous commands of ShowExtraMenus - this is enough to “fool” Audacity into working the SplitLabels command

If you’re interested in the technical details - James wrote:
Add Extra Menus (on/off) TWICE immediately before the Label Split (SplitLabels) command.
That causes the menus and their flags to be regenerated. And then the command will work because the command is not greyed out in the menus anymore after the greying-out flags are re calculated!

I’ve attached an updated copy of the .txt file for the Macro

Peter.

P.S. this has been a useful QA test of the new Macros for me :sunglasses:
silence remove.txt (199 Bytes)

BTW this Macro will only work properly if you don’t have a silence if 1.0 seconds or greater BEFORE the spoken number.

This Macro (as currently parameterized) just deletes the first block of silence of grater that 1.0 (parameterizable) seconds that it finds.

Also you may need to tweak the “silence” level depending on how “silent” you “silence” is :nerd:

Peter.

Thanks so much, all!

**1) are there other 1-plus second silences in each file,**
There would be several pauses per recording. I think almost all of the pauses would be less than a second.

That would be no problem this Macro only removes the first "silence" that it finds

**2) is one second the minimum length of silence between the spoken number and the required audio,**
No, of course I've not listened to all 17,000 of the files, but think the silences would be shorter than a full second.

In which case we would need to tweak down that 1 second minimum silence length

but we have to make sure that whatever we set it at exceeds ant "silence" that may be before the number

**3) what output format are you expecting (audio files, Audacity Projects),**

I'm flexible, but something pretty standard would be best. mp3?
You'll double the compression damage - but you're not making hi-fi audio

**4) are you expecting to write back, overwrite, the existing input file(s) probably not a good idea,**
I'm flexible here, too.
Right now if you have mp3 as input and we change the Macro to export to MP3 then overwrites will occur - James is gonna wortk later to fix this.

**5) can I assume that you would make a safety backup copy of all these files vefore exposing them to Macro processing,**
Yes, I have redundant file copies, so we can be aggressive in experimenting.
Good - so not entiely naive after all ...

**6) would you be prepared to take the (managed) risk of using Beta software for this ?**
In theory, yes. In practice, I'm a total novice to Audacity.
I like bold people  :smiley:

Doing groups of 500 or whatever would be fine.

The big picture is that I’m helping make a dictionary for a language. Along the way I hired a linguist to help. There ended up being about twice as many words as anticipated. We did not realize that the numbers would be added to the audio files. While it’s hard for me to imagine ever breaking even financially on this project, I have used money to get us this far, and would be open to spending some more for a great and quick solution.

I have almost no programming or Audacity experience.

So some hand-holding may be necessary - we can help.

I’ll post some getting-started instructions in a fresh post

Peter

Thanks Peter, Steve, and James!

I’ve been following along without interrupting, but… Does the current Alpha have some form of a Trim (front/back/both) command/Effect which could be run before finding silences?

I’ve just logged the bug: https://bugzilla.audacityteam.org/show_bug.cgi?id=1861

Peter.

  1. In Finder rename your current Audacity folder in the Applications folder to say “Audacity Released”.
    That will preserve you existing Audacity installation
  1. Go to and download the latest Mac alpha DMG

  2. Run the DMG and drag the app over into the Applications folder (do NOT run from the DMG)

  3. copy the attached Macro silence remove.txt file from here to somewhere on your Mac (desktop is fine)

  4. Copy this file into the “Macros” folder - on Mac this is in the following folder: ~/Library/Application Support/audacity/
    note the “~” (tilde) - that’s your user space not system space

  5. Go to the Applications folder and right click on the new Audacity.app

  6. This is unsigned so on first usage you have to permit it to run (it’ll just run on subsequent uses)

  7. Import one of you files into Audacity - have a look to make sure that there is less than one second before the spoken number and more than one second after the spoken number

Now for the good bit :sunglasses:

  1. use Tools >Macros… - you will get the Manage Macros dialog

  2. in the left hand Select Macro box you should see silence remove

  3. left-click on that - it will get highlighted and the Macro commands will show in the right hand pane

  4. Now click on the Project button to the right of the “Apply Macro to:”

  5. This will run the macro

  6. because we’re running non-batch on the project it should ask you where you want to store the output mp3 file

  7. Report back :wink:

Peter

I need a lie-down in a darkened room …

Hmmm - that’s a thought Ed - I’ll have think about that


Peter

I can’t see one - only the old Trim Audio command.

Are you thinking, Ed, Of James’ pre-supplied “Fade Ends” Macro - which does Fades not Trims.

Peter

You can trim a specified number of seconds from each end.
Here’s an example macro script:

SelectTime:End="2" Start="0"
Delete:
SelectTime:End="2" RelativeTo="Project End" Start="0"
Delete:

For more complex “arithmetic” operations, a proper scripting language (such as Python or Nyquist) should be used rather than macros. Like the current “Chains”, Macros are intended for applying a list of command (the difference being that Macros support many more commands than Chains did). The commands available to Macros will also be available to external scripting languages, and eventually also to Nyquist. (Nyquist support will be added later as there are issues with reentrancy that still need to be resolved).

useful - but no help to the original poster here - they don’t know how long an possible front-end silence is but they do know there is a “silence” between the initial spoken number and the audio they want to retain.

WC

Stevethefiddle wrote me a custom “TrimSilence” Nyquist Effect:

;nyquist plug-in
;version 3
;type process
;categories "http://lv2plug.in/ns/lv2core/#UtilityPlugin"
;name "Trim Silence..."
;action "Trimming..."
;info "by Steve Daulton (www.easyspacepro.com). Released under GPL v2.\n\nTrims silence from the beginning and end of the selection.\n"

;control thresh "Silence Threshold (dB)" real "" -38 -100 0

;; TrimSilence.ny by Steve Daulton. Aug 2011.
;; Updated 24 Sept 2012.
;; Released under terms of the GNU General Public License version 2:
;; http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
;; Requires Audacity 1.3.8 or later.

;; RAM USAGE:
;; This plug-in requires the audio to be loaded into RAM.
;; If there is insufficient free RAM Audacity may freeze or crash.
;; The line below limits RAM usage to 1.0 GB (about 47 minutes for
;; a stereo track at 44.1 kHz)
;; If your computer has more than 1GB of physical RAM available, the
;; limit may be increased.

(setq RAM-Limit 2.5) ; RAM limit in GB

; convert threhold to linear
(setq thresh (db-to-linear (min 0 thresh)))

;; Limit of duration in seconds
(setq limit
  (/ (* ram-limit 1000000000)
    (* 4.0 *sound-srate*)))
(when (arrayp s)(setq limit (/ limit 2.0)))

;;; modulo
(defun mod (x y)
  (setq y (float y))
  (round (* y
    (- (/ x y)
      (truncate (/ x y))))))
  
;;; convert to hh:mm:ss
(defun to-hhmmss (seconds)
  (let* ((hh (truncate (/ seconds 3600)))
        (mm (truncate (/ (mod seconds 3600) 60)))
        (ss (mod seconds 60)))
    (format nil "~ah:~am:~as" hh mm ss)))

;;; convert to mono and limit sample rate
(defun convert (sig ratio)
  (if (arrayp s)
      (snd-avg 
        (s-max (snd-abs (aref sig 0))
          (snd-abs (aref sig 1)))
        ratio ratio op-peak)
      (snd-avg sig ratio ratio op-peak)))

;;; find silences
(defun find-sil (sig &aux (start 0)(end 0))
  (do ((new (snd-fetch sig) (snd-fetch sig))
       (flag 0))
      ((not new))
    (if (= flag 0)
        ;; count initial silence
        (if (<= new thresh)
            (setq start (1+ start))
            (setq flag 1))
        ;; count final silence
        (if (<= new thresh)
            (setq end (1+ end))
            (setq end 0))))
  (list start end))


(if (< len (* limit *sound-srate*)) ;max length in samples
  (let* ((start 0)
         (end 0)
         (flag 0)
         ;; ratio provides tighter trimming for short selections
         ;; while maintaining reasonable speed for long selections
         (ratio (max 10 (min 200 (round (/ len 100000.0)))))
         (my-srate (/ *sound-srate* ratio))
         (mysound (convert s ratio)))

    ;loop through samples and mark start and end
    (setf result (find-sil mysound))

    (let ((start (/ (first result) my-srate))
          (end (- (get-duration 1)(/ (second result) my-srate))))
      ;; ensure at least 1 sample remains
      (if (>= start (get-duration 1))
        (setq start (/ (1- len) *sound-srate*)))
      ; trim
      (multichan-expand #'extract-abs start end (cue s))))

  ;; OR print error message
  (format nil "Error.\nMax RAM usage by Trim Silence is set to ~a GB.~%This allows a maximum duration ~
              for a ~a~%track at ~a Hz of ~a.~%Selected track is ~a.~%"
              RAM-limit
              (if (arrayp s) "stereo" "mono")
              (round *sound-srate*)
              (to-hhmmss limit)
              (to-hhmmss (get-duration 1))))

Pay attention to the note on RAM usage:

;; RAM USAGE:
;; This plug-in requires the audio to be loaded into RAM.
;; If there is insufficient free RAM Audacity may freeze or crash.
;; The line below limits RAM usage to 1.0 GB (about 47 minutes for
;; a stereo track at 44.1 kHz)
;; If your computer has more than 1GB of physical RAM available, the
;; limit may be increased.

(setq RAM-Limit 2.5) ; RAM limit in GB

As you can see I have changed this from 1.0 to 2.5 as I have plenty of RAM. However, I have also written a “built-in” Effect in C++; it is much faster, a bit smarter, and allows for three options: trim silence off the front, trim silence off the back & trim silence off both the front and the back. It also gives direct access to the amplitude to consider as “silence”. There is also a checkbox for “Assume a short track”; if checked, it loads the entire track into RAM for processing; otherwise it bites a chunk off the front and/or back (one at a time as needed) does a little checking to verify that you got a good sample (e.g. for front has something that is not “silence” as the final few samples, for back as something that is not “silence” as the first few samples) and if not bites of bigger chunks and retests. The GUI needs a little TLC as statically and looks like this:
Audacity allows you the choice to run one effect (Normalize) every time you import; I have extended that functionality: