Significant duration recordings (~1 month)

[[XAP]Bob]

Hi,

I can't find any posts on this kind of thing in the forum (at least partly because the search facility won't let me search for the phrase "long time") so here goes

Situation:
I have just set up a charity radio station - which broadcasts for the month running to Christmas. Legally we are obliged to record the output and retain it for 4 months.

Setup:
I therefore have a tuner set up at home, connected to my PC which is using audacity to record the signal (22kHz, 16bit, mono).
For completeness - it's an Ubuntu 7.10 box, with the universe/sound repository version 1.3.3-1build1

Request:
This is all well and good - I have enough disk space to record the month...

But I'd like a bit more protection.

I'd therefore like to be able to take the au files, copy them around and convert them to mp3 format for early review and storage (probably as MP3 on DVD).

Obviously I'm generating rather alot of these files (>4000 so far) so I'd like to be able to automate the process somewaht (OK - I'd like it to be completely automated, but I am pretty happy scripting around any manual solution that doesn't involve rodents )

Has anyone here done anything similar before - or know where to start?

[[XAP]Bob]

Hmm - there may be a bigger problem - audacity seemed to stall after ~ 43 hours, it's not out of memory or disk space.

It just decided it didn't want to play ball, and stopped recording.

I've kicked it off again, and set up a a secondary system - but the above questions still stand.

it might be that audacity isn't the right tool, I just happened to have it to hand - if anyone knows of a more appropriate tool then that'd be great too...

[richardash1981]

I certainly wouldn't try and record a month in audacity with anything released so far (most releases will fail to re-load the project at about 26 hours with your settings). What you need is something that streams to disk and then forgets about it, with a way of switching files quite regularly to keep the file size down.

I'm thinking probably either arecord + bash for a simple command-line setup, of flumotion's disker component if you want a GUI system (but a much more complex one). On the plus side, flumotion would also do encoding and streaming if you needed to, but it's probably overkill.

On the arecord route (it's in the alsa-utils package if you don't have it), I'm thinking

Code: Select all

#!/bin/bash
while true
do
# record for one day, file named after today's date
arecord -r 22000 -t wav -c 1 -f S16_LE <other options here> -d 86400 `date +%Y%m%d`.wav
done

To get the mp3s I would probably run a cron job once a day which encodes any WAV files which don't have a corresponding MP3 file in the output directory.

[[XAP]Bob]

arecord seems like a good plan, I just can't pick anything more than silence out of it.

Not sure if that's because audacity is still running though...

If anyone else is following this thread I'd advise the following script instead (it just means that failures in arecord don't overwrite previous files of any significance):

Code: Select all

#!/bin/bash
while true
do
# record for one day, file named after today's date, and a timestamp.
arecord -r 22000 -t wav -c 1 -f S16_LE <other options here> -d 86400 `date +%Y%m%d-%H%M`.wav
done

For info:

$ arecord -L
default:CARD=Intel
HDA Intel, ALC882 Analog
Default Audio Device
front:CARD=Intel,DEV=0
HDA Intel, ALC882 Analog
Front speakers
surround40:CARD=Intel,DEV=0
HDA Intel, ALC882 Analog
4.0 Surround output to Front and Rear speakers
surround41:CARD=Intel,DEV=0
HDA Intel, ALC882 Analog
4.1 Surround output to Front, Rear and Subwoofer speakers
surround50:CARD=Intel,DEV=0
HDA Intel, ALC882 Analog
5.0 Surround output to Front, Center and Rear speakers
surround51:CARD=Intel,DEV=0
HDA Intel, ALC882 Analog
5.1 Surround output to Front, Center, Rear and Subwoofer speakers
surround71:CARD=Intel,DEV=0
HDA Intel, ALC882 Analog
7.1 Surround output to Front, Center, Side, Rear and Woofer speakers
iec958:CARD=Intel,DEV=0
HDA Intel, ALC882 Digital
IEC958 (S/PDIF) Digital Audio Output
null
Discard all samples (playback) or generate zero samples (capture)

$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: Intel [HDA Intel], device 0: ALC882 Analog [ALC882 Analog]
Subdevices: 1/2
Subdevice #0: subdevice #0
Subdevice #1: subdevice #1
card 0: Intel [HDA Intel], device 2: ALC882 Analog [ALC882 Analog]
Subdevices: 2/2
Subdevice #0: subdevice #0
Subdevice #1: subdevice #1

I'm still trying manually...

Code: Select all

arecord -r 22000 -t wav -c 1 -f S16_LE -v -d 86400 `date +%Y%m%d-%H%M`.wav

[[XAP]Bob]

For the sake of completeness, I've got sound-recorder working (I think) as discussed above for arecord.

Code: Select all

cd /directory/on/large/fs
mkdir wav
 while [ 1 ]; do sound-recorder -c 1 -s 22050 -b16 -S 60:00 -f wav wav/`date +%Y%m%d-%H%M`.wav; done

then

Code: Select all

lame -b32 -s 22050 --bitwidth=16 -a wav/<file>.wav mp3/<file>.mp3

[[XAP]Bob]

Hmm - this will be useful for me when I come back to doing this next year - so I'll note it here

As we have an independant news feed (IRN) on the hour for three minutes I'm breaking the recordings in that timeslot - this also corresponds to the usual changeover times for the broadcorpers, so will allow us to easily review shows if needed.
The below scheme gives me a very short break in recording each hour - I can hear it, but it's insignificant.

I've set up a cron job :

Code: Select all

01 * * * * /path/to/music/dir/hourlyrecordings.sh

where /path/to/music/dir/hourlyrecordings.sh is :

Code: Select all

#!/bin/bash
# This script is used to batch record a continuous stream (such as a radio
#  broadcast) into mp3 files. It should be called at appropriate intervals
#  by cron.
# Temporary files and/or long term recording takes up alot of disk space, so a 
#  directory on a large filesystem should be chosen.
#  e.g. a mono 22kHz, 16 bit recording uses ~150MB/hr
#       can be compressed to ~15MB/hour using 32kBit encoding
#  That's an understandable (if not musical) recording at ~10GB per month.

################################################################################
################################################################################
# User edittable defaults:
recorder="sound-recorder"
recorder_options="-c 1 -s 22050 -b16 -S 60:00 -f wav"

encoder="lame"
encoder_options="-b32 -s 22050 --bitwidth=16 -a"

#workingdir=`pwd`
workingdir="/path/to/music/dir/"
wavdir="${workingdir}/wav"
mp3dir="${workingdir}/mp3"
oldwavdir="${workingdir}/oldwav"
################################################################################
################################################################################


################################################################################
function die ( ) { # string
  echo "$0 encountered a problem and is exiting."
  echo "For help see $0 -h or --help."
  echo
  echo >&2 $1
  exit 1
}

################################################################################
function display_help ( ) {
  echo
  echo "This script is used to batch record a continuous stream (such as a"
  echo " radio broadcast) into configurable length mp3 files."
  echo
  echo "Temporary files and/or long term recording takes up alot of disk"
  echo " space, so a directory on a large filesystem should be used."
  echo " e.g. a mono 22kHz, 16 bit recording uses ~150MB/hr"
  echo "      which can be compressed to ~15MB/hour"
  echo " That's an understandable, not musical, recording at ~10GB per month."
  echo 
  echo "Splitting the file may result in a small amount of audio being dropped"
  echo " but the script will then compress the previous file and can therefore"
  echo " free up some disk space."
  echo
  echo " Usage: $0 <options>"
  echo "  -h --help             Display this help."
  echo "  -v --verbose          Be verbose in operation."
  echo "  -r --recorder         Program to use for input capture"
  echo "                         (defaults to sound-recorder)"
  echo "  -R --recorder-options Command line options passed to the recorder"
  echo "  -e --encoder          Program to use for encodingt o mp3"
  echo "                         (defaults to lame)"
  echo "  -E --encoder-options  Command line options passed to the recorder"
  echo "  -k --keep-wavs        Retain the wav files after encoding"
  echo ""
}

################################################################################
################################################################################
# Defaults.
quiet="true"
keep_wav=false

# Parse the arguments
while [ "$1" != "" ] ; do
  arg=$1
  name=`echo $arg | awk -F"=" '{print $1;}'`
  other=`echo $arg | awk -F"=" '{print $2;}'`
  shift
  case "$name" in
    "-h"|"--help")
      display_help
    ;;
    "-v"|"--verbose")
      quiet="echo"
      verbose="--verbose"
    ;;
    "-e"|"--encoder")
      encoder="$1"
      shift
    ;;
    "-E"|"--encoder-options")
      encoder_options="$1"
      shift
    ;;
    "-r"|"--recorder")
      recorder="$1"
      shift
    ;;
    "-R"|"--recorder-options")
      recorder_options="$1"
      shift
    ;;
    "-k"|"--keep-wavs")
      keep_wav=true
    ;;
    *)
      die "ERROR: invalid option [${arg}]"
    ;;
  esac
done

# Sanity check:
cmd_recorder=`which ${recorder}`
if [ -z "${cmd_recorder}" ] || [ ! -x ${cmd_recorder} ] ; then
  die "Could not find the recorder specified [$recorder]."
fi
$quiet "Found recorder [${recorder}] at [${cmd_recorder}]"
cmd_encoder=`which ${encoder}`
if [ -z "${cmd_encoder}" ] || [ ! -x ${cmd_encoder} ] ; then
  die "Could not find the encoder specified [$encoder]."
fi
$quiet "Found encoder [${encoder}] at [${cmd_encoder}]"

for dir in ${wavdir} ${mp3dir} ${oldwavdir} ; do
  if [ ! -d "${dir}" ] ; then
    mkdir -p ${dir}
    $quiet "Created [${dir}] directory."
  fi
done

# Get wav listing for later encoding.
WAVS=`ls ${wavdir}/*.wav`
$quiet "Found existing files to encode: [${WAVS}]"

# Kill existing recordings:
rec=`basename ${cmd_recorder}`
if pkill ${rec} ; then
  $quiet "Killing existing recorders."
  sleep 1
else
  $quiet "No recorders running"
fi

# Start the next:
$quiet "Starting recording:"
${cmd_recorder} ${recorder_options} ${wavdir}/`date +%Y%m%d-%H%M`.wav 2>&1 >/dev/null &

# Encode the wavs that previously existed:
$quiet "Start encoding older files:"
for wav in ${WAVS} ; do
  root=`basename ${wav} .wav`
  $quiet "Encode ${root}"
  ${cmd_encoder} ${encoder_options} ${wav} ${mp3dir}/${root}.mp3
  if $keep_wav ; then
    $quiet "Backup wav"
    mv ${wav} ${oldwavdir}/${root}.wav
  else
    $quiet "Delete wav"
    rm -f ${wav}
  fi
done

$quiet "Done..."

I've done very little testing of the argument parsing above, and it doesn't error nicely if the recording fails for any reason - but it's working for me by default at the moment.

I'll probably add an argument to say "don't bother encoding", just setting $WAVS to blank before the for loop should do the job nicely.

Oh, yes - and thanks to richardash1981 for sending me down the right path - hopefully this thread will be findable by future recorders.

[telocity]

Why not buy a external stand alone solution. You can get digital recorders with extra memory ( I like the olympus ones) that record direct to mp3 along with AC power. There are also more expensive mixing boards that record to harddrive or even ipods. just a idea. I've had problems with recording on older version of audacity for more then 4 hours on ubuntu at church (just upgraded so can't say if new version has same issue.) Just a idea. Please keep us updated on what you did as it is a interesting test for audacity.

[[XAP]Bob]

External standalone solutions tend to be expensive - I already have a tuner and a PC, I even have a cable I can run between the two
And this is a charity - so we're always looking to be efficient in our use of resources.

I also don't know of many external recorders that would be happy for a month at a time - all the ones I've seen look like they're designed for journalists, doing small batches of recording interviews on the road.
As it turns out audacity can do some decent lengths (24 hours wasn't a problem) but it did barf over the 48 hour track (which is *really* annoying).

I've actually dropped audacity in favour of the shell script above - which works with lame and sound-recorder to batch the thing into hourly chunks (I lose about a second or so each hour, but I time that into a news spot).

It would be fairly easy to have it running in a tight loop, so that you could manually break the recordings when silence/applause/white noise occured in any other situation.

I've now got 191 seperate "hours", 2.6G of actually suprisingly good quality recording.

In our situation this also has the advantage that we can look back and pick out individual shows with ease (if we need to look at a particular kids shows then they can be pulled by just choosing the appropriate hours).

[richardash1981]

I would agree that most hardware solutions are not engineered to cope - I've got a Korg D1200, but I wouldn't trust it to do long runs, not least because a power glitch looses everything you have recorded (not even temp files left).

[[XAP]Bob]

I would agree that most hardware solutions are not engineered to cope - I've got a Korg D1200, but I wouldn't trust it to do long runs, not least because a power glitch looses everything you have recorded (not even temp files left).

Ouch - not having temp files is one of the reasons a nice slow reel to reel system might be nice...

I'm just hoping I don't get a power cut!