100% orphaned blocks? I would not call this program reliable

Audacity Help Forum Windows
1

I left my computer recording overnight and have a 25h recording, after stopping the recording, I considered amplifing but decieded to do it later (clicked cancel before starting), I attempted silence removal but stopped that because I couldn’t wait 1.5h (cancelling did not cause a hang), It only hung when I pressed “fit project” zoom, I decided to shut down the computer so I saved and exited. The next day I loaded the save file and it says that 30,000 block files were orphaned.

For some reason “delete” was there as an option, and makes it tempting to press by assuring the user that a block contains only a little bit of information – it was still able to say that even when 100% of the blocks were not loaded (It does not show basic statistics: how many blocks the project uses, how many were loaded or how many that were not found – only how many orphans). The log window was updated so fast listing the orphans, it crashed my graphics card (not that hard to do on this machine, a standby cycle fixes it).

I assume it means it never recorded block order, existance etc, the block’s date stamp appears to give order information. Apparently Audacity 2.02 is not designed to handle this kind of scenario – it does not have an option to “load orphans sequences in a new window” (or in current window as a concurrent soundtrack) as one of the options, using any of the “manual recovery” processes automated. Or let us load audacity data in the file menu (“raw data” does not seem to be it), cannot drag the folder in either. Why is audacity v1.2 “no longer supported”, yet the methods used for data recovery (as old as 1.2) still are good enough? Downloading a third party program for a simple rename task is very dodgy, especially when you can do the timestamp query and sorting in-code.

If you consider the mentioned problems of doing a recovery, solutions of varying value can be applied in audacity to make the process much easier:

Make it easier to fix stereo sound:
*With 3D video, the second image is encoded in the second half of the video. you can do the same with the AU files.
*If wanting to deal with date stamps only, the first piece of the left channel and last of the right channel can be recorded 1.25s shorter, so that the files are written to the drive one at a time rather than in pairs.
*(removed, I suggest two replacements later)

Fixing block order:
*perhaps: rename the files at each save, or: modify the modify/access datestamp to show the file’s order or position in the recording. (the modify/access datestamp can show the next/last save, this will assist in recovery if there is a power outage during the save)
*you have plug ins – use them! Allow plugins for sorting the data. Perhaps study the waveform of the start/end of each block (dc level, fourier transform, etc) and then match the frequency profiles and/or the phase of a few frequencies between the files - not unlike a jigsaw puzzle. The only time a mistake could happen is if the L/R blocks were recorded to disk together and the recorded material switched between mono and stereo sources.
*test-load the save file. Do a quick pass to see if the file is reloadable, the sound data is somewhat identical, or if all the blocks that were required were reloaded. If it fails (as in my case), ask if user wants to try saving as parts (if it was the file length that broke it, allow loading of parts), test each against original.
*As for more modified projects, (since save is obviously untrustworthy) perhaps do not modify the original file, instead record milestones, load the last milestone when user loads. If the milestone is corrupted, ask if the last milestone is sufficient (it can be a bit holey if some of the unneccessary blocks were removed, but that can make re-editing easier) and mention orphans that were added in the broken milestone. Half a pie is better than none. Older milestones could still be useful, allow deleting of them too (but milestone data is small compared to the sound data).

My recording is not exactly important - if audacity fails for my less important recordings, why should I trust it for the more important stuff? Even if I was able to get my recording loaded somehow, I am reluctant to save it again because obviously it could fail again. Since I cannot use audacity for mission critical stuff (because this orphan issue is not taken seriously), what programs could be recommended?

CB.

2

Audacity 2.0.2 (if that is what you mean) is many years obsolete and has the issue you describe if you record over 27 hours at 22050 Hz, 13.5 hours at 44100 Hz, 6.75 hours at 88200 Hz and pro rata according to the sample rate) then save and reopen a project.

If you must use 2.0.2 for recordings over those lengths I would advise you to export to separate WAV files or FLAC files while the project is still open, with a maximum file size of 2 GB each. FLAC files will give you more time for the 2 GB, around 6 hours or more for stereo 16-bit 44100 Hz.

We fixed the bug back in Audacity 2.0.6 and the current version is 2.1.2. I suggest you obtain 2.1.2 from us at http://www.audacityteam.org/download/windows then the issue won’t happen again. There is a also a good chance 2.1.2 will be able to reopen the project correctly, if the only problem is the orphan files.

1.2 is no longer supported because it used a different project format. You can still open projects saved by 1.2 versions of Audacity in Audacity 2.x.x versions. Audacity will often open them without problem, but in case it does not, make a copy of the original AUP file and _data folder into a new folder first. Be sure not to copy into the same folder.


Gale

3

The I think it was at 2.02 for a while… I last installed it in Feb 2014. I tried opening it with 2.12, no go.

If you can suggest a fix for the aup file then maybe it will, but there appears to be a lack of content in it - between:




Also I cannot edit my post, i realise there is a problem with the last stereo idea, so here are better ones:
*record the second channel with a different architecture/language/CAPitals or increment the count backwards (if there is an index in it), or scroll it vertically by 50% so it oscillates to either side of the top/bottom.
*different main folder for each channel

4

I just discovered an unusual way to recover the data… The filenames are always created in the same order (except the last one), to make a working save file for it, I only have to record another of the same length or greater (may need more disk space).
The save file has “min”, “max” and “rms” for each item. V2.02 always had the same min/max for each item, but rms is different. V2.12 has different stuff for all three. Can you tell us what the purpose of each of these is, and what happens if I were to put random stuff in there? With V2.02 only one thing can go wrong, but much more could have had I recorded with V2.12.

CB.

5

None of these (“2.02”, “2.12”) are made by us. If that is really what Help > About Audacity… says, you need to redownload from us at http://www.audacityteam.org/download/windows. And please run a deep anti-virus scan immediately.

2.1.2 supplied by us will not have the issue that it can’t reopen very long projects, so it won’t happen again.


Gale

6

You should now be able to do so now, as long as your posts remain acceptable to the moderators.

Gale

7

The “min” “max” and “rms” are amplitude levels.

The recovery instructions if you have no usable AUP file and a completely unedited recording are at http://manual.audacityteam.org/man/recovering_crashes_manually.html. That method works. You create a new set of WAV files from the data.

I do not recommend other methods, unless you have python and want to use a python script to recreate the AUP file from the data.


Gale

8

If you can avoid it, it’s better not to record such long stretches with any DAW.

There are specialized long time recorders for a reason: no DAW has been thoroughly tested with such long recordings, because it isn’t typical use.

When I need to do real long recordings, there is only one recorder that is reliable: Boom Recorder. But I’m on a Mac. BR is rather primitive (it can’t even open a recording made with it), but it has all kinds of nifty tricks you really, really need when doing long recordings, like SMPTE time stamping and a setting to break up the long recording into seamless manageable pieces. It can also record up to 128 channels, even on fairly mediocre hardware. And it stops when the disk is full…

9

Audacity doesn’t stop when the disk is full but carries on, corrupting the existing recording as it goes. It is a high priority for us to fix that.

The Status Bar at the bottom of Audacity says how much recording time you have remaining.


Gale

10

Making hugely long recordings in any program is an adventure.

Even more adventurous on an unstable machine.

Audacity can make very long recordings (reliably), but there are a number of things that need to be considered.
If you were intending to make a 10,000 km car journey, I expect that you would ensure that the car is in tip-top condition, fully serviced, running perfectly, full tank of gas, oil checked, coolant checked, tyre pressure …
For making huge recordings on a computer, you need to do the same. Ensure that the hardware is running perfectly, plenty of free disk space, “sleep” and “hibernate” turned off, air filters clear and reliable cooling, reliable power (preferably UPS), fully updated drivers, automatic updates turned off…

You also need to consider what you will do with the recording when complete. The maximum size for a WAV file is 4 GB, the capacity of a DVD is about 8 GB, the uncompressed size of a 25 hour stereo 32-bit float recording is about 30 GB.

You also need to consider that doing anything with 30 GB of data is likely to be slooow… Even something simple like zooming or amplifying requires enormously long calculations (25 hours of stereo audio at 44.1 kHz contains around 8,000,000,000 samples).

11

Im moving the 30GB recording to another drive so I have the free space to re-enact the original recording (blank recordings will NTFS compress better, it might allow >2x). This method only works if you choose the right bit rate, and assumes v2.0.2 names the files beyond 13.5h the same as v2.1.2 . I really would like to know how the names get chosen so I do not need to do this (one alternative is to download someone else’s 44100Hz AUP file - a download intended for everyone will need breaks every 256 entries, or audacity can generate a new one for the user). I have tested it with a 40minute recording and it works, so perhaps you can include this method in your manual as a recovery possibility, the only uncertainty is which way the the last piece of each channel goes since the last piece is named different.

Yes, 2.12 = 2.1.2. I still have not gotten used to extended version designations, I have seen too many old programs! (even this forum refers to audacity as being 2.x rather than 2.x.x) “This forum is for Audacity 2.x on Windows.”

Does it impact the product in any way??? Min and max do not seem to be used in 2.0.2 so if it is it will be less for me.
As for those recovery options, stereo is not perfect, I have sorted by date in XP-explorer and the order is messed up (compared to my method). I think that it is a little too hopeful to assume that the dates were correct anyway.

*UPS = laptop :stuck_out_tongue:
*I was recording internet radio. I actually am planning to go on a trip soon, needed something to listen to (new years is a good time to record since they make sure the music is better). I was not intending to make a 25h recording, but I stayed too long at a new years day party.
*Zooming should not need THAT much processing power, my screen is only 1920pix wide (minus whatever it is for the audacity UI) - doesn’t it only need to average a few data spans for each pixel? Spans can be every 100th recorded magnitude (sized to appropriately catch all freqs)… Seeing 25h, you really cannot see much detail anyway so it really can be skimpy - Perhaps to the point of having only a representative magnitude for each piece, remember, I had >30,000 pieces, 30k calcs are easy for a computer (is that what the “rms” or “max” is for in the AUP file?). Alternatively, could use F5 to update the quality if the existing image is no good (Stretch existing drawing if the user pressed once or twice. If more, start redrawing after the user stops pressing zoom and stops scrolling). Perhaps have separate Q+ Q- and updateQ buttons to increase, decrease and zoom-optimise the quality, so its updated separate to the zoom, or allow quality-creep (slowly updates the quality for the zoom as the user works, with a preference for the places between existing details that show a dramatic change, so there is no sustained hanging - only a lag).

It would help if there was a plugin that searches for long (0.5s) silences at around a certain distance apart (ie 1-2h) and cuts it up (or I can leave it as one big rec), but I guess this is half the “fun” of using audacity.

12

I did come across that while trying to find a way to avoid moving the first recording off the drive, it doesn’t save if there is a block missing! (i deleted some as it was recording). It only creates some folders. The solution for that seems kind of simple… let it save anyway and let the user deal with the missing blocks when the project is loaded. It is possible the user only needs to measure the time between two sounds/events that are way too far apart, and he can still save the timing even if the drive is too small, and be able to read the parts he moved off the drive (still aligned to the appropriate time) after changing disks and reloading.

13

Do you mean sample rate, or strictly bit rate (which for PCM is sample rate * bit depth * number of channels)?

The file names are chosen randomly. If you don’t edit the recording, and you have access to sufficient timestamp granularity (which you do with NTFS file system in the recommended recovery method), the randomness does not matter because the file timestamp order matches with the timeline order in Audacity.

I have no intention of doing that without thoroughly testing it, though I have heard of people attempting it. We have two known working methods including the python script I mentioned, so it is not a priority to add a method which is likely to confuse most users or be error prone, given they are directly manipulating the AUP file.

How for example are you changing the file names of the AU files in the newly recorded AUP file (the one to be used for recovery) to the names of the AU files in your original data?

I agree with you about the “2.x” should be “2.x.x”. I’ll change it when I get time.

That is for you to test if you want to use this recovery method instead of the officially recommended one. I think there could be subtle problems at least. I think the “rms” in the AUP file is only used in the “Contrast” effect.

You mean the xplorer2 Pro routine in http://manual.audacityteam.org/man/recovering_crashes_manually.html#Automatic_recovery_tools? I have not tested it many times but it worked perfectly for me. It does rely on you having an NTFS file system, not FAT32.

I don’t think you have said what version of Windows you are using.

The amount of data to be gone through makes a difference.


Gale

14

Audacity needs to know the peak and RMS level for each pixel. A cache system is used to speed this up, which includes, among other things, the Min, Max and RMS values that are written to the XML data. In order to populate and update the caches, the audio samples have to be analyzed, so at some point Audacity has to go through all 8 billion samples, recording the minimum and maximum per block and calculating the RMS per block. Once the caches are populated, drawing the waveform should be fairly quick.

15

I think I changed the “project rate”at the bottom before I began recording (so the answer is “whatever that box changes”). I have only changed 44100 to a lower figure in an attempt to reduce the disk space required for the re-enactment, only to find the file names were different, so I had to keep it at 44100.


The “random” file names are the same every time a new project is recorded, so I do not need to rename anything (providing the “project rate” is the same). Since the last message I posted, I have noticed I can overwrite the d** folders in the temporary folder with the data I desire to recover even while I am recording, and when I zoom out I can see the recovered data appearing at the start (this was another idea I had to save disk space - the 16x d0* folders cover 3.5hrs, so I can do a replace/move that often to keep it simple). If it suddenly did become random, It would be easy to notice because it should become gappy (I hope I am not breaking it by doing the replacement). if I continue, I will have to not lose any piece I remove - else it would forbid saving (and I would have to force-close audacity to do it’s recovery, etc). I still will have to make sure it doesn’t modify any of the pieces that have already been recorded, that’s all.
First 10 for ch0: e0000005 e0000317 e000095c e000059a e0000d2a e00002c9 e0000b5e e00004dd e0000176 e000025b
First 10 for ch1: e0000904 e0000cf0 e00007ad e0000e55 e0000bf2 e0000dbe e0000837 e000003d e00005d4 e00002a7 @ 44100hz

I thought you would know the purpose of these. It would be good to have tools to subtract/add/etc the waveform so I can exactly see (Cannot really check if there are any at this moment because I am recording “nothing” right now and I am not allowed to have another window). I once tried to have two cancelling waves but I could still hear it when I played it back, so listening to it inverted to hear a difference is not going to work well anyway. So what I will do later is produce some AUP files that have really messed up values and see how/if it breaks anything.

Erm, by “Xp-explorer” i meant “windows XP’s explorer”, if ms is recorded in the date stamp, windows explorer is certainly not reading it to perform the sort.

The peak I understand, it’s the scale. RMS, I do not – take your mains voltage for example, that number is the RMS value – the average. 120V, 240V, 415V (three phases of 240V, 120° apart), 480V (split phase, 240V each). 240VDC is the same power as 240V AC power except AC can rise to 339.4V, the only possible use I can see is perhaps for non-parity error checking. Speculation: I do not see how knowing the average will assist in obtaining a value of a specific level - except if you are using RMS to scale the data as well, it will work but I do not think it is unnecessary, it would change the quality of the scaling depending on what is in the recording (especially DC bias), rather than only its peak volume, and a peak can go above the average by many times (But since it is working as is I shouldn’t be complaining. If it is that way it would hinder accuracy further if I happen to lose the “key”). It is possible the RMS is only used for the amplitude display, that is what the second amplitude is for in the editing area is (one is darker blue, other is lighter blue), so maybe something to do with that.

As for the status of the re-enactment, well, after 5 hours, I accidentally clicked [X] on the corner of the window (because I was clearing the screen) and that was sufficient to cause it to stop recording without my permission (similar to it automatically selecting “no” to “do you want to save your changes”). Then after an additional 3h, i was reminded that I forgot to make sure the battery was in the computer when the power went off (yes, I think it is funny too, since it was mentioned earlier!). I keep it out because I know every time i charge it it shortens its lifespan (and in storage it degrades more if it is holding more charge or there is more heat in the storage location) – since I cannot switch off the charger in the laptop (at maybe 30% full), I take the battery out. And I have restarted the recording again this next morning, where I discovered what happens when I replace the temp recording files.

16

Sorry, but that is simply untrue.

Alright so presumably you don’t care about saving this project, which on reopening would have missing and orphan block files.

We already make it clear in http://manual.audacityteam.org/man/recovering_crashes_manually.html#Automatic_recovery_tools that Windows Explorer cannot sort accurately enough to recover a stereo recording with correct left/right channel allocation. xplorer2 Pro can do so, on an NTFS file system.

Again, the link above contains the official instructions (or you can use the python script I mentioned).

Steve knows more about this than I do, but I think RMS, min and max are also stored in the AU files themselves. Whether the AU or XML data gets used depends on the zoom level. He can clarify if he wants to.


Gale

17

Not unless you are using an ancient version of Audacity.

The current blockfile/directory scheme uses two levels of subdirectories - up to 256 ‘eXX’ and up to 256 ‘dYY’ directories within each of the ‘eXX’ dirs, where XX and YY are hex chars. In each of the dXX directories there are up to 256 audio files (e.g. .au or .auf). They have a filename scheme of ‘eXXYYZZZZ’, where XX and YY refers to the subdirectories. The ‘ZZZZ’ component is generated randomly. The XX and YY components are sequential. DirManager fills up the current dYY subdir until 256 are created, and moves on to the next one.

18



I am writing to tell everyone what I did to recover my data. If it is supposed to be random, you can help find what was needed to break it so other’s data can be recovered quickly (audacity would be able to do it by itself, fast), or find some circumstance that is needed for this so that a few people can recover their data more easily. I do not see any GOOD reason why you need to randomise it (unless hindering data recovery is a ‘good’ thing? or preventing overwriting of the data?), audacity only needs the filenames to be unique - they can be jumbled up or deleted later as the user edits it. Since this is obviously good enough for me, it can be good enough for everyone. If some people want the names scrambled, let it be an option in the options menu or /command line.

I did it this way because I backed it up onto my USB drive and didn’t want to wait for it to copy it back. All files moved displaced the existing files. I saw no errors - since the 2 expected orphans that were present were ignored after I pressed save, the temporary folder containing them didn’t get deleted.

As for RMS, MIN and MAX it has no effect on the output (I tested it on the first file on one channel). I expected you guys to be able to tell me their purpose easily since you recently changed the way the AUP file is saved (between 2.0.2 and 2.1.2) so the MIN and MAX is recorded differently. Obviously none of you had a hand in that. If it has no purpose you can still use them to help with zooming as I wrote earlier (or remove them).

This is the last message I needed to write. Don’t expect further replies.

19

We did not change that, or we don’t understand what you mean.

2.0.2 first block of generated tone

<waveblock start="0">
<simpleblockfile filename="e0000bfe.au" len="262144" min="-0.8" max="0.8" rms="0.565686"/>
</waveblock>

2.1.3-alpha first block of generated tone

<waveblock start="0">
<simpleblockfile filename="e00002cd.au" len="262144" min="-0.8" max="0.8" rms="0.565686"/>
</waveblock>



I agree. We’re just talking at cross-purposes.

The moral of the story is clear. Download Audacity only from http://www.audacityteam.org/download/ and keep it updated so you have the latest bug fixes. :wink:


Gale