Can you troubleshoot Missing Audio Data Block?

Jack/Dave:

I have looked into your project and have found that your .aup file seems to be referencing a mostly totally different project than that of your _data directory. None of the .au files referenced are present. It appears that your kingdom may be there but it is highly scrambled, it may not be complete, and the keys to it are lost and gone forever. Here we need to call in the royal locksmith. I can tell this because the zipped file is not negligible, and I can hear tiny tiny segments of both an interviewer and an interviewee.

I have a busy day today and tomorrow, so I’ll call in the “locksmith” early next week. BTW, I’ve heard rumors, but I don’t know anyone who has admitted they were entirely successful at cracking the combination; and if anyone has actually done this, please let me know, I’ll be happy to get you involved!. In the meantime, although it is highly doubtful, I’m wondering if perchance the keys were buried in one of the other .aup files that you had open at the same time. Although this is highly doubtful, it may be worth the time to explore. So if you wish, go ahead and zip up those other files up and send them along.

Note: this will not be a simple job, and complete success is not probable, but for the moment at least, we can remain optimistic.

we can remain optimistic.

I’m not. There is a history of being able to force a clean, original recording back to life, but nobody has ever rescued an edit, much less one with multiple parts.

The person wearing The Producer’s Hat should start planning a replacement show. It might be possible to restore the original show concept if you have all the original interviews, announcements, and voice-overs as backup WAV files.

Koz

jademan, thanks so much for offering to help. I’ve tried to attach the files you asked for – let me know if you were asking for different files.

Let me know if you (or the “locksmith”)(whoever that is!) needs anything else.

With much appreciation,
Jack/Dave
Ethan Herschenfeld intv.aup (103 KB)
Amanda Docter intv.aup (169 KB)
Glenn Slater intv.aup (302 KB)

Will do…

Most of jademan’s ju-ju is in the Audacity Version 3 family and I’m only going to tell you to start planning a replacement show.

The legacy Audacity versions saved a project in a pretty simple way. The _DATA folder contained the actual sound in little 6-second snippets called .au’s. The AUP project manager text file told Audacity where to put the snippets to reconstruct the show.

There were digital deities who, if asked nice, could pull together a simple, clean sound recording from the _DATA folder only—without the AUP file. We kneel at their feet. Once you start editing, file relationships scramble. There is no known way to pull an edit back together, either without the AUP file or after a crash.

So if I wasn’t annoying enough before, all your work should be exported as WAV or other high quality sound files (not MP3) before you start editing. Do Not depend on Audacity to always behave itself, even in the updated version 3.

Koz

koz, Thanks for your comments. As far as I know, they are all true.

koz, I appreciate your anecdotes. I knew from the start that this was a difficult to impossible job. But I told Jack/Dave that I would take a look to see what could be done and I am doing so.

Jack/Dave,

While I am continuing to make progress on your project on different fronts, so far I do not have anything illustrative to present; please know that what koz has been saying is based on experience, and he has far more than I. One of the clues used by the above digital deities was time-stamps - unfortunately, the ones in your project are mostly useless, perhaps due to some global edit, so we have extra difficulties. And, as substantial later edits did take place in your project, this situation is something that we are going to have to deal with. Perhaps in a painful way.

And while I have never rescued an .aup project before, and the outlook is far from positive for this one, I have been successful at rescuing many .aup3 projects, so I am keeping a positive outlook. So the more koz is telling us things that cannot be done and why, he is actually giving me inspiration and clues to novel approaches on how it could be done. :wink:

What’s the deadline for this show?

Was there a show before this? What is its address? Where is it published? It might be handy to know your editing rhythm and technique since we may be trying to reproduce it.

Is there going to be a show after this?

Koz

I did do an April First Design Concept. Actually two, to rescue a clean recording.

We know that in a stereo show, the little .au files alternate Left - Right. Design a software package that tries each au file next to all the others and pick the pair that’s least discontinuous. Go on to the next one. Keep rippling through all of them until 1) you find all the Left - Right pairs, and 2) you find .au files most musically comfortable with their neighbors.

Like most April First designs, it could actually work since the switchover between two adjacent files is by definition perfect. No clicks, pops or holes. So there should only be one match for each file in the whole pile. You may need to leave the machine cranking away on this for a day or two. Make sure of good ventilation.

You might well say, “what happens in silence.” Then the match may not make any difference. Which would you rather have? A show that needs some silence and room tone patching, or needing to call the client because there is no show.

A variation on that is use the file Time and Date stamps. Nice call, but that doesn’t get you close enough. Make baskets of Time and Date compatible files and then go for the musical match. We note you could do this last one manually if you have nothing better to do for the next six months or so.

None of this works on an edited show. There is no fixed, predictable relationship between any snippet and any other.

Koz

koz,

See, you’ve got your thinking cap on… I knew if I asked nicely, you would design this for me. All I have to do now is implement it. :wink:

Jack/Dave,

OK, one of the ideas we were exploring was the possibility that for some reason, the file names in your project got scrambled. I took at look at all file names mentioned by the four aup files you supplied and in the mack_data directory to see if there were duplicates. In reviewing the 7,185 file names, there were 278 duplicate file name names (which duplicates were mostly of a random nature). And I confirmed that none of the files named in the mack aup file were actually present in the mack_data directory. This confirms to me two things we already suspected (or knew): (1) The project files did not seem to be “crossed”, and (2) the file names are assigned completely randomly.

Speaking of random: I did some research on the (now obsolete naming convention used in 2.4.2 and confirmed some things we mostly already knew. The file name consists of eXXYYZZZ (not ZZZZ) held under XX then YY subdirectories. ZZZ is the random number, YY get incremented by 1 after every 256 random files get generated. Ditto for XX. The idea is to keep fewer than 256 files in a directory as certain OS’ had directory performance issues. The “ZZZ” I believe was partly to keep performance relatively consistent by randomizing directory position (generally file “000” could be retrieved faster than “fff”). Of course, today’s file systems have better performance and for Audacity 3.x.x, SQLite3 changes everything so we no longer need be concerned with the directory structure.

One issue that puzzles me is that your AUP file named only 1211 block files, yet the _data directory contains 2375 block files (about double). Also, about the half of these latter looked to be all zeros but when I amplfied them by 45-50X I found (noisy) conversational audio.

Anyway, I am continuing my efforts… remember, koz has told us about the baskets and the musical matching. :wink:

This may prove to be an issue. There were certainly many (not excessive) envelope control points that we’ll probably need to skip.

Jack/Dave,

There is a wonderful new feature being added in a future release of Audacity, hopefully, 3.1.0. The feature is being developed by our musescore friends and allows clips to be easily moved all around on the screen without having to first cut and paste them. This feature is currently under Alpha testing. Linked with this new feature is a clip naming feature by Vitaly Verchinsky available for testing a week ago that allows each separate clip to be assigned a name. Steve and Peter commented on this feature earlier today. I have been developing some routines based on the combined version of this software to allow some new clip rearrangement facilities. My routines allows me/you to select an arbitrary set of (named) clips and programmatically place them anywhere I want. As of last night I have this part working just fine.

I am hoping to combine this with colored clips and koz’ basket sort idea to provide a solution to our current dilemma. Watch this space.

Hey, I don’t know how to say “Thank you” enough. I so much appreciate all the effort you’re going through for me.

No one ever said it was going to be easy. :wink: I’m just crossing my fingers that we can come up with something that you can use. :wink:

And while we’re crossing our fingers, the person wearing the Producer hat should start building the shows over again. Or at very least set a time limit to stop waiting. Rescuing an edit has a perfect track record. Zero, and it’s never a good sign when a computer goes down.

This was a different complaint, but the solutions are the same. Make sure the computer is in good health. Clean Shutdown twice after you find out why it crashed. Once to clean up the Windows crash damage and then again to make sure of a fresh start.

https://forum.audacityteam.org/t/acx-check-problems-in-v2-4-2-win10/62578/2

Koz

Well I’m happy to report that I have finished development on this project. I must admit that the results have fallen short of my expectations. koz was predicting much of this.

But more importantly we have been able to recover all of the data from those .au files that you sent me. And I have provided some tools to help you wade through the “mess”. More on this tomorrow.

Rather than send all of your data back to you, I am simply sending you the tools to help you recover the data yourself. I am sending eight (8) .aup files, and an custom experimental untested version of Audacity 3.1.0. I have PM’d the particulars to you.

Because of the size of your project, I have divided it into eight (8) smaller projects. They are in order by the date the clip was modified. Each of the .aup files contains instructions to load about 1/8th of your existing data (from the .au files in your _data directory).

First backup the one and only K M intv.aup file you have. You can backup your _data directory if you desire but I believe it is already backed up and I don’t expect we will be writing to it.

Unzip the experimental Audacity into its own directory (you can keep it in downloads if you prefer). Do NOT run this program from the zipped file; instead, run it from an unzipped directory so the associated files are also exposed. Note that it is always safest to run a virus checker on any programs you download.

Then successively rename K M intv(1-8).aup to K M intv.aup to agree with your exiting K M intv_data directory. Do NOT double-click on these files - if you do, you will simply bring up the old version of Audacity and possibly cause further damage.

Load each file one at a time with my test version of Audacity and load the project. You will see what appears to be a mostly black screen. This is to be expected. Then immediately save it as, say, K M intv(1-8).aup3. Exit Audacity, then repeat this for each of the eight files.

Do not edit any of the projects. Just save them to the four unique names. I’ll post more instructions tomorrow as it is getting late today. :wink:

Also, please keep me up to date on your progress with this…

My wife’s Windows 10 PC just crashed

Do you know why the machine crashed? I know Windows used to have a reputation for poor stability, but it really is hard to make a Windows 10 machine fall over. What steps are you taking to make sure after all this fuss it doesn’t happen again?

Koz

So I had a little booboo. It is not my first and I am sure it is not my last. I’ve posted another version of my Audacity Recovery program, and I’ve sent you a link. The booboo won’t affect anything you’ve already done, so just change it out now and you’ll be fine.

OK, so your _data directory had about 2400 individual sound clips of 6 seconds or less. The file names are all “randomized”, the .aup file was not helpful to me, and the only information available to help us is the file’s last modification time. (The creation date and access date are also not helpful).

Because of the size of your project, I have split the file up into eight (8) parts, part 1 being the oldest and part 8 being the newest. So each project contains, on average, 300 files. Every one of the 211 clips in part 1 was created on August 11th at 6:21:48 PM. Every one of the 379 clips in part 2 was created on August 11th at 6:21:50 PM. So these first two projects are going to prove to be difficult to work with. So let’s start with the last one, part 8 for illustration.

Open the special Recovery version of Audacity; then under recent files, open your project 8. You will see a single (mono) track, and a lot of black lines over toward the right. When you loaded the clips into the project earlier, each clip went to an assigned space for it, each space about seven seconds apart. So the clips in project 8 start at about the 4 hour mark. On the Tool Transport Bar, skip to the end of the project, then zoom in about 8 times. At the top of each clip, you will find the date and time the clip was last modified, in this case 8/23|12:58:58 in military time. The last four (4) clips all have the same time and color (blue). The first two of these clips are extremely close to zero. I’m not sure what these are - it appears to be some kind of echo, yet at about 40-50dB lower volume. Perhaps the recording was originally in stereo.
First Track005.png
.
We’ll skip these two clips for now. So drag the cursor over these last two clips then select Edit > Clip Boundaries > Sort Clips by Content. (This menu item appears only in my experimental Audacity.)
Tracks009.png
When you play these two clips together, you can tell that they go together. In this case, the developmental sort routine put the tracks in the correct order. We are not always so lucky.



The previous three clipss are black, red, and green so we know they were edited at different times. (Check the date/time stamps at the tops of these clips):
Tracks010.png
.
. Perhaps they are related. We’ll just pull them down manually: click on the top bar of the clip, then drag it down and slide it over:
Tracks011.png
Now play the clips. What does it sound like to you? (To me one of the clips sounds like an edited clip of the former). So there is going to be a little manual work and judgement required here…

The sorting routine does not work as nicely as I had hoped. But I wanted to avoid further delays in this project by introducing scope creep. I’ll be taking a closer look at it during the next few days to see if I can come up with any refinements. If you have any specific suggestions in this area, I would be happy to take a look at them.

We’ll just pull them down manually: click on the top bar of the clip, then drag it down and slide it over:

Correct me but this is one of the tricks plain Audacity can’t do. That works a lot more like a video editor. You want a clip over there, click on it and pull it over.

Now all you need is Mark-In and Mark-Out and you can double the editor’s production speed. If you’ve never met those before, they’re deceptive. You only get two at a time. Mark-In and Mark-Out. They’re not labels.

I don’t remember the keystrokes, something like Control-I and Control-O. Start the clip and scrub when you get close to the IN point. Scrub is built-in and it’s always active. It’s not something you need to specially select and configure. Another time saver.

Control-I.

Play or scrub forward to the OUT point and Control-O. Delete or Copy, or whatever else you want to do with that marked portion. They’re sticky. They will not move by themselves and you only get two at a time, so there is no confusion as to which one is which.

If you need to move one of the marks slightly, grab it and pull. No menu options or hidden controls. There is a way to simply remove the marks, but I don’t remember the keystrokes. I had a stunned moment when I found that I could move the Out Point before the In Point. The application just didn’t give a flying care.

I remember vividly pulling together a fifteen minute video presentation at break-neck speed with just those tools.

Video editors work on frame boundaries, so there will always be a time when you need to do precision sound editing in Something Else. Audacity is the Something Else.

Then we explain to the editors that Audacity doesn’t have a lot of those handy, rapid tools they like…

Koz

Missed one. The editors I used have Get INFO built-in. No moving a clip to an external application to figure out what you have. I know that’s hard in Audacity because of the special internal format, but still.

Koz

Which is why we are using a special EXPERIMENTAL version of Audacity. I am hoping that most of these features will be coming out in 3.1.0. At least the clip moving parts, that is. :wink: The other stuff, perhaps I will make available after some more refinements, including a possible automatic .au file puller - but I am getting ahead of myself…

Dear jademan,

Thank you thank you thank you for all your amazing work. I have never had anyone on any discussion forum spend as much time trying to help me as you have. (And I have posted problems on dozens of discussion forums during my loooooong life.)

I will take it from here. I’ll go through all your work and see if I can piece together the interview.

My wife is just starting to re-edit the interview, but she knows she did some great work on the last one (that she lost) and so piecing together what you have done will help her.

Again, let me thank you from the bottom of my heart.