Can you troubleshoot Missing Audio Data Block?

[jademan]

koz,

See, you’ve got your thinking cap on… I knew if I asked nicely, you would design this for me. All I have to do now is implement it.

Jack/Dave,

OK, one of the ideas we were exploring was the possibility that for some reason, the file names in your project got scrambled. I took at look at all file names mentioned by the four aup files you supplied and in the mack_data directory to see if there were duplicates. In reviewing the 7,185 file names, there were 278 duplicate file name names (which duplicates were mostly of a random nature). And I confirmed that none of the files named in the mack aup file were actually present in the mack_data directory. This confirms to me two things we already suspected (or knew): (1) The project files did not seem to be “crossed”, and (2) the file names are assigned completely randomly.

Speaking of random: I did some research on the (now obsolete naming convention used in 2.4.2 and confirmed some things we mostly already knew. The file name consists of eXXYYZZZ (not ZZZZ) held under XX then YY subdirectories. ZZZ is the random number, YY get incremented by 1 after every 256 random files get generated. Ditto for XX. The idea is to keep fewer than 256 files in a directory as certain OS’ had directory performance issues. The “ZZZ” I believe was partly to keep performance relatively consistent by randomizing directory position (generally file “000” could be retrieved faster than “fff”). Of course, today’s file systems have better performance and for Audacity 3.x.x, SQLite3 changes everything so we no longer need be concerned with the directory structure.

One issue that puzzles me is that your AUP file named only 1211 block files, yet the _data directory contains 2375 block files (about double). Also, about the half of these latter looked to be all zeros but when I amplfied them by 45-50X I found (noisy) conversational audio.

Anyway, I am continuing my efforts… remember, koz has told us about the baskets and the musical matching.

None of this works on an edited show. There is no fixed, predictable relationship between any snippet and any other.

This may prove to be an issue. There were certainly many (not excessive) envelope control points that we’ll probably need to skip.

[jademan]

Jack/Dave,

There is a wonderful new feature being added in a future release of Audacity, hopefully, 3.1.0. The feature is being developed by our musescore friends and allows clips to be easily moved all around on the screen without having to first cut and paste them. This feature is currently under Alpha testing. Linked with this new feature is a clip naming feature by Vitaly Verchinsky available for testing a week ago that allows each separate clip to be assigned a name. Steve and Peter commented on this feature earlier today. I have been developing some routines based on the combined version of this software to allow some new clip rearrangement facilities. My routines allows me/you to select an arbitrary set of (named) clips and programmatically place them anywhere I want. As of last night I have this part working just fine.

I am hoping to combine this with colored clips and koz’ basket sort idea to provide a solution to our current dilemma. Watch this space.

[JackBauer4Prez]

Hey, I don’t know how to say “Thank you” enough. I so much appreciate all the effort you’re going through for me.

[jademan]

No one ever said it was going to be easy. I’m just crossing my fingers that we can come up with something that you can use.

[kozikowski]

And while we’re crossing our fingers, the person wearing the Producer hat should start building the shows over again. Or at very least set a time limit to stop waiting. Rescuing an edit has a perfect track record. Zero, and it’s never a good sign when a computer goes down.

This was a different complaint, but the solutions are the same. Make sure the computer is in good health. Clean Shutdown twice after you find out why it crashed. Once to clean up the Windows crash damage and then again to make sure of a fresh start.

https://forum.audacityteam.org/t/acx-check-problems-in-v2-4-2-win10/62578/2

Koz

[jademan]

Well I’m happy to report that I have finished development on this project. I must admit that the results have fallen short of my expectations. koz was predicting much of this.

But more importantly we have been able to recover all of the data from those .au files that you sent me. And I have provided some tools to help you wade through the “mess”. More on this tomorrow.

Rather than send all of your data back to you, I am simply sending you the tools to help you recover the data yourself. I am sending eight (8) .aup files, and an custom experimental untested version of Audacity 3.1.0. I have PM’d the particulars to you.

Because of the size of your project, I have divided it into eight (8) smaller projects. They are in order by the date the clip was modified. Each of the .aup files contains instructions to load about 1/8th of your existing data (from the .au files in your _data directory).

First backup the one and only K M intv.aup file you have. You can backup your _data directory if you desire but I believe it is already backed up and I don’t expect we will be writing to it.

Unzip the experimental Audacity into its own directory (you can keep it in downloads if you prefer). Do NOT run this program from the zipped file; instead, run it from an unzipped directory so the associated files are also exposed. Note that it is always safest to run a virus checker on any programs you download.

Then successively rename K M intv(1-8).aup to K M intv.aup to agree with your exiting K M intv_data directory. Do NOT double-click on these files - if you do, you will simply bring up the old version of Audacity and possibly cause further damage.

Load each file one at a time with my test version of Audacity and load the project. You will see what appears to be a mostly black screen. This is to be expected. Then immediately save it as, say, K M intv(1-8).aup3. Exit Audacity, then repeat this for each of the eight files.

Do not edit any of the projects. Just save them to the four unique names. I’ll post more instructions tomorrow as it is getting late today.

Also, please keep me up to date on your progress with this…

[kozikowski]

My wife’s Windows 10 PC just crashed

Do you know why the machine crashed? I know Windows used to have a reputation for poor stability, but it really is hard to make a Windows 10 machine fall over. What steps are you taking to make sure after all this fuss it doesn’t happen again?

Koz

[jademan]

So I had a little booboo. It is not my first and I am sure it is not my last. I’ve posted another version of my Audacity Recovery program, and I’ve sent you a link. The booboo won’t affect anything you’ve already done, so just change it out now and you’ll be fine.

OK, so your _data directory had about 2400 individual sound clips of 6 seconds or less. The file names are all “randomized”, the .aup file was not helpful to me, and the only information available to help us is the file’s last modification time. (The creation date and access date are also not helpful).

Because of the size of your project, I have split the file up into eight (8) parts, part 1 being the oldest and part 8 being the newest. So each project contains, on average, 300 files. Every one of the 211 clips in part 1 was created on August 11th at 6:21:48 PM. Every one of the 379 clips in part 2 was created on August 11th at 6:21:50 PM. So these first two projects are going to prove to be difficult to work with. So let’s start with the last one, part 8 for illustration.

Open the special Recovery version of Audacity; then under recent files, open your project 8. You will see a single (mono) track, and a lot of black lines over toward the right. When you loaded the clips into the project earlier, each clip went to an assigned space for it, each space about seven seconds apart. So the clips in project 8 start at about the 4 hour mark. On the Tool Transport Bar, skip to the end of the project, then zoom in about 8 times. At the top of each clip, you will find the date and time the clip was last modified, in this case 8/23|12:58:58 in military time. The last four (4) clips all have the same time and color (blue). The first two of these clips are extremely close to zero. I’m not sure what these are - it appears to be some kind of echo, yet at about 40-50dB lower volume. Perhaps the recording was originally in stereo.

.
We’ll skip these two clips for now. So drag the cursor over these last two clips then select Edit > Clip Boundaries > Sort Clips by Content. (This menu item appears only in my experimental Audacity.)

When you play these two clips together, you can tell that they go together. In this case, the developmental sort routine put the tracks in the correct order. We are not always so lucky.



The previous three clipss are black, red, and green so we know they were edited at different times. (Check the date/time stamps at the tops of these clips):

.
. Perhaps they are related. We’ll just pull them down manually: click on the top bar of the clip, then drag it down and slide it over:

Now play the clips. What does it sound like to you? (To me one of the clips sounds like an edited clip of the former). So there is going to be a little manual work and judgement required here…

The sorting routine does not work as nicely as I had hoped. But I wanted to avoid further delays in this project by introducing scope creep. I’ll be taking a closer look at it during the next few days to see if I can come up with any refinements. If you have any specific suggestions in this area, I would be happy to take a look at them.

[kozikowski]

We’ll just pull them down manually: click on the top bar of the clip, then drag it down and slide it over:

Correct me but this is one of the tricks plain Audacity can’t do. That works a lot more like a video editor. You want a clip over there, click on it and pull it over.

Now all you need is Mark-In and Mark-Out and you can double the editor’s production speed. If you’ve never met those before, they’re deceptive. You only get two at a time. Mark-In and Mark-Out. They’re not labels.

I don’t remember the keystrokes, something like Control-I and Control-O. Start the clip and scrub when you get close to the IN point. Scrub is built-in and it’s always active. It’s not something you need to specially select and configure. Another time saver.

Control-I.

Play or scrub forward to the OUT point and Control-O. Delete or Copy, or whatever else you want to do with that marked portion. They’re sticky. They will not move by themselves and you only get two at a time, so there is no confusion as to which one is which.

If you need to move one of the marks slightly, grab it and pull. No menu options or hidden controls. There is a way to simply remove the marks, but I don’t remember the keystrokes. I had a stunned moment when I found that I could move the Out Point before the In Point. The application just didn’t give a flying care.

I remember vividly pulling together a fifteen minute video presentation at break-neck speed with just those tools.

Video editors work on frame boundaries, so there will always be a time when you need to do precision sound editing in Something Else. Audacity is the Something Else.

Then we explain to the editors that Audacity doesn’t have a lot of those handy, rapid tools they like…

Koz

[kozikowski]

Missed one. The editors I used have Get INFO built-in. No moving a clip to an external application to figure out what you have. I know that’s hard in Audacity because of the special internal format, but still.

Koz

[jademan]

Correct me but this is one of the tricks plain Audacity can’t do.

Which is why we are using a special EXPERIMENTAL version of Audacity. I am hoping that most of these features will be coming out in 3.1.0. At least the clip moving parts, that is. The other stuff, perhaps I will make available after some more refinements, including a possible automatic .au file puller - but I am getting ahead of myself…

[JackBauer4Prez]

Dear jademan,

Thank you thank you thank you for all your amazing work. I have never had anyone on any discussion forum spend as much time trying to help me as you have. (And I have posted problems on dozens of discussion forums during my loooooong life.)

I will take it from here. I’ll go through all your work and see if I can piece together the interview.

My wife is just starting to re-edit the interview, but she knows she did some great work on the last one (that she lost) and so piecing together what you have done will help her.

Again, let me thank you from the bottom of my heart.

[jademan]

Jack/Dave,

You are welcome. I am glad you were able to make some use of my results. I am keenly interested in this project as I feel that Audacity had dealt you a great injustice by locking away all of your hard work and also I know that a number of other Audacity 2.4.2 users are stuck with this problem . Thank you for giving me the encouragement to push forward with this project which would probably have never otherwise been done.

To complete this project for these other users, I need to:

1. Automate the process of locating and sorting the .au files and
2. Further refine the auto-collating/sorting feature “Sort by Content”. Currently it only uses a single point matching system. I need to expand it to a broader curve matching system.

I will continue to post updates here as they are available.