Orphaned Blockfiles

[orgelquaeler]

I have been experiencing the bug which results in a file with orphaned blockfiles, and the gaps filled with blank space. I will try to be specific about the scenarios.

My specs: Pentium Dual Core Allendale 1.8GHz, 4GB RAM, Windows 7 64 Bit, 250GB Hard drive
The files: Usually 1.5hour long lectures, mono

1.) I only ever have this problem when working with .aup files. Working with WAV's and Mp3's seems to be fine.
2.) Until recently, this problem only reared its ugly head if I left a project file open for a long period of time (several days). If I saved the project, then opened it again, the file would be missing little pieces.
3.) Now, what happens is I open the file, edit certain specific sections of audio, save it, then close the file. The result is that there will be two files: One edited file which is fine, and one other file which has missing block files, but only in those specific sections of audio where there were edits. The remainder of the file remains intact. I cannot account for it's creating a separate file, in the same directory as the original.
4.) In addition to this, I will occasionally encounter a scenario where I will open a file and it will be corrupted, without another file having been created as I mentioned above.

This is the directory where the files are. Note I only intended one copy of each, and there is only one aup directory for the project files.

directory.png (46.12 KiB) Viewed 2273 times

This is what the file looks like when I open it.

AC-4.png (86.52 KiB) Viewed 2273 times

[Gale Andrews]

Thanks, orgelquaeler. Leaving the project open a long time is another scenario I have heard before. Do you have these projects open in an Explorer window? That is also under suspicion as a cause.

What is the full three-section version number of your Audacity version (see Help > About Audacity)? Is it an old Beta like 1.3.6 or something that creates .aup.bak backup files? There should be only one .aup.bak per project (each backup ovewrites the last), but opening the backup does create orphans because it uses the same _data folder that the main .aup file uses. For that reason we stopped writing .aup.bak files several Betas ago. If you are not using 1.3.13 already, please upgrade.

What is the original name of the project you provide an image of? Are you doing File > Save Project As? Once you have saved a project in the window, Save Project As will save that project with the new name you choose as a separate .aup file and _data folder. That scenario has been tested (but not dozens of times) and not found to create an issue. Or do you save with File > Save Project, or by File > Close or File > Exit and say "Yes" to "Save Changes"?

Also as it stands your image (given the file extensions are hidden) makes no sense because it shows Audacity Project Files with the same name which is impossible. Can you open Explorer, click "Organize", then "View" and in "Advanced settings", choose "Show hidden files, folders and drives"? Click OK then post your image again with file extensions. Also there is an "09 AC 4" file with no _data folder. Do you know what happened there?

And so I am clear, if you reopen the exact project you last closed, it reopens without an error dialogue or audible/visible glitches? But the "copies" of it that you didn't explicitly save have "orphan files" and gaps in the waveform? Do the copies also have "missing blockfiles"?

In your other scenario (2), is it the same as (3), that is:
* the intended project is not corrupted
* you intend multiple ,aups and _data folders that are different versions of the same project
* there are also unwanted copies of the .aup project file but no unwanted data folders?

And in scenario (4), where you get corruption in the project audio, do you get missing blockfiles, orphan blockfiles, or any other displayed error when you launch the project?


Gale

I have been experiencing the bug which results in a file with orphaned blockfiles, and the gaps filled with blank space. I will try to be specific about the scenarios.

My specs: Pentium Dual Core Allendale 1.8GHz, 4GB RAM, Windows 7 64 Bit, 250GB Hard drive
The files: Usually 1.5hour long lectures, mono

1.) I only ever have this problem when working with .aup files. Working with WAV's and Mp3's seems to be fine.
2.) Until recently, this problem only reared its ugly head if I left a project file open for a long period of time (several days). If I saved the project, then opened it again, the file would be missing little pieces.
3.) Now, what happens is I open the file, edit certain specific sections of audio, save it, then close the file. The result is that there will be two files: One edited file which is fine, and one other file which has missing block files, but only in those specific sections of audio where there were edits. The remainder of the file remains intact. I cannot account for it's creating a separate file, in the same directory as the original.
4.) In addition to this, I will occasionally encounter a scenario where I will open a file and it will be corrupted, without another file having been created as I mentioned above.

directory.png

AC-4.png

[orgelquaeler]

Hello Gale,

I will try to answer your questions to the best of my ability.

My Audacity version is 1.3.13.

The original file was saved from the WAV file using File > Save As, but any subsequent savings were done using Ctrl + S, or by closing the project and choosing to save the file. The name change was done post-edits, using explorer, which, now that I think about it, could be an issue depending on how the files are linked to their folders.

Most of the time, I can open the original file just fine with no missing block files. The copies all have missing block files. However, sometimes the original will become corrupted as well, and I have had to redo projects as a result. In no case do I ever intend to have multiple .aup's in the same directory. Any further edits get saved in other folders, and usually as WAV's. Usually, multiple Audacity windows being open is not an issue, since there is only one open window during the editing process. I usually close the files, then late come back to do any splitting/splicing. It is when I re-open the file that I find the problem. There does not seem to be an extra folder.

Anytime this error occurs, I get the missing blockfile message, except for possibly once, and I cannot be sure, where the computer went to sleep and when it came up, the project had missing blockfiles, without having closed the file. I cannot be sure this is what happened (it was a long time ago).

I will upload a picture of the directory with more information later.

[Gale Andrews]

The original file was saved from the WAV file using File > Save As, but any subsequent savings were done using Ctrl + S, or by closing the project and choosing to save the file.

That cannot (should not) produce multiple .aup files and _data folders.

The name change was done post-edits, using explorer, which, now that I think about it, could be an issue depending on how the files are linked to their folders.

If you did not want the duplicate .aups and _data folders as you say below, I'm not sure why you were renaming them - do you recall how Audacity was naming these duplicates originally?

Renaming .aup files and data folders in Explorer is a bad idea as explained at http://wiki.audacityteam.org/w/index.ph ... s#together but it would (should) not of itself cause orphaned blockfiles. You just would not be able to open the _data folder for the project if you got it wrong.

If you do want to create multiple versions of the same project (each version having its own .aup file and project folder), File > Save Project As (once a project is saved) is the best way.

Most of the time, I can open the original file just fine with no missing block files. The copies all have missing block files. However, sometimes the original will become corrupted as well, and I have had to redo projects as a result. In no case do I ever intend to have multiple .aup's in the same directory. Any further edits get saved in other folders, and usually as WAV's. Usually, multiple Audacity windows being open is not an issue, since there is only one open window during the editing process. I usually close the files, then late come back to do any splitting/splicing. It is when I re-open the file that I find the problem. There does not seem to be an extra folder.

Your topic subject says "orphaned blockfiles" (which means files in the _data folder which are not listed in the .aup file), but you now say "missing block files". "Missing audio data blockfiles" are files which are listed in the .aup file but absent from the _data folder:

1.3.13 Missing Audio Data Blockfile(s) dialogue

missing_blockfiles.png (16.5 KiB) Viewed 2260 times

Is that what you see? This would be more expected because missing files must lead to silences in the audio. Can you supply the .aup file and the log (Help > Show Log then Log > Save) for any intended projects where you see these errors?

In these cases when you close the project, do you File > Close or File > Exit and save the changes?

Anytime this error occurs, I get the missing blockfile message, except for possibly once, and I cannot be sure, where the computer went to sleep and when it came up, the project had missing blockfiles, without having closed the file. I cannot be sure this is what happened (it was a long time ago).

This is again a scenario that one user has mentioned before, though I cannot make it produce a problem.

Also are you intentionally adding any silences to parts of the project before you close it, if so with what command? This is another scenario that is suspected.

And are you saving to the root of the drive (for example, in "C:" instead of "C:projects")? This is another suspected scenario.

As things are at the moment, I would strongly recommend exporting a WAV file before you close any project (to a location outside that holding the project).

It might be very useful if you have the patience to make copies of the project just before and then after closing a project.

To do this, create a new folder other than in the project location (for example, "proj1_before_close"), then before closing, copy the .aup file and _data folder into that new folder using right-click > Copy then right-click > Paste. This avoids renaming issues. Then create another new folder (for example, "proj1_after_close") and copy and paste the (intended) .aup file and _data folder in the same way to that new folder. Then if the close produces duplicates in the original project folder or reopening the project from its original folder produces errors, we should be able to replicate it if you send us those new folders. Obviously don't open the projects from those new folders yourself.

Thanks for your help.


Gale

[orgelquaeler]

Hello Gale,

Sorry about the break in correspondence here. Things have been more hectic here than usual.

I will try to answer your questions:

I was renaming the files depending on the content. I hadn't considered that it could screw with the way they open, and will not do this again.

The dialog I get says "missing/orphaned blockfiles".

I always close projects by "X"ing them out. I usually do Ctrl + S to save beforehand.

I am not adding intentional silence to any part of the audio.

I am not saving to the root of the drive.

I am going to try working another series of files to see if i can duplicate the problem. If so, I will let you know. I hope I have provided enough information. I have very limited time to spend on here, but i want to help the cause any way I can.

Best Regards,
David

[Edgar]

Hi David!

I have very limited time to spend on here, but i want to help the cause any way I can.

Every little bit of help is welcome!

Just remember, we want to see Log files but you must save them before you close a Project because they are not cumulative--every time you exit the Log file (with the errors listed which have occurred during the current run of Audacity) is lost.

Most often we will want two Log files, the first from just before you exit Audacity after Saving the Project (which will have "problems" when you open it again) and the second from right after you Open the Project which has "problems". This way we can see what caused the "problem" in the first place and what the "problem" is in the second case.

I realize this really means saving a Log for every Project, as there is no way to tell in advance if a Project will have "problems" on reopening.

[WARNING rambling thoughts follow!]
Team Audacity has been batting around some ideas on how to automate this Log saving process but there is no real consensus on how to implement the idea--it is messy. One problem is that the Log is full of obsolete debugging statements so if we stored an "ongoing" Log somewhere it would grow rapidly.

Another problem is that if a user does not save a Project, then a Project-specific Log has no suitable on-disk storage folder in which to be stored; if Audacity stores such a Log for every time it is run but no Project is saved. then, again, lots of storage will be consumed and the user will need to clean out the old as needed.

[Gale Andrews]

Hi David,

As Edgar says, we're grateful for any help, but naturally the more the better.

It's OK if easier to get in the habit of saving a log before each exit of the program, as long as you don't use Log > Clear in the log itself. If there are errors, all the errors persist in the log until exit. But the Missing/Orphaned error dialogues will suggest you look at the log anyway.

The dialog I get says "missing/orphaned blockfiles".

The dialogue has a sequence of messages if there are multiple errors, but each message has its own dialogue. So I assume you get both "missing" and "orphaned" warnings in two separate dialogues?

If you get a chance it would really be helpful to open Explorer, click "Organize", then "View" and in "Advanced settings", choose "Show hidden files, folders and drives" so we can see what extension Audacity was producing for those files that have the same name.

I am going to try working another series of files to see if i can duplicate the problem.

Thanks. As well as saving the log before exit, saving a copy of the .aup and _data folder before exit then a second copy after exit would be really helpful (as I described above by copying them into new folders). If you don't touch the copies but the original has a problem, we can probably see it happening by using your copies (definitely, if a problem occurs on re-opening rather than exit).

We've realised meantime that if you make a copy before exit then there will be orphan files used for Undo which Audacity won't have been able to delete, but we can figure out which orphans are due to that.


Thanks


Gale

[Brash Blusher]

Hello, Gale,

I'm new to recording with Audacity (I only started 4 days ago). Today I suddenly got an error message warning me that there were 4 orphan files that were the result of some bug or crash, and that these orphan files should be deleted immediately.

I opened a new tab in Internet Explorer and Googled "orphan files ..." and before I even finished typing the words, my super smart computer suggested that these words went with Audacity. This forum was the first result I clicked on and read. As I skimmed through the info here, I noticed that someone mentioned having changed a file name based on the content.

Having been a word processor in New York City for many years, I am thoroughly trained to expect that changing the name of a file is a harmless thing to do, and that all related files will maintain their correct connections to whichever file is being renamed (e.g., data files, tag files, etc).

But I instantly realized that Audacity is apparently not programmed to do this. (I was also a computer programmer for a time, so I know that it could either be a hassle to add a chunk of code to do that, or it could be a breeze if the right chunk of code could be 'borrowed' from somewhere else and plunked down in the right place in the Audacity software code.)

I had recorded an old tape recording of myself playing "Have Yourself A Merry Little Christmas" on a Yamaha DX7 recorder ('KybdHaveYourselfAMerryLittleXmas.aup'). As it was a rehearsal tape and I didn't try to make it perfect back in 1994, it had one bass note that came in a hair too soon. Could I fix that in Audacity? Yes! And so I did.

I began by Saving Project As ('KybdHaveYourselfAMerryLittleXmasA.aup'). By adding 'A' at the end, I signaled to myself that this was the edited version. I copied a sliver of the timeline just before the early note and then pasted that sliver back in. Perfect.

However, due to the format in which I was naming the .aup, .wav and .mp3 files I was creating, I wanted to maintain my naming plan.

So I renamed ('KybdHaveYourselfAMerryLittleXmas.aup') as ('KybdHaveYourselfAMerryLittleXmasOrig.aup') to mark it as the original recording.

Then I renamed ('KybdHaveYourselfAMerryLittleXmasA.aup') as (''KybdHaveYourselfAMerryLittleXmas.aup') so that it wouldn't have that ugly 'A' stuck to the end.

Just to make things even more tidy, I renamed the newly edited version, ('KybdHaveYourselfAMerryLittleXmasA_data'), as ('KybdHaveYourselfAMerryLittleXmas_data'). That took out the nasty 'A' and gave me a data file whose name matched my perfectly edited .wav and .mp3 files, all of whose names were now massaged into being what I wanted them to be.

Thank goodness I didn't tell Audacity to delete those 'orphan files!' They were my children: the precious files with all my hard work, only orphaned because I, their parent, withdrew my name from them and gave them the 'wrong' name.

This all happened at once -- having the problem and finding this forum, so I was able to easily remember which file should be named what. I changed the .aup, .wav, and .mp3 file names back to their original names. However, I then got a message that the ('KybdHaveYourselfAMerryLittleXmasA_data') file could not be found. So I changed ('KybdHaveYourselfAMerryLittleXmas_data') back to ('KybdHaveYourselfAMerryLittleXmasA_data'), and now Audacity and I are both happy again.

Evidently, there is no underlying identifier to anchor the file independent of its surface name, and therein lies the problem at this stage.

Have I described anybody else's problem with orphan files? More importantly, could somebody tell me if backing out of renaming files restores Audacity's sanity for them, too?

[steve]

This is certainly an important issue Brash Blusher, and that is why it is highlighted in red in this "frequently asked question" entry: http://manual.audacityteam.org/man/FAQ: ... s#aufolder as well as being described in detail in the manual here: http://manual.audacityteam.org/man/Audacity_Projects and in the "Quick Help" here: http://manual.audacityteam.org/man/Saving
It is also discussed in several places in the Audacity wiki including here and here.

There is also a feature in recent versions of Audacity 1.3 to help people avoid the problem altogether: http://manual.audacityteam.org/man/Dependencies_Dialog
Also, recent versions of Audacity 1.3 provide this error message when you import uncompressed audio files:

warning.png (47.91 KiB) Viewed 2158 times

and by default the user is given the option to resolve dependencies when they save the project:

save.png (68.47 KiB) Viewed 2158 times

However the issue of dependent files is not unique to Audacity - the same issue exists with PowerPoint, PageMaker, Cakewalk, Sonar, Cubase, Logic and many other programs that work with projects that have linked files.

The problem is not as simple as maintaining links between a couple of files - if you have a look in an AUP file for a project that has dependencies (open it BUT DON'T CHANGE IT, in a text editor), you will see that there can be hundreds of links to different fragments of audio data. I doubt that there is a robust, cross-platform way to protect people that don't read the manual, or the Quick Help, or the Frequently Asked Questions, or the pop-up warning messages.

[Brash Blusher]

Hi, Gale,

Wow. Thanks for all of those links. What a gold mine, all in one place. I'm going to keep coming back to this page until I thoroughly understand everything in each link.

Like this: "The .aup file and _data folder must always be kept together as created by Audacity. Never move or rename them manually." Never rename these files manually. I wonder how long it would've taken me to run across that in the manual. And if I did see that while reading the entire manual before beginning my first project, would it have registered in my mind if not at a need-to-know point in my process? I'm still looking up terms and acronyms.

I thought of a suggestion to make for this file-renaming problem, which triggers the "delete orphan files" warning: that pop-up warning could first state, "Manually renaming files can cause this problem, so manually restore the original names. Otherwise, ..."

OK, I accept your scolding ... "people that don't read the manual, or the Quick Help, or the Frequently Asked Questions, or the pop-up warning messages."

The manual - After only 4 days, I'm still working on reading the entire manual (er ... pedalling as fast as I can here).
The Quick Help - my Bible.
The FAQ - my best friend, but only if I know I need to ask a question and then know the precise question to ask.
Pop-up warning messages - Ah! Like the one that said to delete my orphan files, and that I immediately put on hold till I could investigate it.

But never fear, I'll keep coming back, though probably only to shriek that I need help on something, not to broadcast how proud I am that I did something right.

And Gail, I apologize if my observation on the Audacity code sounded, um, brash. It wasn't meant in that spirit. Just a week or so ago I wondered if I had some way on my new computer to convert my old audio cassette tapes. I discovered the Windows Sound Recorder and made several .wma sound files using a $15 stereo cable from my little SONY tape recorder headphone jack to the blue line-in port on the back of my PC.

That's when I discovered and downloaded Audacity in order to convert the .wma files to .mp3 files. I had no idea at first that Audacity was much more than just a tool for MP3 conversions. I emailed a couple of MP3 files with apologies for the sound quality coming from the old cassette tapes.

Imagine my utter delight when I started to focus on Audacity and realized that I had such a powerful free tool in my hands. After setting the line-in power to -6.0 on the Meter Toolbar and recording directly into Audacity, I produced absolutely beautiful recordings of those "old" tapes. I almost cheered, especially when I was able to correct that one note that came in early on "Merry Little Xmas." On my 4th day.

My modus operandi (like that of a lot of new users, I suspect): (1) Start out with something really simple. (2) Spend loads of time reading up on how to do the most basic thing. (3) Next, try to add one new wrinkle at a time. (4) When overloaded with incomprehensible geek-speak till the brain and the eyes glaze over, take a break. (5) Come back when the brain is fresh and start again at Step 1.

I appreciate your help. Thanks.