Need to recover project, all files orphaned?

I just spent 3-4 hours recording my podcast, and now I can’t open the project. I have the .aup file and the data folder, but the logs seem to think that all the files are orphaned.

My usual process is to record all the tracks, save the project to my desktop, QUIT the app, and then transfer the .aup file and data folder to my file server. I record on my laptop in my studio, but edit on my big iMac - which is why I have to copy it. However, in this case, I think I may have copied the files while the project was open - I’m guessing that’s the problem. The data folder appears to have all the files in it, so I’m hoping there’s some clever way to reconstruct the project. When I tried to open without deleting the orphan files, I got an empty window (no tracks). When I tried to open and delete the orphaned files, the entire data folder was emptied.

I’m running Audicity 2.1.2, I believe I installed from .dmg. I’m on a MacBook Pro (2016, USBC) running Sierra (10.12.6).

I’ve attached my logs. I tried several things there, which you will probably see. I tried saved compressed, but abandoned it once I realized it was just converted to Ogg. (I was hoping it was some sort of zip format where all the project was in a single file - easier to transfer around.) I do still have a copy of the .aup and full data folder on my file server. The one on my laptop was emptied. I tried copying it back and had the same orphan issue.
log.txt (500 KB)

Open the _DATA folder and you should see collections of sub-folders with lots of little AU sound files. Each of those AU files should be six seconds of sound and they should open and play in Audacity.

If you open the AUP file, is it filled with XML programming language similar to this?

This one has been spread out with extra spaces for “graphic beauty,” but the actual programming is real and does open.

Third line down and two thirds to the right, see where it says projname="? That should match exactly the name of your project or show.

My usual process is to record all the tracks,

Describe the show. Do you have to use Projects?

Koz

Yes, I have a bunch of those files… are you saying I need to stitch them together manually to recover all my audio? Six seconds at a time?

Also not sure what the show description will tell you. Why wouldn’t I want to use projects?

At any rate, yes, it appears the audio files work. Now how do I fix the aup file so that this project is sane again? I’ll have a look into the XML, see if it’s something obvious. What specific problem with the XML causes the orphan file warning?

Okay, my aup file is definitely screwed up. This is all it contains for ‘imports’. These are NOT the files in my data folder (below). I’m pretty sure I do not use Ogg by default, unless that is the default out of the box. I see no “wavetrack” or anything “track”. I had 5-6 tracks. The file listing is attached.

Are the track names in alpha/hex order? If I knew the proper XML format, I could probably reconstruct a valid aup file.

[data-files.txt|attachment](upload://qEDP7NjSjfvSHTYOQiq9pBtHGtK.txt) (78.9 KB)

Why wouldn’t I want to use projects?

Because of this exact problem. If anything goes wrong, you have no show even though it looks like you do. If you have multiple edited tracks one above the other, you have no choice but to use a Project.

If I knew the proper XML format, I could probably reconstruct a valid aup file.

See my graphic example. This was a simple stereo show. The top third is the header, the next third is Left and the bottom third is Right. The AU files alternate Left/Right and they are intentionally named out of sequence.

Scroll down for manual recovery. If you have a mono clean recording (no edits), it’s sometimes possible to recover using the time stamps.

http://manual.audacityteam.org/man/recovery.html
http://manual.audacityteam.org/man/recovering_crashes_manually.html

Open up one of your working projects or make a simple duplicate of the failed one and read the AUP Project Manager file. I’ve never seen a detailed description of the format past this.

It’s generally accepted that if you lose the AUP file on an edited show, that’s the end of the world. The AUP file keeps track of the edits and discontinuities.

We should wait for a senior forum elf.

Koz

The contents of that AUP file is for the “compressed” project you tried to save then trashed. If you can find a _data folder that contains those OGG files then you might be able to reconstruct the project.
– Bill

Okay. So I found this wiki/doc for recovering Audacity projects, and posting it here in case someone else wants to try it.
http://manual.audacityteam.org/man/recovering_crashes_manually.html

However, I must say, Audacity makes this ridiculously difficult. The files are randomly named?? Why not name them sequentially? One of the first steps in this recovery process is to literally rename all the files sequentially based on timestamp. Why wouldn’t they start out that way? Sorry, just frustrated…

I wrote myself a simple python script that renamed all the files sequentially based on timestamp. I was able to recover most of the audio, but two of my d0x folders failed to recover using the audacity_recovery app : failed with “OverflowError: long int too large to convert to int”. (screenshot attached) Each d0x folder only contains about 250 files. I had six d0x folders (d00-d05) and 4 of them converted cleanly. Two of them failed, but did produce a single wav file with partial audio at least.

So I’ve recovered most of the audio, and I’ll just have to re-record portions.

If anyone knows how I might fix the OverflowError, let me know.
Screen Shot 2017-10-29 at 11.43.13 AM.png

The files are randomly named??

In older Audacity versions, they were. I remember when they did that random thing they were solving a very real production problem whose description escapes me. It’s been plaguing performers ever since.

Sorry, just frustrated…

I can make it worse. The developer daddies don’t think this is a big problem and I don’t think it’s even on the Problems To Be Solved list.

Audacity 2.2.0 will shortly be out. I asked if this was the version with the single file Project format (hopefully).

Ummmmmmm.

No.

Koz

I guess it’s good to ask if you know why the show failed? I believe the AUP file is written last, so it’s the most likely to go if there is a timing or writing error.

The up side is you can publish your rescue process if it works better than ours. You’ll be a hero.

Koz

I don’t know all the technical details, but I do recall that when block file writing changed from sequential numbering to partially random numbering, Audacity suddenly became much more stable and automatic data recovery improved immensely.

It wasn’t much of a process. I just wrote a python script to rename the files sequentially based on timestamp. If you think that’s valuable, I can clean it up and publish it. Then I used the audacity_recovery app to recover the audio from the renamed files, which only mostly worked (I got that OverflowError).

I’m really not trying to throw shade on Audacity here - I really do appreciate the app. Just wish that it would package the project files into a single unit, and I wish it was a little more robust against project corruption (and easier to repair).

BTW… where is auto save enabled? I can’t find it in the preferences anywhere. That might have saved my butt.

I’m pretty sure there is no Auto Save. The very last thing you need on a borderline powered machine is for it to start managing a save cycle in the middle of a recording—or worse, an overdubbing session.

Koz

There is, but it’s a machine readable file, not a human readable file.

I’d guess that’s why it is dumped to disk as binary data rather than encoding and formatting it into a human readable form.

That is being more than a little harsh on the developers who, as you well know Koz, are very few in number on this volunteer Open Source Project.

Yes it has been on the radar for a fair while now, has been much discussed by Team members (including yourself at times) - it has a formal Proposal in the Audacity Wiki, which was started by Steve back in 2011:

It’s also more recently come on to James Crook’s (key developer) “Problems To Be Solved list” in his umbrella Proposal for “Low Hanging Fruit”:

The problem is that as he states there: “Not quite low hanging fruit, but very good value for the work that would be expended.” and “This one is potentially a lot of work, but the benefit in reduced support of keeping all audio in one file is HUGE.” - i.e. it will be a fair bit of work to code (and test) and we are but few in number - so the trade-off is do we get new additional features or do we expend effort on this? And given that we are a volunteer project we don’t have a management structure to choose and direct …



And the reason it won’t be is because we have very few developers - and none of them has, so far, volunteered to work on this. Rather for the upcoming 2.2.0:

a) James gives us choosable themes, revised menu structures and other new stuff

b) Paul Licameli has worked hard on making Audacity more secure and safe to use (especially in out of space situations) and has helped top work on the new MIDI playback with PokeChu22 (a long-standing feature request)

c) Steve has been working hard to add help links to the Manual for all the Effects, Generators & Analyzers - and from some of the key (and tricky) error messages

And those really are all we have right now, sadly.

Peter

has been much discussed by Team members (including yourself at times)

I didn’t discuss it. I just complained about it. I’m not a developer. I’m modeling the Lowly User.

All those tasks are difficult and the goals are important, but to back out to 10,000 feet for a second, the goal of Audacity is to make a product or show. Involving split-file Projects seems to not be the best, most stable way to do that.

How would you advise the poster at the top of this thread? As near as we can tell, they did nothing wrong and yet, we just spent the last three days sweeping his show up from the floor, and they’re still going to have to re-record some of the non-recoverable segments.

It’s completely by accident I didn’t get swept up in this. I knew from the first day I wanted WAV files and took steps to get them. I’ve never had a show crash because I’ve never used Projects.

Koz

It may not seem like the best, most stable way when looking from the ruins of a broken project, but it’s virtually universal for DAW type applications. Pick any of the big names: Cubase, Sonar, Logic, Reaper, Audition, even the lowly Garage Band, they all save multi-file projects.

As might be expected from Apple’s usability geniuses, Garage Band hides the fact that the project has multiple parts, so that it looks like one file, until you move it to a new computer when it shatters into a million fragments.

The real reason that Audacity does not yet have a unitary project format, is that it very difficult for a DAW type application to use a single file project format in an efficient and reliable manner. When Audacity gets a single file project format, I think that will be a unique feature.

Several DAW type application provide some means of ‘zipping’ the project into a single file bundle, but the user has to know that they need to do that on top of saving the project. It also does not help in the event of a crash, because these applications work with multi-part projects, and the single file version of the project is just a packed version of the multi-part project that has to be unpacked before it can be used.

Yes you have. Audacity always works with Projects, never directly on files.

You may not have ever “saved” Audacity Projects, but you always use them. The fact that you have never had a show crash, indicates how reliable the Project format is in normal circumstances. Each time we read a post about a broken project, we are looking at an exceptional circumstance, usually brought about by machine failure, user error, or interference from a third party application.

I don’t recall the last time that I lost data from a project - it was way back in the 1.3.x beta versions.

Interesting info on projects - thanks for posting.

Would it be possible to have an option to zip a project built in? Because I move my projects around constantly, it would be handy to have that. I know I can do this manually, but it would be nice if the tool had this built in and understood how to open the zip when I double-click it. (I’m not looking for compression here, just consolidation.)

Another option might be to keep the project file within the folder, so it’s still logically one “unit”. So… saving new project called “my_project” creates a folder like this:

my_project:

  • my_project.aup
  • e00
    – d00
    – d01
    – d02

This would help with clutter, too. I find myself doing 2-3 related projects at once on my desktop, with similar names, and it gets crowded and confusing real fast. And also maybe you could maintain a project file backup, or some sort of change history, that might help with recovery.

It would also be nice to have a tool that will reconstruct a valid (if minimal) .aup file from a data folder. I realize you would lose a lot of info for a highly-edited project, but for a simple set of raw tracks, it seems like that would be pretty straightforward. This would help not only for corrupt project files, but missing project files (someone forgot to copy them together, or accidentally deleted the project file but not the data folder, etc).

Just some thoughts. Thanks for all the detailed replies!

That’s more or less how I work:

my_project

  • myproject.aup
  • myprodect_data
    • e00
      • d00
      • d01
      • d02
        my_project_resources
  • my_project_notes.txt
  • my_project_lyrics.pdf
  • my_project_imported files
    • file1.wav
    • file2.wav

Yeah, good call. I’ll start doing that, too.