Suddenly Corrupt Aup3 file

Audacity Help Forum Windows
1

I’m using Audacity 3.1.2 on Windows 10 Home but also have access to Audacity 3.1.2 on macOS Big Sur.

I run a vocal music group and have been editing together an a cappella number for a few weeks now waiting for more submissions from members of the group. I was editing earlier today during lunch, saved, left, came back, and when I went to open the file again, I received the following error message (with different timestamps since I’m reproducing it for this forum message):
{
“timestamp”: 1639596449,
“event_id”: “4fa991eb236e9147bc839c309c2ab09b”,
“platform”: “native”,
“release”: “audacity@3.1.2”,
“contexts”: {
“os”: {
“type”: “os”,
“name”: “Windows”,
“version”: “10.0.19043”
}
},
“exception”: {
“values”: [
{
“type”: “Error_Opening_Project”,
“value”: “Project is corrupt\n(Unable to work with the blockfiles)”,
“mechanism”: {
“type”: “runtime_error”,
“handled”: false,
“data”: {
“sqlite3.query”: “DELETE FROM sampleblocks WHERE NOT inset(blockid);”,
“sqlite3.rc”: “11”,
“sqlite3.context”: “ProjectGileIO::GetBlob”,
“log”: “14:18:31: Error: Couldn’t find symbol ‘av_packet_alloc’ in a dynamic library (error 127: The specified procedure could not be found.)\n14:18:31: Error: Couldn’t find symbol ‘av_packet_free’ in a dynamic library (error 127: The specified procedure could not be found.)\n14:18:31: Error: Couldn’t find symbol ‘avcodec_free_context’ in a dynamic library (error 127: The specified procedure could not be found.)\n14:27:29: sqlite3 message: (11) database corruption at line 71416 of [1b256d97b5]\n14:27:29: sqlite3 message: (11) statement aborts at 8: [DELETE FROM sampleblocks WHERE NOT inset(blockid);] database disk image is malformed\n14:27:29: DBConnection SetDBError\n\tErrorCode: 11\n\tLastError: Project is corrupt\n(Unable to work with the blockfiles)\n\tLibraryError: database disk image is malformed\n”
}
}
}
]
}
}

All I did was try to open the file I’d been working in for days. I immediately Googled the error and came across the forum and many other similar issues of this nature. I then tried to open the file on my Mac, but received a similar error message (once again reproduced, so different time stamps):

{
“timestamp”: 1639592339,
“event_id”: “32bff051ed374267b5098d3e05950539”,
“platform”: “native”,
“release”: “audacity@3.1.2”,
“contexts”: {
“os”: {
“type”: “os”,
“name”: “Macintosh”,
“version”: “10.16.0”
}
},
“exception”: {
“values”: [
{
“type”: “Error_Opening_Project”,
“value”: “Project is corrupt\n(Unable to work with the blockfiles)”,
“mechanism”: {
“type”: “runtime_error”,
“handled”: false,
“data”: {
“sqlite3.query”: “DELETE FROM sampleblocks WHERE NOT inset(blockid);”,
“sqlite3.rc”: “11”,
“sqlite3.context”: “ProjectGileIO::GetBlob”,
“log”: “13:18:55: Error: Could not parse file "/". \nError: Project is corrupt\n(Unable to work with the blockfiles)\n13:18:59: sqlite3 message: (11) database corruption at line 71416 of [1b256d97b5]\n13:18:59: sqlite3 message: (11) statement aborts at 8: [DELETE FROM sampleblocks WHERE NOT inset(blockid);] database disk image is malformed\n13:18:59: DBConnection SetDBError\n\tErrorCode: 11\n\tLastError: Project is corrupt\n(Unable to work with the blockfiles)\n\tLibraryError: database disk image is malformed\n”
}
}
}
]
}
}

One forum said to restart my computer and try opening the file again to no avail. I couldn’t recover any temp files, nor had I been duplicating saves at multiple stages for I had no reason to think that Audacity 3 could present itself to be this unstable despite it being open source. I just upgraded when I was told to upgrade. I had just started using Audacity again after leaving Garageband because there are more functionalities, but the migration to Sqlite3 databases from XML files and directories made it difficult for me to access any of the raw files that I had quickly recorded into Audacity and used as guides for all of the other vocal parts I wanted to line up against.

I saw that some people had success uploading a zip folder with the files and seeing if anyone was able to recover some of the work, so I preemptively did that: https://drive.google.com/file/d/1kQJElzFHc65fhGNEoP3lM-PUm82WjrVn/view?usp=sharing

However, I am going to start over tonight in Garageband because I can move faster there and don’t want to chance this’ll happen and lose all this time again. This was number we were releasing for the holidays, so I’ve just gotta redo all this soon. But if anyone’s able to assist on how I can uncorrupt this file that I can’t figure out how it became corrupt, help with recovering any of scaffolding I’d created in this original file, and/or tell me how to prevent this from happening moving forward, that would be greatly appreciated.

Thank you all very much!
-Vince
Executive Director
Noteworthy Philadelphia
https://www.noteworthyphila.com

2

This could be similar to a problem I had that was fixed by opening it up in the beta version and/or opening it up in a prior version of Audacity. Noot sure if it’s the same issue, but you could try that.

Here is the original thread - Help - Internal error at D:\a\audacity .... - #6 by mafg1953

Mike

3

recover some of the work, so I preemptively did that:

That’s so Jademan can download and look at the whole show at once and try to unscramble it. It’s possible he’s going to be able to give you pieces and segments back, not an intact edit.

The only thing you can do directly on the forum is a tiny sound sample as an illustration, not a large edited production.

Koz

4

Vince - I am glad to see that you have a healthy attitude regarding this error. I have good news :smiley: and bad news :frowning: . The good news :smiley: is that your project “looks” to be completely recoverable. The bad news is that your project is consuming gobs and gobs of computing power and storage and wall clock time on my computer :frowning: . I have managed to recover your project - the only thing is it seems to be about 20GB in size :frowning: ! So, obviously something is seriously wrong here :frowning: . I have some tools that I can continue to use to isolate the issue, but unfortunately, I am out of time today :frowning: and I am out of town tomorrow :frowning: . So I would continue with your current plan and cross your fingers that I’ll have something you can make use of before your projects is due. :slight_smile:

5

All I did was try to open the file I’d been working in for days.

Any patterns yet?

"I’ve been working on my show for (Some Large Number) of (Time Divisions) and I went to open it up today and I got (Some Error Message) instead of my show.

Some counter somewhere is running out of juice (technical term)? Are all of these damaged shows on Windows? I know there are some damaged shows that the poster was editing in Windows, had it fail, and then tried to open it on a Mac. That doesn’t count.

Any shows where the poster recorded, say, ten minutes, saved the work and watched it crash? I guess that wouldn’t make news. Let’s say someone recorded a short but super important sound track and watched it fall over?

I’m fascinated by being able to go either up a version or down and have a recovery.

Koz

6

OK, so I was able to spend a little more time with this. I was wondering why your “recovered” project was so… big! As I delved into it more I found that your first clip alone has been duplicated 51 times. Why this is, I don’t know - it could have been caused by the corruption; or it could have been caused by a bug in the software, or it could have been caused by user application error - the latter of which is quite common since 3.1.x has been introduced.

As I scan your project, you have around 500 duplicated clips which, until I am able to upgrade my program - perhaps even this summer, I have to deal with manually and this would take me many many hours. Based on your comments you know what I am talking about.

I believe you have all of the original audio for this project. If there is any missing audio that is somehow lost in this project, let me know, and I’ll try to extract the appropriate track - it does seem to be all there, just far too much of it to be useful to anyone in its current form. Otherwise, I am going to quit this project and cut my losses. Sorry to have given you false hope. :frowning:

So now I need to get back to my real life (I am merely a lowly volunteer - trying to help those in trouble).

If you decide to restart your work in audacity, I strongly recommend, for you, going back to 3.0.5 (Old Audacity versions download) where the clips cannot be so easily duplicated. I am hopeful that a future software update will make these duplicate clips easier to deal with.

However, I did forward your project to the developers - I think they stand to learn something from it.

7

or it could have been caused by a bug in the software

Let me clarify some technical details. Starting from 3.1.0, Audacity supports non-destructive clip trimming. This is implemented using two additional offsets per clip. In XML they are represented with trimLeft and trimRight attributes of the waveclip. For example, when you split a clip (using Cmd-I), you get two clips referencing all the data from the original clip but with the offsets adjusted appropriately. This is very common for Editors and DAW, and it is quite useful as you can easily adjust the clip boundaries after the split. From the disk space perspective, it doesn’t have any overhead attached. The only issue, which will be addressed in future releases (3.2+), is that there is currently no easy way to remove the hidden data.

Some effects and commands indeed can generate lots of “full” clips. For example, “detach at silences” will generate as many “full copies” of the track as silences detected. This does not increase the storage requirements for the wave data but can result in a long load time, which was addressed in 3.1.2 and 3.1.3. However, this specific project is not the case. Audacity 3.1.0 loads it within a second on my PC. For 3.1.3-beta, it’s 32 ms (this value is logged now). My PC can be considered a “beast” to some extent, but I do not expect any significant delay for computers capable of running Windows.

the latter of which is quite common since 3.1.x has been introduced



I strongly recommend, for you, going back to 3.0.5

Error 11 was introduced in Audacity 3.0. It is not related to 3.1 releases in any way. Similar corruptions were possible previously but mostly were unnoticed, especially for the block files. SQLite database is a relatively complex binary format, which is very sensitive to corruption in the BTree pages. Storage fails and fails unexpectedly. SQLite tries to do the best possible job to keep the integrity, but it cannot address failures coming from OS and hardware. Quite some time ago, not with Audacity, we had a MacMini in our CI setup, which replaced data with random bytes in some locations. We were trying to figure up why tests are failing for a long time before we noticed :slight_smile:

This specific project had a very minor corruption, which was fully recoverable with:

sqlite3 .recover project.aup3 | sqlite3 recovered.aup3
sqlite3 recovered.aup3 "PRAGMA application_ID = 1096107097; PRAGMA user_version = 50397184;" ".exit"

Usually, luck is not on our side, and we end up having a “lost_and_found” table, which contains any kind of orphaned data. Unfortunately, we haven’t yet come to a consensus on how to make the recovery process accessible for users in the general case.

8

Quite some time ago, not with Audacity, we had a MacMini in our CI setup, which replaced data with random bytes in some locations. We were trying to figure up why tests are failing for a long time before we noticed

This is the exact place I would be doing a long-form memory test. Entertainment production uses extended memory unlike Photoshop jobs and spreadsheets. And I don’t mean one-and-done testers. I mean the one that runs all night and halts on errors. Racing Jackrabbit Test failed at memory location XX on pass number 304.

So the Mini was broken.

it cannot address failures coming from OS and hardware.
Storage fails and fails unexpectedly.

Is this why Audacity fails more and more likely as the drives get further and further away from the production office? I can see where network negotiation would drive a brittle service nuts.

Any way to do a pre-production service test? Run this to see if your system is up to the job? This would be far preferable to producing a Jademan Event after several days of editing.

Koz

9

:smiley: Thank you very much for your assistance. :smiley:

10

I understand your frustration well. However, handling disk write errors originating from the hardware or OS level is usually beyond what user-space software can do. This problem is not exclusive to Audacity 3. It happens to all the software. It happened to Audacity 2. It happens to my local git clones at least twice a year. It happens to enterprise drives with ECC.

Such events are random. SQLite3 is one of the most well-tested pieces of software in the world. It worked countless hours for billion of people in the world. Your browser uses it internally. Most likely, your phone uses it for every app. Still, “error 11,” aka SQLITE_CORRUPT, is entirely internal to the library. This specific project had three (!) wrong bytes on a single page (64kb for Audacity) for 1,226,244,096 of bytes inside.

For the last five months, we have received around 5.8k user reports with SQLITE_CORRUPT errors. This is a vast amount indeed. However, it comes from millions of users and even more Audacity launches. Unfortunately, this is very well on par with the failure rate for storage drives.

The only real solution is a constant replication, preferably to different hardware. This is not feasible for Audacity, at least not as a default option. The disk space requirements, especially when using effects are very high already.

Indeed, sometimes we fail. When we do, we work really hard to make a timely hotfix, especially if there is no easy workaround for the issue we introduced. On top of that, we work on improving testing procedures a lot too.

Entertainment production uses extended memory unlike Photoshop jobs and spreadsheets.

Audacity uses far less RAM than Photoshop. From the storage perspective, 40 Gb PSD is quite a common thing.

11

I have enormous respect for programmer/developers. I just scraped past programming in my technical training. I know the theories and I know the process, I just don’t have a feel for it.

I knew this wasn’t the career path for me when the guy next to me in the classes routinely got done with a project—correctly—and was sunning himself at the beach while I was still in class.

However, I also know that telling our Producer their show is a thing of flaming wreckage is an experience not to be repeated. I’ll call you next time this happens. It’s a life-changing event.

This is a vast amount indeed.

It gets our attention when many forum posters arrive with nearly the same problem and different from historical postings. We work backwards. One forum posting is worth many failures in the field. So I would not fall in love with that 5.8k user reports number. When you just flushed an interview with the governor is not a good time to compose a user report.


OK, failing a pre-production facility test and given there’s nothing inherently wrong with SQLite3, how about a software Jademan. We know two ways to help with the pain of a crash. Open the show in the next/or the last Audacity, and post it to Jademan for Human Rescue.

Being able to rescue parts of a show, the show out of order, or get a definitive certification of death is far better than helplessly watching a project file that refuses to open.

Koz