Corrupted file

I now I should have exported a mp3 or done a backup - unfortunately I didn’t.

Can anything be done to save this? The drive did have space. There is 1 G on the file.

13:58:45: Using internal LAME
14:09:48: sqlite3 message: (11) database corruption at line 67525 of [7ebdfa80be]
14:09:48: sqlite3 message: (11) database disk image is malformed in “PRAGMA main.synchronous = NORMAL;PRAGMA main.journal_mode = WAL;PRAGMA main.wal_autocheckpoint = 0;”
14:09:48: Failed to set mode on /Volumes/Seagate Portabl/ACX/chp4.aup3
Error: database disk image is malformed
SQL: PRAGMA main.busy_timeout = 5000;PRAGMA main.locking_mode = SHARED;PRAGMA main.synchronous = NORMAL;PRAGMA main.journal_mode = WAL;PRAGMA main.wal_autocheckpoint = 0;
14:09:48: DBConnection SetDBError
ErrorCode: 11
LastError: Failed to set safe mode on primary connection to /Volumes/Seagate Portabl/ACX/chp4.aup3
LibraryError: database disk image is malformed

If you care to share your .aup3 file on a public server, you can PM the address to me. I’ll take a look at it.

hello,

I am having the same issue with one of my files. Do you have any recommendations on how to resolve?

Thank you

I saw your post on the other thread: audacity failed to set safe mode , and I have nothing to offer there. sandrecord’s report on this thread and yours look identical, otherwise, I might have separated these posts. These particular error messages look very nasty - it is quite possible there is nothing that can be done. I was interested in sandrecord’s .aup3 file mostly for forensic information (I have not seen this error before). However, if you would like to zip it up and upload it to a public file server, then post or PM me a link, I’ll try to take a look at it early this coming week.

Hello, thank you for your response. Which public server do you suggest I use.

I have used dropbox and google.

https://drive.google.com/file/d/1P3tq2G556wWcI_5gaMLd5La9TPJMBXRL/view?usp=sharing

Thanks you

You’ll need to provide access.

Caprice,

OK, I have been looking at your file. As I said before, “these particular error messages look very nasty - it is quite possible there is nothing that can be done.”

I have looked into your particular .aup3 file with all of the Audacity tools I have at my disposal - none of them will get anywhere with your file. Audacity makes use of an underlying sqlite database where it stores all of your data. In this case, the underlying sqlite process returns an error 11 = Corrupt file, and refuses to do anything else.

I have also explored several sqlite utilities that claim to have some success recovering databases from these errors. None of these utilites have proved fruitful.

However, when I look at the raw data buried within your .aup3 file, I can see what appears to be chunks of audio data interspersed between unintelligible and damaged control data. When I take your .aup3 file and import it as RAW 32-bit float data, Little-endian, I can almost see (and hear) your audio.

I am thinking if we can remove the control data from this result, we may be able to recover some/much of your data.

For this reason, I am asking a Nyquist developer if he can assist with a program that might be able to remove these data points. It may be of general value. Stay tuned.

It is easy to remove all envelope control points IF you can open the project in Audacity (“Extra menu > Scriptables I > Set Envelope”).

Apply to the entire project and all envelope controls points should be removed:
envelope.png
If the project won’t open, then how do you access the audio data?

Thanks. No. Not envelope control points. I mean actual data points or samples. Sorry I didn’t make myself clear. So the file would actually end up a little bit shorter. (It is too long to begin with).

So looking at the screenshot, the invalid data point is not only wrong, it is extra and needs to be removed altogether. The invalid point should be treated as no data, and everything to the right needs to be shifted one sample point to the left. I don’t know if you can do this in nyquist.

It can be tricky.

If the “invalid data point” is a valid 32-bit float value, then:

  1. It’s unlikely to be a problem
  2. You could force it down into a valid range by clipping the audio to +/- 1.0
  3. A script could probably be written to export as a valid WAV file (with the invalid data points removed)
  4. For short tracks (less than about 30 minutes, mono, @ 44100 Hz) it’s probably not too difficult to remove the “invalid data points” with a conventional Nyquist “Effect” plug-in.

Option 3 is not simple for long tracks, but I think it would be possible.

The bigger problem comes with “invalid data points” that are not valid 32-bit float numbers, aka "NaN"s (NaN - Wikipedia)
I did some work with Nyquist a few years ago to help Nyquist deal with NaNs safely - I’ll need to test to see what the current situation is.

Thank you, Steve. :smiley: Thanks for pointing out the issues.

Mono tracks only, and quite slow.
Tested with 1 hour audio selection (took about 2.5 minutes) with 64-bit Audacity on Linux:

This code replaces any invalid / out of range samples with silence:

;version 4
;type process

(setf step 100000)

(defun process (sig)
  (let ((out (s-rest 0))
        (ar-out (make-array step))
        (sr *sound-srate*)
        (tn (/ step *sound-srate*)))
    (do ((ar (snd-fetch-array sig step step) (snd-fetch-array sig step step))
         (i 0 (1+ i)))
        ((not ar) out)
      (dotimes (j step)
        (if (and (<= (abs (aref ar j)) 1)      ;valid range
                 (= (aref ar j) (aref ar j)))  ;not a NaN
            (setf (aref ar-out j) (aref ar j))
            (setf (aref ar-out j) 0)))
      (setf out
          (sim out
              (at-abs (* i tn)
                      (cue (snd-from-array 0 sr ar-out))))))))

(process *track*)

Thank you very much Steve, your program was very helpful and illuminating. I see you found a solution to the NAN issue. :smiley: I had been hoping we could find a Nyquist solution as this could be something anyone could use easily. :nerd: Other than what you mentioned, your program has two limitations: (1) It zeroes out the erroneous data, rather than remove it, and (2) it terminates without an error message when a very large section of data is selected.

Nonetheless, there is a distinct flaw in my concept. :frowning: While the program does remove most of the noise that occurs every 1/3 of a second or so :smiley: , there is still noise at 14 second intervals that needs to be addressed :frowning: . Back to the drawing board for me and thanks again for you assistance. :frowning:

Before running your program:


.

After running your program:


.
You can see that a lot of the data has been cleaned up…

Both of those could have been fixed if it was sufficiently useful to spend time on it :wink:

Can you tell if the bad samples are “extra” samples, or, if they were all removed, would there be bits missing from the sound?

Yes, I know, :wink: - but I think we have bigger fish to fry.

Yes, they are “extra” samples. If they were all removed, then the sound would be perfect - except for the possibly bigger issues I have outlined above. :smiley: :wink: :nerd:

It looks like the remaining “bad” data is mostly much higher amplitude than the “good” data.
If you want to see what’s left after removing all samples that have an absolute value > 0.5, change this line in the Nyquist script

(if (and (<= (abs (aref ar j)) 1)      ;valid range

to

(if (and (<= (abs (aref ar j)) 0.5)

First Track004.png
.
Thanks, but I think it is going to need a different approach. :wink: Importantly, we’ve got your mind churning. :smiley:

Caprice,

Just so you know, I am still looking into your project.