Data lost bug - workaround?

[Newbis]

Hi,
I have a problem with Data loss in 1.3 Beta Unicode on Windows XP.

Here's how to possibly duplicate the behavior:

In a project with a lot of tracks using data saved by Audacity (in a data folder that Audacity created when you created the project), paste in another track from a wav file opened in Audacity. Use the "move up" feature to move the track higher up so it's in the position you want relative to the other tracks. At some point, close the Wav file that you pasted from without saving it as an Audacity project. Use the envelope tool to change dynamics in the new track that contains the freshly pasted data. Save the main project as many times as you like throughout the process. While the project is still open, the data can be seen, heard, and used to export audio. But once that project is closed and reopened, only the envelopes and possibly some of the track data is there; the rest is just a flat line.

Currently, when this happens, my only option seems to be to re-paste a new track from the WAV file, then very meticulously apply the envelope tool based on what it looks like in the track that's lacking the data, then delete the empty track and apply any other effects from memory. Then, either the chance of the paste saving this time is 50-50, or possibly if one does a "Save As" instead of a "Save," maybe that's more insurance that the problem won't recur.

Now here's the question: Until they fix this bug, are there any other workarounds? For example, is there a way to paste data into the track without losing the envelope information for that track? Or is there a way to "find" the information among all those "au" and "auf" files and "reconnect" the track to its data? Or if not, is there a way to take the wav file and somehow connect the track to it? Or conversely, is there a way to paste the envelope information (and possibly other modifications that were done) onto another track? What I want to avoid is that very time-consuming process of making the envelopes look exactly like the previous track, as this is very tedious, and also prone to error (since it's just visual).

Thanks!

[bgravato]

In the Preferences -> Import / Export which option have you selected for "When importing audio files"?

If you use the "faster" option and you import a wav file into the project, the audio isn't copied into the project (at least until you change it), rather it's linked from the original wav. If you meanwhile move or delete the original wav, next time you open the project if audacity can't find the wav all you see is a flat line.

The best workaround is to use the "safer" option.

[Newbis]

In the Preferences -> Import / Export which option have you selected for "When importing audio files"?

If you use the "faster" option and you import a wav file into the project, the audio isn't copied into the project (at least until you change it), rather it's linked from the original wav. If you meanwhile move or delete the original wav, next time you open the project if audacity can't find the wav all you see is a flat line.

The best workaround is to use the "safer" option.

Thanks...I had assumed that when I had saved the project originally and had selected the "safer" option, that this would take care of it. But you're right...The "faster" option is still checked in the preferences, even though when I created the project and it asked if I wanted "safer" I had said "safer" so that it created a directory with lots of auf and au files.

Of course, it shouldn't really matter because the WAV file was never renamed or removed...And besides, when you copy and paste data, that ought to transfer the data itself to the project, shouldn't it? I mean, I understand that if I opened a WAV file and selected not to have Audacity make a copy, and then I moved or renamed the WAV file, then I would have an empty reference to a file that's no longer there. But if I copy and paste data, the data should be there, I would think. I guess perhaps Audacity thinks that copying and pasting is still merely creating a reference to something else....but to me that's a bug. (Since when is copying & pasting the same as importing?)

At least I think I understand what's happening now. Basically, it seems that "preferences" are attached to the program instance, not to the file itself. So if I do a SaveAs, and it asks me if I want the "safer" option, that doesn't change the "preferences." Then later, if I open a WAV file in Audacity and copy and paste data from it into another project, Audacity probably thinks it should create a reference to the "project" that contained the WAV file. Then if I close the WAV file without saving it as an Audacity project (or maybe if I merely pasted from it before saving it as a project), the reference is probably broken and can't be repaired by merely re-opening the WAV file and saving with the default name, since it probably had a reference to some sort of empty value or ID number pointing to a handle to some window that can never be duplicated. (Re-opening the WAV file in an unsaved project and then opening the main project while the other one is open doesn't work either.)

So then the only way around this is probably to always save the project "AS" something else after copying&pasting any data into a project, so that it makes a copy of everything. Still....Bug bug bug bug bug...

Anyhow, I would like to know if there's any workaround "after the fact"...that is, if there's anyway one can "force" a reference, that is, either modify a blank reference to go somewhere, or find what it's looking for so one can put a file with the name it wants? Or it would be nice if one could overlay an envelope map with wave data.

[bgravato]

Yes, what you describe sounds like a bug... I could not reproduce this on Audacity 1.3.13 on Mac.

Which exact audacity version are you using? Check in Help menu -> About

How many tracks did you have in the project? mono or stereo tracks? How long is the project?

If you change to "Safer" option in the preferences does this still ocurr?

Have you been able to reproduce this more than once? with different projects, wavs?

How did you copy paste the track? Using the keyboard shortcuts? Or the mouse?

It could be a windows specific bug, so lets wait for the windows elves to pop up and see if any of them can reproduce this...

[steve]

There's quite a few questions bundled into this post - let's see if I can separate this tangled web a little.

Data lost when copy/paste between Projects.
This seems to be the main problem here.
Yes I have seen this problem in some older versions of Audacity 1.3 beta. There has been some recent major reworking of how Audacity handles missing data, so this problem is hopefully fixed in the current 1.3.13 version, but I have not yet tested 1.3.13 extensively enough on Windows to confirm if it is totally fixed or not.
I have not seen this bug with 1.3.13 alpha, which is the most recent version that I have used extensively on Windows.

This bug can be difficult to reproduce - the only times I've seen it have been in the middle of large complex projects, (exactly when you don't want to see a bug like this), which makes it very difficult to back track and see exactly what chain of events caused the problem.

If you were using Audacity 1.3.13 beta when this occurred we will need to try and find a repeatable method of reproducing the bug.

How to safely copy data between projects.
The safest way to copy audio between projects is to Export the audio from one project as a WAV file and Import the WAV file into the other Project using the "Copy In" (safer) method to Import. After importing, use the "Check Dependencies" feature (File menu) to ensure that the file has been copied into the Project. This is the "belt and braces" method and is as close to 100% reliable as anything that involves computers.

What happened to the data?
This is a bit of conjecture on my part.
If the "copy from" Project is Project "A", and the "paste to" Project is Project "B".
I think what is happening is that data that only exists in a temporary folder associated with Project A is being "copied" to Project B. When Project A is closed, it clears its temporary data (as it should). When Project B is saved it copies its temporary data into its project _data folder. My conjecture is that there is a mix-up about which data belongs to which project, so either the data is being incorrectly deleted by Project A, or not saved by Project B. Either way, after both projects have been closed the data that should be in Project B has gone.

Can the lost data be recovered from the .AU data files in either of the _data folders?
I very much doubt it.

How to replace missing audio.
There is currently no mechanism for replacing missing audio data other than Importing or pasting in a new copy of the data. Essentially this means that audio track (or section of the track) is trash. There have been discussions about developing a means to replace missing audio data within a track, but this feature does not yet exist.

Copy envelope data from one track to another
There is no "official" way to do this yet, but it is possible to do by hacking the AUP file.
There have been discussions about making Envelopes more portable so that they can be copied/pasted between tracks, but this feature does not yet exist.
To hack an AUP file so that an envelope is copied from one track to another:
IMPORTANT - DO THIS ON A COPY OF THE PROJECT. If you get this wrong the project may be permanently and totally destroyed.
The Audacity Project file (AUP) is written in XML. If you have no idea what that means, please ignore the rest of this section, otherwise proceed with caution.

Open the AUP file in a text editor (for example NotePad or NotePad++)
Each audio track begins with:

Code: Select all

<wavetrack name="Audio Track" 

For each audio track there is also a closing tag

Code: Select all

</wavetrack>

If the audio track has envelope points, the envelope data is between the opening and closing tags for that wavetrack.
The envelope begins with:

Code: Select all

<envelope numpoints=

and ends with

Code: Select all

</envelope>

If a track contains no Envelope points, there will be a single <envelope numpoints="0"/> line shortly before the closing </wavetrack> tag.
Here's an example from a mono audio track:

Code: Select all

			</sequence>
			<envelope numpoints="0"/>
		</waveclip>
	</wavetrack>

To copy an envelope from one track to another track.
Copy the envelope data (everything between and including <envelope numpoints= to </envelope>) and paste it between the destination tracks <wavetrack> tags, replacing that tracks original <envelope> data.

Note: Stereo tracks have 2 envelopes - one for the left channel and one for the right channel. The envelopes for left and right channels must be identical.

Save the file, and open it in Audacity. If you have done it correctly, the envelope will now appear on the new audio track. If you have made a mistake, delete the project and start again.

The Envelope feature fiddly and time consuming to use
Yes it is.
After the release of Audacity 2.0 I very much hope than this feature can be developed further.
There are currently a number of suggestions about Envelopes on the "Feature Request" page of the Audacity wiki. If you feel strongly enough about the issues you may want to start a discussion on the "Adding Features" section of the forum and we can see if we can come up with some definite proposals about envelopes that we can then put to the developers.

Of course, it shouldn't really matter because the WAV file was never renamed or removed...

It does matter, because (from your first post) you were not directly copying the WAV file, you were copying an audio track from within an Audacity copy. We're looking at an abstraction here and it's difficult to see if the visual representation of the waveform that was being copied was "attached" to an actual WAV file, or to a temporary copy of the WAV file. The actual data linking and moving can be very complex and has clearly been going wrong in certain situations. If the problem is occurring in Audacity 1.3.13 beta, then we need to try and pin down exactly what those "situations" are.

[kozikowski]

Bug bug bug bug bug...

One of the features of a "bug" is multiple people can reproduce the problem on multiple computers. In order to do this, we need very specific file data -- or possibly the files themselves posted and the very specific steps and conditions to reproduce. As above, unless a Developer can reproduce the problem at will, it could live forever.

The other possibility, however unlikely, is that your Audacity is acting funny. That is, if nobody can reproduce the error. That's when it loses its "bug" certification.

See Private Messaging.

Koz

[Newbis]

Yes I have seen this problem in some older versions of Audacity 1.3 beta. There has been some recent major reworking of how Audacity handles missing data, so this problem is hopefully fixed in the current 1.3.13 version, but I have not yet tested 1.3.13 extensively enough on Windows to confirm if it is totally fixed or not.
I have not seen this bug with 1.3.13 alpha, which is the most recent version that I have used extensively on Windows.

Thanks for the insights and the explanation about how to tweak envelopes in XML.
Here's what I get for the version under "About": Audacity ® 1.3.10-beta (Unicode)
Does this mean that what I'm describing is simply an old problem that's already known for that version?

...in which case, it sounds as if my two courses of action are 1) upgrade to 1.3.13, or 2) just use my SaveAs workaround until the next official release.

In the meantime, I'll try to see if I can duplicate the problem on a smaller project, as the data folder for my current project is 270 MB.

[bgravato]

I can't guarantee that upgrading to 1.3.13 will solve this problem, but you should definitely do it...

Audacity development branch (1.3.x) has been in freeze mode for a while... that means no new feature are being added... mostly bug correction is being done until the release of the next stable series (2.0). Audacity 1.3.13 has several improvements and bug corrections over the previous versions, so you should really upgrade.

[steve]

Does this mean that what I'm describing is simply an old problem that's already known for that version?

I can't find this problem specifically listed in the release notes, but Audacity 1.3.10 is about the version when I noticed this problem.
Since then I've updated several times (now using 1.3.13 beta on Windows XP) but also, I usually use the "ultra-safe" method for transferring audio between projects.

I would recommend that you upgrade to 1.3.13. I think there's a good chance that this issue may have been fixed as a result of fixing some other bug.
After updating, keep an eye out for this problem, and let us know if you see it in the current version.

In the meantime, I'll try to see if I can duplicate the problem on a smaller project,

If you are able to reproduce the problem in 1.3.13, then that would be very useful. Reproducing the problem in 1.3.10 would probably not be so useful.