Why re-compress unchanged audio tracks

I often save a compressed project to work on later, eventually doing Export Multiple to MP3. Typically all I do in these projects is filtering or noise removal and then add or edit a label track. When I re-save it as a compressed project, it updates all the audio tracks. They ‘always’ get smaller, so I must be losing something each time. If I just Save it (Ctrl-S), my .aup file references an uncompressed e0 directory, which appears in the folder. Also, before I started using compressed projects, I copied pieces of audio liberally into new tracks and exported by track name rather than label knowing that Audacity wouldn’t actually copy the data to use it in a new track. Now I keep it all in one track.

What I’m requesting is this: If the audio in a track has not been altered, then don’t update the .ogg file. My workaround for this has been to save the compressed project somewhere else, copy the .aup file back from there and delete the newly saved project. This wastes a lot of time generating .ogg files that will be discarded.

One more request: A warning when saving a project that was loaded from a compressed project.

Why are you saving compressed projects? Are you short on disk space? Every time you compress (using OGG, MP3 or any other lossy format) you lose quality. Forget compressed projects and just save the Audacity project. Just remember to keep the .aup file and the _data folder together. Some people find it convenient to put them together in their own folder.

– Bill

I use compressed to squeeze several church services onto a thumb drive 'cause my laptop is getting full. Since I’m exporting MP3’s anyway for this I don’t think the loss in the ogg will amount to much. Also there are parts I don’t export for the podcast and I want to keep the projects around for a few weeks in case I’m asked for something I might have recorded but didn’t put in the podcast.

:astonished: Just opening the project replaces the .aup file with one that references the uncompressed e0 folder it just made.
Before I found out about compressed projects I was exporting a flac and label file. I could recreate the project from just those. Save Compressed Project seemed simpler.

You can get quite big thumb drives these days.

And you can get (relatively cheaply) portable 0.5TB & 1TB pocket USB disks - of a size that easily fits in a jacket pocket. Here’s an example found at random on Amazon: http://www.amazon.co.uk/including-Software-assembled-CBA-Digital/dp/B002FOCNRM/ref=sr_1_10?s=electronics&ie=UTF8&qid=1290079668&sr=1-10

My son has one (not the one referenced above) for his work use and finds it very rugged and reliable in use.

With bigger storage you can export the recorded project to an uncompressed WAV file on the portable drive.

WC

DickN mentioned he was already exporting as FLAC+labels file before founding about compressed projects, so I guess that’s not the issue here.

Regarding external hard-drives I’m pretty happy with my iomega 2.5" 320GB (they make it in 500GB now too). It’s quite small and light. Nonetheless it’s still considerably bigger than a flash drive and harddisks always require more careful handling than flash.

Yes, but the poster did go on to say that he was using <<compressed to squeeze several church services onto a thumb drive 'cause my laptop is getting full>>. Hence the bigger external drive suggestion :slight_smile:

And yes, I like Iomega drives too. I have one of their 1TB desktop USB disks (my wife has two, bouught after seeing mine), but it is certainly small enoungh to be portable in a briefcase, or a laptop bag 20cm x 12cm x 3cm. I carry it to my son’s house occasionally to make an offsite backup of all my production/archive WAV files.

WC

Actually the thumb I use for this currently has a couple of uncompressed projects taking up most of it. I only compress the less critical ones (i.e. those that will wind up as MP3’s). I apply effects (noise removal, filtering, volume compression etc.) before making a compressed copy. Thanks for the USB HD suggestion though, will look into it.

My original post was not about why one would ever use the feature, but about already-compressed sound files that didn’t need updating getting rewritten by compressing Audacity’s working files which were derived from them, as I often edit the label track before exporting and may do this in multiple sessions.

Also, as I discovered recently, I lose the “compressed project” version of the .aup file just by opening the project. The workarounds for these issues are simple but one has to know about the behavior and remember to use the workaround. If one does not use the workaround, which is to make a copy of whatever files one is not going to alter and copy them back later, then the degradation due to lossy compression is compounded unnecessarily. Moreover, the needless rewriting of “unchanged” files takes a long time due to the compression algorithm.

I haven’t used the compressed project feature yet, but if it behaves like DickN said, then I must generally agree with what he says. If this goes to the wiki add a +1 vote for me please.

That suggests that Audacity is accessing compressed audio files within the project, but that is not the case. Audacity always works with uncompressed audio and can only work with uncompressed audio. This means that when you open a “Compressed Copy” of a project, all of the audio data is immediately uncompressed. When you save the project again, it is the current audio data (the uncompressed audio data) in the project that is saved.

So how about if there was an option to select FLAC as the compression format for compressed projects rather than Ogg?

I understand that when saving a Compressed Project, Audacity is compressing its current audio data regardless of its original source, but what I’m saying is that if a track’s audio data has not been altered and the original source for this session was a Compressed Audacity Project, it should not update the .ogg file for that track.

I’m in favor of having that option, and I would use it quite a lot. My previous statement would still apply, however, if only because of the time involved in generating the compressed data.

My other issue was losing the “compressed project” version of the .aup file when opening a compressed project.

How would Audacity know if the track had been altered? The source file itself is not being used by Audacity - Audacity is using a copy.

Are you suggesting that Audacity should compare the audio data in the project with the original source file and see if it matches, or that Audacity should check through the History to see if any operations have occurred that would have caused the data to change, or something else?

Would Audacity do this only if the current project had been opened from a compressed project file, or for all projects?

There’s a clue in the name. “Save Compressed Copy of Project”
Compressed projects were never designed as an alternative to saving a full, uncompressed project. The idea is just that it creates a more portable copy of the project. There are a number of drawbacks to compressed projects. You are highlighting just one of them, but I would have thought that a more important drawback was the loss of audio quality through saving in a lossy compression format.

There’s no need to compare audio data or check history… This is much easier to do: just have a flag associated with each track. If at any point in the session that track is altered that flag is set. At saving time only the tracks which have that flag set are recompressed.

I think this could apply to all projects compressed or not.

Thinking about this raised a doubt… I haven’t work with this before so if someone could clarify I’d be thankful… What happens when I import a WAV file into a project? I believe that there’s an option not to save that into the project data right? Like when I open the project it will load that track from the wav file, not from audacity project. Am I correct on this? If so, then what happens if I change that track in Audacity and then save? What will it do? Will it overwrite the linked wav file with the updated version and delete the original wav? Or will it save that new data into the audacity project and no longer link it to the external wav?

Yes.
There is an option in “Edit menu > Preferences > Import/Export” >>> “Read uncompressed audio files directly (faster)”.
There is an option in “Edit menu > Preferences > Projects” >>> “When saving a project that depends on other audio files > Do not copy any audio”.

“Read uncompressed audio files directly (faster)” will allow Audacity to reference imported uncompressed audio files without copying them into the project. It does this by creating “virtual data blocks” (my terminology) instead of using .AU files (which would either be in the temporary folder if the project has not yet been saved, or in the _data folder if the project has been saved).

These “virtual data blocks” are listed in the Audacity Project file (.AUP) as “pcmaliasblockfile” and are given a .AUF extension.

If/when the pcmaliasblockfiles are edited, the corresponding data from the linked audio file (WAV, AIFF or FLAC files) are copied into the project as .AU files.

If this project is Saved and the option “Do not copy any audio” has been selected, the project will still depend on the external WAV/AIFF/Flac file for all unchanged “virtual blocks”. These will be shown in the AUP file as .AUF files (pcmaliasblockfiles). Any data blocks that have been changed will be shown in the AUP file as .AU data blocks.
The first time that a project is saved, any .AU files that are in the temporary folder that are required by the project will be copied into the project _data folder. Any unused .AU files will be discarded when the project is closed. The uncompressed audio file that is linked to the project is called a “dependency” because the project depends on data within that file.

If all of the data from the linked dependency file is modified, then all of the data will have been copied into the project as .AU block files and the original uncompressed audio file will no longer be a dependency. This can be checked from “File menu > Check Dependencies”.

The underlying problem with (lossy) compressed audio files is that there is not a direct correlation between data blocks in the compressed audio file and data blocks in the Audacity project. The only way that Audacity can know that a specific sample belongs at a specific location in a specific data block is if the entire file has been decoded (uncompressed). This means that “On Demand” reading of data from compressed files is not (currently) possible, and imported compressed files must be copied into the project.

The only way that I can see this proposed feature working is for Audacity to copy data from the compressed file and raise a flag. Then if any data in that track is changed for the flag to change state. While the flag is in its original state the compressed source file would be listed as a Dependency, but if the flag state changed then it would be removed from the list of Dependencies. On “Save”, if the compressed file is listed as a Dependency, then the associated track data would be discarded and a link would be made to the original source (compressed) file so that when the project is reopened the compressed file is imported (copied) back into the project.

While I think that such a scheme is probably possible, it is potentially very dangerous. For example, if the user saves the project with a different name, then the dependent source file will still be in the old project _data folder, and if the old project is moved, renamed or (most likely) deleted, then the new project will be destroyed. I guess that this could be avoided by only linking to the original source file IF the data is unchanged AND the option to “not copy data” is selected in Preferences AND the project is Saved with the original project name, but this is getting extremely complicated for the rare and not-recommended case that the user is opening a compressed copy of the projects AND the user has re-saved the project as a compressed copy AND has not made any changes to the track AND has not saved as a full (uncompressed) project.

Even if this proposal is implemented, the next question is “I only truncated the silence at the end of the track. Why can’t Audacity split the original file without decoding - like you can do with MP3Split?” The simple answer is that Audacity is primarily designed for high quality editing and processing of uncompressed audio - as soon as you start trying to work with compressed audio there is inevitably loss of sound quality. Personally I’d rather that the developers continued to concentrate their efforts on high quality audio work :wink:

Much easier and safer to recommend that users always Save the actual (uncompressed) project if they wish to continue working on it later, and only use “Save Compressed Copy” as a copy. This recommendation is in line with the recommendation that users should always (if possible) work with uncompressed audio until the final export.

I see this gets to be a conundrum. I suggest another way to avoid at least one of the issues I’m having: Add “Save compressed copy of project as” to the file menu. That avoids the split personality issue of the original version’s .aup file toggling between compressed file references and .au file references since I could then save compressed directly to the thumb drive. In the “…as” case, the e0 folder should not be written. If the file name initializes to the original when I’ve navigated to the destination folder, that would also be useful (maybe it does already - I don’t remember using “save project as” except with a new project) since I append “_ogg” to the original file name when I compress it.

Arguably, “save compressed copy of project” is a misnomer, since the original .aup file is overwritten.

A reminder/warning, when saving compressed copy of project without a name change, that it was loaded from a compressed copy would also be useful, since I’d want to “save compressed copy of project as” to avoid further degrading the existing compressed file(s) if I hadn’t altered any audio data. I’d still have to copy the .aup file back from the “as” location to the working location. A “compressed project” flag in the .aup file might simplify keeping track of this.

Actually, wouldn’t it be simple to remember that audio data has been changed? Doesn’t Audacity do this already so it remembers to give the “save changes” prompt on exit? OK, there’s probably one flag that gets set for every change including labels, but there could be two flags, right? Or would that get too complicated when keeping track of "undo"s?

I think there should be a warning if overwriting an existing project with a compressed copy.
Perhaps this could include a reminder that overwriting a compressed project will re-compress the audio data.

Given the complexities involved with compressed projects, how would it suit if there was such a warning, and also an option to use FLAC files in a compressed project? As far as I’m aware (but I’m not a programmer) such features would be possible and a real benefit for many users. I’d probably use FLAC compressed projects myself :slight_smile:

Then this feature is already implemented to some extent…

Even if this proposal is implemented, the next question is “I only truncated the silence at the end of the track. Why can’t Audacity split the original file without decoding - like you can do with MP3Split?” The simple answer is that Audacity is primarily designed for high quality editing and processing of uncompressed audio - as soon as you start trying to work with compressed audio there is inevitably loss of sound quality. Personally I’d rather that the developers continued to concentrate their efforts on high quality audio work > :wink:

So why create the compressed project format in the first place? :stuck_out_tongue:

I think the solution here might be to change the way compressed projects are saved… I’m not going to discuss the way ‘normal’ uncompressed projects are saved… but when it comes to compressed projects a different approach could have be done… like one file per track? Something more similar to the way external linked files are managed…

That’s what does happen with compressed projects.

Interestingly, in Audacity 1.3.12 you are not allowed to overwrite an existing project with “Save Project As”.
I wonder why it is allowed to overwrite an existing project with “Save Compressed Copy” :confused:
I’ll see what I can find out.

It already saves one file per track. But there is no opportunity to select the type and level of compression.

I have several problems with the way compressed projects are managed.

  1. When you save a compressed project you are allowed to use the same name as the uncompressed project. This merely creates the OGG files for each track within the _data folder and does not delete the .au files, so the overall space for the project is actually increased. Furthermore you lose any clips, envelopes, etc. you have in your uncompressed project. I think you should not be allowed to overwrite an uncompressed project with a compressed project. The compressed version should be an archive or transmission format.
  2. When you open a compressed project, it creates the e00 and dxx folders and recreates the .au files within the _data folder. So now you’ve got a _data folder with the compressed tracks in OGG format, and your uncompressed data. If you Save your project without intent to recompress, you still have those compressed track files hanging around. I think Audacity should create a new .aup file and a new _data folder when opening a compressed project, prompting the user for a unique name.

The above changes would keep compressed and uncompressed projects completely separate.

These limitations and gotchas are documented here http://manual.audacityteam.org/index.php?title=File_Menu#savecompressed but we are encouraging/recommending that users go into the _data folder and delete things which seems to me a big no-no.

@Steve: while I was composing this you posted similar concerns in this thread and to -quality!

– Bill

That sounds to me like a very good idea.
Not only would it avoid the “bloated” projects that you mention, but it would also make it a lot more clear to the user what is actually going on.

Thanks for the link. (does that mean that any discussion should be on the manual discussion page :frowning: )

I was just about to mention that to you in case you wanted to comment there :stuck_out_tongue: