Making bit-exact WAV file copies of DATs

[PatMcGee]

I'm trying to make bit-exact WAV file copies of a bunch of DATs. So far, I haven't succeeded.

I record a DAT using Audacity, then trim off the leader and trailer and export it to a WAV. I record the same DAT again, trim to the same spots relative to the music and export as a WAV again. When I compare the two WAV files, I see differences between parallel samples of about 3-4 (out of 65536). That is, if the sample in one file is 00DF, the parallel sample in the other file might be 00E2. Or it might be 00DA.

I've seen the same thing happen on two different hardware setups, except that the differences between samples is slightly higher using the Echo than the Impact.

Setup A:
TASCAM DA-20 mk II, using coax S/PDIF out.
Inpact Coaxial to Optical converter
Toslink cable into audio input port on my Mac Mini
Audacity 1.3.13
OS X 10.6.8

Setup B:
TASCAM DA-20 mk II, using coax S/PDIF out.
Echo Audiofire 4
Firewire to Mac Mini
Audacity 1.3.13
OS X 10.6.8

My two theories so far:
1) The input mixer in the OS X sound input driver is dithering the audio streams to make it "sound better".
2) The input mixer in the OS X sound input driver is resampling the input in order to mix different input streams together.

(Why there might be an input mixer in the sound driver, I don't know. But I know there is an output mixer so I can't dismiss the possibility that there is an input mixer.)

As far as I know, I can dismiss the second theory. When I open "Audio MIDI Setup" utility and view the Format on the Input tab, I can see the sample rate and I can see it change as I play DATs recorded at different sample rates. When I play a DAT recorded at 44.1k, the Setup utility displays 44100.0 Hz. When I play a 48k DAT, the Setup utility displays 48000.0 Hz. When I try to change the sample rate while a DAT is playing, the Setup utility changes it back after a few seconds.

According to tech support at Echo, the bits that come in to the S/PDIF input are precisely the same as the bits sent out the Firewire port, with just the format converted. And the Impact pretty much must just do conversions from electrical to optical; it's cheap enough that there can't be anything in it that mucks with the bits. Since the same thing happens with both, I'm inclined to believe that the problem is elsewhere.

So, anybody got any hints or suggestions?

Alternately, does anyone know of any Mac recording programs that will read directly from the optical input port and bypass the driver? Or the FireWire port?

Thanks,
Pat

[kozikowski]

I'm believing that if you plug anything into the optical port, it will show up in Apple > System Preferences > Hardware > Audio > Input and you can select it. The audio will be 48000, 16-bit, Stereo in the case of "normal" DAT.

Set Audacity preferences to that and restart Audacity. Turn off Dithering in Quality.

That should take are of the bit damage.

Alternately, you can do the editing directly in QuickTime 7 (available on your Leopard install disk under utilities or additional installs).

Koz

[PatMcGee]

Major progress, but not total success. Thanks, Koz.

Argh! I thought I had set dithering to None, but when I looked closer, I found it in two places on the Quality panel. I've now got both set to None, not just one of them.

Now, instead of every sample differing by 3-4 or so, I see every second or third sample differing by 1. And that's true on both setup A and setup B. I see the same amount of difference between the two setups also.

I tried recording using QuickTime, and it saved to a .mov file. So far I haven't found a good way to extract a .wav file from that. QT Pro can do it, but that costs money, and I haven't succeeded at getting mplayer to accept command line options.

So, what might I try next?

BTW, the reason I'm doing this is that I've got several hundred DATs to transfer for an archive to send to the Library of Congress. I want to automate the QC process instead of having to listen closely to each one. So, I want to transfer it twice and diff the two files. If they come out bit-identical, then I know that I've got a good transfer.

[billw58]

DATs are imperfect. DAT players apply three levels of error correction. At the first level the error correction codes completely and accurately recreate the sample. At the second level the hardware interpolates the bad sample using known good samples. At the third level it mutes the output. If you can get a service manual for the DA20 you may be able to find the pins on the main microprocessor that indicate first and second level error correction. You could put a counter or oscilloscope on the second-level error pin and see if any second-level error correction is taking place.

-- Bill

[kozikowski]

With the dithering off, you should be able to process one file and the input and output should match. That's a good as we can do.

DAT has quite a reputation in the sound world. Have you looked inside your machine? It's a spinning head videotape machine in there with all the errors that suggests.

What you do is take your DAT machine out for a shoot and just as a whim, take a backup system as well. You might have just left the DAT on the kitchen table at home because you always use the backup recording for the show. So that's the kind of fragility you have.

Koz

[PatMcGee]

Bill, thanks for the clue. I did some checking and found out a lot that I hadn't known before. It was an interesting journey, but in the end, I think it doesn't solve the problem.

It turns out on the DA-20 Mk II, there's a way to display the error rate on the front panel. According to the manual, this is the number of errors per 10,000, and it displays as a 4-digit number. The deck can display the rate for head A, for head B, and the mean of the two. I set it to display the mean.

When I played my test tape with this info displayed, the number changed every two seconds or so. In that two seconds, the tape played about 100,000 samples (at 48k), so I assume that the display probably updates every 100,000 samples and shows the actual count for that period divided by 10 and either truncated or rounded. At least, if I were programming it, I'd probably do that, and I'd probably truncate it.

I got a bunch of zeros, a few numbers between 1 and 5 or so, with an occasional number higher. The highest I saw was 17. I played the same tape again, and I saw different numbers, especially right at the beginning of the tape. This time, the highest was 27.

I think that what this means is that, in that play, I should expect to see interpolated samples somewhere between almost never (in the times when the display showed zero) and about one in 300 (when the display showed 27).

What I see is that the samples are different about once in every 3 or so.

So either I've made a bad assumption somewhere or what I see is caused by something other than the level 2 interpolation.

Does this make sense to you? Am I reading too much into what I see here?

Thanks,
Pat

[PatMcGee]

Hi Koz,
I'm not sure that I understand your last comment. I think what you're saying is that, when I use Audacity to record the same DAT twice with dithering off, I should indeed expect to see two bit-identical files.

When I combine that with Bill's comment, I should see differences when the tape has a minor (level 2) or major (level 3) dropout. For level 2, I expect to see about as many samples different as what the player displays in the error counter (times 10). For level 3, I expect to see whole frames of all zero samples. And of course the dropouts will differ between different plays of the same tape because of dust, random tracking errors, and all sorts of other issues.

If so, that is indeed what I do expect but that's not what I get.


Yeah, I know DATs have a reputation for being pretty fragile, but I'm having pretty good success. These tapes were made starting about 1995. (I've got 10-1/2" reel-to-reel tapes for the festivals before that. Which is a whole 'nother can of worms.) Some of the DATs have lots of dropouts; others are pretty good. And it doesn't appear to be age-related. At least the DATs seem to be less fragile than the CD-Rs that I started getting for some stages around 2001. I use Exact Audio Copy for those. Some CDs show essentially zero errors, but about one in 10 or 15 are completely unreadable. Again, it doesn't seem to be age related.

Thanks, Pat

[kozikowski]

when I use Audacity to record the same DAT twice with dithering off, I should indeed expect to see two bit-identical files.

No. When you open up a music file -- not capture a DAT recording -- you can expect bit tracking. As we repeatedly point out, Audacity is a complete slave to the computer and the environment. Nowhere is it written that you can't have transfer errors between the DAT machine and the Mac.

Music CDs have the same technology as wet process color photographs. Don't leave either one in the sun or office lighting for along time. They even age when protected in a file cabinet. Average success depends on the quality of the burn and the quality of the blank media. You can do a lot worse than average. Never use blanks with the words "Hi Kwality" or "HiQ" in the title.

Audio systems as a general group are designed to be sloppy. If you burn a Music CD on your Mac, it's a lot quicker than if you forced the Mac to do a Data CD recording of the same work. In the later case, it optimizes the burn and then goes back and checks it.

Koz

[billw58]

I've never tried to do what you're doing, so I'm kind of working in thought-experiment land.

Potential sources of error (in no particular order):
1) Transmission over the coax S/PDIF cable (does the S/PDIF format include error-correcting codes in the transmission frames?)
2) Coax to optical conversion (unlikely)
3) Coax to Firewire conversion (unlikely)
4) Transmission over the Toslink cable
5) Transmission over the Firewire cable (impossible?)
6) DAT interpolation
7) Mac digital input to Audacity input (unlikely?)

Do we know that the error rate displayed on the DA-20 involves level-2 errors?

While I commend your diligence, I think we can agree that due to level-2 errors in DAT playback you will never get a bit-for-bit copy of the DAT recording. The LSB errors you are seeing are certainly inaudible. They are definitely below the ambient noise of the recording environment. If I were you, I would get on with the task of dubbing the DATs.

-- Bill

[PatMcGee]

Hi Bill, Thanks for the comments.

I like your list. I think sources 1-5 in your list are unlikely in this case, mostly because the errors always show up in the low order bit(s) and random errors in those places would probably show up as errors in all the bits, not just the low-order ones.

I also thought of another possible source: 8) Mac OS X Driver. (Unless you meant to include this in your #7). I know that the output driver has a mixer and I'm making the guess that it has both potential resampling and dithering. Based on what I have found out recently, I think the input driver does neither of these things, but I don't know that.

I think we don't know for sure that the error rate display on the mk II indicates level-2 errors, but I'm going to make the guess that it does. As a user, that's what I'd want to know. Probably includes level-3 also, but those are easily audible.

I'm going to this trouble because these files will go into the collection at the Library of Congress. After that, no one will have access to the physical tapes again, as the tapes don't belong to the LoC. So I'm trying to make the transfers as flawless as possible. I have to admit that I really wouldn't go to this effort for anyone else.

I've found that, once every few tapes (10? 5? 20?), there's an audible dropout in the transfer. Many times, when I play that same tape again, there's no dropout at that point. So I guess that a stray piece of dust or maybe just random physical tape movement caused a level-3 issue. I can usually hear these, but only if I listen intently for the whole two hours. What with hundreds of tapes to do, I'd prefer to find another way to do QC on them. Thus my idea of transferring twice and comparing files. (If something is wrong at the same place both times, then I think replaying a third time is unlikely to help, so I wouldn't bother.)

I hoped to get bit-identical files because 'diff' will compare files and show differences at the bit level. If I can't get that, then I need to write a program that will compare files but ignore small differences. With such a program, only the level-3 issues would show up, which is just what I want. I'm just about to give up on trying to get bit-identical files and do this, but I want to push just a little bit more before I do.

The thing that keeps me from giving up right now is not understanding why I see very different error rates between that reported by the mk II and the difference samples in the files. Since I don't understand that yet, I hold out the forlorn hope that I can find out and then make a change that will make things better.

Thanks, Pat