Possible bug with importing large raw audio files?

[meowsqueak]

I am generating 10 seconds of raw audio with a python script. I am using numpy to write out 480,000 float32 samples, little endian.

Code: Select all

    import numpy as np
    SAMPLE_RATE=48000
    DURATION=10.0
    audio_out = []
    # a simple ramp over the entire duration
    for t in range(int(SAMPLE_RATE * DURATION)):
        audio_out.append(t / (SAMPLE_RATE * DURATION))

    raw_out = np.array(audio_out, dtype=np.float32)
    data.astype('float32').tofile("result10.raw")

When I import it into Audacity (2.0.3, OSX 10.7.5, installed from .dmg) the data is scrambled badly. But if I reduce the number of samples by a factor of 10 (by reducing SAMPLING_RATE, or DURATION, or both), to 48,000, then it loads correctly. There seems to be a file size somewhere above 48,000 * 4 = 192kB for which it imports files larger than this incorrectly.

Also, if I convert the raw file to a wav using sox, it also loads the resulting wav correctly:

Code: Select all

 $ sox -r 48000 -e float -b 32 -c 1 result10.raw result10.wav

Screenshots show the converted WAV alongside the imported RAW:

RAW vs. WAV (converted from RAW by sox)

Screen Shot 2013-04-22 at 12.28.35 PM.png (45.55 KiB) Viewed 1681 times

Zoomed in

Screen Shot 2013-04-22 at 12.28.59 PM.png (57.44 KiB) Viewed 1681 times

I have checked the obvious - endianness - yes, my raw files are in little-endian, and I'm selecting this when importing. I tried big-endian too, just to make sure it wasn't something simple like that. I also checked that my python code isn't producing values too small to represent with a float32, it's not.

This seems like a definite bug to me, because if I keep my files small (about a tenth of the size I actually need to use) then it imports correctly. It's only for files above a size somewhere between 192kB and 1.92MB that things break.

[kozikowski]

So that wasn't a typo. You really are trying to manage sound files with a 480KHz sample rate.
Koz

[meowsqueak]

So that wasn't a typo. You really are trying to manage sound files with a 480KHz sample rate.
Koz

If you're referring to the first sentence, it's not a typo, however it's 480k samples over 10 seconds, so that's 48kHz. When I try to import it into Audacity, I set the target sample rate at 48kHz, not 480kHz.

So I'm not doing anything weird with 480kHz audio - I'm just trying to import raw audio at 48kHz of around 10 seconds, and it's not really working in this case. I don't know what makes this case special.

[steve]

Note that the Import Raw settings are not saved, so even if you have just imported a raw 32 bit float file, the next time you go to import a raw file the sample format setting will probably be wrong and you will need to manually re-set them.
I've just tried a RAW file: 1.5 million sample 48 kHz mono 32 bit float little endian and it worked as expected (tested on Linux).

[meowsqueak]

Yes, I'm painfully aware that the Import settings need to be manually set each time . I've done this perhaps 100 times in the last 2 days so it's something I'm very familiar with now. I'm absolutely sure the import isn't working for my 480ksample file. I just checked it again, made sure I used the right settings - it's still not working.

I've attached the raw file (float32, little-endian) I'm trying to import. It is simply 480,000 float32 values starting at 0.0 and ramping linearly up to 1.0. A 10 second positive-side ramp. Not a listenable signal, but it's a sequence of valid audio samples. When I import it, I don't get a nice ramp, I get this sort of mess:

failed import (10 seconds)

Screen Shot 2013-04-23 at 9.49.41 AM.png (36.88 KiB) Viewed 1665 times

If I alter my python script to generate this same ramp over just 1 second - very simply by changing DURATION to 1.0 - then it imports fine (with exactly the same import settings):

successful import (1 second)

Screen Shot 2013-04-23 at 9.51.33 AM.png (14.59 KiB) Viewed 1665 times

I was concerned that perhaps numpy's 'tofile()' function was misbehaving with larger array sizes. However if I convert the 10-second raw to wav with sox, it looks fine to me:

Code: Select all

$ sox -L -r 48000 -e float -b 32 -c 1 result10.raw result10.wav

I can't show a screenshot as 3 attachments is the limit, but it's exactly the ramp I'd expect to see.

@steve, I'd be interested to know whether that file imports correctly in the Linux version.

[steve]

I've attached the raw file

Terrific, that's what I was going to ask you to do

@steve, I'd be interested to know whether that file imports correctly in the Linux version.

If I use the default "offset" (which comes up as "Start offset: 1 byte", I get the same result as you.
If I set the offset to zero, I get what I think you are expecting - a ramp from 0.0 to 1.0

Anticipating your next question - how Audacity decides on its default settings for Import RAW.
I don't know C++ well enough, but you can perhaps work it out from the Audacity source code: /src/import/RawAudioGuess.cpp

[meowsqueak]

Uh, *blink* *blink*... offset... that's it. For some reason I was completely blind to that parameter. I have no idea why. That's exactly the cause of the problem. Thank you for pointing that out!

I think it would be better if Audacity didn't try to guess the RAW parameters (it always gets it wrong in my experience), but either presented the last-used or exact same default parameters each time (so that changing them is manual but easily repeated as it's the same procedure each time). I think I've been caught out here by not noticing that offset was automatically changing between files. I feel rather sheepish now, and I've also wasted other people's time as well as a huge amount of my own on this. My apologies.

[steve]

I feel rather sheepish now,

You shouldn't. I also missed it at first, but I'm probably more familiar with tracking down such problems in Audacity.

I think it would be better if Audacity didn't try to guess the RAW parameters (it always gets it wrong in my experience), but either presented the last-used or exact same default parameters each time (so that changing them is manual but easily repeated as it's the same procedure each time).

The important word in that is "guess". As these are headerless I don;t think that it can ever be much more than a guess.

If you have time to do so it may be worth raising this suggestion on the "Adding Features" board ( http://forum.audacityteam.org/viewforum.php?f=20 ) so that the idea can be discussed.

[Gale Andrews]

I think it would be better if Audacity didn't try to guess the RAW parameters (it always gets it wrong in my experience), but either presented the last-used or exact same default parameters each time (so that changing them is manual but easily repeated as it's the same procedure each time).

The important word in that is "guess". As these are headerless I don;t think that it can ever be much more than a guess.

If you have time to do so it may be worth raising this suggestion on the "Adding Features" board ( http://forum.audacityteam.org/viewforum.php?f=20 ) so that the idea can be discussed.

I'll just add the vote sometime as a discussion wasn't started.

The most popular idea seems to be to save and load presets.


Gale

[steve]

I'll just add the vote sometime as a discussion wasn't started.

Shame the discussion wasn't started, but thanks for registering the vote.

The most popular idea seems to be to save and load presets.

I'm not sure how you could have presets, as most combination are equally valid. The most common data formats are well defined (such as Microsoft PCM WAV), but also have headers, so importing as RAW is not required. I can see the point in saving user presets, as users of Import RAW are likely to use the same, very similar, or small selection of settings each time. I'd have thought that in many cases, saving the last setting used would be sufficient.

I tend to agree with meowsqueak that the auto-detection is often more trouble than it's worth. My vote is that auto-detection could be enabled or disabled and when disabled the last setting used would be saved.