snd-xform BUG!!

[Paul L]

I have been trying to write an effect that first cuts the input sound into ranges, then transforms certain of the pieces, then reassembles changed and unchanged pieces into a sequence.

I use snd-xform to select ranges of time and shift them to 0:

Code: Select all

(defun extract-global (snd start end)
  (let* ((t0 (snd-t0 snd))
	 (result
	  (snd-xform snd (snd-srate snd) 0 (- start t0) (- end t0) 1.0)))
    result))

This should have the effect of making a little sound that "throws away" samples outside of the range.

Then I have a list of these little sounds, some of them further transformed and some not, and then I use shift-time (which is a thin function around snd-xform if you look at it with grindef) to put the pieces back in place and then I sum them.

It appears that some, but not all, of the un-transformed sounds improperly "remember" cycles that the first snd-xform should have cut off. And strangely, it looks like the end of the lengthened piece is not the original end of the selection, but what should be the end of some other one of the unchanged pieces. I suspect that when these twice snd-xform'd sounds evaluate, and they share the original sound in their data structures, there is some side effect on the original sound causing bad interactions.

I found that I can work around this by (1) returning

Code: Select all

(prod -1.0 result)

instead from the function above and then (2) negating the final summation of pieces. I found that I do NOT get a successful workaround, if I just return

Code: Select all

(prod 1.0 result)

or even if I try

Code: Select all

(prod -1.0 (prod -1.0 result))

which makes me suspect that Nyquist is trying to do some "clever" simplifications of the sound objects before evaluating them, like detecting that multiplying by 1.0 is identity. But I suspect the cleverness goes wrong somewhere when sounds are supposed to be clipped by snd-xform, and I can foil this cleverness by negating the sound.

This explanation might not be clear, I haven't found a simpler example that you can do in Nyquist prompt.

[Paul L]

Here's an easy Nyquist prompt experiment. It reveals that the bug depends not only on how my extracted sounds are defined and transformed, but also, THE SEQUENCE IN WHICH THEY ARE THEN PASSED TO sum.

First make a sine wave of 100 Hz, 1 second duration, 0.25 amplitude, and view it as Waveform. Select all and do this in Nyquist prompt.

Code: Select all

    (defun extract-global (snd start end)
      (let* ((t0 (snd-t0 snd))
        (result
         (snd-xform snd (snd-srate snd) 0 (- start t0) (- end t0) 1.0)))
        result))

(let* (

(s1 (extract-global s 0.5 0.51))
(s2 (extract-global s 0.6 0.61))
(s3 (extract-global s 0.7 0.71))

(t1 s1)
(t2 s2)
(t3 s3)

(result (sum t1 t2 t3))

) (sum result (s-rest (get-duration 1))))

That picks out three cycles of the sound from various times, shifts all of them to time 0, and sums them with a silence that leaves the length of the selection unchanged. So, one cycle at 0 with amplitude 0.75 as expected.

Change three lines to this:

Code: Select all

(t1 (shift-time s1 0))
(t2 (shift-time s2 0))
(t3 (shift-time s3 0))

And the effect is the same, as expected. Now try this:

Code: Select all

(t1 (shift-time s1 0.5))
(t2 (shift-time s2 0.5))
(t3 (shift-time s3 0.5))

The lone tall cycle is at 0.5, as expected. But now try different shifts.

Code: Select all

(t1 (shift-time s1 0.1))
(t2 (shift-time s2 0.2))
(t3 (shift-time s3 0.3))

Three separated cycles. Still unsurprising. But now make another change:

Code: Select all

(result (sum t3 t2 t1))

There's the bug! The sequence of evaluation of these sounds seems to matter. It looks like t1 is as it should be, but t2 ends where t3 should end, and t3 overlaps t2 for one cycle, and it ends... where the extract s2 began!

If the arguments to shift-time are all 0.5, then the result is correct with either sequence of arguments to sum. Or if you take the buggy version and change each of the t's to (prod -1.0 (shift-time ... ) ) then the behavior is as expected, no matter which sequence in sum.

[steve]

[Update] This was written in response to the first post (you posted again while I was typing).

Normally you would use sound transformations rather than using snd-xform directly, so here's a simple example of the same "bug" using the "at" and "extract" transformations.

Code: Select all

(setf s1  (extract 0 0.5 s))
(setf s2  (extract 0 0.5 s))

(sim
  (at 0 (cue s1))
  (at 1 (cue s2)))

One would probably expect that the result should be the first half of "S", followed by silence for half the duration of S, followed by the first half of S again, but that does not happen

Although we appear to have created two new sound objects, S1 and S2, these are really just pointers to S.
Also. Nyquist uses lazy evaluation, so S1 and S2 are not evaluated until the end - we can prove this by modifying the code a little:

Code: Select all

(setf s1  (extract 0 0.5 s))
(setf s2  (extract 0 0.5 s))

(sim
  (at 0 (cue s1))
  (at 1 (cue s2)))
(print "Hello World")

If we run this code with a very long selection, and watch the computer resources, we will notice that RAM and CPU usage are virtually nothing. S1 and S2 are not evaluated at all in this second example!

We know that "extract" returns the sound between "start" and "stop" relative to the current *warp*, but because, in our first example, it is not being evaluated until the end, "stop" is not known until after the expression has been evaluated.
There is a brief note in the manual regarding this:

Note: Due to some internal confusion between the specified starting time and the actual starting time of a signal after clipping, *stop* is not fully implemented.

One way that we can work round the ambiguity of the stop time is to force S2 to be evaluated before evaluating the final expression, for example:

Code: Select all

(setf s1  (extract 0 0.5 s))
(setf s2  (sum 0 (extract 0 0.5 s)))

(sim
  (at 0 (cue s1))
  (at 1 (cue s2)))

Another way is to explicitly define "stop" in the final expression, for example:

Code: Select all

(setf s1  (extract 0 0.5 s))
(setf s2  (extract 0 0.5 s))

(extract 0 1.5
  (sim
    (at 0 (cue s1))
    (at 1 (cue s2))))

[Paul L]

You are not trying to tell me this bug is not a bug?

I found an example where the sound returned by sum depends on the sequence of the summands. If I am encouraged to think about sound objects in mostly "functional programming" terms (so long as I don't use snd-fetch, snd-fetch-array, or snd-fft), then this is certainly not expected behavior. Nor do I understand how the behavior with the summands scaled by -1.0, and without the scaling, can both be correct.

[Paul L]

I tried transforming my t1, t2, t3, with (sum 0 ...) and that does fix the "bug." But (prod 1 ...) does not fix the "bug." Justify that behavior.

Can we drop the quotation marks from "bug" please?

[steve]

Can we drop the quotation marks from "bug" please?

I was quoting your use of the term, but I'm not convinced that it is a bug (and certainly not a capitalised BUG )

I totally agree that there is some very confusing behaviour when using snd-xform related functions, but the whole issue of *warp*, Environment, and Behaviours is rather confusing, and more so due to the way that Nyquist interacts with Audacity.

One example of a related behaviour that I think IS a bug can be demonstrated with the following code (applied to a stereo track):

Code: Select all

(vector (aref s 0)(aref s 0))

but this is a bug in Audacity rather than in Nyquist.

By the way, regarding the issue that you raised about Audacity freezing when Nyquist returns an empty array; I submitted a patch that has now been committed so that returned arrays are validated. In the current SVN alpha version, returning an empty array will generate an error rather than freezing - this will be in the next release version of Audacity. I've not got a fix for the issue with negative sample rates, but I also raised that on the developers mailing list and Roger Dannenberg has said that he'll take a look.

[Paul L]

You have not persuaded me that there is a sensible user model justifying all of this observed behavior. Explain why wrapping terms in (sum 0 ...) fixes the "bug" but (prod 1 ... ) does not. I do not see a coherent user model here.

I find the global variables and environments and the local/global time distinctions more annoyance than help, so I am trying to go around them to the documented behavior of the snd- functions whenever I can to get done what I need done. And I tell you snd-xform is not behaving as documented if it has these problems with order of evaluation.

With the function grindef in the Nyquist prompt, you can see how extract and cue and cue-sound and shift-time are implemented in terms of snd-xform. There is nothing in extract that sets the global variable *STOP*. There is only a use of the parameter called STOP. You confuse these. Your last example, while hacking around the bug (bug, BUG!!! I say) in snd-xform, does not correctly explain WHY it works. I say it's a mysterious accident that it works, just as it's a mysterious accident that (sum 0 ... ) works and (prod 1 ... ) does not, none of it explicable by the manuals.

I thought the global variables and environments that I avoid using affect only the layer of stuff written in LISP that I can dump with grindef, and not the internals of the snd- functions.

[Paul L]

I see that cue calls cue-sound which uses *START* and *STOP* which default to minus and plus 10^21.

Extract does not use them. shift-time uses different variables called MIN-START-TIME and MIN-STOP-TIME (no stars in names) as infinities to pass into snd-xform. They have the same default values as *START* and *STOP*.

None of these functions works by MODIFYING those globals.

Again this suggests that that these globals are used to give values to snd-xform, but are not supposed to influence what happens inside snd-xform or what happens at the later time of evaluation of the samples.

I understand snd-xform creates an object embodying a rule for lazy evaluation of samples. I understand that the rule is supposed to be completely well defined at the time snd-xform returns and is not supposed to change at the evaluation time. What I observe fails to conform to that. I don't believe what I am observing is sensible intentioned behavior and not a bug.

[steve]

OK, let's take a different approach.

The source code says:

snd_xform -- return a sound with transformations applied.

The "logical" sound starts at snd->time and runs until some
as yet unknown termination time. (There is also a possibly
as yet unknown logical stop time that is irrelevant here.)
The sound is clipped (zero) until snd->t0 and after snd->stop,
the latter being a sample count, not a time_type.
So, the "physical" sound starts at snd->t0 and runs for up to
snd->stop samples (or less if the sound terminates beforehand).

The snd_xform procedure operates at the "logical" level, shifting
the sound from its snd->time to time. The sound is stretched as
a result of setting the sample rate to sr. It is then (further)
clipped between start_time and stop_time. If initial samples
are clipped, the sound is shifted again so that it still starts
at time. The sound is then scaled by scale.

To support clipping of initial samples, the "physical" start time
t0 is set to when the first unclipped sample will be returned, but
the number of samples to clip is saved as a negative count. The
fetch routine SND_flush is installed to flush the clipped samples
at the time of the first fetch. SND_get_first is then installed
for future fetches.

An empty (zero) sound will be returned if all samples are clipped.

If I run the code (as a test example):

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 0)
      (start 1.25)
      (stop 3.5)
      (scale 1))
  (snd-xform sound sr time start stop scale))

then SND-XFORM behaves exactly as advertised. Do you agree?

Now what happens if we change "time"?
This gets a bit peculiar in Audacity because the start time for returned sounds is always zero. (Note that Nyquist was written as a standalone programming language that was shoehorned into Audacity to provide a simple but powerful tool for rapid development of experimental plug-ins).

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 1.25)
      (stop 3.5)
      (scale 1))
  (snd-xform sound sr time start stop scale))

In this case, the start time is shifted by Nyquist to 2.0, but when the sound is returned to Audacity, the start time is zero. The act of returning the sound to Audacity has shifted the sound backward to zero.

We can counteract this shift by giving Audacity a "time = 0" reference:

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 0)
      (stop 5)
      (scale 1))
  (sum (s-rest 1)
  (snd-xform sound sr time start stop scale)))

Now we get the expected behaviour.

As an example - generate a 10 second "Chirp" and apply the above code and the result is (as expected):

tracks001.png (15.52 KiB) Viewed 2925 times

Let's modify the code a little:

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 0)
      (stop 5)
      (scale 1))
  (sum (s-rest 0.5)
    (snd-xform sound sr time start stop scale)))

Again we get the expected result:

tracks002.png (15.59 KiB) Viewed 2925 times

Now let's shorten the "s-rest" a little more:

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 0)
      (stop 5)
      (scale 1))
  (sum (s-rest 0.4)
    (snd-xform sound sr time start stop scale)))

tracks003.png (15.84 KiB) Viewed 2925 times

We now have an appearance of "the bug"!
We expected that "sound" would stop at 5.0 seconds because we tried to clip it to 5.0 seconds. When we did this without SIM it worked fine, but for some reason SIM has messed it up! Why?

Simultaneous Behavior
http://www.cs.cmu.edu/~rbd/doc/nyquist/part4.html#27

Strictly speaking, this is wrong!
SIM returns a sound which is the sum of the given behaviours and as the manual stresses "sounds are not behaviors!"- usually we can get away with it, but not this time.

What happens if we (correctly) use CUE in the last example?

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 0)
      (stop 5)
      (scale 1))
  (sum
    (cue (s-rest 0.4))
    (cue (snd-xform sound sr time start stop scale))))

tracks005.png (15.47 KiB) Viewed 2925 times

It's not what we wanted, but it has done the right thing. SIM (or SUM) plays both sounds starting at the same time.
What we actually wanted was to start "sound" at 2.0 seconds, so the "correct" code would be:

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 0)
      (stop 5)
      (scale 1))
  (sum
    (cue (s-rest 0.4))
    (at-abs 2 
      (cue (snd-xform sound sr time start stop scale)))))

but this will not work!
We want snd-xform to set the start and stop time of "sound", but we are overriding it with (SIM (AT-ABS (CUE .... so snd-xform does not get evaluated.

To get this to work (and still be "correct") we need to evaluate SND-XFORM using the start time and stop time of "sound".
If we use SIM (or SUM) with a number and a sound, SND-OFFSET is used to perform the operation, and this will then evaluate our SND-XFORM function using the start time and stop time of "sound".

Code: Select all

(let ((sound s)
      (sr *sound-srate*)
      (time 2)
      (start 0)
      (stop 5)
      (scale 1))
  (sum
    (cue (s-rest 0.4))
    (at-abs 2 
      (cue (sum 0 (snd-xform sound sr time start stop scale))))))

[Paul L]

I agree that sim is another name for sum.

I agree that your fifth example is the first one that surprises me.

I do not agree that what you then link to explains the behavior. In that example the "surprising" but explicable behavior involves seq, not sim. The "surprise" was that the actual start of one of the sounds is not delayed as expected. The surprise in our example is that the ending time of the extract is later than expected, not that the start time is earlier than expected.

I gave you an example in which the sequence of the arguments given to sum had an effect on the result. This is still not explained.

I gave an example in which the "bug" as I define it can be "fixed" by negating each summand with (prod -1.0 ...) and then applying the same to the sum. No (sum 0 ...) is used. I do not understand why this "works."