Onset detection based on mel flux

[sirnotto]

I have not yet concluded which would be a best compromise for capturing many different kinds of material. But for now:

• The standard FFT is quite reliable, but tends to underperform when there is no clear transient in the onset.
• The yin works very well for pitch related transitions, but tends to skip some onsets where there is no pitch change.
• The mel seems to works better than FFT on continuous pitch changes, but not as good as yin. However, mel captures more onsets that are not related to pitch change than does yin.

[ctrapani]

Another example here, with REALLY encouraging results via MEL separation: A passage of overlapping notes and diads on Fender Rhodes

You can see that the YIN analysis is useless for segementation, and FFT gets some attacks but not all — but the mel flux segmentation does the job very well:

[sirnotto]

Wow, that is encouraging!

[schwarz]

Hi Notto and Chris, fantastic thread, thanks for the amazing contributions!
I mean to write a “segmentation lab” patch to compare several methods, can I base it on your great patch, @ctrapani?
I’d also like to add numeric analysis of segmentation quality based on human ground truth. Would you mind sending me a hand-corrected “perfect” segmentation of your respective example sounds?
Cheers!

[sirnotto]

Hi Diemo,

Happy to know the thread is useful to you. Here is my ground truth for the two flute examples.

Octave arpeggio:
Onset markers at 0, 590, 1020, 1495, 1920, 2370, 2820, 3270, 3690, 4120, 4580, 5020, 5410, 5870, 6310

This example has a final breath intake at around 6850 which the mel onset interpreted as a new final onset. I would say it is not necessarily “wrong” to isolate the inhalation as its own onset, but with the flute note still reverberating in the background it makes more sense to ignore this.

Same notes repetition:
Onset markers at 0, 400, 620, 880, 1140, 1380, 1640, 1920, 2170, 2660, 2890, 3150

[schwarz]

Thanks! If you still have it in the patch, you can just do writetrack and save as sdif or txt.

[schwarz]

I looked into some details of onseg:
First, it is astonishing that it works so well, even without using the abs delta. You can actually compare this with onseg.odfmode square or rms which use the square difference to the median.
Second, the sum hides a lot of information from onseg. Without it, each band’s difference to its median is taken and then summed (onseg.numcols starting at onseg.colindex).
More to come…

[ctrapani]

Dear Diemo — Of course, it is fine with me to use me patch as a starting point.

Here is the .mubu and a screenshot of adjusted Mel parameters for the Rhodes example.

Rhodes-Mel-segmentation.mubu (1.0 MB)

Regarding yin segmentation: Actually I am surprised that the flute example works so well because I thought that onseg was only looking for upward changes when pointed to the frequency column (and thought I had verified that!).

Can I propose a tougher example? I have trouble with this one and would be curious to know if either of you have suggestions or success:

[sirnotto]

Hi Chris,

That last one was tough! I didn’t get satisfactory results with any of the detectors.

Right now, I’m on the verge of going full circle and concluding that, for all its flaws, the standard FFT detector is the best compromise when dealing with large amounts of audio. Because the yin and mel based detectors are more specialised for dealing with the type of onsets which FFT can’t handle.

Wouldn’t it be great to have a machine learning algorithm that could decide which algorithms to use based on categories of input? Now that’s a project…

[ctrapani]

Thanks for having a look. Yes, this is where it gets complicated… Maybe another option, more within reach than machine learning, would be some system that performs different types of analyses and compares results - confirming clear onsets when data points coincide or rejecting those that seem not to work.

For me, the ideal would be to somehow replicate the slider in AudioSculpt:

Of course this is kind of equivalent to changing the threshold for onseg segmentation, but the dream would be a slider that allows one to move between combined (weighted!?) data from FFT / YIN / and spectral differencing analyses…

[schwarz]

Yep, that’s called fusion. Would it be a workable first step to combine all of the segmentations into one stream? In your experience, does yinseg catch stuff which loudness doesn’t, or would it produce too many false positives for too many sound classes?

[sirnotto]

Based on my limited experience, yin tends to capture less overall than loudness, but sometimes it captures continuous pitch changes which loudness ignores.

The mel detection algorithm, on the other hand, as per now, is more tricky to control. There is a very fine line (dependent on threshold) between capturing many false positives and nothing at all.

[ctrapani]

Exactly: yinseg can catch slurred or continuous notes better than loudness detection. But yinseg is not useful if there is contrapuntal motion, for instance, but mel can catch changes in individual voices.

If we can get all segmentations into one stream, that would be helpful, but weeding out the false positives will be tricky.

One idea might be to create a series of defined presets: “midrange percussive,” “treble legato,” “contraupuntal” that would give different weights to the various detection algorithms.

[schwarz]

So here is a first version of the Segmentation Laboratory:

Here you can run 4 segmenters, and look at the data in various processing steps. It should use only features in v1.9.14, but we found that the editor is kind of stubborn, so you’ll have to resend the view setup quite often.

segmentation-lab.maxpat (244.0 KB)
attrchanged.js (1.1 KB) [EDIT: this file doesn’t seem to be downloadable as .js, here is its source:]

// attrchanged.js: listen to change in obj of given list of attrs

function valuechanged (data)
{
    if (data.attrname)
    {
	outlet(0, data.attrname, data.value);
    }
}

function loadbang ()
{
    this.bang();
}

function bang ()
{
    if (jsarguments.length < 2)
	error("first argument must be scripting name of listened to object");

    var ob = this.patcher.getnamed(jsarguments[1]); // get the object in 1st arg
    var obattrs = ob.getattrnames();
    var listento = [];
    var listeners = [];

    if (jsarguments.length == 2)
	listento = obattrs; // no attr arg: listen to all
    else
	listento = jsarguments.slice(2);

    for (var i = 0; i < listento.length; i++)
    {
	if (obattrs.indexOf(listento[i]) >= 0)
	    listeners[i] = new MaxobjListener(ob, listento[i], valuechanged);
	else
	    error(ob.maxclass +" object "+ jsarguments[1] +" does not have attribute "+ listento[i]);
    }
    gc();
}

Have fun!
…Diemo

[sirnotto]

Very nice work! This should be fun

[ctrapani]

Thanks, Diemo! Strangely I can’t download "attrchanged.js"from the forum site; I get this odd error message:

Perhaps you could upload a .zip (or email me an attachement)? Thanks and more soon

[schwarz]

must be some policy to disallow .js, I put the short source code with the patch.

[schwarz]

Here’s a new version of segmentation-lab with more stats and hz to MIDI scaling of yin before the delta: seglab v2.zip (28.4 KB)

Next version will use new features and fixes of mubu 1.9.15.

[ctrapani]

Thanks, Diemo. I am finding myself with less time than I’d hoped to dig into this right away, but I am very much looking forward to it — C

[beller]

Hi guys, why not using a project at this point ?
You can git it and do versionning…

Let me know if I can help