< Back to IRCAM Forum

MuBu and JavaScript + project general overview

MuBu
#1

Hi everyone,

I’m a bit in a swamp with a project I’m working on and I figured I could try to get some help here.

The project I’m working on is for my master’s degree and, ideally, should be a device/patch that continuously listens to the local environment it’s placed in, does an analysis of the signal that’s coming in and decides if there’s something “worth” saving, based on the values of certain audio descriptors. If there is, it would save it and categorize it in pre-defined classes (I’m talking more about “compositional” classes than “acoustical” classes, so I don’t really care if it’s “human voice” or “dog barking”, more “background drone”, “foreground element”, “iterative”, ecc…). Each of these classes would then have their own rules for processing, spatialization, and so on (I’m planning to use JavaScript for the rules-related aspects).

That’s the general idea. It seems like MuBu can do pretty much all the things I need, and I started patching, hacking some of the help patches and some of the recent tutorials made by Matthew Ostrowski (thanks a lot for those by the way!!).
I’m currently using a few pipo modules (pipo.descr, pipo.onset, ircam descriptors, I’m possibly going to use pipo.mfcc) to analyze the incoming signal and to save the data in a mubu. So far so good, even if I’m starting to wonder how can I manage the fact that the different analysis work on different sample rates. (I guess I could timetag everything but I’m not sure that would solve the issue. Maybe first using an onset segmentation and then running the analysis on the different chunks would be better? Then I could choose a few descriptors I’m interested in, calculate let’s say the median value for the duration of the chunk, and categorize that chunk in the “right” class).

Anyways, at the moment I’m simply trying to find out a way to send the values that come out of the pipo analysis to a javascript that checks the analysis values and puts the segments in the proper classes. I’ve seen you can dump a mubu.track content as a list, so that might work, but I’m not sure that’s the best way to do that. I also noticed that there are a few js objects in the pipo section (if I’m not mistaken), but it’s not clear to me how they work.

And obviously, while the idea still seems good and interesting to me, now that I’ve started patching I’m a bit puzzled about it all and I’m not even really sure it’s feasible (for example, how could it constantly record and analyze and save the proper chunks? Using a ring buffer is the first thing that comes to mind, but again I’m not really sure about it all).

So, for now I guess the real question would be what’s the best way to output values of data tracks from an imubu to a java script patch in order to be able to analyze and sort them?

I hope this makes some kind of sense and sorry for the super long explanation, I’m just thinking out loud for the moment as this helps me to formulate the many issues I’ll halve to solve. I’d be really happy with even just a general brainstorming, as I think the general prospect of the project is pretty good but I’m really struggling with its implementation.

Thanks again!

Tobia

#2

Hi Tobia, what a great project!

Let me ask some questions first:

  • how much information you need for deciding the classes?
    1. just the segment statistics (mean / max / stddev of descriptors)?
    2. full segment descriptors?
    3. the segment audio?
  • are you mainly interested in detecting events?
  • do you think you might need a self-adapting detection (thresholds kind of following what’s going on)?

If it is 1. above, the pipo.js module could output an additional class column in one unique pipo chain. Otherwise, loop through one or several mubu tracks in js to analyse it as you wish.

In general, look at mubu.process with @process 1, which will run continuously as soon as there’s enough data. Instead of a complex pipo chain, you can also have several mubu.process depending on each other, to keep intermediate data to see what’s going on.

Cheers,
Diemo

#3

Hi Diemo, and thank you so much for the quick reply!

Yes, the project is great, perhaps even TOO great, as the whole process of implementing it seems a bit daunting!

To answer your questions:

  • I think the segment statistics would be just fine. I could set ranges for the different values and use those to classify the segments. I don’t really see the need to have a more in-depth use of each descriptor.
    At the moment I’m thinking of using just a few descriptors (Onset segmentation? Loudness? Spectral spread/skewness/etc? Zero Crossing Rate? Maybe the first mfcc values?. Especially since I’m in the first stages of building the patch, I think I will focus on just one or two descriptors for now and then I’ll choose what else to use once I know everything is working and by doing some trial and error).
    I will DEFINITELY need to also save the segment audio, as the whole idea is to have a sort of constantly-on “field recorder”, resulting in a bunch of in-situ recordings organized into classes and in a set of environmental data which could then be turned into an algorithmic composition or used as a starting point for a more live approach.

  • I guess I am, but I’m not sure what the alternative would be to be honest. The main problem I can see by focusing strictly on the events is that, if the environment is quiet and there’s not so much going on, focusing only on sudden increases of loudness or energy would probably mean losing a lot of potentially interesting material.

  • That’s something that I have thought of, yes. I would say that, for now, it’s really not that important, and I would be extremely happy to just have the whole system in place and working. But in a further iteration of the project yes, having the system adapt itself to the environment would be really cool! (One thing I forgot to mention, I’m also planning to use a bunch of sensors to collect data from the environment, which could then be used to guide some of the functions in the patch).

I’ll have a look at the other things you mentioned in your answer, and the idea of having several mubu.process instead of a long pipo chain is really good.

Thanks for now!
Tobia