Structural problems with the SDIF file written by MUBU

jojocello · February 5, 2021, 7:31am

Hello,

I have an SDIF file containing the motion data of a cellist with its instrument (https://we.tl/t-NcJSFjf0X2).
This file has been written from a MUBU container containing 2 tracks :

markers (the X,Y,Z coordinates of each marker) with the info table
SDIF.StreamId 1 SDIF.FrameSignature XMMK SDIF.MatrixSignature XMMK SDIF.MatrixName Markers
rigidbodies (the X,Y,Z,Yaw,Pitch,Roll of the bow and the cello) with the info table
SDIF.StreamId 2 SDIF.FrameSignature XMRB SDIF.MatrixSignature XMRB SDIF.MatrixName Rigibodies

I’m facing two problems with this MUBU-generated SDIF file :

1/ I cannot read the SDIF file with other softwares.

Actually, the frame/matrix SDIF structure seems inconsistent regarding the second stream (XMRB).
I tried the SDIF-Edit program (https://github.com/j-bresson/SDIF-Edit/releases) and PYSDIF3 Python library (pysdif3 · PyPI)… Both give the same error at reading :
*Sdif* Warning 20: SdifFile: data/cellist.sdif (byte: 1224=0x04c8=0002310), TextFile: (null) FramH : XMRB Size: 0x0050 NbMatrix: 1 NumID: 2 Time: 0.0020466 MtrxH : XMRB DataWidth: 0004 Rows: 2 Columns: 6 --> Matrix is not in frames components : XMRB

2/ I cannot add a 1NVT table in the SDIF file.

Actually I’d like to get the matrix column names of my 2 SDIF streams and I thought to do it by adding two key/value properties in the 1NVT table produced by the MUBU container… Something like that :

marker_names = « head_x, head_y, head_z, etc… »
rb_names = « bow_x, bow_y, bow_z, bow_yaw, bow_pitch, bow_roll, etc… »

But I don’t know how to do this with MUBU (whereas it was quite straightforward with the ftm.sdif.write object).
I tried to add my key/value pairs in the info table of the buffer containing my MUBU tracks, but without more success… This buffer info table doesn’t seem to be exported as a 1NVT table in the resulting SDIF file.

Please would you have some suggestions regarding these two MUBU/SDIF issues ?

Best,
Jojo

schwarz · February 5, 2021, 8:44am

Hi Jojo, what a great power-user question!

Ad 1: the file wrongly defines types which use the XMMK Matrix in the XMRB frame, instead of the XMRB Matrix, although it is written correctly with #2:XMRB/XMRB. Can you post an extract of your patch that shows how the SDIF.* info is set? You can also post the .mubu you get with writeall.

Ascii chunks of file /Users/schwarz/Downloads/cellist.sdif:

1TYP
{
  1MTD  XMMK    {x, y, z}
  1MTD  XMRB    {x, y, z, yaw, pitch, roll}
  1FTD  XMMK
        {
          XMMK  Markers;
        }
  1FTD  XMRB
        {
          XMMK  Markers;
        }
}

Data in file /Users/schwarz/Downloads/cellist.sdif (6108992 bytes):
11397 XMMK frames in stream 1 between time 0.002047 and 94.963730 containing
   11397 XMMK matrices with  34 -- 34 rows,   3 --  3 columns
11397 XMRB frames in stream 2 between time 0.002047 and 94.963730 containing
   11397 XMRB matrices with   2 --  2 rows,   6 --  6 columns

Ad 2:: I thought any other entry in the track info table will be written to the files’ Name–Value Tables, but in fact, this happens only with track info SDIF.FrameSignature 1NVT, but then no data will be written…
The workaround here is to create another track specifically for the NVT info. I suspect this limitation is because NVTs can not be formally associated to a stream in SDIF.

Best, Diemo

jojocello · February 5, 2021, 10:53am

Hi Diemo, thanks !
Sure that powerful technologies inspire great questions

1/ : I send you the extract of my patch which sets the SDIF.* infos of the tracks (I’ve a little bit of Odot inside). As well as the cellist.mubu file obtained by a writeall command on the container : https://we.tl/t-oB6yHXPkjx
I checked again my track infos, and I don’t understand how the generated 1TYP struct can reference the XMMK matrix within the XMRB frame. It’s weird… By the way, what tool did you use to get this raw ascii chunks of the SDIF file ?

2/ : I also tried the SDIF.FrameSignature 1NVT in a track info, but in this case actually, we cannot preserve the frame/matrix structure of the stream. Ideally, it might be nice to store additional key/value entries of the track as NVT infos of the SDIF file (especially to keep the column names). I will try your solution of additional empty track dedicated to the NVT fields. Will I be right if I format the NVT infos in this way ? :
SDIF.FrameSignature 1NVT
marker_names "head_x, head_y, head_z, …"
rb_names "bow_x, bow_y, bow_z, bow_yaw, bow_pitch, bow_roll,…"

Cheers,
Jocelyn

jojocello · February 22, 2021, 12:21pm

Hello Diemo,
Did you get my little patch and could you open the cellist.mubu file ?
I’m still hoping that a solution may be found regarding the SDIF structure written by a MUBU container with multiple tracks…
Do you think it would be too much work to adapt the MUBU code for taking into account this functionality ?
I planned to use it quite a lot in my architecture, as SDIF is a powerful way to keep synchronized informations of the real-time performance.
Or should I forget this format to prefer individual text files (one by MUBU track) ?
Best,
Jocelyn

schwarz · February 24, 2021, 9:59am

Hi Jocelyn,
sorry, I missed the wetransfer. You can attach files to posts here directly.

querysdif on the command line.

these are in the STYP header and wind up as mubu column names, but yeah, additional info can go into the NVTs as you’ve shown. You might want to use just space as separator, without comma, then Max can parse the value string into a list.
Best

jojocello · February 24, 2021, 11:23am

Hi Diemo,
Thanks for your reply.
My .mubu file is too big to be directly attached to the post.
So I provide you with a new wetransfer link : https://we.tl/t-pIgvlSqn7p
Jo

schwarz · February 25, 2021, 11:15am

Hi Jocelyn, thanks for the patch, I can confirm that there is a bug mubu SDIF export user-type definition when there are more than 1. The data is exported correctly, though, but readers will complain.
There are 2 possible workarounds until we fix this:

edit the SDIF file and change 2 bytes in the STYP chunk: from

1FTD  XMRB
        {
          XMMK  Markers;
        }

to

1FTD  XMRB
        {
          XMRB  Markers;
        }

(no one cares about the matrix name). This could be done simply with a text editor or with a sed command

use sdiftotext / texttosdif and modify the text representation in between

BTW, do you have documentation about your cello mocap project, which is very interesting?

Best,
Diemo

jojocello · February 28, 2021, 7:51pm

Hi Diemo,
Thanks a lot for the trick, it works
I didn’t find the texttosdif command in the SDIF distribution, but a tosdif which makes the job.
Just an additional remark about the contents of the SDIF file generated by MUBU : I notice that a lot of frames have the same timetag, whereas their matrix content is different. For some marker data, I counted until more than 10 occurrences of the same timestamp… Is there a reason for these repetitions or is it something that might also be corrected for the next MUBU release ?
Regarding my cello project, I’m interested in understanding how the cellists’ sound features connect to their motor control. To this aim, I explore some timbre and space deformations to assess how the musicians adapt their movements in real-time. I didn’t get some experimental results at the present time because the technical architecture is quite complex to set up, but you will find some useful details of the latter in my proceeding for the SFA 2020 congress.
Cheers,
Jocelyn

schwarz · March 2, 2021, 4:08pm

Hi Jocelyn,

I see that, too, in the .mubu file. Can you please check if the timestamps are different in the mubu container right after recording?

Best.

jojocello · March 5, 2021, 12:54pm

Hi Diemo,
Sorry for the delay, I could only get access to the mocap room today.
I’ve recorded some live data in the MUBU container (last release 1.9.16).
Unfortunately the problem of timetag occurrences persists (cf attached screen capture).
Maybe some bug to correct for the next release ?
Cheers,
Jocelyn
timetag repetitions

schwarz · March 5, 2021, 1:01pm

Hi Jocelyn, so how do you record the data? mubu.record?
If so, it is a problem on the network side which transmits data in bursts, in the same logical max time interval. You could see if the mocap system includes a time tag with the point data, and use that in an
append tt data message to mubu.track markers.
What is the expected framerate of your system?

jojocello · March 5, 2021, 8:39pm

Bravo Diemo, it seems that you’re right with the way our mocap system transmits the data on the network.
During the analysis of Optitrack data frames (Python code), I extracted the timestamp and inserted it in the OSC frame to forward to Max (with the library Osc4Py3).
The expected Optitrack framerate is 120 Hz, so Max should receive a frame each 8 ms in average (with udpreceive).
It seems actually to be correct (cf first screen capture), excepted for some moments : I underlined a reception of 5 frames in one shot after 40 ms more or less…
So the problem can come from our network or from the Python library I use to forward the frames to Max.
I will investigate this further but in any case, it seems that the writing of MUBU frames with the same timetag is caused by these Mocap streaming bursts.
How should I declare my MUBU track to manually record the timestamp I got from Optitrack (I record my data with mubu.record) ?
Should I keep the attributes « @timetagged yes » and « @samplerate 120 » (cf second screen capture) ?
Thanks a lot for your intuition of the burst, it was very helpful
Jocelyn

schwarz · March 18, 2021, 12:17pm

Hi, synchronisation is always a tricky thing. Is the optitrack timestamp the number after /frame/ and is it in ms (and is there a more precise fractional timestamp available)? It looks much more regular, with deltas of 7–9, even when it arrives in a burst. Use this time, relative to the start time, in the append message to mubu.track ... @timetagged yes @samplerate 120 and don’t use mubu.record.

jojocello · March 23, 2021, 7:01am

Hi Diemo,
Yes the optitrack timestamp is the number after /frame in my OSC packets.
This is an information I extracted from the binary data streamed by Motive (the Optitrack software).
It seems to be a kind of timestamp in ms, with deltas of 7-9 as you noticed.
Yesterday, I made the test of append messages with this timestamp information (relatively to the start recording time) and it works fine. It solved my multiple MUBU timetag occurences
Thanks a lot for your precious help.
Have a nice day.
Jocelyn