Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart...

Motion–Sound Mapping by Demonstration

Jules Françoise

{Sound Music Movement} Interaction

UMR STMS

IRCAM, CNRS, UPMC

Catherine Achard

Olivier Chapuis

Thierry Dutoit

Rebecca Fiebrink

Sergi Jordà

Marcelo Wanderley

INRIA, Orsay

U. of Mons

Goldsmiths U. of London

UPF, Barcelona

U. McGill, Montreal

(examinatrice)

(examinateur)

(Rapporteur)

(examinatrice)

(examinateur)

(Rapporteur)

Advisors:

Frédéric Bevilacqua (Ircam)

Thierry Artières (UPMC, AMU)

phd defense — Ircam — 18/03/2015

Augmented Violin

Pedagogy

New Interfaces

Gaming

Music Games

May 13, 2011

For$a$Violinist,$Success$Means$A$New$Low$Point$By MATTHEW GUREWITSCH$

Librado Romero/The New York Times More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of �subharmonics,� the violinist Mari Kimura has been exploring the new horizons opened up by an electronic motion sensor, a half-glove equipped with electrodes that monitor the angle and speed of her bowing arm.

SINCE!Pythagoras,!musicians!and!scientists!have!known!(or!thought!they!knew)!that!the!lowest!pitch!a!string!stretched!taut!can!produce!�!the!fundamental!�!is!the!pitch!it!�� d!the!vibrating!section!sounds!at!a!higher!pitch.!Theoretically,!there!is!no!ceiling.!As!with!�� !

For!the!last!five!centuries,!give!or!take,!the!range!of!a!violin!bottomed!out!on!the!G!below!middle!C,!the!pitch!of!the!open!G!string.!But!fiddlers!are!not!like!dancers!after!all.!For!nearly!two!decades,!the!Japanese!violinist!Mari!Kimura,!48,!has!been!exploring!unsuspected!subterranean!sounds!as!much!as!an!octave!deeper.!!

Complementing!the!familiar!concept!of!harmonics!(pitches!drawn!from!the!overtones!of!a!given!fundamental),!Ms.!Kimura!has!named!her!freshly!discovered!sonorities!�� astered!the!subharmonic!octave,!third,!second!and!

Movement Learning

Music Rehabilitation

Background | {Sound Music Movement} Interaction team

Motion & Sound

Relationships

mapping

outline

Related Work

Mapping-by-Demonstration

Probabilistic Movement Models

Probabilistic Models for Sound Parameter Generation

Playing Sound Textures

Motion–Sound Interaction through Vocalization

Background & Concepts

Models

Applications

outline

Related Work

Models

Applications

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

Related Work | motion-sound mapping: from wires to models

feedback

Hunt, A., Wanderley, M. M., & Kirk, R. (2000). Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference.

Explicit Mapping

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

feedback

Motion parameters directly wired to sound control parameters

Hunt, A., Wanderley, M. M., & Kirk, R. (2000). Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference.

Implicit Mapping

Use an intermediate model of interaction

> Physical models

> Geometrical approaches

> Machine Learning

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

Mapping Model

feedback

[Momeni et al., 2006]

[Van Nort et al., 2014]

[Caramiaux, 2013]

Machine Learning

Discrete gesture recognition

Continuous gesture recognition & temporal mapping

Regression

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

Mapping Model

User Examples

feedback

[Gillian, 2011]

[Bevilacqua al., 2011]

[Caramiaux, 2013]

Related Work | interactive machine learning

Machine Learning

Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.

Machine Learning

Training DataMachine Learning

training

Machine Learning

training

new input

predictions

Machine Learning

training

new input

predictions

evaluate results

Interactive Machine Learning

create training

examples

training

new input

predictions

adjust parameters

& train

evaluate results

create training

examples

adjust parameters

& train

evaluate results

outline

Related Work

Models

Applications

Related Work | mapping through listening

Caramiaux, B., Françoise, J., Schnell, N., & Bevilacqua, F. (2014). Mapping Through Listening. Computer Music Journal, 38(3), 34–48.

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

feedback

Related Work | mapping through listening

listening as a starting point for mapping design

Caramiaux, B., Françoise, J., Schnell, N., & Bevilacqua, F. (2014). Mapping Through Listening. Computer Music Journal, 38(3), 34–48.

Sound Synthesis

Motion Capture

Motion Param.

SoundParam.

listening

mapping through listening

interactive machine learning

Robot Programming by Demonstration

Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. Handbook of Robotics.

Mapping by Demonstration | Overview

demonstration performance

Listening

Acting

Listening

Acting

Mappingtraining

Listening

Acting

Mappingtraining

Performance

Listening

Acting

Mappingtraining

Performance

Feedback

Listening

Acting

Performing

training

example

Françoise, J., Schnell, N., & Bevilacqua, F. (2013). A Multimodal Probabilistic Model for Gesture-based Control of Sound Synthesis. In Proceedings of the 21st ACM international conference on Multimedia (MM’13) (pp. 705–708). Barcelona, Spain.

Mapping by Demonstration | requirements

quick training

from 1 to several examples

continuous & causal inference

(real-time)

Recognition Paradigm

Generation Paradigm

Mapping by Demonstration | 2 criteria

Generation Paradigm

Probabilistic Models

Synthesisers

modelX

modelY

modelZ

SynthX

SynthY

SynthZ

Generation Paradigm

movement parameters

Synthesisers

modelX

modelY

modelZ

SynthX

SynthY

SynthZ

motion

Generation Paradigm

generatedsound parameters

movement parameters

Synthesisers

modelX

modelY

modelZ

SynthX

SynthY

SynthZ

motion

Generation Paradigm

Instantaneous

Temporal

inputmotion

instantaneous

Generation Paradigm

Instantaneous

Temporal

temporal

inputmotion

Recognition Paradigm Generation Paradigm

Mapping by Demonstration | 4 models

Gaussian Mixture Models gmm

Gaussian Mixture Regression gmr

Hierarchical Hidden Markov Models

Hierarchical Hidden Markov Regression

Recognition Paradigm Generation Paradigm

outline

Related Work

GMM HHMM

GMR HHMR

Models

Applications

GMR HHMR

outline

Related Work

GMM HHMM

GMR HHMR

Models

Applications

GMR HHMR

outline

Related Work

GMM HHMM

Models

Applications

outline

Related Work

GMM HHMM

GMR HHMR

Models

Applications

Probabilistic Movement Models | Gaussian Mixture Models

likelihoods

motionparameter

inputmotion

Probabilistic Movement Models | Hidden Markov Models

motionparameter

likelihoods

inputmotion

motionparameter

likelihoods

inputmotion

predictedsound

parameters

Probabilistic Movement Models | GMMs & HMMs for interaction

Training

> 1 to several examples

User-defined complexity

> number of states/gaussians

Training

User-defined Regularization

> numerical errors

> generalization on few examples

Training

> numerical errors

Training

> numerical errors

Continuous Recognition

> Forward inference in HMMs

Probabilistic Movement Models | Hierarchical Hidden Markov Models

Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.

transitionprior

hidden state

exit state

SignalLevel

SegmentLevel

transitionprior

hidden state

exit state

SignalLevel

SegmentLevel

Dynamic Bayesian Networks

Linear time inference

Forward algorithm

> causal estimation of signal & segment state probabilities

transitionprior

hidden state

exit state

SignalLevel

SegmentLevel

Dynamic Bayesian Networks

Linear time inference

Forward algorithm

> causal estimation of signal & segment state probabilities

P A S R

Example gesture

PASR decomposition

Preparation

Attack

Sustain

Release

> anticipation gesture

> retraction gesture

4 Phases:

Probabilistic Movement Models | HHMM — PASR

P A S R

p = 0.5 p = 0.5

p = 1.0 p = 1.0 p = 1.0

p = 0.5

p = 1.0

Possibility of authoring the transition parameters

> prior : P et A states

> exit : S et R states

2 playing modes :

> entire gesture with preparation and release

> quick transitions from sustain to attack

example: demonstration

example: performance

outline

Related Work

GMM HHMM

GMR HHMR

Models

Applications

Probabilistic Motion-Sound Models | background: speech & motion

Similar applications from Speech processing

Acoustic-articulatory Inversion

Silent Speech Interfaces

Speech-driven animation

Robotics & Animation

Programming-by-Demonstration (Motor behavior)

Animation

Probabilistic Approaches

GMM-based / HMM-based

[Zhang & Renals, 2008]

[Hueber and Badin, 2011]

[Ding, 2014]

[Calinon, 2010]

[Tilmanne, 2013]

Probabilistic Motion-Sound Models | General Approach

35training performancedemonstration

Motion Features

Sound Features

Joint Features

Motion Features

Sound Features

Joint Features

Training Joint density distribution

Motion Features

Sound Features

Joint Features

Conditional density

distribution

training performancedemonstration

Motion Features

Sound Features

Joint Features

Conditional density

distribution

training performancedemonstration

Probabilistic Motion-Sound Models | Gaussian Mixture Regression

observations

x = [x (m) , x (s)]

observations

x = [x (m) , x (s)]

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

covariance

observations

x = [x (m) , x (s)]

Conditional pdf

p�x (s) | x (m),�

k=1�k (x (m)) N

�x (s) ; µ̂(s)

k (x (m)),�̂(ss)k

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

covariance

responsibility

covariance

�k (x (m))

µ̂(s)k (x (m)) =µ(s)

k +�(sm)k

��(mm)

��1 �x (m) �µ(m)

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

observations

x = [x (m) , x (s)]

Conditional pdf

p�x (s) | x (m),�

k=1�k (x (m)) N

�x (s) ; µ̂(s)

k (x (m)),�̂(ss)k

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

covariance

responsibility

covariance

�k (x (m))

µ̂(s)k (x (m)) =µ(s)

k +�(sm)k

��(mm)

��1 �x (m) �µ(m)

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

observations

x = [x (m) , x (s)]

x̂ (s) =K�

k=1�k (x (m))µ̂(s)

k (x (m))

Least Squares EstimateConditional pdf

p�x (s) | x (m),�

k=1�k (x (m)) N

�x (s) ; µ̂(s)

k (x (m)),�̂(ss)k

joint pdf

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

x = [x (m) , x (s)]

p(x i | θ) =K∑

k=1wkN

(x i ; µk ,Σk

)prior

covariance

responsibility

covariance

�k (x (m))

µ̂(s)k (x (m)) =µ(s)

k +�(sm)k

��(mm)

��1 �x (m) �µ(m)

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

Probabilistic Motion-Sound Models | GMR — number of gaussians

Probabilistic Motion-Sound Models | GMR — regularization

Probabilistic Motion-Sound Models | Hidden Markov Regression

movement parameter

likelihoods

movement parameter

movement parameters

predictedsound

parameters

likelihoods

movement parameter

movement parameters

Joint Features

Training

HMM with Joint

Observation probability distribution

joint observation pdf:

µk =�µ(m)

k ;µ(s)k

�(1)

�k =��(mm)

k �(ms)k

�(sm)k �(ss)

covariance

p (x t | zt = k) =N�x t ;µk ,�k

Joint Features

Training

HMM with Joint

HMM with Conditional Observation probability distribution

covariance

x (s)t | x (m)

t , zt = k�=N

�x t ; µ̂k (x (m)),�̂

�conditional observation pdf:

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

��(mm)

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

Joint Features

Training

HMM with Joint

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

��(mm)

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

online estimation (forward)

p(x (s)t |x (m)

1:t ,�) =

Joint Features

Training

HMM with Joint

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

��(mm)

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

p(x (s)t |x (m)

1:t ,�) = N�

x (s) ; µ̂(s)i (x (m)

t ),�̂(ss)i

Joint Features

Training

HMM with Joint

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

��(mm)

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

p(x (s)t |x (m)

1:t ,�) = N�

x (s) ; µ̂(s)i (x (m)

t ),�̂(ss)i

��(m)

t (i )

Joint Features

Training

HMM with Joint

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

��(mm)

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

p(x (s)t |x (m)

1:t ,�) = N�

x (s) ; µ̂(s)i (x (m)

t ),�̂(ss)i

��(m)

t (i )N�

Joint Features

Training

HMM with Joint

covariance

µ̂(s)k (x (m)

t ) =µ(s)k +�(sm)

��(mm)

��1 �x (m)

t �µ(m)k

�(1)

�̂(ss)k =�(ss)

k ��(sm)k

��(mm)

��1�(ms)

x̂ (s)t =

i=1�(m)

t (i )µ̂(s)k (x (m)

Least Squares Estimate

Interactive Machine Learning implementation

Number of Hidden States

> complexity / temporal grain

Regularization

> smoothness / generalization

Hierarchical Extension

> high-level structure

Evaluation on Tai Chi Movement Sequences

outline

Related Work

GMM HHMM

GMR HHMR

Models

Applications

Playing Sound Textures | siggraph’14 Studio installation

Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.

Corpus-based Descriptor-driven Sound Synthesis

CataRT (Schwarz)

Sound Corpora from DIRTI

User Studio, Diemo Schwarz, Roldan Cahen

Playing Sound Textures | siggraph’14 Studio installation

Corpus-based Descriptor-driven Sound Synthesis

CataRT (Schwarz)

Sound Corpora from DIRTI

User Studio, Diemo Schwarz, Roldan Cahen

Advantages

Complex gestures

> implicit sonification strategies

Adaptive

> personalize gestures

> personalize sonification

Playing Sound Textures | texture sonification for gesture learning

Protocol

> reproduce a gesture as precisely as possible

> training from audio-visual demonstration

> reproduction: 3 recording blocks (10 trials)

silent vs sonified

> 11 participants

> 4 gestures (2 sonified / 2 silent)

Analysis

> correlation distance to reference

Playing Sound Textures | texture sonification for gesture learning

outline

Related Work

GMM HHMM

GMR HHMR

Models

Applications

Sonification through Vocalization | imitation game

MicrophoneWireless Motion Sensor

Imitation Game: adaptation

Evaluation

Sequence game

> 59 participants recorded at Siggraph’14

Analysis

> investigate log-likelihoods during gesture reproduction

Results

> sensitive to user expertise

> fast adaptation with sound feedback

==> learning

Sonification of movement

qualities

HMR System trained with Expert

performances

Exploratory workshop with dancers

> integration in teaching session

Promising for pedagogy

> Further evaluation planned

Vocalization | Application in dance

Françoise, J., Fdili Alaoui, S., Schiphorst, T., & Bevilacqua, F. (2014). Vocalizing Dance Movement for Interactive Sonification of Laban Effort Factors. Designing Interactive Systems (DIS’14), Vancouver, Canada.

Vocalization| movement sequence analysis (tai chi)

MO Sensors(Accel. + Gyro.)

(b) (c)

Tai Chi

> long sequences

> expertise (consistency over trials)

Motion Capture

> 3 inertial sensors (6DOF: acc + gyro)

> 10 performances of 45s sequence

> compare expert & student

Françoise, J., Roby-Brami, A., Riboud, N. & Bevilacqua, F. (2013). [SUBMITTED]

(b) (c)

Tai Chi

> long sequences

Motion Capture

(b) (c)

Tai Chi

> long sequences

Motion Capture

Movement Sequence Analysis | vocalization in performance

conclusions

Conclusions

Mapping by Demonstration

> mapping through listening + interactive machine learning

> powerful tool for designing sonic interactions

> leverages expertise

Conclusions

GMM HHMM

GMR HHMM

recognition paradigm

generative paradigm

instantaneous temporal

General framework for designing

motion & sound relationships

with probabilistic models

> learning from few examples

> continuous & causal inference

> generative approach

Conclusions

GMM HHMM

GMR HHMM

recognition paradigm

generative paradigm

instantaneous temporal

XMM library

open-source C++ library

Max implementation

General framework for designing

motion & sound relationships

with probabilistic models

> learning from few examples

> continuous & causal inference

> generative approach

Modeling and sonifying complex movements

> music

> dance

> rehabilitation

MOCO: International Workshop on Movement & Computing

Vocalization

Domain-specific Expertise

> Sonic Interaction Design

> Dance & Movement Practice

Applications & Perspectives

(Greg Beller’s Synekine project)

(Moving Stories Project, Thecla Schiphorst)

(LEGOS project & Labex Smart)

(SkAT-VG EU project)

thank you

… and many thanks to collaborators (incomplete pseudo-alphabetical list):

Frédéric Bevilacqua, Omid Alemi, Pablo Arias, Thierry Artières, Caroline Barbot, Greg

Beller, Riccardo Borghesi, Éric Boyer, Amélie Briaucourt, Baptiste Caramiaux, Fabien

Cesari, Olivier Chapuis, Sarah Fdili Alaoui, Masha Fedorova, Emmanuel Fléty, Sylvain

Hanneton, Olivier Houix, Stacy Hsueh, Ianis Lallemand, Jean-Philippe Lambert,

Benjamin Matuszewski, Nicolas Rasamimanana, Natacha Riboud, Sébastien

Robaszkiewicz, Agnès Roby-Brami, Victor Saiz, Kevin Sanlaville, Thecla Schiphorst,

Norbert Schnell, Diemo Schwarz, Alejandro Van Zandt-Escobar, …

Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart...

Documents

Isha Schwaller de Lubicz - livresnumeriquesgratuits.comlivresnumeriquesgratuits.com/data/documents/Her-Bak-Pois-Chiche... · Isha Schwaller de Lubicz HER-BAK « POIS CHICHE » VISAGE

The Theory of Sound 1

Sound Bar - Elektronica | Entertainment | Sony NL · Sound Bar Sound Bar. 2FR N’installez pas cet appareil dans un endroit confiné, comme sur une étagère ou un meuble TV intégré

Dox acoustics - D-sound

Cible / Média Scop sound

Textile Aiguilleté U3 P3 Textile Floqué showtime...U3 P3 ΔLw = 19 dB déclarée 931202 HER 06-17 HER 06-18 HER 06-19 HER 06-14 HER 06-15 HER 06-16 HER 06-10 HER 06-12 HER 06-13

Compact Sound Speaker System - …resource.boschsecurity.com/documents/Compact_Sound_Speake... · Introduction Le Compact Sound Speaker System est un haut-parleur large bande très

Sound Make Here #2

Traduction du manuel de Little Sound DJ alias LSDj en ... · Traduction du manuel de Little Sound DJ alias LSDj en version 3.7.4 Little Sound Dj v3.7.4 Auteur original: Johan Kotlinski

Eugenia's Escape: The Written Word in Frances Burney's Camilla · reveal her inner nature: 'Her form and her mind were of equal elasticity' (15). When displeased by a perceived flaw

Projet Sound Touch - soutenance d'avancement

Les Fruits With Sound Ym Blog

Bluetooth-модуль wave.com4 sound · Bluetooth-модуль wave.com4 sound Динамики, аудиосистема или wave.com4 sound&light Место установки

Rebecca Skeeles and Her Horses

Déesses musicales - Ana Mighty Sound

écouter Click on one of the ten sounds. sound 1 sound 2 sound 3 sound 4 sound 5 sound 6 soundsspelling sound 7 sound 8 sound 9 sound 10 Exit

PORTABLE SOUND SYSTEM SYSTEME DE SONORISATION … · 2017. 1. 13. · combo208-vhf 1. portable sound system . systeme de sonorisation portable . tragbare beschallungsanlage . compact,

DON’T LET HER BE GONE - parigny-country.e-monsite.comparigny-country.e-monsite.com/medias/files/don-t-let-her-be-gone... · DON’T LET HER BE GONE Chorégraphe : Marie-Claude GIL

Textile Aiguilleté U3 P3 Textile Floqué showtime...930202 HER 05-17 HER 05-18 HER 05-19 HER 05-14 HER 05-15 HER 05-16 HER 05-10 HER 05-12 HER 05-13 HER 05-07 HER 05-08 HER 05-09

Toute l afrique her jpb