Social Signal Processing

Humans are social animals by nature (Aristotle, Politika ca. 328 BC) and social interactions are an essential part of their day-to-day life. Social interactions are an arena for affective displays and experiences which have a large influence on one's physical and mental well-being. For example, the relevance of affect for understanding employee behavior and organizational functioning at large is profound [1]. Moreover, previous theorizing has described meetings as affective events that have important consequencesfor employee attitudes and behaviors beyond the meeting context [1]. This has given rise to a new interdisciplinary research field, Social Signal Processing (SSP) [2], where researchers from the field of computer science and social psychologists have come together to build the next generation of socially intelligent computing systems. With advances in the field of SSP, affective sciences research has shown increasing prominence in high-critical and socially relevant domains, e.g. health, social robotics, security and employee well-being.

One way in which affect is expressed in social interactions and therefore accessible for social signal processing concerns speech signals. In this research, we investigate speech-based machine learning techniques to automatically detect individual- and group- level affect expressions in social interactions. Being an interdisciplinary research, the goals of this project stem from both the informatics per-spective and the psychology perspective. The reseach work is carried out in close collaboration with the "Industrial and organizational psychology" research group headed by Nale Lehmann-Willenbrock.

Challenges

The field of social signal processing and speech affect recognition has spanned roughly two decades [3], with improving state-of-the-art performance in automatic recognition. Nevertheless, there is still a wide range of open research questions to be answered. In our research we focus on three such open research questions,

Subjectivity and Label uncertainty in affect,
Affect as a diffuse and dynamic construct, and,
Emergent affect dynamics in social interaction.

Subjectivity in Emotion Expressions

As opposed to affect and feelings as inner subjective experiences, we focus on emotional expressions as visible behaviors that others subjectively perceive and can respond to. A common framework for analyzing the expression of emotion in social interactions is pleasure-arousal theory, which describes emotional experiences in two continuous, bipolar, and orthogonal dimensions: pleasure-displeasure (valence) and activation-deactivation (arousal). A crucial challenge in studying emotional expressions within the arousal and valence framework concerns the significant degree of subjectivity surrounding them [3-5].

Figure 1: Overview of the proposed architecture. Blue denotes layerswith deterministic weights and yellow with probabilistic weights.

In order to better represent subjectivity in emotional expressions, we introduced a Bayes by Backpropogation (BBB) [6] based architecture that learns a distribution of weights to capture the inherent uncertainty related to subjective emotion annotations [9]. To the best of our knowledge, this work is the first to use Bayesian neural networksfor speech emotion recognition. In contrast to [4-5], the model can be explicitly trained on a distribution of emotion annotations. For this, we introduce a KL divergence-based loss term $\mathcal{L}_{\text{KL}}$ that promotes capturing aleotoric uncertainty (label uncertainty) rather than exclusively capturing epistemic uncertainty (model uncertainty) [7]. An overview of the proposed model can be seen in Figure 1. The research work is accepted at Interspeech 2022: Navin Raj Prabhu, Guillaume Carbajal, Nale Lehmann-Willenbrock, Timo Gerkmann, "End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks", Interspeech, Incheon, Korea, Sep. 2022, https://arxiv.org/abs/2110.03299 [9].

By assuming that the emotion annotation distribution $\mathcal{Y}_t$ follows a Gaussian distribution at time frame $t$, the goal of our approach is to obtain an estimate $\widehat{\mathcal{Y}}_t$ of $\mathcal{Y}_t$ and infer both $m_t$ and $s_t$ from realizations of $\widehat{\mathcal{Y}}_t$ [9]. Our approach [9], learns a variational posterior of weights $P(w|\mathcal{D})$ given the training data $\mathcal{D}$ and outputs a predictive distribution $\widehat{\mathcal{Y}}_t$ given by $P (\widehat{y}_t|x_t) = \mathbb{E}_{P(w|D)}[P(\widehat{y}_t|x_t, w)]$, where $\widehat{y}_t$ are realizations of $\widehat{\mathcal{Y}}_t$. The model is trained on the end-to-end loss function,
$$\mathcal{L} = \mathcal{L}_{\text{CCC}}(m) + \mathcal{L}_{\text{BBB}} + \mathcal{L}_{\text{KL}},$$

where,

$\mathcal{L}_{\text{CCC}}(m)$ is concordance coefficient which optimizes for mean subjective annotations $m$,
$\mathcal{L}_{\text{BBB}}$ is the stochastic negative evidence lower bound (ELBO) which optimizes for BBB weight distributions $ P(w|\mathcal{D}) $, and,
$\mathcal{L}_{\text{KL}} = f(\mathcal{Y}_t\|\widehat{\mathcal{Y}}_t)_{\text{KL}} = \int \mathcal{Y}_t(x) \: \log\frac{\mathcal{Y}_t(x)}{\widehat{\mathcal{Y}}_t(x)} dx$ optimizes for the label distribution $\mathcal{Y}_t$.

Results: Estimation of subjective emotion annotation distribution $ \mathcal{Y}_t $

Figure 2: Results obtained for a test subject for arousal

Figure 3: Results obtained for a test subject for valence.

The key insights which can be drawn from Figure 2 and Figure 3 are as follows,

From Figure 2, we see that the proposed label uncertainty approach can fit quite accurately the ground truth distribution $\mathcal{Y}_t$ on the arousal dimension.
For arousal, the approach can fit the ground-truth emotion annotation distribution $\mathcal{Y}_t$ with an estimate $\widehat{\mathcal{Y}}_t$ with 0.258 $\mathcal{L}_{\text{KL}}$. $\mathcal{L}_{\text{KL}} = 0$ denotes identical distribtutions.
Moreover, for arousal, the model achieves a $\mathcal{L}_{\text{CCC}}(s) = 0.340$, where $\mathcal{L}_{\text{CCC}}(s)$ is the CCC between the standard deviations of $\mathcal{Y}_t$ and $\widehat{\mathcal{Y}}_t$. $\mathcal{L}_{\text{CCC}}(s) = 1$ denotes indentical standard deviation estimates.
However, performs poorly on the valence dimension. Note that it is commonly observed in emotion recognition literature that the audio modality insufficiently explains valence.

Future works: Subjectivity in emotional expressions

Multimodal label uncertainty model for improved valence annotation distribution estimations.
In the model we assumed that $ \mathcal{Y}_t $ follows a Gaussian distribution, and had only $ a=6 $ annotations available to model the distribution. This might sometimes lead to unstable $ \mathcal{L}_\text{KL} $ during approximation of $ \mathcal{Y}_t $, thereby affecting the training processes. Acknowledging that gaining more annotation is resource inefficient, as future work, we will investigate techniques to model stable $ \mathcal{Y}_t $ with limited annotations.

Future Works

Affect as a diffuse and dynamic construct:

Affect is a volatile construct that can momentarily change and dynamically influence interacting partners' behaviors, making them a diffuse and dynamic construct. With this in mind, as a future work, we will aim to operationalize individual affect as a diffuse, and dynamic construct. For example, we will extend individual affect to dyadic relationships, in terms of paralinguistic mimicry [8] between meeting attendee pairs. Data-driven mimicry which directly capture the diffuse and dynamic nature of affect is a knowledge gap in exisitng literature.

Emergent affect dynamics

Building on the insights from dyadic relationships, subsequently we will focus on modeling the group-level affective emergence across a range of social interactions. Following research tasks will be focussed on as part of this research direction,

Understand the temporal dynamics behind emergent processes and devising annotation schema to best capture the fine-grained dynamics of group-level affective dynamics.
Tailor or device AI techniques to best model the temporal, diffuse and emergent nature of group-level affective emergence.

Research on such group-level constructs have key applications for collaborative tools and virtual meeting platforms. Moreover, by expanding our research focus to the group as a whole, we hope to find essential antidotes to draining meeting experiences.

References

C. Scott, J. A. Allen, S. G. Rogelberg, and A. Kello, “Five theoretical lenses for conceptualizing the role ofmeetings in organizational life,” in The Cambridge handbook of meeting science, Cambridge University Press New York, NY, 2015, pp. 20–46.
A. Vinciarelli, M. Pantic, and H. Bourlard. Social signal processing: Survey of an emerging domain. Image and vision computing, 27(12):1743–1759, 2009.
B. W. Schuller, “Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends, ”Communications of the ACM, vol. 61, no. 5, pp. 90–99, Apr. 2018.
J. Han, Z. Zhang, Z. Ren, and B. Schuller, “Exploring percep-tion uncertainty for emotion recognition in dyadic conversationand music listening, ”Cognitive Computation, pp. 1–10, 2020.
K. Sridhar and C. Busso, “Modeling uncertainty in predictingemotional attributes from spontaneous speech,” in ICASSP - IEEE Int. Conf. on Acoustics, Speech and Signal Processing,2020, pp. 8384–8388.
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra,“Weight uncertainty in neural network,” in International Conference on Machine Learning. PMLR, 2015, pp. 1613–1622.
R. Zheng, S. Zhang, L. Liu, Y. Luo, and M. Sun, “Uncertainty in bayesian deep label distribution learning,” Applied Soft Computing, 2021.
M. C. Nanninga, Y. Zhang, N. Lehmann-Willenbrock, Z. Szlávik, and H. Hung, “Estimating verbal expressionsof task and social cohesion in meetings by quantifying paralinguistic mimicry,” in Proc. of the 19th ACM Int.Conf. on Multimodal Interaction - ICMI 2017, pp. 206–215.

Publications

Navin Raj Prabhu, Guillaume Carbajal, Nale Lehmann-Willenbrock, Timo Gerkmann, "End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks", Interspeech, Incheon, Korea, Sep. 2022. [arxiv]
Navin Raj Prabhu, Nale Lehmann-Willenbrock, Timo Gerkmann, "Label Uncertainty Modeling and Prediction for Speech Emotion Recognition using t-Distributions", Affective Computing and Intelligent Interaction (ACII), Nara, Japan, Oct. 2022. [arxiv]