Published Research

Scientific discoveries pave the way for technology that responds appropriately to our expressions.

We’ve introduced new datasets and statistical methods to explore the dimensions of meaning that explain the feelings we report in different situations, the patterns of brain activity they evoke, physiological responses like goosebumps, and nuanced expressions in the face, body, and voice.

Facial Expressions

Speech Prosody

Vocal Bursts

PublicationYearBibTeXPDF
Brooks, J.A., Tzirakis, P., Baird, A., Kim, L., Opara, M., Fang, X., Keltner, D., Monroy, M., Corona, R., Metrick, J., & Cowen, A.S. (2023). Deep learning reveals what vocal bursts mean to people in different cultures. Nature Human Behaviour, 7, 240-250.2023Cite📄
Tzirakis, P., Baird, A., Brooks, J.A., Gagne, C., Kim, L., Opara, M., Gregory, C., Metrick, J., Tiruvadi, V., Schuller, B., Keltner, D., & Cowen, A.S. (2023). Large-Scale Nonverbal Vocalization Detection Using Transformers. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1–5.2023Cite📄
Alice Baird, Panagiotis Tzirakis, Jeffrey A Brooks, Chris B Gregory, Björn Schuller, Anton Batliner, Dacher Keltner, and Alan Cowen. 2022. The ACII 2022 Affective Vocal Bursts Workshop & Competition. In 2022 10th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 1–5.2022Cite📄
Baird, A., Tzirakis, P., Gidel, G., Jiralerspong, M., Muller, E.B., Mathewson, K., Schuller, B., Cambria, E., Keltner, D., & Cowen, A.S. (2022). Proceedings of the ICML 2022 expressive vocalizations workshop and competition: Recognizing, generating, and personalizing vocal bursts. arXiv.2022Cite📄
Baird, A., Tzirakis, P., Brooks, J.A., Kim, L., Opara, M., Gregory, C.B., Metrick, J., Boseck, G., Keltner, D., & Cowen, A.S. (2022). State & trait measurement from nonverbal vocalizations: A multi-task joint learning approach. Proceedings of Interspeech 2022, 2028-2032.2022Cite📄
Cowen, A.S., Elfenbein, H.A., Laukka, P., & Keltner, D. (2019). Mapping 24 emotions conveyed by brief human vocalization. American Psychologist, 74(6), 698-712.2019Cite📄

Multimodal/Other

PublicationYearBibTeXPDF
Christ, L., Amiriparian, S., Baird, A., Kathan, A., Müller, N., Klug, S., Gagne, C., Tzirakis, P., Stappen, L., Meßner, E., König, A., Cowen, A., Cambria, E., & Schuller, B. (2023). The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation. Proceedings of the 4th International Conference on Multimodal Sentiment Analysis Challenge and Workshop, 1–10.2023Cite📄
Amiriparian, S., Christ, L., König, A., Cowen, A., Meßner, E., Cambria, E., & Schuller, B. (2023). MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects. Proceedings of the 31st ACM International Conference on Multimedia (MM '23), 9723–9725.2023Cite📄
Brooks, J.A., Tiruvadi, V., Baird, A., Tzirakis, P., Li, H., Gagne, C., Oh, M., & Cowen, A.S. (2023). Emotion expression estimates to measure and improve multimodal social-affective interactions. ICMI ‘23 Companion: Companion Publication of the 25th International Conference on Multimodal Interaction, 353-358.2023Cite📄
Christ, L., Amiriparian, S., Baird, A., Tzirakis, P., Kathan, A., Müller, N., Stappen, L., Meßner, E., König, A., Cowen, A.S., Cambria, E., & Schuller, B.W. (2022). The MuSe 2022 multimodal sentiment analysis challenge: Humor, emotional reactions, and stress. MuSe ‘22: Proceedings of the 3rd International Conference on Multimodal Sentiment Analysis Workshop and Challenge, 5-14.2022Cite📄
Horikawa, T., Cowen, A.S., Keltner, D., & Kamitani, Y. (2020). The neural representation of visually evoked emotion is high-dimensional, categorical, and distributed across transmodal brain regions. iScience, 23(5), 101060.2020Cite📄
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A.S., Nemade, G., & Ravi, S. (2020). GoEmotions: A dataset of fine-grained emotions. arXiv.2020Cite📄
Cowen, A.S., Fang, X., Sauter, D., & Keltner, D. (2020). What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures. Proceedings of the National Academy of Sciences, 117(4), 1924-1934.2020Cite📄
Cowen, A.S., & Keltner, D. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114, E7900–E7909.2017Cite📄

Background/Theoretical

BibTeX

@InProceedings{Kollias_2023_CVPR,
    author    = {Kollias, Dimitrios and Tzirakis, Panagiotis and Baird, Alice and Cowen, Alan and Zafeiriou, Stefanos},
    title     = {ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection \& Emotional Reaction Intensity Estimation Challenges},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2023},
    pages     = {5889-5898}
}

@Article{Cowen2021-be,
  title    = "Sixteen facial expressions occur in similar contexts worldwide",
  author   = "Cowen, Alan S and Keltner, Dacher and Schroff, Florian and Jou,
              Brendan and Adam, Hartwig and Prasad, Gautam",
  abstract = "Understanding the degree to which human facial expressions
              co-vary with specific social contexts across cultures is central
              to the theory that emotions enable adaptive responses to
              important challenges and opportunities1--6. Concrete evidence
              linking social context to specific facial expressions is sparse
              and is largely based on survey-based approaches, which are often
              constrained by language and small sample sizes7--13. Here, by
              applying machine-learning methods to real-world, dynamic
              behaviour, we ascertain whether naturalistic social contexts (for
              example, weddings or sporting competitions) are associated with
              specific facial expressions14 across different cultures. In two
              experiments using deep neural networks, we examined the extent to
              which 16 types of facial expression occurred systematically in
              thousands of contexts in 6 million videos from 144 countries. We
              found that each kind of facial expression had distinct
              associations with a set of contexts that were 70\% preserved
              across 12 world regions. Consistent with these associations,
              regions varied in how frequently different facial expressions
              were produced as a function of which contexts were most salient.
              Our results reveal fine-grained patterns in human facial
              expressions that are preserved across the modern world.",
  journal  = "Nature",
  volume   =  589,
  number   =  7841,
  pages    = "251--257",
  month    =  jan,
  year     =  2021
}

@Article{Cowen2020-tn,
  title     = "What the face displays: Mapping 28 emotions conveyed by
               naturalistic expression",
  author    = "Cowen, Alan S and Keltner, Dacher",
  abstract  = "What emotions do the face and body express? Guided by new
               conceptual and quantitative approaches (Cowen, Elfenbein,
               Laukka, \& Keltner, 2018; Cowen \& Keltner, 2017, 2018), we
               explore the taxonomy of emotion recognized in facial-bodily
               expression. Participants (N = 1,794; 940 female, ages 18-76
               years) judged the emotions captured in 1,500 photographs of
               facial-bodily expression in terms of emotion categories,
               appraisals, free response, and ecological validity. We find that
               facial-bodily expressions can reliably signal at least 28
               distinct categories of emotion that occur in everyday life.
               Emotion categories, more so than appraisals such as valence and
               arousal, organize emotion recognition. However, categories of
               emotion recognized in naturalistic facial and bodily behavior
               are not discrete but bridged by smooth gradients that correspond
               to continuous variations in meaning. Our results support a novel
               view that emotions occupy a high-dimensional space of categories
               bridged by smooth gradients of meaning. They offer an
               approximation of a taxonomy of facial-bodily expressions,
               visualized within an online interactive map. (PsycInfo Database
               Record (c) 2020 APA, all rights reserved).",
  journal   = "Am. Psychol.",
  publisher = "American Psychological Association (APA)",
  volume    =  75,
  number    =  3,
  pages     = "349--364",
  month     =  apr,
  year      =  2020,
  language  = "en"
}

@Article{Cordaro2020-oa,
  title     = "The recognition of 18 facial-bodily expressions across nine
               cultures",
  author    = "Cordaro, Daniel T and Sun, Rui and Kamble, Shanmukh and Hodder,
               Niranjan and Monroy, Maria and Cowen, Alan and Bai, Yang and
               Keltner, Dacher",
  abstract  = "An enduring focus in the science of emotion is the question of
               which psychological states are signaled in expressive behavior.
               Based on empirical findings from previous studies, we created
               photographs of facial-bodily expressions of 18 states and
               presented these to participants in nine cultures. In a
               well-validated recognition paradigm, participants matched
               stories of causal antecedents to one of four expressions of the
               same valence. All 18 facial-bodily expressions were recognized
               at well above chance levels. We conclude by discussing the
               methodological shortcomings of our study and the conceptual
               implications of its findings. (PsycInfo Database Record (c) 2020
               APA, all rights reserved).",
  journal   = "Emotion",
  publisher = "American Psychological Association (APA)",
  volume    =  20,
  number    =  7,
  pages     = "1292--1300",
  month     =  oct,
  year      =  2020,
  language  = "en"
}

@Article{
doi:10.1126/sciadv.abb1005,
author = {Alan S. Cowen  and Dacher Keltner },
title = {Universal facial expressions uncovered in art of the ancient Americas: A computational approach},
journal = {Science Advances},
volume = {6},
number = {34},
pages = {eabb1005},
year = {2020},
doi = {10.1126/sciadv.abb1005},
URL = {https://www.science.org/doi/abs/10.1126/sciadv.abb1005},
eprint = {https://www.science.org/doi/pdf/10.1126/sciadv.abb1005},
abstract = {Analyses of facial poses in ancient American sculpture reveal parallels with modern Western conceptions of emotional behavior. Central to the study of emotion is evidence concerning its universality, particularly the degree to which emotional expressions are similar across cultures. Here, we present an approach to studying the universality of emotional expression that rules out cultural contact and circumvents potential biases in survey-based methods: A computational analysis of apparent facial expressions portrayed in artwork created by members of cultures isolated from Western civilization. Using data-driven methods, we find that facial expressions depicted in 63 sculptures from the ancient Americas tend to accord with Western expectations for emotions that unfold in specific social contexts. Ancient American sculptures tend to portray at least five facial expressions in contexts predicted by Westerners, including “pain” in torture, “determination”/“strain” in heavy lifting, “anger” in combat, “elation” in social touch, and “sadness” in defeat-supporting the universality of these expressions.}
}

@InProceedings{10.1145/3581783.3612835,
author = {Schuller, Bj\"{o}rn W. and Batliner, Anton and Amiriparian, Shahin and Barnhill, Alexander and Gerczuk, Maurice and Triantafyllopoulos, Andreas and Baird, Alice E. and Tzirakis, Panagiotis and Gagne, Chris and Cowen, Alan S. and Lackovic, Nikola and Caraty, Marie-Jos\'{e} and Montaci\'{e}, Claude},
title = {The ACM Multimedia 2023 Computational Paralinguistics Challenge: Emotion Share \& Requests},
year = {2023},
isbn = {9798400701085},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3581783.3612835},
doi = {10.1145/3581783.3612835},
abstract = {The ACM Multimedia 2023 Computational Paralinguistics Challenge addresses two different problems for the first time in a research competition under well-defined conditions: In the Emotion Share Sub-Challenge, a regression on speech has to be made; and in the Requests Sub-Challenges, requests and complaints need to be detected. We describe the Sub-Challenges, baseline feature extraction, and classifiers based on the 'usual' ComPaRE features, the auDeep toolkit, and deep feature extraction from pre-trained CNNs using the DeepSpectRum toolkit; in addition, wav2vec2 models are used.},
booktitle = {Proceedings of the 31st ACM International Conference on Multimedia},
pages = {9635–9639},
numpages = {5},
keywords = {challenge, emotion share, computational paralinguistics, requests, benchmark, complaints},
location = {<conf-loc>, <city>Ottawa ON</city>, <country>Canada</country>, </conf-loc>},
series = {MM '23}
}

@Article{Cowen2019-pu,
  title    = "The primacy of categories in the recognition of 12 emotions in
              speech prosody across two cultures",
  author   = "Cowen, Alan S and Laukka, Petri and Elfenbein, Hillary Anger and
              Liu, Runjing and Keltner, Dacher",
  abstract = "Central to emotion science is the degree to which categories,
              such as Awe, or broader affective features, such as Valence,
              underlie the recognition of emotional expression. To explore the
              processes by which people recognize emotion from prosody, US and
              Indian participants were asked to judge the emotion categories or
              affective features communicated by 2,519 speech samples produced
              by 100 actors from 5 cultures. With large-scale statistical
              inference methods, we find that prosody can communicate at least
              12 distinct kinds of emotion that are preserved across the 2
              cultures. Analyses of the semantic and acoustic structure of the
              recognition of emotions reveal that emotion categories drive the
              recognition of emotions more so than affective features,
              including Valence. In contrast to discrete emotion theories,
              however, emotion categories are bridged by gradients representing
              blends of emotions. Our findings, visualized within an
              interactive map, reveal a complex, high-dimensional space of
              emotional states recognized cross-culturally in speech prosody.",
  journal  = "Nature Human Behaviour",
  volume   =  3,
  number   =  4,
  pages    = "369--382",
  month    =  apr,
  year     =  2019
}

@ARTICLE{Brooks2023-jm,
  title    = "Deep learning reveals what vocal bursts express in different
              cultures",
  author   = "Brooks, Jeffrey A and Tzirakis, Panagiotis and Baird, Alice and
              Kim, Lauren and Opara, Michael and Fang, Xia and Keltner, Dacher
              and Monroy, Maria and Corona, Rebecca and Metrick, Jacob and
              Cowen, Alan S",
  abstract = "Human social life is rich with sighs, chuckles, shrieks and other
              emotional vocalizations, called `vocal bursts'. Nevertheless, the
              meaning of vocal bursts across cultures is only beginning to be
              understood. Here, we combined large-scale experimental data
              collection with deep learning to reveal the shared and
              culture-specific meanings of vocal bursts. A total of n = 4,031
              participants in China, India, South Africa, the USA and Venezuela
              mimicked vocal bursts drawn from 2,756 seed recordings.
              Participants also judged the emotional meaning of each vocal
              burst. A deep neural network tasked with predicting the
              culture-specific meanings people attributed to vocal bursts while
              disregarding context and speaker identity discovered 24 acoustic
              dimensions, or kinds, of vocal expression with distinct
              emotion-related meanings. The meanings attributed to these
              complex vocal modulations were 79\% preserved across the five
              countries and three languages. These results reveal the
              underlying dimensions of human emotional vocalization in
              remarkable detail.",
  journal  = "Nature Human Behaviour",
  volume   =  7,
  number   =  2,
  pages    = "240--250",
  month    =  feb,
  year     =  2023
}

@InProceedings{10095294,
  author={Tzirakis, Panagiotis and Baird, Alice and Brooks, Jeffrey and Gagne, Christopher and Kim, Lauren and Opara, Michael and Gregory, Christopher and Metrick, Jacob and Boseck, Garrett and Tiruvadi, Vineet and Schuller, Björn and Keltner, Dacher and Cowen, Alan},
  booktitle={ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
  title={Large-Scale Nonverbal Vocalization Detection Using Transformers}, 
  year={2023},
  volume={},
  number={},
  pages={1-5},
  doi={10.1109/ICASSP49357.2023.10095294}
}

@InProceedings{10086002,
  author={Baird, Alice and Tzirakis, Panagiotis and Brooks, Jeffrey A. and Gregory, Chris B. and Schuller, Björn and Batliner, Anton and Keltner, Dacher and Cowen, Alan},
  booktitle={2022 10th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)}, 
  title={The ACII 2022 Affective Vocal Bursts Workshop & Competition}, 
  year={2022},
  volume={},
  number={},
  pages={1-5},
  doi={10.1109/ACIIW57231.2022.10086002}}

@misc{baird2022proceedings,
      title={Proceedings of the ICML 2022 Expressive Vocalizations Workshop and Competition: Recognizing, Generating, and Personalizing Vocal Bursts}, 
      author={Alice Baird and Panagiotis Tzirakis and Gauthier Gidel and Marco Jiralerspong and Eilif B. Muller and Kory Mathewson and Björn Schuller and Erik Cambria and Dacher Keltner and Alan Cowen},
      year={2022},
      eprint={2207.06958},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}

@inproceedings{inproceedings,
author = {Baird, Alice and Tzirakis, Panagiotis and Brooks, Jeff and Kim, Lauren and Opara, Michael and Gregory, Chris and Metrick, Jacob and Boseck, Garrett and Keltner, Dacher and Cowen, Alan},
year = {2022},
month = {09},
pages = {2028-2032},
title = {State & Trait Measurement from Nonverbal Vocalizations: A Multi-Task Joint Learning Approach},
doi = {10.21437/Interspeech.2022-10927}
}

@Article{Cowen2019-iw,
  title     = "Mapping 24 emotions conveyed by brief human vocalization",
  author    = "Cowen, Alan S and Elfenbein, Hillary Anger and Laukka, Petri and
               Keltner, Dacher",
  abstract  = "Emotional vocalizations are central to human social life. Recent
               studies have documented that people recognize at least 13
               emotions in brief vocalizations. This capacity emerges early in
               development, is preserved in some form across cultures, and
               informs how people respond emotionally to music. What is poorly
               understood is how emotion recognition from vocalization is
               structured within what we call a semantic space, the study of
               which addresses questions critical to the field: How many
               distinct kinds of emotions can be expressed? Do expressions
               convey emotion categories or affective appraisals (e.g.,
               valence, arousal)? Is the recognition of emotion expressions
               discrete or continuous? Guided by a new theoretical approach to
               emotion taxonomies, we apply large-scale data collection and
               analysis techniques to judgments of 2,032 emotional vocal bursts
               produced in laboratory settings (Study 1) and 48 found in the
               real world (Study 2) by U.S. English speakers (N = 1,105). We
               find that vocal bursts convey at least 24 distinct kinds of
               emotion. Emotion categories (sympathy, awe), more so than
               affective appraisals (including valence and arousal), organize
               emotion recognition. In contrast to discrete emotion theories,
               the emotion categories conveyed by vocal bursts are bridged by
               smooth gradients with continuously varying meaning. We visualize
               the complex, high-dimensional space of emotion conveyed by brief
               human vocalization within an online interactive map. (PsycINFO
               Database Record (c) 2019 APA, all rights reserved).",
  journal   = "Am. Psychol.",
  publisher = "American Psychological Association (APA)",
  volume    =  74,
  number    =  6,
  pages     = "698--712",
  month     =  sep,
  year      =  2019,
  language  = "en"
}

@inproceedings{10.1145/3606039.3613114,
author = {Christ, Lukas and Amiriparian, Shahin and Baird, Alice and Kathan, Alexander and M\"{u}ller, Niklas and Klug, Steffen and Gagne, Chris and Tzirakis, Panagiotis and Stappen, Lukas and Me\ss{}ner, Eva-Maria and K\"{o}nig, Andreas and Cowen, Alan and Cambria, Erik and Schuller, Bj\"{o}rn W.},
title = {The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation},
year = {2023},
isbn = {9798400702709},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3606039.3613114},
doi = {10.1145/3606039.3613114},
abstract = {The Multimodal Sentiment Analysis Challenge (MuSe) 2023 is a set of shared tasks addressing three different contemporary multimodal affect and sentiment analysis problems: In the Mimicked Emotions Sub-Challenge (MuSe-Mimic), participants predict three continuous emotion targets. This sub-challenge utilises the Hume-Vidmimic dataset comprising of user-generated videos. For the Cross-Cultural Humour Detection Sub-Challenge (MuSe-Humour), an extension of the Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset is provided. Participants predict the presence of spontaneous humour in a cross-cultural setting. The Personalisation Sub-Challenge (MuSe-Personalisation) challenge is based on the Ulm-Trier Social Stress Test (Ulm-TSST) dataset, featuring recordings of subjects in a stressed situation. Here, arousal and valence signals are to be predicted, whereas parts of the test labels are made available in order to facilitate personalisation. MuSe 2023 seeks to bring together a broad audience from different research communities such as audio-visual emotion recognition, natural language processing, signal processing, and health informatics. In this baseline paper, we introduce the datasets, sub-challenges, and provided feature sets. As a competitive baseline system, a Gated Re-current Unit (GRU)-Recurrent Neural Network (RNN) is employed. On the respective sub-challenges' test datasets, it achieves a mean (across three continuous intensity targets) Pearson's Correlation Coefficient of .4727 for MuSe-Mimic, an Area Under the Curve (AUC) value of .8310 for MuSe-Humour and Concordance Correlation Coefficient (CCC) values of .7482 for arousal and .7827 for valence in the MuSe-Personalisation sub-challenge.},
booktitle = {Proceedings of the 4th on Multimodal Sentiment Analysis Challenge and Workshop: Mimicked Emotions, Humour and Personalisation},
pages = {1–10},
numpages = {10},
keywords = {humour detection, affective computing, multimodal sentiment analysis, multimodal fusion, challenge, benchmark, emotion recognition},
location = {Ottawa ON, Canada},
series = {MuSe '23}
}

@inproceedings{10.1145/3581783.3610943,
author = {Amiriparian, Shahin and Christ, Lukas and K\"{o}nig, Andreas and Cowen, Alan and Me\ss{}ner, Eva-Maria and Cambria, Erik and Schuller, Bj\"{o}rn W.},
title = {MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects},
year = {2023},
isbn = {9798400701085},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3581783.3610943},
doi = {10.1145/3581783.3610943},
abstract = {The 4th Multimodal Sentiment Analysis Challenge (MuSe) focuses on Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects. The workshop takes place in conjunction with ACM Multimedia'23. We provide three datasets as part of the challenge: (i) The Hume-Vidmimic dataset which offers 30+ hours of expressive behaviour data from 557 participants. It involves mimicking and rating emotions: Approval, Disappointment, and Uncertainty. This multimodal resource is valuable for studying human emotional expressions. (ii) The 2023 edition of the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset comprises German football press conference recordings within the training set, while videos of English football press conferences are included in the unseen test set. This unique configuration offers a cross-cultural evaluation environment for humour recognition. (iii) The Ulm-Trier Social Stress Test (Ulm-TSST) dataset contains recordings of subjects under stress. It involves arousal and valence signals, with some test labels provided to aid personalisation. Based on these datasets, we formulate three multimodal affective computing challenges: (1) Mimicked Emotions Sub-Challenge (MuSe-Mimic) for categorical emotion prediction, (2) Cross-Cultural Humour Detection Sub-Challenge (MuSe-Humour) for cross-cultural humour detection, and (3) Personalisation Sub-Challenge (MuSe-Personalisation) for personalised dimensional emotion recognition. In this summary, we outline the challenge's motivation, participation guidelines, conditions, and results.},
booktitle = {Proceedings of the 31st ACM International Conference on Multimedia},
pages = {9723–9725},
numpages = {3},
keywords = {affective computing, challenge, multimodal fusion, summary paper, multimodal sentiment analysis, emotion mimics, cross-cultural humour detection, emotion recognition},
location = {<conf-loc>, <city>Ottawa ON</city>, <country>Canada</country>, </conf-loc>},
series = {MM '23}
}

@article{keltner2022emotions,
  title={How emotions, relationships, and culture constitute each other: advances in social functionalist theory},
  author={Keltner, Dacher and Sauter, Disa and Tracy, Jessica L and Wetchler, Everett and Cowen, Alan S},
  journal={Cognition and Emotion},
  volume={36},
  number={3},
  pages={388--401},
  year={2022},
  publisher={Taylor \& Francis}
}

@inproceedings{10.1145/3581783.3610943,
author = {Amiriparian, Shahin and Christ, Lukas and K\"{o}nig, Andreas and Cowen, Alan and Me\ss{}ner, Eva-Maria and Cambria, Erik and Schuller, Bj\"{o}rn W.},
title = {MuSe 2023 Challenge: Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects},
year = {2023},
isbn = {9798400701085},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3581783.3610943},
doi = {10.1145/3581783.3610943},
abstract = {The 4th Multimodal Sentiment Analysis Challenge (MuSe) focuses on Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects. The workshop takes place in conjunction with ACM Multimedia'23. We provide three datasets as part of the challenge: (i) The Hume-Vidmimic dataset which offers 30+ hours of expressive behaviour data from 557 participants. It involves mimicking and rating emotions: Approval, Disappointment, and Uncertainty. This multimodal resource is valuable for studying human emotional expressions. (ii) The 2023 edition of the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset comprises German football press conference recordings within the training set, while videos of English football press conferences are included in the unseen test set. This unique configuration offers a cross-cultural evaluation environment for humour recognition. (iii) The Ulm-Trier Social Stress Test (Ulm-TSST) dataset contains recordings of subjects under stress. It involves arousal and valence signals, with some test labels provided to aid personalisation. Based on these datasets, we formulate three multimodal affective computing challenges: (1) Mimicked Emotions Sub-Challenge (MuSe-Mimic) for categorical emotion prediction, (2) Cross-Cultural Humour Detection Sub-Challenge (MuSe-Humour) for cross-cultural humour detection, and (3) Personalisation Sub-Challenge (MuSe-Personalisation) for personalised dimensional emotion recognition. In this summary, we outline the challenge's motivation, participation guidelines, conditions, and results.},
booktitle = {Proceedings of the 31st ACM International Conference on Multimedia},
pages = {9723–9725},
numpages = {3},
keywords = {affective computing, challenge, multimodal fusion, summary paper, multimodal sentiment analysis, emotion mimics, cross-cultural humour detection, emotion recognition},
location = {<conf-loc>, <city>Ottawa ON</city>, <country>Canada</country>, </conf-loc>},
series = {MM '23}
}

@article{HORIKAWA2020101060,
title = {The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain Regions},
journal = {iScience},
volume = {23},
number = {5},
pages = {101060},
year = {2020},
issn = {2589-0042},
doi = {https://doi.org/10.1016/j.isci.2020.101060},
url = {https://www.sciencedirect.com/science/article/pii/S2589004220302455},
author = {Tomoyasu Horikawa and Alan S. Cowen and Dacher Keltner and Yukiyasu Kamitani},
keywords = {Neuroscience, Cognitive Neuroscience, Techniques in Neuroscience},
abstract = {Summary
Central to our subjective lives is the experience of different emotions. Recent behavioral work mapping emotional responses to 2,185 videos found that people experience upward of 27 distinct emotions occupying a high-dimensional space, and that emotion categories, more so than affective dimensions (e.g., valence), organize self-reports of subjective experience. Here, we sought to identify the neural substrates of this high-dimensional space of emotional experience using fMRI responses to all 2,185 videos. Our analyses demonstrated that (1) dozens of video-evoked emotions were accurately predicted from fMRI patterns in multiple brain regions with different regional configurations for individual emotions; (2) emotion categories better predicted cortical and subcortical responses than affective dimensions, outperforming visual and semantic covariates in transmodal regions; and (3) emotion-related fMRI responses had a cluster-like organization efficiently characterized by distinct categories. These results support an emerging theory of the high-dimensional emotion space, illuminating its neural foundations distributed across transmodal regions.}
}

@misc{demszky2020goemotions,
      title={GoEmotions: A Dataset of Fine-Grained Emotions}, 
      author={Dorottya Demszky and Dana Movshovitz-Attias and Jeongwoo Ko and Alan Cowen and Gaurav Nemade and Sujith Ravi},
      year={2020},
      eprint={2005.00547},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@inproceedings{10.1145/3610661.3616129,
author = {Brooks, Jeffrey A. and Tiruvadi, Vineet and Baird, Alice and Tzirakis, Panagiotis and Li, Haoqi and Gagne, Chris and Oh, Moses and Cowen, Alan},
title = {Emotion Expression Estimates to Measure and Improve Multimodal Social-Affective Interactions},
year = {2023},
isbn = {9798400703218},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3610661.3616129},
doi = {10.1145/3610661.3616129},
abstract = {Large language models (LLMs) are being adopted in a wide range of applications, but an understanding of other social-affective signals is needed to support effective human-computer-interaction (HCI) in multimodal interfaces. In particular, robust, accurate measurements of human emotional expression can be used to tailor responses to human values and preferences. In this paper, we present two models available from an API-based suite of emotional expression models that measure nuanced facial and vocal signals, providing rich, high-dimensional emotional expression estimates (EEEs). We demonstrate the ability of EEEs to provide insight into two established datasets and present methods for integrating EEEs into large language model (LLM) applications. We discuss how this approach is a step towards more reliable tools for clinical screening and scientific study, as well as empathic digital assistants that can be used in therapeutic settings.},
booktitle = {Companion Publication of the 25th International Conference on Multimodal Interaction},
pages = {353–358},
numpages = {6},
keywords = {Emotion Science, Multimodal Sentiment Analysis, Affective Computing, Mental Health, Emotion Recognition},
location = {Paris, France},
series = {ICMI '23 Companion}
}

@article{
doi:10.1073/pnas.1702247114,
author = {Alan S. Cowen  and Dacher Keltner },
title = {Self-report captures 27 distinct categories of emotion bridged by continuous gradients},
journal = {Proceedings of the National Academy of Sciences},
volume = {114},
number = {38},
pages = {E7900-E7909},
year = {2017},
doi = {10.1073/pnas.1702247114},
URL = {https://www.pnas.org/doi/abs/10.1073/pnas.1702247114},
eprint = {https://www.pnas.org/doi/pdf/10.1073/pnas.1702247114},
abstract = {Emotions are centered in subjective experiences that people represent, in part, with hundreds, if not thousands, of semantic terms. Claims about the distribution of reported emotional states and the boundaries between emotion categories—that is, the geometric organization of the semantic space of emotion—have sparked intense debate. Here we introduce a conceptual framework to analyze reported emotional states elicited by 2,185 short videos, examining the richest array of reported emotional experiences studied to date and the extent to which reported experiences of emotion are structured by discrete and dimensional geometries. Across self-report methods, we find that the videos reliably elicit 27 distinct varieties of reported emotional experience. Further analyses revealed that categorical labels such as amusement better capture reports of subjective experience than commonly measured affective dimensions (e.g., valence and arousal). Although reported emotional experiences are represented within a semantic space best captured by categorical labels, the boundaries between categories of emotion are fuzzy rather than discrete. By analyzing the distribution of reported emotional states we uncover gradients of emotion—from anxiety to fear to horror to disgust, calmness to aesthetic appreciation to awe, and others—that correspond to smooth variation in affective dimensions such as valence and dominance. Reported emotional states occupy a complex, high-dimensional categorical space. In addition, our library of videos and an interactive map of the emotional states they elicit (https://s3-us-west-1.amazonaws.com/emogifs/map.html) are made available to advance the science of emotion.}}

@article{doi:10.1177/09637214221150511,
author = {Dacher Keltner and Jeffrey A. Brooks and Alan Cowen},
title ={Semantic Space Theory: Data-Driven Insights Into Basic Emotions},

journal = {Current Directions in Psychological Science},
volume = {32},
number = {3},
pages = {242-249},
year = {2023},
doi = {10.1177/09637214221150511},

URL = { 
    
        https://doi.org/10.1177/09637214221150511
    
    

},
eprint = { 
    
        https://doi.org/10.1177/09637214221150511
    
    

}
,
    abstract = { Here we present semantic space theory and the data-driven methods it entails. Across the largest studies to date of emotion-related experience, expression, and physiology, we find that emotion is high dimensional, defined by blends of upward of 20 distinct kinds of emotions, and not reducible to low-dimensional structures and conceptual processes as assumed by constructivist accounts. Specific emotions are not separated by sharp boundaries, contrary to basic emotion theory, and include states that often blend. Emotion concepts such as “anger” are primary in the unfolding of emotional experience and emotion recognition, more so than core affect processes of valence and arousal. We conclude by outlining studies showing how these data-driven discoveries are a basis of machine-learning models that are serving larger-scale, more diverse studies of naturalistic emotional behavior. }
}

@article{KELTNER2021216,
title = {A taxonomy of positive emotions},
journal = {Current Opinion in Behavioral Sciences},
volume = {39},
pages = {216-221},
year = {2021},
issn = {2352-1546},
doi = {https://doi.org/10.1016/j.cobeha.2021.04.013},
url = {https://www.sciencedirect.com/science/article/pii/S2352154621000991},
author = {Dacher Keltner and Alan Cowen},
abstract = {Within social functionalist theory (SFT), emotions structure attachment relations, cooperative alliances, hierarchies, and collectives. Within this line of thinking, a rich array of positive emotions enable the formation and negotiation of these relationships. Guided by these arguments, we synthesize how top-down confirmatory studies and data-driven, computational studies converge on evidence for 11 positive emotions with distinct experience, expression, and physiology. This taxonomy includes amusement, awe, compassion, contentment, desire, love, joy, interest, pride, relief, and triumph. We conclude by considering how recent taxonomic efforts will advance emotion science in mapping the distinct forms and functions of the positive emotions.}
}

@Article{Cowen2021-bh,
  title     = "Semantic space theory: A computational approach to emotion",
  author    = "Cowen, Alan S and Keltner, Dacher",
  abstract  = "Within affective science, the central line of inquiry, animated
               by basic emotion theory and constructivist accounts, has been
               the search for one-to-one mappings between six emotions and
               their subjective experiences, prototypical expressions, and
               underlying brain states. We offer an alternative perspective:
               semantic space theory. This computational approach uses
               wide-ranging naturalistic stimuli and open-ended statistical
               techniques to capture systematic variation in emotion-related
               behaviors. Upwards of 25 distinct varieties of emotional
               experience have distinct profiles of associated antecedents and
               expressions. These emotions are high-dimensional, categorical,
               and often blended. This approach also reveals that specific
               emotions, more than valence, organize emotional experience,
               expression, and neural processing. Overall, moving beyond
               traditional models to study broader semantic spaces of emotion
               can enrich our understanding of human experience.",
  journal   = "Trends Cogn. Sci.",
  publisher = "Elsevier BV",
  volume    =  25,
  number    =  2,
  pages     = "124--136",
  month     =  feb,
  year      =  2021,
  language  = "en"
}

@Article{Keltner2019-lh,
  title     = "Emotional expression: Advances in basic emotion theory",
  author    = "Keltner, Dacher and Sauter, Disa and Tracy, Jessica and Cowen,
               Alan",
  abstract  = "In this article, we review recent developments in the study of
               emotional expression within a basic emotion framework. Dozens of
               new studies find that upwards of 20 emotions are signaled in
               multimodal and dynamic patterns of expressive behavior. Moving
               beyond word to stimulus matching paradigms, new studies are
               detailing the more nuanced and complex processes involved in
               emotion recognition and the structure of how people perceive
               emotional expression. Finally, we consider new studies
               documenting contextual influences upon emotion recognition. We
               conclude by extending these recent findings to questions about
               emotion-related physiology and the mammalian precursors of human
               emotion.",
  journal   = "J. Nonverbal Behav.",
  publisher = "Springer Science and Business Media LLC",
  volume    =  43,
  number    =  2,
  pages     = "133--160",
  month     =  jun,
  year      =  2019,
  language  = "en"
}

@Article{Cowen2019-or,
  title     = "Mapping the passions: Toward a high-dimensional taxonomy of
               emotional experience and expression",
  author    = "Cowen, Alan and Sauter, Disa and Tracy, Jessica L and Keltner,
               Dacher",
  abstract  = "What would a comprehensive atlas of human emotions include? For
               50 years, scientists have sought to map emotion-related
               experience, expression, physiology, and recognition in terms of
               the ``basic six''-anger, disgust, fear, happiness, sadness, and
               surprise. Claims about the relationships between these six
               emotions and prototypical facial configurations have provided
               the basis for a long-standing debate over the diagnostic value
               of expression (for review and latest installment in this debate,
               see Barrett et al., p. 1). Building on recent empirical findings
               and methodologies, we offer an alternative conceptual and
               methodological approach that reveals a richer taxonomy of
               emotion. Dozens of distinct varieties of emotion are reliably
               distinguished by language, evoked in distinct circumstances, and
               perceived in distinct expressions of the face, body, and voice.
               Traditional models-both the basic six and affective-circumplex
               model (valence and arousal)-capture a fraction of the systematic
               variability in emotional response. In contrast, emotion-related
               responses (e.g., the smile of embarrassment, triumphant
               postures, sympathetic vocalizations, blends of distinct
               expressions) can be explained by richer models of emotion. Given
               these developments, we discuss why tests of a basic-six model of
               emotion are not tests of the diagnostic value of facial
               expression more generally. Determining the full extent of what
               facial expressions can tell us, marginally and in conjunction
               with other behavioral and contextual cues, will require mapping
               the high-dimensional, continuous space of facial, bodily, and
               vocal signals onto richly multifaceted experiences using
               large-scale statistical modeling and machine-learning methods.",
  journal   = "Psychol. Sci. Public Interest",
  publisher = "SAGE Publications",
  volume    =  20,
  number    =  1,
  pages     = "69--90",
  month     =  jul,
  year      =  2019,
  keywords  = "affect; emotion; expression; face; semantic space; signal; voice",
  language  = "en"
}