Advancing Sentiment and commonsense-aware multimodal dialogue systems

Dialogue systems have multifaceted applications in customer service, virtual assistants, education, mental health, and many more areas. Such applications are often executed in the form of chatbots due to its flexibility. Its market value evidences the popularity of chatbots–– according to Revechat, 2.6 billion USD in 2019, that is projected to grow to 9.4 billion USD by 2024. Furthermore, 80% of businesses are expected to employ some form of a chatbot by 2020. Such adoptions are often driven by cost savings of up to 30% in customer service.

Creating a human-like conversational system is a long-standing goal of artificial intelligence (AI). However, it is not a trivial task as we, human beings, count on several variables such as emotions, sentiment, prior assumptions, intent, or personality traits to participate in dialogues and monologues (refer to Fig 1). These variables control the language that we generate and the way we understand the language that we hear. Hence, it is instead an overstatement to claim that a generic framework such as seq2seq can generate near-perfect natural language.

At the DeClare Lab at SUTD (refer to Fig 2), Asst Prof Soujanya Poria and his team focus on developing cutting-edge neural models, based on sound linguistic concepts to solve challenging Natural Language Processing (NLP) tasks. One of the primary goals of the research lab is to create a human-like dialogue understanding system by leveraging key factors such as pragmatics, affect, empathy, multimodal cues, and commonsense.

The lab’s multi-disciplinary research team, which comprises researchers and collaborators from different parts of the world, is not limited to developing deep-learning models, but instead embraces the curation of various datasets to advance the research infrastructure. This includes creating multimodal monologue and dialogue datasets — MOSEI, MELD dataset, and Mustard — for sentiment, emotion, dialogue act, and sarcasm classification.

The research focus of this lab started back in 2015 when Asst Prof Poria proposed a novel multi-staged fusion approach powered by deep neural network-based feature extraction from text, facial expressions, and speech for sentiment analysis and emotion recognition in monologues and dialogues. This research called multimodal sentiment analysis is a more complex form of sentiment analysis that extends textual sentiment analysis to vision and speech, particularly useful to analyze sentiment in real-time scenarios such as conversations, YouTube videos. The core concept of multimodal sentiment analysis is multimodal representation learning by fusing multiple modalities to improve the performance of unimodal systems (refer to Fig. 3).

This approach, published by Empirical Methods in Natural Language Processing (EMNLP), was one of the first deep learning approaches to multimodal sentiment analysis. Asst Prof Poria continued to author several research papers on this topic which were published by top conferences including the Association for the Advancement of Artificial Intelligence (AAAI), Association for Computational Linguistics (ACL), North American Chapter of the Association for Computational Linguistics (NAACL),

He worked alongside respected research collaborators from the University of Michigan (Prof Rada Mihalcea), the Carnegie Mellon University (Prof L.P. Morency), the Instituto Politécnico Nacional (Prof Alexander Gelbukh and Dr. Navonil Majumder), and the Adobe Research (Dr. Niyati Chhaya) to advance the field of multimodal sentiment analysis to new heights. Today, the domain of multimodal language understanding is one of the most critical research areas in AI.

Asst Prof Poria and his teammates embarked on a major research direction that focused on creating a human-like dialogue understanding system by leveraging key factors such as pragmatics, affect, empathy, multimodal cues, and commonsense.

Upon realizing the importance of the context to improve the performance of the multimodal classification systems, they came up with an autoregressive network implemented using long short-term memory (LSTM) to capture the context in dialogue and monologue for sentiment classification.

Although simple, it was one of the early models to showcase the power of contextual information in aiding the sentiment classification benchmark performance. This idea was published by one of the most reputed NLP conferences – the Association of Computational Linguistics (ACL).

Inspired by this success, Asst Prof  Poria and the team started inventing new ways of capturing context in a more complex scenario, for instance, in conversations where the contextual information depends on the interlocutors’ relationship.The dialogue understanding concept proposed by Asst Prof Poria and his co-authors from Instituto Politécnico Nacional, University of Michigan, and Carnegie Mellon University was published in IEEE Access 2019 (refer to Fig 4).

The vision of this research has since paved the way for several new research directions, such as Emotion Recognition in Conversations (ERC), Multimodal Sentiment Analysis, and Affective Dialogue Systems.

The field of ERC (refer to Fig 5), a term coined by Asst Prof Poria et al. in 2019, deals with the problem of emotion recognition in dialogs or multi-party conversations by modeling speaker-specific context, interlocutors’ dependency, etc. ERC is an essential task to understand emotions and sentiments in conversations for sentiment-aware dialogue generation.  The research team has published several works on ERC at top NLP and AI venues including EMNLP, ACL, AAAI.

The idea of context and speaker state modeling for emotion recognition and other classification tasks in dialogues has emerged as an essential task in the Natural Language Processing (NLP) research and gained significant attention from the NLP and AI community. In just one and a half years, this research field has produced more than 60 research papers and two workshops were dedicated to it.

One of the methods to address emotion recognition in conversations using context that has received much attention from the AI community is DialogueRNN. A paper on it was published by the Association for the Advancement of Artificial Intelligence (AAAI) 2019. It uses a complex hierarchical recurrent neural network to model interlocutors’ latent states and relationships to shape the context for dialogue understanding and classification tasks such as state and emotion recognition.

This idea has great potential for applications in many other sequential and contextual classification tasks. At the 2019 Empirical Methods in Natural Language Processing (EMNLP) conference, Asst Prof  Poria together with his students and collaborators, published a graph neural network to model context in dialogues, called DialogueGCN, which demonstrated the efficacy of graph neural networks for context modeling in solving a wide range of dialogue understanding tasks.

Detecting emotions in conversations will lead to more emotion-aware and empathetic response generation. Based on this theory, Asst Prof Poria’s team developed an empathetic chatbot. The team aims to deploy this platform for various use cases, such as e-learning and patient care.

Recently, Asst Prof Poria has started working on the commonsense knowledge infusion in deep learning models to improve NLP tasks. In his latest work published by Association for Computational Linguistics (ACL) 2020, he has demonstrated the efficacy of commonsense knowledge to improve domain adaptation.

At present, he is doing active research to extend this idea to enhance dialogue understanding and eventually enrich the generation performance. The illustration below (refer to fig 6) depicts two scenarios where a chat-bot is engaging with users of two different age groups (a young man and a child). When posed with the same query, the chat-bot performs commonsense reasoning to understand the concepts of toys to be played primarily by children and utilizes the age information of the user (present as metadata) to generate tacky and engaging replies. This whole reasoning process requires the bot to infer the relevant concepts and verify whether they are synergistic together, for instance, to verify if they follow common sense.

Commonsense knowledge can substantially improve the dialogue understanding and generation as depicted in the figure below (refer to fig 7). It was found that commonsense knowledge also helps in predicting emotion shifts and understanding the difference between closely related emotion classes such as anger and frustration.

Looking forward, Asst Prof Poria believes more exciting research can be done towards creating human-like multimodal conversational agents that would be common sense-aware and empathetic and could understand the human sentiment.

To find out more on the latest research outcomes from Asst Prof Poria and the team, head to the DeCLaRe lab website.

The dialogue understanding research at DeCLaRe lab has created a substantial impact on both academia and industry. These are a few examples:

The DeCLaRe lab has also been featured and mentioned by various media outlets and blogs:

Soujanya’s research interests include sentiment analysis, multimodal language understanding, dialogue understanding, distilling commonsense information for natural language processing, and semantics. Some of his notable works include aspect extraction, multimodal sentiment analysis using deep learning, emotion recognition in conversations for affective, and empathetic dialogue generation.

To date, Asst Prof Poria’s research articles have been cited almost 8000 times with an h-index of 43, according to Google scholar.

At SUTD, Asst Prof Poria founded the DeCLaRe lab, a research group doing cutting-edge advanced research in NLP.

For more info on Asst Prof Poria, click here.