Improving Addressee Recognition in Multi-party Dialogue with LLMs

Among the many challenges of multi-party dialogue systems, addressee recognition—identifying who a speaker is addressing—remains a critical task that needs more attention.

In this paper, the authors present a novel approach to addressing addressee recognition within multi-modal, multi-party dialogue systems. They introduce a dataset focused on triadic (three-participant) conversations and benchmark the performance of a large language model (GPT-4o) in this complex task. Their findings shed light on the difficulties inherent in this problem and highlight the need for further innovations to improve LLMs' ability to navigate multi-party conversational dynamics.

Understanding the Core Concepts

Before diving deeper, let's unpack the key concepts explored in this research:

• Multi-party Dialogue: This refers to conversations involving more than two participants. Unlike dyadic interactions, multi-party dialogues feature more complex turn-taking and coordination between speakers, as each participant must decide when to speak and who they are addressing.

• Addressee Recognition: In a multi-party dialogue, it’s essential for a system to identify who the current speaker is addressing. This could be a specific participant, a group, or no one in particular. In many cases, the addressee is indicated through explicit cues like names or direct questions. However, subtle non-verbal cues, such as gaze or tone, can also play a significant role.

• Large Language Models (LLMs): These models, like GPT-4o, are capable of generating and understanding human-like text based on vast amounts of training data. However, LLMs have yet to master the complexities of multi-party dialogue, such as correctly identifying who is being addressed in a conversation.

The Challenge of Addressee Recognition

Understanding who is being addressed in a multi-party conversation is a foundational element for any multi-party dialogue system. Unlike in dyadic interactions, where the addressee is usually clear, multi-party dialogues involve intricate social dynamics where the recipient of a message can change rapidly. The key challenges for addressee recognition include:

1. Explicit vs. Implicit Addressing: In many multi-party dialogues, the addressee is not explicitly mentioned. Instead, participants rely on subtle contextual cues such as gaze, tone, or even the position of the speaker. Identifying these implicit cues is challenging for models like GPT-4o, which may not fully capture non-verbal context.

2. Dynamic Turn-taking: In multi-party interactions, participants often interrupt, speak over one another, or quickly switch topics. These dynamics make it hard for systems to determine who is the intended recipient of a message, especially when multiple individuals may be involved in the conversation at the same time.

3. Language and Visual Cues: In real-world conversations, addressee recognition involves more than just the text—visual cues like gaze and body language play an important role. This is even more relevant when dialogues are multimodal, where the system must consider both the spoken words and visual signals to identify the addressee correctly.

The TEIDAN Corpus: A Resource for Multi-party Dialogue Research

To address the challenge of addressee recognition, the authors created a novel multi-modal, multi-party dialogue corpus called TEIDAN. This corpus focuses on triadic (three-participant) conversations, providing a rich resource for studying the complexities of multi-party interactions. Key features of the TEIDAN corpus include:

• Spontaneous, Natural Conversations: Unlike other datasets that rely on scripted dialogues or task-based conversations, TEIDAN captures free-flowing discussions on topics such as city planning, island survival items, and weekend travel plans. This provides a more authentic reflection of real-world dialogue dynamics.

• Multi-modal Data: Each conversation was recorded with both visual (facial expressions and gaze) and audio data, allowing the model to learn from both spoken words and non-verbal cues.

• Addressee Annotation: The authors annotated a subset of the corpus to indicate who was being addressed in each conversational turn. Approximately 20% of the turns in the dataset explicitly specify an addressee, while the remaining 80% involve turns where no specific person is directly addressed.

Benchmarking Addressee Recognition with GPT-4o

To evaluate how well a large language model can handle addressee recognition in multi-party dialogues, the authors tested GPT-4o, a multimodal LLM, on the TEIDAN dataset. The task involved predicting which participant was being addressed in each conversational turn. GPT-4o was given a prompt with multiple turns and was tasked with identifying the addressee for the last utterance. The results were quite revealing:

• Accuracy Results: GPT-4o achieved an accuracy of 80.9%, which was only slightly above chance (80.1%). This highlights the difficulty of the task and the model's struggle to correctly identify the addressee, especially when cues are implicit or non-verbal.

• Types of Errors: The errors made by GPT-4o were predominantly false negatives, where the model incorrectly marked the addressee as ‘O’ (no specific addressee). In cases where the addressee was not explicitly stated, the model had difficulty identifying the intended recipient.

Insights from the Benchmark

The findings from this experiment offer several insights into the challenges of addressee recognition in multi-party dialogue systems:

1. Explicit Addressing is Easier: As shown in one of the examples, GPT-4o was able to correctly identify the addressee when the conversation included explicit cues, such as a direct question to a specific participant. However, when the turn-taking was more fluid and implicit, the model struggled.

2. The Complexity of Non-verbal Cues: Many conversational turns in the TEIDAN dataset rely on non-verbal cues, like gaze direction, which GPT-4o cannot easily interpret without visual data. This highlights the importance of multimodal systems that can integrate both audio and visual signals for more accurate addressee recognition.

3. Need for Further Research: The performance gap between the model and human-level recognition emphasizes the need for further research into improving the understanding of multi-party dynamics. Enhancing the ability of LLMs to capture implicit cues, such as gaze or social context, will be crucial in advancing multi-party dialogue systems.

Conclusion: Advancing Multi-party Dialogue Systems

This research introduces a crucial step forward in building effective multi-party dialogue systems by focusing on the task of addressee recognition. The TEIDAN corpus and the benchmark provided for GPT-4o demonstrate the complexities involved in understanding multi-party interactions and the limitations of current LLMs in this area. While GPT-4o performed reasonably well in explicit cases, its struggles with implicit addressee recognition suggest that multimodal systems, combining both verbal and visual cues, will be essential in moving forward.

As research continues in this area, we can expect more sophisticated models that will not only handle multi-party dialogue more effectively but also engage in more natural, context-aware conversations that mimic human social dynamics.

What’s Your Opinion?

How do you think multimodal LLMs can be enhanced to better recognize addressees in multi-party dialogues? What other challenges do you think exist in scaling these models for real-world applications? Let us know your thoughts!