logo EmuAgent: An Evidence-Augmented Multimodal Agent for Meme Understanding

Bo Xu1, Chenyuan Wang1, Xinyu Chen1, Hongfei Lin1 Feng Xia†2
1Dalian University of Technology
2RMIT University
†Corresponding authors.
Code Data


Abstract

Memes serve as an important medium of online communication, often conveying rich emotions and intentions. In recent years, vision-language models (VLMs) have been widely applied to meme understanding tasks. However, due to the diverse forms of meme expression and their continuous evolution driven by trending events and popular culture, VLMs with limited knowledge updating capabilities still face significant challenges in accurately interpreting memes. 🌈 To address these challenges, we propose EmuAgent, the first evidence-augmented multimodal agent for meme understanding, which automatically collects and verifies relevant evidence during the reasoning process to generate more accurate and reliable meme interpretations. In addition, we develop a scalable automated data generation pipeline and release a large-scale, high-quality dialogue dataset for meme understanding, termed EmuDialog. Furthermore, we introduce a comprehensive evaluation benchmark, EmuBench, covering multiple themes to systematically assess the performance of existing VLMs on meme understanding. Extensive experiments and analyses demonstrate that EmuAgent substantially improves the accuracy of meme interpretation, highlighting the role of evidence-augmented reasoning in meme understanding.

🔥Highlight

  • We propose EmuAgent, the first visual–language reasoning framework for meme understanding that integrates a dynamic, agent-style tool invocation mechanism. EmuAgent incorporates four key tools specifically designed for meme understanding, ensuring timely and reliable evidence acquisition during reasoning.
  • We develop a scalable data generation pipeline that combines tool usage with reasoning chains to automatically produce structured meme understanding data. Based on this pipeline, we construct EmuDialog, a high-quality dataset consisting of approximately 12,000 instances, providing a reliable foundation for training evidence-augmented models.
  • We introduce EmuBench, a new evaluation benchmark for meme understanding that covers four representative themes, enabling comprehensive assessment of state-of-the-art (SOTA) VLMs. Furthermore, extensive experiments demonstrate the potential of EmuAgent in advancing meme understanding.


The EmuDialog Dataset

Existing meme datasets mainly focus on harmful content detection, and meme interpretation data are often derived by re-annotating such datasets, which limits their timeliness and thematic diversity. To mitigate these issues, we extend several existing datasets, including MemeCap, M3H, ClimateMemes, and MOOD, and construct EmuDialog, a multi-theme dataset for meme understanding. we analyze the distribution of invalid action occurrences and interaction trajectory lengths in EmuDialog, as shown in Figure 2.

teaser

Figure 2. Data statistics of EmuDialog.

Overall Framework of EmuAgent

The overall framework of EmuAgent is illustrated in Figure 3. EmuAgent incorporates four key tools specifically designed for meme understanding, ensuring timely and reliable evidence acquisition during reasoning.

description

Figure 3. An overview of EmuAgent. Left: an example of agent–environment interaction for meme understanding. Right: the overall EmuAgent framework.

Main Results


EmuBench Result

Table 1: Performance comparison on EmuBench.

description

Theme Analysis

Table 2: Model performance across different themes.

description

Human Evaluation

Table 3: Results of human evaluation.

description

Ablation Study

Table 4: Ablation study on the effect of the agent framework and supervised fine-tuning (SFT) for meme understanding.

description

Case Study

As illustrated in Figure 4, we conduct a case study to analyze meme interpretation results produced by different models. MiniGPT-v2 exhibits substantial hallucinations during interpretation, with its reasoning deviating significantly from the underlying meme semantics. For MiniCPM-V 2.6, although the overall interpretation direction is generally reasonable, the model fails to sufficiently leverage key textual and visual evidence in the meme, resulting in relatively coarse and high-level explanations. Qwen2.5-VL-Instruct, by contrast, tends to introduce additional subjective inferences (e.g., “similarly complacent”), which partially exceed the information explicitly conveyed by the meme itself. These observations indicate that existing models still face limitations in jointly understanding memes and generating accurate interpretations. In comparison, EmuAgent explicitly acquires and verifies relevant evidence during reasoning, enabling more comprehensive coverage of critical information. This further validates the effectiveness of evidence-augmented reasoning for meme understanding.

teaser

Figure 4. Qualitative examples of responses from different models.

The EmuAgent Dataset Example

teaser