ProKDA Learn Before You Judge: Progressive Knowledge-to-Decision Alignment
for Explainable Hateful Meme Detection

Bo Xu 1, Chenyuan Wang 1, Xinyu Chen 1, Quanhao Zhu1, Rui Lin1, Liang Zhao 1, Hongfei Lin 1, Feng Xia †2
1 Dalian University of Technology 2 RMIT University
† Corresponding author
Paper Preprint Code
01 — Overview

Hateful memes often convey abusive intent through implicit image-text interactions, making both accurate detection and explanation challenging. Existing explain-then-detect methods jointly optimize explanation generation and label prediction, which can introduce task interference and hurt detection performance.

Challenge Harmful meanings are often expressed through subtle interactions between images, text, and external context.
Limitation Jointly learning explanations and labels can compromise the detection objective.
Key Idea Separate knowledge acquisition from final decision learning.
Result ProKDA achieves state-of-the-art detection performance while providing evidence-supported explanations.
Project introduction figure
Comparison of ProKDA with existing MLLM-based methods.
02 — Method

Progressive Knowledge-to-Decision Alignment

ProKDA follows a progressive learning paradigm that transforms meme-related background knowledge into accurate and explainable hatefulness decisions.

How ProKDA Works

ProKDA consists of two major steps. First, it constructs a meme-specific background knowledge corpus through an agentic pipeline. A query agent identifies relevant background concepts, external knowledge is retrieved from Wikipedia, and an answer agent organizes the retrieved evidence into textual, visual, and multimodal explanations for each meme.

Based on this knowledge, ProKDA progressively trains the multimodal language model in three stages. The model first learns background knowledge and its relation to meme semantics, then adapts to direct hatefulness detection, and finally focuses on low-confidence boundary samples for DPO-based alignment. By separating these learning objectives across stages, ProKDA reduces interference between explanation generation and label prediction while progressively converting external knowledge into reliable detection decisions.

Overview of the ProKDA framework
03 — Results

Main results

ProKDA consistently improves hateful meme detection across HMC, MAMI, and PrideMM, outperforming both direct detection and explain-then-detect baselines across different model scales.

Comparison with Baseline Methods

Accuracy (ACC), macro-F1 (M-F1), and weighted-F1 (W-F1) are used as evaluation metrics (%). Bold and underlined values denote the best and second-best results, respectively.

# Model HMC MAMI PrideMM
ACC M-F1 W-F1 ACC M-F1 W-F1 ACC M-F1 W-F1
MLLMs as Auxiliary Modules
1 ExplainHM (LLaVA-v1.5-13B) 75.60 75.39 — — — — — — —
2 M2KE (LLaVA-v1.5-13B) 75.76 75.62 — 75.85 75.71 — — — —
MLLMs as Core Detectors (Training-Free)
3 U-CoT+ (Qwen2.5-14B) 72.50 72.41 — 79.90 79.89 — 71.60 71.37 —
4 LoReHM (LLaVA-34B) 65.60 65.59 — 75.40 75.28 — — — —
5 MIND (LLaVA-v1.6-34B) 66.40 68.38 — 73.60 75.38 — — — —
6 ALARM (Qwen2.5-VL-72B) 75.80 75.79 — 81.28 81.25 — — — —
MLLMs as Core Detectors (Training-Based)
7 MemeReason (Qwen3-VL-8B) 81.20 79.00 81.00 — — — — — —
8 RA-HMD (Qwen2.5-VL-7B) 80.80 80.10 — 81.00 81.00 — 78.00 77.80 —
9 ExPO-HM (Qwen2.5-VL-7B) — 81.10 — — 82.30 — — 78.70 —
10 Qwen2.5-VL-3B
11 Zero-shot 63.10 61.99 62.12 61.80 57.44 57.44 65.09 64.24 64.38
12 SFT (Explain-then-Detect) 67.00 65.22 65.38 69.70 68.63 68.63 66.86 66.81 66.84
13 SFT (Direct Detection) 72.20 71.07 71.18 77.50 76.99 76.99 74.36 74.35 74.36
14 DPO 71.60 70.24 70.36 76.40 75.65 75.65 74.75 74.75 74.73
15 ProKDA 80.20 80.19 80.20 78.70 78.13 78.13 75.74 75.72 75.70
16 Qwen2.5-VL-7B
17 Zero-shot 63.40 60.19 60.42 56.30 47.31 47.31 58.19 51.10 51.58
18 SFT (Explain-then-Detect) 72.00 70.93 71.04 75.00 74.34 74.34 76.53 76.48 76.51
19 SFT (Direct Detection) 75.60 75.15 75.22 78.90 78.37 78.37 77.71 77.71 77.72
20 DPO 74.30 73.65 73.73 77.30 76.60 76.60 76.53 76.53 76.53
21 ProKDA 83.50 83.41 83.44 83.10 83.08 83.08 80.28 80.27 80.28
04 — Examples

Qualitative Examples

Representative examples illustrate how ProKDA leverages background knowledge to produce accurate and evidence-supported hateful meme decisions.

01

Implicit Hateful Meaning

ProKDA identifies implicit hateful intent by connecting visual and textual cues with relevant background knowledge.

Implicit hateful meaning example
02

Background Knowledge Matters

External knowledge helps ProKDA resolve references beyond the meme image and text alone.

Background knowledge example
03

Evidence-Supported Decision

ProKDA aligns background knowledge with final decisions, enabling more reliable explanations.

Evidence-supported decision example
05 — Citation

Cite this work

If you find ProKDA useful in your research, please consider citing our work.

BibTeX 
@article{prokda2026,
  title   = {Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detection},
  author  = {Bo Xu and Chenyuan Wang and Xinyu Chen and Quanhao Zhu and Rui Lin and Liang Zhao and Hongfei Lin and Feng Xia},
  journal = {arXiv preprint arXiv:2609.19778},
  year    = {2026}
}