Can AI recognise persuasion in Ukrainian wartime memes?
Not reliably. On 200 native Ukrainian memes, the best micro hierarchical F1 was 0.384 and macro F1 stayed below 0.20. Models often over-predicted persuasion, while explanations could agree on the same words and still assign the wrong rhetorical label.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1Researchers collected 200 native Ukrainian memes and created a 158-item Ukrainian translation of an English benchmark, then tested four models without task-specific training.
- 2The strongest micro hierarchical F1 on native memes was 0.384 and macro F1 peaked at 0.198, showing weak performance on rarer persuasion categories.
- 3Two native speakers achieved only moderate agreement across multi-label annotations, and explanation methods could focus on the same cues while the model still chose the wrong label.
Research topic
How accurately vision-language models detect persuasion techniques in native Ukrainian wartime memes and whether their explanations identify evidence that truly supports the prediction
The answer: performance was too weak for automated judgement
The best reported micro hierarchical F1 on the 200 native Ukrainian memes was 0.384, achieved by GPT-4.1-mini with image and text. Macro F1, which gives more weight to rare categories, peaked at only 0.198. Qwen3-VL-8B's image-and-text result was 0.342 micro F1 and 0.193 macro F1. Those scores show partial recognition of rhetorical families, not reliable identification of the specific techniques present in a meme.
The models frequently over-predicted persuasion when visual and textual signals were combined, especially in emotionally charged wartime material. That matters because a moderation or monitoring system can turn a difficult cultural judgement into a false label at scale. The paper explicitly treats the labels as an analytical lens rather than assuming that Ukrainian grassroots memes are state propaganda. The results do not support using a model to decide that an individual post is propaganda or that its author is acting deceptively.[1][2]
The Impact Brief · Free
Follow the evidence in society & media.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
Two datasets separated translation from native culture
UkrMeme contains 200 organically produced Ukrainian-language wartime memes collected from public Telegram channels and the SUCHO Meme Wall in early 2026. Two trained native speakers independently assigned labels from a 22-technique taxonomy plus a No Propaganda label, and a curator resolved disagreements. The second dataset, PTM-UA, began with an English propaganda-meme benchmark. Researchers extracted and removed English text, translated it into Ukrainian with GPT-5, rendered it back onto the image and manually filtered poor translations or damaged layouts, leaving 158 items.
That design creates a useful contrast. PTM-UA keeps the imagery and political discourse of an English benchmark while changing the written language. UkrMeme reflects Ukrainian wartime culture, humour and resilience. Performance was consistently higher on the translated set: GPT-4.1-mini reached 0.613 micro hierarchical F1 with image and text, compared with 0.384 on native memes. Translation therefore did not reproduce the cultural and rhetorical difficulty of authentic local material.[1][2]
What was measured and why agreement matters
The four zero-shot models were Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, GPT-4.1-mini and Gemma 3 4B. Tests compared text-only, image-only and combined inputs. Hierarchical F1 awards partial credit when a predicted label and the gold label share a parent in the taxonomy, such as related emotional or credibility appeals. Even under that more forgiving metric, rare techniques remained difficult: macro F1 stayed below 0.30 on PTM-UA and below 0.20 on UkrMeme.
Human annotation was not perfectly settled. MASI-weighted Krippendorff's alpha was 0.406 and mean per-class Cohen's kappa was 0.404, both indicating moderate agreement. Agreement was strongest for No Propaganda, smears and appeals to fear or prejudice, and very low for loaded language. This is not merely annotator error; rhetorical categories overlap and context changes interpretation. A model evaluated against an uncertain boundary cannot be treated as discovering an objective fact that people themselves consistently recognise.[1]
Explanations did not guarantee a correct label
For the two open Qwen models, the researchers compared generated token explanations with Layer-wise Relevance Propagation, which traces internal relevance through the network. Verbal explanations tended to emphasise emotional or rhetorical cues, while internal relevance concentrated more on content words. Native multimodal memes showed higher median overlap between the two methods than translated examples, suggesting more stable cue selection in some settings.
But agreement between explanations did not mean the answer was right. In several low-scoring cases, both methods focused on the same words yet the model mapped those cues to the wrong technique—for example confusing a smear, loaded language, name-calling or an appeal to emotion. Other cases showed divergence that may reflect optical-character-recognition noise or translation artefacts. A plausible highlight or rationale is therefore not proof that the classification is faithful or correct.[1]
Impact on journalists, moderators and communities
For journalists and researchers, the dataset fills a real gap: most persuasion benchmarks are English-centred, while memes combine language, imagery, shared history and humour. Tools may help organise a corpus or surface items for review, but the low scores and uncertain labels argue for human analysis. Investigators should retain the original post, source and context rather than relying on extracted text or a model-generated category.
For platforms and public agencies, automated enforcement would be especially risky. A system that over-predicts persuasion could suppress satire, coping, political speech or documentation of war. The harms are not evenly distributed: lower-resource languages often receive weaker models and fewer appeal mechanisms. Any operational use should be limited to research or triage, with native-speaking reviewers, transparent uncertainty and a route for affected people to challenge a label.[1]
Evidence limits and what would change the assessment
Both datasets are small. Two hundred native memes cannot represent the range of Ukrainian regional language, political positions, platforms or changing wartime references. PTM-UA keeps English-source imagery and may preserve pretraining familiarity, while machine translation and re-rendering introduce artefacts. The model comparison is zero-shot and limited to four systems; task-specific training, new model versions or different prompts could change the ranking. The accessible paper does not report a funding or competing-interest statement.
Confidence would rise with a larger time-stratified dataset, more native annotators from varied backgrounds, clear treatment of disagreement and evaluation on newly collected material hidden from developers. Studies should separate descriptive research from moderation decisions and measure real false-positive consequences. For now, the strongest conclusion is constructive but cautious: culturally grounded Ukrainian data exposes weaknesses that translated English benchmarks hide, and neither a confident label nor an attractive explanation is enough to automate judgement.[1][2]
What this means for people
- Ukrainian journalists and researchers gain a culturally grounded dataset but still need native-speaking human judgement.
- Creators could be mislabelled if platforms confuse wartime humour or resilience with coordinated propaganda.
- Moderators should use the models only for cautious triage, with uncertainty and appeal paths visible.
Global context
The gap between translated English material and native Ukrainian memes is a warning for every lower-resource language. Benchmark translation can preserve familiar images and rhetoric while missing local culture, humour and political context. Governments, platforms and researchers should not present an English-derived model as globally reliable without native data, community review and evidence about real moderation outcomes.
What the evidence does not yet show
- The study is an unreviewed preprint and had no independent replication at publication.
- The native dataset has 200 memes and the translated control 158, limiting statistical power and cultural coverage.
- Human agreement was moderate because persuasion labels are subjective and overlapping.
- PTM-UA retains English-source imagery and may contain translation, OCR and rendering artefacts.
- Only four zero-shot models were tested; the results do not establish performance after task-specific training or on future models.
What to watch next
- Larger native-language datasets collected over time and across platforms
- Evaluation by broader groups of Ukrainian annotators with disagreement preserved
- False-positive consequences in newsroom, research and moderation workflows
- Whether model explanations remain stable and faithful on genuinely unseen cultural material
Living evidence record
Impact record IAI-1MRF1A6
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
12 October 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 12 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Society & Media
Can people reliably spot AI-written text?
Not in this Japanese experiment. Across 3,120 judgements, four reported reading strategies produced only 41% to 45% accuracy. The result cautions against accusing a person from prose style alone, but one human text, one prompt and retrospectively grouped explanations sharply limit its reach.
9 min · 1 source
Society & Media
Can image-aware AI catch fake news before it spreads?
A new peer-reviewed model improved two fixed benchmark tests by combining article text, images and an AI-generated image description. It did not verify claims or face a live, changing news stream.
7 min · 2 sources
Society & Media
Former Google safety chief warns AI could amplify harms seen in social media
Reuters reports testimony from a former Google safety executive who argued that companion-like AI and persuasive systems could expose children to harms more directly than conventional social feeds.
2 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.