Benchmarking LLMs and Fine-Tuned Models for Multilingual Policy Agenda Annotation in European Parliamentary Speeches

About the project

**Principal Investigador:** Bastián González-Bustamante **Research Team:** Tom Bellens, Christopher Klamm, and Marta Koch This project benchmarks zero-shot, few-shot and reasoning LLMs against fine-tuned Transformer baselines for multilingual policy agenda annotation using European parliamentary speeches as the core corpus. We will construct a reproducible evaluation pipeline comparing open and closed LLMs with fine-tuned BERT-family models across multiple European languages, quantifying trade-offs in performance, robustness, computational costs, and carbon footprint. Models will be assessed on a fixed, stratified test split with uniform prompts and controlled prompt variations. Decoding will be controlled through deterministic settings and contrasted with stochastic runs. Human-in-the-loop validation will be triggered by uncertainty and model disagreement to adjudicate complex cases and iteratively improve label quality. Outputs include a peer-reviewed article, fine-tuned model releases, and a working paper with practical guidelines.