Computer Science > Computation and Language
arXiv:2604.13899v4 (cs)
[Submitted on 15 Apr 2026 (v1), last revised 17 Jun 2026 (this version, v4)]
Title:Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection
View a PDF of the paper titled Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection, by Ahmad Dawar Hakimi and 3 other authors
View PDF
HTML (experimental)
Abstract:Instruction-tuned LLMs can annotate thousands of instances at low cost. This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be cheaply labeled? We investigate both on a new dataset of 277,902 German political TikTok comments (25,974 LLM-labeled, 5,000 human-annotated), comparing LLM and human annotation across seven conditions, four encoders, and 10 random seeds. Under a two-question interface that mirrors the human annotation task, LLM annotation at scale outperforms human-supervised classifiers at roughly one-tenth the cost (\$28 for GPT-5.2 Batch API vs. \$316 for Prolific). The advantage holds for both a closed-source (GPT-5.2) and an open-weight (Qwen3.5-122B-10B) LLM, is robust under soft-label evaluation, and is unlocked specifically by the two-question decomposition; a holistic single-prompt baseline only ties with human supervision. AL provides no reliable advantage over random sampling under either LLM annotator. However, error structure varies sharply: only GPT-5.2 under the two-question interface produces classifiers with near-human FP/FN balance, while other LLM variants over-flag border-control and economic competition discourse. We release the dataset and code.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.13899 [cs.CL] |
| (or arXiv:2604.13899v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.13899
arXiv-issued DOI via DataCite
|
Submission history
From: Ahmad Dawar Hakimi [view email][v1] Wed, 15 Apr 2026 14:10:58 UTC (2,603 KB)
[v2] Wed, 22 Apr 2026 13:35:57 UTC (2,603 KB)
[v3] Tue, 16 Jun 2026 13:15:14 UTC (2,747 KB)
[v4] Wed, 17 Jun 2026 07:33:22 UTC (2,759 KB)
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
View a PDF of the paper titled Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection, by Ahmad Dawar Hakimi and 3 other authors
References & Citations
Loading...
BibTeX formatted citation
Data provided by:
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.