Dr. Christian M. Meyer

DKPro Agreement

An Open-Source Java Library for Measuring Inter-Rater Agreement

Abstract. In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, π, κ, α, etc., we particularly support unitizing tasks by measuring αU as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.

Eingereicht: 16.05.2014 | Veröffentlicht: 23.08.2014
Datenmodell zur Berechnung der Inter-Rater Reliabilität<br/> in Coding- und Unitizing-Studien
Datenmodell zur Berechnung der Inter-Rater Reliabilität
in Coding- und Unitizing-Studien