Dr. Christian M. Meyer

DKPro Agreement

An Open-Source Java Library for Measuring Inter-Rater Agreement

Abstract. In this paper, we introduce a novel Java implementation of multiple inter-rater agreement measures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, π, κ, α, etc., we particularly support unitizing tasks by measuring αU as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.

Submitted: 16.05.2014 | Published: 23.08.2014
Data model for measuring inter-rater agreement of<br/> coding and unitizing studies.
Data model for measuring inter-rater agreement of
coding and unitizing studies.