. . . . "To measure the performance of the studied crowdsourcing approaches (contest and microtasks), we report on: i) inter-rater agreement computed with the Fleiss’ kappa metric in order to measure the consensus degree among raters (experts or MTurk workers); ii) precision to measure the quality of the outcome of P each crowd, computed as T P T +F P ." . . . . "2019-11-10T18:05:11+01:00"^^ . .