http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#Head http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo http://www.nanopub.org/nschema#hasAssertion http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#assertion http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo http://www.nanopub.org/nschema#hasProvenance http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#provenance http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo http://www.nanopub.org/nschema#hasPublicationInfo http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#pubinfo http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://www.nanopub.org/nschema#Nanopublication http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#assertion http://purl.org/np/RA6f9C7xRc395-jmjLo3hSBCtMALLrwteL6DqWhjyx1fA http://purl.org/nanopub/x/introduces http://purl.org/np/RA6f9C7xRc395-jmjLo3hSBCtMALLrwteL6DqWhjyx1fA#figure http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#paragraph http://purl.org/spar/c4o/hasContent In the Verify stage of this type of task, the crowd outperformed the precision of the Find stage, achieving values of 0.5510 for the ‘first answer’ setting and 0.8723 with ‘majority voting’. This major improvement on the precision put in evidence the importance of having the a multi-validation pattern like Find-Fix-Verify in which initial errors can be reduced in subsequent iterations. Congruently with the behavior observed in the first workflow, MTurk workers perform well when verifying language-tagged literals. Further- more, the high values of inter-rater agreement con- firms that the crowd is consistently good in this par- ticular scenario. Figure 7b depicts the results of the ‘majority voting’ setting when classifying triples correctly, i.e., true positives (TP) and true negatives (TN), vs. misclassifying triples, i.e. false positives (FP) and false negatives (FN). We can observe that the crowd is exceptionally successful in identifying correct triples that were classified as erroneous in the previous stage (true negatives). This can be confirmed by the high value of accuracy 17 (0.9531) achieved by the crowd in this stage with ‘majority voting’. A closer inspection to the six false positives revealed that in three cases the crowd misclassified triples whose object is a proper noun, for instance, (Tiszaszentimre, name, Tiszaszen- timre@en) and (Ferrari Mythos, label, Ferrari Mythos@de) ; in the other three cases the object of the triple corresponds to a common noun or text in the following languages: Italian, Portuguese, and English, for example, (Book, label, Libro@it) . http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#paragraph http://www.essepuntato.it/2008/12/pattern#contains http://purl.org/np/RA6f9C7xRc395-jmjLo3hSBCtMALLrwteL6DqWhjyx1fA#figure http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#paragraph http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://purl.org/spar/doco/Paragraph http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#provenance http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#assertion http://www.w3.org/ns/prov#hadPrimarySource http://dx.doi.org/10.3233/SW-160239 http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#assertion http://www.w3.org/ns/prov#wasAttributedTo https://orcid.org/0000-0003-0530-4305 http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo#pubinfo http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo http://purl.org/dc/terms/created 2019-09-20T18:05:11+01:00 http://purl.org/np/RAtwDxBF1MJoX9AzkW5vkAFZ3mcprtiP6-MGybEB2Zevo http://purl.org/pav/createdBy https://orcid.org/0000-0002-7114-6459