. . . . "Given these insights, RQ3 was formulated: How can we design better crowdsourcing workflows using lay users or experts for curating LD sets, beyond one-step solutions for pointing out quality flaws? To do so, we adapted the crowdsourcing pattern known as Find-Fix- Verify, which has been originally proposed by Bern- stein et al. in [3]. Specifically, we wanted to know: can (i) we enhance the results of the LD quality issue de- tection through lay users by adding a subsequent step of cross-checking (Verify) to the initial Find stage? Or is it (ii) even more promising to combine experts and lay workers by letting the latter Verify the results of the experts’ Find step, hence drawing on the crowds’ complementary skills for deficiency identification we recognized before?”" . . . . "2019-11-12T12:34:11+01:00"^^ . .