Conflation of expert and crowd reference data to validate global binary thematic maps

François Waldner*, Anne Schucknecht, Myroslava Lesiv, Javier Gallego, Linda See, Ana Pérez-Hoyos, Raphaël D'andrimont, Thomas De Maet, Juan Carlos Laso Bayas, Steffen Fritz, Olivier Leo, Hervé Kerdiles, Mónica Díez, Kristof Van Tricht, Sven Gilliams, Andrii Shelestov, Mykola Lavreniuk, Margareth Simões, Rodrigo Ferraz, Beatriz BellónAgnès Bégué, Gerard Hazeu, Vaclav Stonacek, Jan Kolomaznik, Jan Misurec, Santiago R. Verón, Diego De Abelleyra, Dmitry Plotnikov, Li Mingyong, Mrinal Singha, Prashant Patil, Miao Zhang, Pierre Defourny

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

14 Citations (Scopus)


With the unprecedented availability of satellite data and the rise of global binary maps, the collection of shared reference data sets should be fostered to allow systematic product benchmarking and validation. Authoritative global reference data are generally collected by experts with regional knowledge through photo-interpretation. During the last decade, crowdsourcing has emerged as an attractive alternative for rapid and relatively cheap data collection, beckoning the increasingly relevant question: can these two data sources be combined to validate thematic maps? In this article, we compared expert and crowd data and assessed their relative agreement for cropland identification, a land cover class often reported as difficult to map. Results indicate that observations from experts and volunteers could be partially conflated provided that several consistency checks are performed. We propose that conflation, i.e., replacement and augmentation of expert observations by crowdsourced observations, should be carried out both at the sampling and data analytics levels. The latter allows to evaluate the reliability of crowdsourced observations and to decide whether they should be conflated or discarded. We demonstrate that the standard deviation of crowdsourced contributions is a simple yet robust indicator of reliability which can effectively inform conflation. Following this criterion, we found that 70% of the expert observations could be crowdsourced with little to no effect on accuracy estimates, allowing a strategic reallocation of the spared expert effort to increase the reliability of the remaining 30% at no additional cost. Finally, we provide a collection of evidence-based recommendations for future hybrid reference data collection campaigns.
Original languageEnglish
Pages (from-to)235-246
JournalRemote Sensing of Environment
Early online date23 Nov 2018
Publication statusPublished - Feb 2019


  • Accuracy assessment
  • Crowdsourcing
  • Data quality
  • Photo-interpretation
  • Stratified systematic sampling
  • Volunteered geographic information


Dive into the research topics of 'Conflation of expert and crowd reference data to validate global binary thematic maps'. Together they form a unique fingerprint.

Cite this