The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial

This study examined the effects of feeding back experts’ initial ratings on three Delphi outcome measures: (1) the percentage of items on which experts changed their opinion; (2) the degree to which experts changed their ratings towards the group response; and (3) the increase in the level of agreement among experts. Additionally, two conformity indices were developed. Within a real-world Delphi study, experts were randomly assigned to one of two conditions: either their initial ratings were included in feedback (IN) or excluded from feedback (EX). Results showed that experts in the EX condition changed their opinion relatively more often than experts in the IN condition. Results also suggested that experts in the EX condition changed their ratings to a greater degree towards the group response than experts in the IN condition. No difference between conditions was found regarding the increase in the level of agreement.