The effect of controlled opinion feedback on Delphi features: Mixed messages from a real-world Delphi experiment

A real-world Delphi experiment was conducted to investigate the effect of two controlled opinion feedback conditions on the drop-out rate, experts' degree of opinion change, and the increase in the level of agreement among experts. Additionally, experts' perceived usefulness of feedback was explored. In the first and second Delphi round experts received a questionnaire which consisted of two sections. Within each section experts were asked to rate several items. In round 2, experts in one condition received feedback consisting of summary statistics and rationales (S&R condition), whereas experts in the other condition received rationales only (R condition). Results showed that drop-out of experts was greater in the S&. R condition than in the R condition. No difference between conditions was found concerning experts' degree of opinion change. The increase in the level of agreement across the items in the second section of the questionnaire differed significantly between conditions. This difference was mainly due to a decrease in agreement in the R condition, suggesting that feedback of rationales may increase disagreement among experts. In round 3 experts preferred to receive both summary statistics and rationales, although they tended to perceive rationales as somewhat more useful than summary statistics.