Probing the power of Apollo: methodological challenges and opportunities of the Delphi methods for developing rankings

The Delphi method is a structured data-collection process aimed at facilitating experts to achieve a certain level of agreement on a complex problem. In several subsequent rounds experts give their opinion on the problem of interest, usually by means of a standardized questionnaire. After each round experts receive controlled opinion feedback in the form of a summary of the findings. Based on this feedback experts are allowed to reconsider their opinion in the next round.

Although the Delphi method seems to provide opportunities for developing rankings, research into its application to the development of rankings seems to be non-existent. Furthermore, the Delphi method has several unresolved methodological issues, in particular regarding the measurement of agreement and the provision of controlled opinion feedback. Therefore, the general objective of this PhD project was to find out what the methodological challenges and opportunities are of the Delphi method for developing rankings.

To achieve this objective, four studies were conducted. In the first study, simulations were performed to find out how various agreement indices behave within and across the rounds of a Delphi study. In the second study, the Delphi method was applied to develop a ranking of research domains in landscape architecture. An experiment within the study investigated the effect of two types of controlled opinion feedback on the drop-out rate, experts’ degree of opinion change, and the level of agreement among experts. The third study investigated the methodological characteristics of six urban sustainability rankings, which resulted in a recommendation to define and operationalize the concept ‘urban sustainability’ using the Delphi method. Therefore, in the fourth and final study the Delphi method was applied to find out which components experts find most relevant for defining and measuring urban sustainability. An experiment within this study provided insight into the effect of feeding back experts’ initial ratings on the degree of opinion change and the level of agreement.

Overall, this PhD project showed how the Delphi method may be used to: (1) obtain a ranking of objects on a ranking attribute and (2) obtain a definition and operationalization of a complex ranking attribute. It was also shown that these applications of the Delphi method do not come without challenges. First, selection criteria and search strategies need to be developed by which sufficient numbers of different types of experts may be found. Second, for the first Delphi questionnaire a parsimonious list of items (e.g. ranking objects or components of a ranking attribute) needs to be drawn up that largely covers the potentially great diversity of existing items. Third, careful considerations need to be made about which types of information (i.e. summary statistics, rationales, experts’ own initial ratings) to feed back to experts after every round as this may influence various Delphi outcome measures. Fourth, the choice for a particular consensus, agreement, or association index needs to be carefully made and justified as it determines the level of agreement among experts that is obtained. Finally, this project showed that by means of a simulation study and two real-world Delphi experiments new knowledge about the functioning of the Delphi method could be acquired. More of these studies are needed to establish evidence-based guidelines and to uncover the full potential of the Delphi method for developing rankings.