Best of the best: Who makes the most accurate decisions in expert groups?
Does a mass on a mammogram indicate breast cancer? Will Serbia be a member of the EU by 2025? Will there be more floods in Germany in five years’ time? The diagnoses and predictions made by doctors, scientists, and experts often have far-reaching consequences. And in many cases, it is only years later that it is possible to say which expert made the right call most often.
An interdisciplinary research team from the Max Planck Institute for Human Development and the Leibniz Institute of Freshwater Ecology and Inland Fisheries has developed a simple new method that can be used to identify the best decision-makers from a group of experts without having to know whether their decisions — past or present — are correct or incorrect. “Providing that at least half of all decisions made within the group are correct — which is typically the case in expert groups — and that each person has made about 20 yes/no decisions, this method has proved to work very well,” says Max Wolf, researcher at the Leibniz Institute of Freshwater Ecology and Inland Fisheries and co-author of the study.
The method was developed on the basis of insights into collective intelligence. It rests on a simple assumption: Those individuals in a group of experts who make decisions that are most similar to the decisions of others also make the best decisions. For yes/no decisions, this assumption is easily confirmed by means of mathematical modeling. To test whether the method also works in real groups, the researchers analyzed published predictions and diagnoses made by various groups in different fields.
For example, the researchers examined the diagnoses made by 100 radiologists in the USA. In the early 2000s, the radiologists interpreted the mammograms of 155 women to determine whether or not they had breast cancer. The research team analyzed the data to identify the radiologists whose decisions were, on average, most similar to the decisions of the others. As they had access to follow-up information on the health status of the 155 women screened, the researchers were also able to determine which radiologists made the most accurate and thus best diagnoses. They were the same radiologists as those identified using the new statistical method.
“It has been shown time and again that experts who are good in their field are good in a similar way, whereas poor performers are bad in very different ways. Working on the basis of this observation, we developed this new method and tested it in various areas,” says Ralf Kurvers, lead author and researcher at the Center for Adaptive Rationality at the Max Planck Institute for Human Development.
In addition to radiologists’ diagnoses, the research team analyzed skin cancer diagnoses made by 40 Italian dermatologists; geopolitical predictions made by 90 forecasters on the online platform Good Judgment Project; and the results of a simple general knowledge test, in which 100 participants were asked to identify the larger of two American cities.
“We believe that the relationship between similarity and accuracy of decisions can be an effective tool for practice. The method can be used to improve collective and individual decision-making processes in medical diagnostics, environmental risk analyses, and the business world,” says co-author Stefan Herzog, also a researcher at the Center for Adaptive Rationality.