In October 2022, a collective-intelligence research group shared a study suggesting that organisations should reward disagreement to avoid herd behaviour. On 21 October, Émile Servan-Schreiber, who has run prediction markets since 2000, posted a pointed reply. He warned against academic articles promising methods that outperform prediction markets, which he said are often written by people who would not recognise a real one.
The exchange is worth unpacking because it turns on a practical question: how should a forecasting system pay the people who feed it information?
What the paper modelled
The study is “Optimal incentives for collective intelligence” by Richard P. Mann of the University of Leeds and Dirk Helbing of ETH Zurich, published in PNAS in May 2017. It is a mathematical model, not an experiment with people. Simulated agents each pay attention to one of many pieces of information, cast a vote on an outcome, and get rewarded if they are right. Over time, agents copy the strategies that pay best. The group’s answer is its majority vote.
The authors compared three ways of paying correct agents. The first two come from earlier work by Hong, Page and Riolo.
- Binary rewards: everyone who is right gets the same fixed reward.
- “Market” rewards: a fixed pot is shared equally among everyone who is right, so each share is larger when fewer people are right. The authors write that this closely mimics the reward system of actual prediction markets.
- Minority rewards, their proposal: a correct agent is rewarded only if fewer than half of the others were also correct.
In their simulations, market rewards pushed agents to crowd onto the same few pieces of information, and on complex problems the group did worse than if attention had simply been spread evenly. Minority rewards kept agents spread out and gave near-perfect collective accuracy. The authors concluded that this “sheds doubt” on the accuracy of existing markets.
real-world systems should reward those who have demonstrated accuracy when majority opinion has been in error
Mann and Helbing, PNAS (2017)
The objection: that is not how markets pay
Servan-Schreiber’s argument is that the paper’s labels are the wrong way round. A fixed pot split equally among everyone who picked the right answer is how pari-mutuel betting works, as at a racetrack. It is not how a prediction market works.
In a prediction market, what you earn depends on the price you paid. A contract that pays 100 if an event happens might trade at 20 when most traders think it unlikely and at 90 once most agree. Someone who buys at 20 and turns out to be right gains 80; someone who buys at 90 gains only 10. In his words, to win you must be right before most others. That, he argues, is close to the paper’s minority rewards, and the same holds for prediction polls that score forecasters against the crowd.
Read that way, he wrote, the paper does not cast doubt on prediction markets. It shows that they are more accurate than pari-mutuel betting and may be close to optimal at pooling scattered information, which he called “interesting but hardly novel”.
Where the two sides agree
Servan-Schreiber endorsed the paper’s conclusion that accuracy against the majority deserves the reward. His objection was to the claim of novelty: prediction markets, he wrote, “have been doing that since 1988”. That is the year the University of Iowa launched the Iowa Electronic Markets, a real-money market best known for its election contracts.
The authors themselves left the door open. They called for experiments on how real people respond to minority rewards, and for work on how such schemes could be applied to science funding, reputation systems and prediction markets.
