On 1 June 2016, Émile Servan-Schreiber announced that Hypermind had won the Republican nomination race, in the sense that matters to a forecaster. The Associated Press had counted Donald Trump past the delegate threshold on 26 May. Looking back over the primaries, Hypermind’s blog compared its forecasts with those of three better-known markets and found them clearly more accurate. The rest of 2016 would test that confidence, and the lessons Servan-Schreiber drew from it are as instructive as the win.
The nomination race
The comparison ran from 25 January to 3 May 2016 and covered a four-way question: Ted Cruz, Marco Rubio, Trump, or someone else. The three rivals were Betfair, a UK betting exchange that the blog described as the world’s largest prediction market, and two US real-money markets: the long-running Iowa Electronic Markets (IEM) and the newer PredictIt. Hypermind is a play-money market. Its traders bet virtual currency, and cash prizes go to those whose forecasts prove most accurate.
The charts show the difference in behaviour. According to the blog, Hypermind was consistently more bullish on Trump and less inclined to overreact when he stumbled. The clearest case came in April, when the idea of denying him the nomination at a contested convention gained ground in every market, but much less on Hypermind.
Accuracy was measured with the Brier score, which adds up the squared differences between the probabilities given and what actually happened, so lower is better. For a four-way question, a perfect forecast scores 0, a chance forecast of 25% on each option scores 0.75, and a forecast that is totally wrong scores 2. Hypermind computed each market’s score every day and averaged them.
- Hypermind was 35% more accurate than Betfair.
- It was 40% more accurate than the IEM and PredictIt.
These are Hypermind’s own figures. The blog attributed them to how its traders were chosen: a few thousand people, selected and rewarded only on performance. The market had grown out of the collaboration between Lumenogic, as the company behind Hypermind was then called, and the Good Judgment Project, which won a US intelligence research forecasting tournament.
Brexit: one chance in four
Three weeks later came the UK referendum on leaving the European Union. On 16 June, Servan-Schreiber reported that the market had tightened to 45% for Brexit, the same day he told France 24 that prediction markets were still betting on Remain. By the eve of the vote, Hypermind gave Brexit 25%. Leave won with 51.9% of the vote, according to the BBC.
Servan-Schreiber’s response, in a blog post titled Lessons from Brexit, was that a 25% event happening is not a failure of the forecast. It is what 25% means: it should happen about one time in four.
Only those who make the mistake of confusing 25% (unlikely) with 0% (not a chance) could blame Hypermind.
Émile Servan-Schreiber, Hypermind blog, June 2016
The test of a probabilistic forecaster, he argued, is calibration: across many questions, do events given 25% happen about a quarter of the time? He reported that over two years and 181 questions, Hypermind’s probabilities had broadly matched observed frequencies.
November: the FBI letter and the result
On Sunday 6 November, two days before the presidential vote, FBI director James Comey told Congress that a review of newly found emails had not changed the bureau’s conclusions about Hillary Clinton’s private email server. The next morning, Servan-Schreiber reported that Clinton’s chances on Hypermind had risen overnight from 65% to 75%, adding that Trump still had at least one chance in five.
Trump won. In Did Donald Trump the Wisdom of Crowds?, Servan-Schreiber called it a failure across the board for prediction markets, as Brexit had been. He wrote that on election day Hypermind had given Trump at most 25%, more than any other crowd-based system it compared, including Betfair and PredictIt. Clinton did win the popular vote, which Hypermind and the IEM had given her about 80% chance of doing, he noted. He also pointed out that only 47% of Hypermind traders who had bet on Brexit also bet on a Trump presidency.
An independent comparison in The Washington Post, by Pavel Atanasov and Regina Joseph, both formerly of the Good Judgment Project, scored the major forecasts on the Electoral College. Every method had a Brier score above 0.50. On the last day of the campaign, Good Judgment and Hypermind were the most accurate, at 0.56 and 0.61 respectively, or, as the headline put it, the least wrong.
What the year shows
Hypermind’s 2016 contains both a clear win and two high-profile misses, and Servan-Schreiber’s argument covers all three. A probability forecaster should be judged on many questions, not on the most memorable one. On the nomination, that meant months of daily scores. On Brexit and the White House, it meant accepting that one-in-four events do happen, and checking calibration over hundreds of outcomes instead.
