Nervous, Nervous Roy

“Whit is so much cheaper than I am,” he says. “We were going to fly him out first class for a meeting. He said, ‘I can’t fly first class if I’m going to make this movie.’ He flew coach. His first bill for location scouting was $13.42.”

As lunch was winding down, Stillman filled a cup with coffee from a Dunkin’ Donuts Box O’ Joe. He spotted a dime on the floor. “Money!” he said reaching for the coin. “I can finance my next film!”

The dismal performance of the experts inspired Mr. Tetlock to turn his case study into an epic experimental project. He picked 284 people who made their living ‘commenting or offering advice on political and economic trends,’ including journalists, foreign policy specialists, economists and intelligence analysts, and began asking them to make predictions. Over the next two decades, he peppered them with questions: Would George Bush be re-elected? Would apartheid in South Africa end peacefully? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the pundits rated the probability of several possible outcomes. By the end of the study, Mr. Tetlock had quantified 82,361 predictions.

How did the experts do? When it came to predicting the likelihood of an outcome, the vast majority performed worse than random chance. In other words, they would have done better picking their answers blindly out of a hat. Liberals, moderates and conservatives were all equally ineffective. Although 96% of the subjects had post-graduate training, Mr. Tetlock found, the fancy degrees were mostly useless when it came to forecasting.