Saturday, May 3, 2014


James Surowiecki at The New Yorker had a nice column last month on "Punditonomics," the tendency of much public discussion to focus on individuals who seem to have forecast one or two big events in the past.

The economic incentive is clear:
Experts in a wide range of fields are prone to making daring and confident forecasts, even at the risk of being wrong, because when they're right the rewards are immense. An expert who makes one great prediction can live off the success for a long time; we assume that the feat is repeatable. 
But, being right once is pretty meaningless

The most comprehensive study in this field was done by the psychologist Philip Tetlock. [JC: link here.] Over many years, he tracked some three hundred experts, asking them to estimate the probability of various geopolitical events. He found that, though a given expert may foretell one extreme event, dosing so consistently was next to impossible. Experts who foresaw the breakup of Yugoslavia also thought, wrongly, that Hungary and Romania would slide into civil war. Being spectacularly right once doesn't guarantee being right in the future. In fact, the opposite may be true. In one fascinating study by the business school professors Jerker Denrell and Cristina Fang, [JC: Link here] people who successfully predicted an extreme event had worse overall forecasting record than their peers.
Most of James' examples are financial
The history of forecasting is littered with examples of experts who were acclaimed as visionaries, only to disappoint. Two weeks before the Great Crash of 1929, Irving Fisher, one of the pioneers of economic forecasting, declared that stock prices had reached a "permanently high plateau." In the late seventies, the market-timing abilities of the investment guru Joe Granville were legendary, but he completely missed the beginning of the bull market in 1982. Elaine Garzarelli, who correctly called the crash of 1987, pronounced in October of 2007 that she was "absolutely bullish" on the stock market. That year, the banking analyst Meredith Whitney became famous for her bearish but accurate prediction that Cititgroup would have to slash its dividend and take billions in writedowns. But she was woefully wrong when, just a few years later, she warned, on "60 minutes," that cities in the U.S. were likely to default, resulting in "hundreds of billions of dollars in losses to investors. [JC: so far!] ... 
Criticizing financial forecasts is, I think, a bit too easy. Blog readers will have more in mind the many debates over who saw the financial crisis coming or didn't, who called the housing "bubble" or didn't, who thought the recession would turn in to another great depression or didn't, who saw the current endless slump coming or didn't, who saw the european debt crisis or didn't, who saw its end or didn't, who has been expecting inflation, who has been expecting deflation, who said that reduced government expenditures would lead to a new recession, and so on and so on. The whole Paul Krugman - Niall Ferguson debate over who said what when comes to mind too.

I think this little article makes clear why these are such hopeless and profoundly unscientific debates. (Which, you may have wondered, is why I have completely ignored them so far.)  Mining old blog posts for successes -- or damning people and their "models"  for selected failures -- proves nothing.   To learn something about  about economic logic and the ability of economic ideas to understand cause and effect, you at least have to assemble an entire forecast record.

More deeply, any serious forecast, reflective of the worth of an economic theory, must be written down and divorced from the judgment of the forecaster. Even if, say, Bob Shiller turned out to be a psychic who could tell when bubbles were happening, and never got a forecast wrong, that is fairly useless knowledge unless Bob can somehow write down his process so that someone else can do it too.  Otherwise, this is like saying to a climate scientist, "well you thought it would rain last weekend, so surely 'your model' is wrong."

Academic economics does this. We wrote down models, and test them by whether the models' predictions, in anyone's hands, agrees with the data.  It's interesting that the policy debates, even by ex academics, goes back to such solidly pre-scientific witch-doctor evaluation.

Source link 
Another cool link on this subject was sent to me by a colleague. The graph on the left comes from a speech given by Masaaki Shirakawa, Governor of the Bank of Japan.  We usually think of economists as fallible, but demography, well, that should be easy to forecast. Apparently not so.

I include this picture for its beautiful art. Forecasts of GDP, inflation, budget deficits, and interest rates all look about this way. Forecasts should also always include standard errors, but past histories in this graphical form might be more informative and easier to communicate.


  1. Is it really true that finance is too easy? It seems terribly hard to me. For example, I don't think either Krugman or Ferguson or anyone else for that matter can predict what the US long bond rate will be 12 months from now. But you are surely right that a valid prediction requires a reproducible methodology.

    1. I meant that criticizing financial forecasts is too easy -- and forecasting financial prices too hard! I clarified that, thanks.

  2. I love the Chart. Clearly the Japanese are unique in their inability to forecast important demographic and economic variables. I'm confident that the US does not suffer from such lack of skill. :-)

    Moral of the story? A person confident in his ability to make such forecasts or who quotes such forecasts as authority sufficient to win a debate can, on Bayesian grounds, be largely ignored.

  3. If events cannot be consistently predicted it tells us something about the experts but it also tells us a lot about the nature of the processes that lead to the events.

    There are some "experts" I would be more likely to trust than others:

  4. If events cannot be consistently predicted it tells us something about the experts but it also tells us a lot about the nature of the processes that lead to the events.

    There are some "experts" I would be more likely to trust than others:

    (Professor Cochrane - my browser is acting up. If this comes up as a double post please eliminate one of the copies. I apologize for any confusion.)

  5. I know this is OT, but a lot of attention is being given to the new Pikketty book, largely because of its support for Obama's talking points about the economy. I hope that some economists who are more friendly to free markets, like you, will read it and explain it to the unwashed like me.

    1. Marginal revolution is doing a good job of collecting all sorts of thoughtful Pikketty reviews.

    2. I still would like your thoughts.

    3. A compendium of MR links to reviews/thoughts on Piketty can be found here (note that the page does not have a favorable view of MR's 'crusade', but it's still a useful compilation):

      For what it's worth, Fat Man, I think you have the relationship backwards. Inequality was presumed to be non-issue, and, for better or worse, work from Saez, Piketty et al brought it to the limelight. Obama's talking points are the effect, not the cause. I also hope you aren't suggesting that only the opinions of economists who are "friendly to free markets" are the opinions that matter.

    4. Yes, Piketty wrote the book before Obama made it an issue. OTOH, if the White House had not decreed it to be an important issue, the book would be debated in back pages of an obscure journal of economic history.

      Further, I assert as matter of faith and morals that only the opinions of free market economists are worthwhile. The rest of them are a bunch of partisan hacks.

    5. As a matter of faith and morals, you only listen to folk who say what you want to hear? A fascinating philosophy. You should actually consider also subscribing to Supply Side Liberal, it's a great blog for moderates like myself.

      Anyways, inequality debates started years before the White House ever mentioned it, namely with Occupy, too big to fail, etc. You can even find economic opinion pieces going back to the late 1990's discussing it. You are therefore clearly mistaken here.

    6. Jeff: Commenters like you do not make a case for seeking out leftist viewpoints. Further, in modern America it is impossible to ignore the constant repetition of left-wing propaganda in the mainstream media.

      Further, it is true that the Democrat party (or at least the Wallaceite wing) has loved redistribution-ism since the the Great Depression of the 1930s. Obama was clearly a redistributionist when he was elected. I would never accuse him of having had an original thought.

      But, the current level of publicity is determined by a White House desperate to distract the electorate from their own multiple disasters in domestic and foreign policy. The media attention is based on the media's own habit of saying: "Yes, Sir, How high Sir?" when ever the White House says frog.

    7. I think you've entrenched yourself too deeply into tribal politics. I recommend you stop thinking in terms of "Leftists" vs. "Free Marketers" and instead take things issue by issue. Don't pay attention to parties. There's no such thing as a party taking an action, or a party belief. Persons take actions. Persons have beliefs, and they vary independent of partisan politics more often than you think.

      My abbreviated reason for seeking view points that don't neatly fit into "Free Market" thinking is summarized nicely by Mr. Albert Einstein.

      "Whoever undertakes to set himself up as a judge of Truth and Knowledge is shipwrecked by the laughter of the gods."

    8. Jeff: Don't worry. When I want your advice, I will send you a handwritten letter asking for it.

      Just so our genial host, does not forget it, I still would like his thoughts about Piketty.

      Jeff: Not your thoughts. Such as they are.

  6. One of the toughest things about finance is that there's a difference between inevitable and imminent. Predicting the correct outcome without the right timing frequently has the same effect as being wrong. The JGB widowmaker trade comes to mind, as well as the people who shorted tech too early in the 90s and capitulated. Meredith Whitney may turn out to be right but if you followed her advice on the timing the outcome was in distinguishable from "wrong".

    BTW, it was Elaine Garzarelli, Mr Surowhiskey.

  7. Often, comparing models to data is subjective. Suppose someone comes along with a statistical relationship between inflation and M1 growth that, over the past 50 years, performs pretty well. Someone else comes along with a qualitative model that is not good at making quantitative predictions, but is able to recognize when we will have a rapidly growing money supply and not much inflation. After how many years of rapid money growth without the usual amount of inflation do we need to say that the first model is not applicable to the current macroeconomy? Are the past 50 years the relevant data or just the past 5 years?

  8. I PREDICT that this essay, accurate as it is, will not be well received by many!

    (Favorite sentence award goes to: "It's interesting that the policy debates, even by ex academics, goes back to such solidly pre-scientific witch-doctor evaluation.")

  9. "Blog readers will have more in mind the many debates over who saw the financial crisis coming or didn't, who called the housing "bubble" or didn't..."

    "We wrote down models, and test them by whether the models' predictions, in anyone's hands, agrees with the data..."

    I would add the importance, as stressed by Tetlock, of making these predictions testable. So much of "financial prediction" resembles fortune-telling and palm-reading: it's sufficiently vague that it can accommodate completely different outcomes.

    Even if you can get testable hypotheses, I suspect you'll run into the same issues of "value adjustments" and arguments about "close call counterfactuals" encountered by Tetlock.

    Lastly,the Niall Fergusons and Paul Krugmans of the world strike me as prototypical extreme "hedgehogs". I just keep that in mind if I get stuck reading something of theirs...

  10. Prof Cohrane : great post; somewhat related -- if you're interested your colleague Reid Hastie at Booth has done some interesting work in building models of individual expert judgment.


Comments are welcome. Keep it short, polite, and on topic.

Thanks to a few abusers I am now moderating comments. I welcome thoughtful disagreement. I will block comments with insulting or abusive language. I'm also blocking totally inane comments. Try to make some sense. I am much more likely to allow critical comments if you have the honesty and courage to use your real name.