Signal and Noise: Why So Many Predictions Fail – but Some Don't

Citation

Subedi, D.K. (2013), "Signal and Noise: Why So Many Predictions Fail – but Some Don't", Competitiveness Review, Vol. 23 No. 4/5, pp. 426-430. https://doi.org/10.1108/CR-02-2013-0010

This is a book on forecasting. This is an important, interesting and timely book written by Nate Silver.

The introduction of this book begins by pointing out the difference between information and knowledge. While there are 2.5 trillion gigabytes of information added to the World Wide Web every day, most create confusion and spread myths rather than propagate knowledge. Unfortunately, we do not have an innate ability to extract knowledge from this ever growing din of “noise”. This book then goes on to discuss “Bayes theorem”. As per the theorem, there is nothing like a perfect forecast, but odds of success can be improved significantly by adjusting the initial forecast (or forecasting model) with relevant new information (introduction).

Nate Silver writes this book against a backdrop of his avid interest and experience in (mostly successful) forecasting related careers across various disciplines. He is currently the blogger for the New York Times' political blog, FiveThirtyEight.com. His claim to fame came from his successful prediction of the outcome of the 2008 presidential election (Chapter 2). However, even before that, as an avid baseball fan, he developed software, known as PECOTA, to forecast future performances of baseball players. He sold this software to Baseball Prospectus. When he started working on this system, he was still employed as an economic consultant with KPMG (an accounting firm) (introduction). What is more, in between his baseball software and political blogging career, he was a fairly successful gambler, playing poker online (Chapter 10).

This book has 13 chapters besides the introduction and conclusion. About half of it is devoted to explaining Bayes theorem. The introduction describes the difference between information and knowledge. It also gives an outline of the book. Chapters 1 through 3 discuss the fundamentals of forecasting, using, as examples, failure of economic models in forecasting the recent financial crisis (Chapter 1), successes and failures in political predictions (Chapter 2) and comparing the successes of data driven forecasting (such as his own PECOTA) and scouts who use gut feelings based on their observations to identify potential superstars (Chapter 3). Chapters 4‐7 extend the fundamentals of forecasting to more complex and dynamic systems. Chapter 4 discusses weather forecasting and Chapter 5 examines earthquake forecasting. Chapter 6 discusses economic forecasting and Chapter 7 examines the issues in modeling the spread of contagions such as flues.

Following these chapters, Silver turns to Bayes theorem. Chapter 8 uses the backdrop of a successful gambler (who bets on basketball games to make money) to introduce the concepts and mechanisms behind Bayes theorem. It is a process where the forecasts are “less and less wrong” in every step. Then, Chapter 9 uses the example of chess. It discusses how Garry Kasparov lost to IBM's supercomputer “deep blue”. It uses that example to discuss the uses of heuristics in forecasting. Chapter 10 is about his own experience playing poker. He explains the Bayesian style of thinking of successful gamblers. Then, in Chapters 11‐13, Bayes theorem is applied in the context of more dynamic systems – financial markets (Chapter 11), climate forecast (Chapter 12) and terrorism (Chapter 13).

Going through all the chapters, we see that this is a book on the practice of forecasting – not theory. There are no formulae here, except for a simple one on Bayes theorem. But many times its explanations of (both successful and unsuccessful) forecasting (drawn from the author's own works, as well as those of other professionals) sound like admonitions to sincerely follow the steps (directives and insights) found in standard statistics textbooks. At the same time, the author is also aware of the role of judgment in forecasting. This book goes on to discuss the relevant cognitive and behavioral limitations of human beings (including the so‐called experts) in detail. In total, this book shows that good forecasting is an outcome of sound judgment, as well as technical skills.

The arguments of this book are in agreement with those of Amos Tversky and Daniel Kahneman, as written in “Judgment under uncertainty: heuristics and biases” in 1974, and elaborated in Kahneman's book, Thinking Fast and Slow. The book reports on how people make inferior judgments because of their inability to follow the basic tenets of Bayes' logic in real life (Kahneman, 2011, Appendix A). The authors, both trained psychologists, used various experiments in lab setting to make their point. In Silver's book, real life examples from gambling to economic forecasting are used to make his point. More importantly it also reports cases where better forecasts are produced by understanding and applying Bayes' logic to given situations.

Kahneman states that humans are more likely to draw conclusions first and seek evidence later. In the process, they tend to discard information contrary to their beliefs (Kahneman, 2011, Chapter 3). This is evident in Silver's discussion on information and knowledge (introduction). More information does not always mean more knowledge. On the contrary, it can also provide enough data to support people's preconceived biases. People on both sides of arguments, be they political (Democrat vs Republican) or climate issues (presence or absence of “greenhouse effect”, for example), can dig up information, and argue endlessly. To look at all evidence without only looking to confirm one's preconceived notions requires vigilance, as well as lots of self‐control and discipline.

Predictions done by so‐called experts, many times, demonstrate their failure to process information in an unbiased manner. The experiments conducted by Philip Telock (a professor of psychology and political science) are cited in this context both by Kahneman (2011, Chapter 20) and this book (Chapter 2). Telock asked experts in politics to make predictions on a variety of issues. Their forecasts did not turn out to be much better than those produced by random chance alone. Either they were pandering to their biases, or they may have bought into too much of their own grandiose notions on how the world should look, making them unable to objectively evaluate the available information. Chapter 6 shows that economists are not so different. Their forecasts are not much better. On the other hand, political scientists who were not as biased were able to produce better forecasts. Their ability to learn from new information was definitely better.

We all like good stories and generally do not look at how they are structured. And neither are we good at understanding probability (Kahneman, 2011, Chapter 7). That is why we fall for persuasive arguments given by experts on TV shows and ignore information with likelihood estimates. People who have to communicate generally avoid discussing ranges and uncertainties and that can be misleading. For example, Silver explains, when the Obama administration stated that the unemployment rate could go down to 8 percent, it ignored the fact that such estimates come with an error of + and − 2 percent. Later, they faced criticisms about not hitting the mark, even though they were within the range (Chapter 1). In another case, when flood levels were forecasted to be 49 feet, people took that literally. Such forecasts are generally correct within + and − 9 feet (Chapter 6).

One of the important traits of human beings is pattern recognition. It enables us to store a vast amount of information despite the limited capacity of our brain. Yet this very trait, experts point out, makes us see patterns where there are none. Many times, people see patterns even in random numbers and take that as a “signal” when it is merely “noise” (Silver, introduction, Chapter 11; Tversky and Kahneman, 1974; Kahneman, 2011, Chapter 1). Technical analysts have a tendency to see such patterns in financial data. Fama introduced the efficient market hypothesis, which states all the information that is available is already incorporated in the stock price, when he found out that analyzing these patterns did not help him become a better stock picker (Chapter 11). On the other hand, there are instances where people have failed to see patterns. For example, there were plenty of signals given by the terrorists before 9/11 and the Japanese before the Pearl Harbor attack, which only became obvious after the fact. Connecting the dots requires theory (Chapter 13).

Spurious correlation is another closely related issue. When there is an abundance of data, much of it can have visible but false patterns and (based on the patterns) some of them can also have high correlations with each other. But presence of correlation should not mean there is causality as well. Theory is required to figure that out. Data can be used only to confirm what theory suggests. Economists produce large data and lots of indices and correlations, as well as many incorrect forecasts, based on many correlations. The standing joke is that economists have “called nine out of the last six recessions correctly”. One reason experts give for this dismal record in forecasting is that economics lacks solid theories required to understand cause and affect relationships (Chapter 6).

The most important issue in this book is related to Bayes theorem. It is about how new information can be incorporated to make forecasting better. Even in weather forecasting, which is based on solid science, with lots of data and huge computer support for modeling and data crunching, incorporation of human judgment from information not directly captured by data and models can improve the forecast by up to 25 percent (Chapter 4). Economic forecasts can also be improved by adjusting models with judgment. One study showed that when done properly economic forecasting improved by about 15 percent. However, judgments by economists can be fraught with biases. For example, economic forecasts produced by the White House are generally less accurate (Chapter 6).

A closely related field to economics is finance. Questions have come up like why did analysts in Moody's and S&P (credit rating agencies) fail to incorporate increasing evidence of the housing market bubble in their models (Chapter 1)? And, why did so many analysts fail to communicate the true value of many of the dot com stocks during the tech bubble era (Chapter 11)? One knowledgeable source told Nate Silver that “it's in everyone's interest to keep the market going up”. Traders are individually acting rationally. They make money by letting the ongoing bubble continue (Chapter 11). This is a classic case of agency problem. Forecasts are poor when forecasters have axes to grind.

Important insights in this book also come from its chapters on weather forecasting (Chapter 4) and economic forecast (Chapter 6). The most important part for me was the comparison of weather forecasting with economic forecasting. Weather and the American economy both produce enormous data. And, like weather, interactions between elements in the American economy are nonlinear and dynamic. However, despite these complications, weather forecasting is becoming very reliable. For example, 25 years ago a three day advanced forecast of the landfall of a hurricane would be within a range of 350 miles. Now that is reduced to a 100 mile radius (Chapter 4). On the other hand, the accuracy of economic forecasting has not improved that much. If you consider the forecasters' ability to call the recent recessions, it has not improved at all (Chapter 6). Economic forecasting has additional problems though. Unlike weather, its data are not very reliable. For example, data on the GDP or employment rate are revised after a few months. Second, physics and chemistry behind the cause and effect relationships in weather are well understood. Economics theory is not that established.

Looking at the dismal performances of economic (and financial) forecasting (and modeling), some experts dismiss the whole discipline of economics and (financial) modeling as useless (for example, Taleb, 2012). But, not this book. First, modeling can have other important roles besides forecasting. For example, a good model on the potential spread of the flu virus may not be a good predictor of the spread of the flu epidemic itself. Once the model is known, people change behavior (take flu shots/close schools, etc.) and stop the potential epidemic (Chapter 7). In economics, also, once recession is predicted, congress and governments can act to change the very outlook (Chapter 6). I would argue that if Moody's and S&P had successfully incorporated the housing bubble in their models, the bubble would not have been so big and widespread (Chapter 1).

Weather forecasting (looking from the landfall radius of a hurricane) was not very accurate 25 years ago. Now we should all be glad that nobody called for abandoning weather forecasting. Of course, we all agree that economic forecasting has to improve before it becomes reliable. However, as frustrating as it is at present, stopping the progress altogether is not the answer. More importantly, it is not pleasant reading an assessment of financial analysts' behavior from a New York Times journalist (i.e. Nate Silver) (Chapters 1 and 11). The serious ethics and accountability issues should be addressed. It would at least make the financial models more reliable, in the future. Of course, there is more to it than improving forecasts.

I enjoyed reading this book. It would be a good read for all my colleagues (i.e. business/economics professors) and students and also former students who are managing business or government institutions.