Prediction, being popular and being right
Looking at the categories listed on the right-hand side of this blog, I noticed that the two I've used most are "prediction" and "mobile computing". This seemed ironic given that prediction and mobile computing are things that I mostly avoid. Of course the tags reflect the number of posts in which I've taken issue with buzzword-driven predictions and the black box fallacy. Reading the works of Duncan Watts and Daniel Kahneman recently gave me further justification for staying out of the prediction business, refined my dim view of it, and gave me some constructive thoughts on how to deal with forecasting.
According to both Watts and Kahneman, predictions made by supposed experts in a variety of fields are no better than random guesses. This, they suggest, is due to the prevalence of "unknown unknowns" in complex systems like politics, human societies, the economy, and the environment: in systems like this, there are invariably numerous unforeseeable influences that would-be forecasters can't even know that they ought to be thinking about, let alone predict. Experts nonetheless make bold predictions because confidence sells: few people have much interest in "experts" who can only give them uncertain and equivocal advice, so experts are under great social and financial pressure to make bold and confident predictions that, by and large, they won't be held accountable for anyway.
Reading the chapter of Kahneman's Thinking, Fast and Slow on this, I was reminded of a misgiving that I had about Martin Seligman's work on positive psychology. Seligman observes that optimists frequently blame failure on specific changeable circumstances, giving the optimist hope that he or she can still succeed when trying again under (hopefully) other circumstances. Seligman contends that this is good for the optimist, who will persist, but seems to be largely uninterested in its consequences for everyone else. His favourite example seems to be the selling of insurance through cold-calling, in which the heroes are those optimists who persist despite being knocked back by nine out of ten potential customers. Seligman shows no interest in the fate of the nine out of ten people who had to endure an unasked-for sales pitch for a product they didn't want.
So it seems that being a bold and confident personality might be beneficial for an individual while being detrimental to the societies or organisations to which they belong. Kahneman cites the example of optimistic CEOs who make bold and impressive-sounding decisions that wind up losing money for their companies. (He also observes that the opposite might happen: the economy as a whole benefits from the optimism of would-be business owners who start ventures even though statistics indicate that only one-third of such ventures actually succeed).
Reading Kahneman's book (and maybe this blog), it's easy to feel like all prediction is hopeless. Of course there are plenty of things we can predict with great confidence: engineers, for example, routinely make accurate predictions about the behaviour of structures, machines and computers based on well-tested models of how the the world works. But few people find such predictions very interesting, perhaps because they are so routine. There's simply more glory to be had in making hopelessly inaccurate predictions and ignoring their consequences.
