Library / Behavioural Finance
Date of review: October 2021
Book author: Philip Tetlock and Dan Gardner
Вook published: 2015
Book author: Philip Tetlock and Dan Gardner
Вook published: 2015
Superforecasting. The Art & Science of Prediction by Philip Tetlock and Dan Gardner (2015)
I finally re-read the book which I first bought and read about 5 years ago. I have been hearing and reading very positive comments about this book from many professionals I highly respect. In parallel to that, I have become very interested in the subject of decision making, especially in investing.
Learning the key concepts in investing, learning about various sectors, business models, valuation tools, and so on - is only half of what is needed. The other half is how you apply this knowledge, how you make your judgement. Essentially in investing, like in many other parts of life, you deal with two issues: incomplete information (you do not have all necessary information in advance) and luck (few things are certain, we deal with probabilities). How do you incorporate this into your thinking process, and especially what do you do when you receive some negative information, how do you decide if you need to change your view or if this is just noise to be ignored? I have a few books in a queue that I plan to read in the near term, and perhaps I will form a more universal approach/technique to making better decisions.
So far, I will share some key ideas that I was able to pick up in the book.
Firstly, to be objective, I would like to share what I consider a drawback. The book is not very clear, in my view. The author has done a fantastic job of real-life experiments with forecasting over a 20-year period. But I would have preferred if he just focused on this study, takeaways and conclusions.
Instead, there are many different stories from real life, as well as science and and a bit of fiction. This makes following the author quite difficult. Perhaps, it is the result of having actually two authors working on the book, perhaps they thought that if they had simply presented the key ideas, the book would be too dry.
So far, I will share some key ideas that I was able to pick up in the book.
Firstly, to be objective, I would like to share what I consider a drawback. The book is not very clear, in my view. The author has done a fantastic job of real-life experiments with forecasting over a 20-year period. But I would have preferred if he just focused on this study, takeaways and conclusions.
Instead, there are many different stories from real life, as well as science and and a bit of fiction. This makes following the author quite difficult. Perhaps, it is the result of having actually two authors working on the book, perhaps they thought that if they had simply presented the key ideas, the book would be too dry.
Now, to the essence of the book
As you may have guessed, the author (Wharton professor) has carried out a multi-year competition asking different people to make predictions about specific events like 'Will US launch a military operation against Syria next year?'. He identified a small group of people (Superforecasters) who have done much better, many well-regarded experts in political science and international affairs. At the same time, the majority of 'experts' delivered results that were "roughly as accurate as dart-throwing chimpanzee".
The central message of the book is that you do not need super intelligence to make better-than-average predictions, it is how you use your skills, how you treat new information, how much you practice and how much you learn to get better.
I tried to summarise key qualities of Superforcasters below as well as key weaknesses of ordinary people which prevent them from making better judgements about the future. There are some practical suggestions on how to improve your forecasting skills.
The central message of the book is that you do not need super intelligence to make better-than-average predictions, it is how you use your skills, how you treat new information, how much you practice and how much you learn to get better.
I tried to summarise key qualities of Superforcasters below as well as key weaknesses of ordinary people which prevent them from making better judgements about the future. There are some practical suggestions on how to improve your forecasting skills.
Key qualities of Superforcasters
1. Avoid beliefs and big ideas, keep your ego low and away.
To make accurate forecasts, it is important to remain open-minded and be willing to change your views with new information. It is much harder to do if you have publicly stated certain things or associated yourself with a particular group (Green, Liberal, Christian), in which case you would focus on defending your beliefs rather than seeking truth. Accepting you are wrong is difficult, but it is even more difficult if you have stated things publicly over a long period of time and hence have a lot of ego invested into a specific idea. I think this is what traders are so good at compared to more fundamental investors. Traders can change their minds, accepting they were wrong, while many fundamental investors would hide their bias behind such virtues as patience and focus on fundamentals. This can lead them to keep an eye on just a narrow group of indicators and expose them to the risk of missing bigger and more important factors. Being disciplined and following some key rules could help here.
2. Fox vs Hedgehog: Fox is better.
Hedgehogs tend to focus on one big idea while Foxes would look at different concepts, seek different kinds of information and try to understand various viewpoints of different people. This has produced better forecasts historically.
3. Keeping score, hard work and practice.
Superfocasting is not about exceptional skills but rather the result of long practice and efforts. The first important step in the process is keeping the score of your forecasts, for which your forecasts have to be time-bound and relatively specific, although not deal with events influenced by too many random factors. It is important to understand what led you to wrong decisions, how much of it was due to judgement rather than pure luck and what you can improve in the future. I think maintaining an investment diary, writing investment theses in advance and then reviewing them is critical for successful investing. The only question is to what extent one has to share this in public, as this could make it more difficult to change one's views later.
4. Ability to update your forecasts (Bayesian equation).
There is a section in the book about how to change/update your assessment of probabilities with new information based on the Bayesian equation. I feel like I did not fully understand it in order to easily apply it to various real-world situations, so I am not going to provide all details. However, this is something on my list to research better. I remember well how I was struggling to solve a famous Monti Hall puzzle (and in the end got the wrong answer) not so far ago. I now understand the logic behind it, but I think there are more complex situations in investing when updating your views once new information becomes available can raise your odds of winning dramatically.
5. Outside view and watching the base rate.
When faced with a problem, a typical person tries to analyse the situation paying attention to all kinds of facts and specific details. This is called the Inside View. But often, a better way of assessing the problem would be to look at general standard features about the case and then look back at history to see what has been an average outcome of such cases. In investing, this could be applied to fast-growth companies - rather than trying to analyse what makes those companies grow so fast, a better way to estimate for how long this fast-growth phase would continue would be to check statistics. The average life of a public company is also a useful starting point (base rate). The book also discusses the idea of taking an outside view as a way to remove yourself from your beliefs and prior work (sunk costs) to take a fresh look.
6. Group work and seeking counter-arguments.
Superforecasters are good at taking the role of outsiders looking at the situation with a fresh pair of eyes or even looking at their own prior analysis as being completely new to it. But working in a group can help even more, and even superforecasters can benefit from it. You should constantly seek alternative views and opinions of others who may find flaws in your own judgement. Understanding a different perspective on the same situation is much more helpful than getting feedback from the people who think the same. The author briefly discusses the AOM concept (Active open-mindedness) introduced by the psychologist Jonathan Baron.
7. The world is a much more uncertain place than we think.
Poor forecasters (the majority of people) tend to have a deterministic view. They think most events happen for a reason. This helps to have peace of mind as opposed to living with the idea that the world is a completely random place and anything can happen tomorrow. But the downside of such a traditional view is that you miss many elements of the world view and your final assessment becomes flawed if you try too hard to fit facts into a narrative/framework. Tetlock writes that the study revealed a significant correlation between the precision of forecasts and individuals' beliefs in fate - "the more a forecaster inclined toward it-was-meant-to-happen thinking, the less accurate her forecasts were. Or, put more positively, the more a forecaster embraced probabilistic thinking, the more accurate she was". "So finding meaning in events is positively correlated with well-being but negatively correlated with foresight".
8. Answer the correct question.
Many people make poor predictions because they answer different questions. Instead of thinking about whether an event can happen, they try to decide what SHOULD happen. This often happens when individuals put emotions into a particular situation. For example, when asked whether rebels in Syria could succeed against Assad's army, more people thought they would because they kept images of all the 'sins of the dictator' in their heads and, thus, answered the question 'Why rebels should win', or 'What they wish would happen to Assad'. Thinking that way, forecasters missed the fact that Assad's army was way much bigger. Another useful point made by Tetlock is to twist or reverse the question. Answering the opposite question allows you to take other details into consideration and get a better result (e.g. instead of answering 'Will North Korea launch a missile strike?' try answering 'Will North Korea keep it peaceful'?
9. Tips for making use of new information.
Updating your forecast when new information comes up can be even more difficult than making an original one. Two issues often prevent us from being good at adjusting our estimates: we either under-react to new information or over-react. In the first instance, we may have made a big statement and developed the whole philosophy around a particular subject and made all this public. In this case, "superforecasters may have a surprising advantage: they're not experts or professionals, so they have little ego invested in each forecast". On the other hand, when we are presented with too much information, we may lose confidence in our original judgement, which means we over-react. This often happens with traders reading dozens of news articles a day, most of which have limited new information but add to the overall confusion. One good suggestion is to think first, then take a break and come back to the same problem the next day. This looks quite useful in investing, too, developing your thesis over time and adjusting position size accordingly rather than putting 100% of your portfolio.
There are two other interesting pieces in the book
One deals with leadership. Obviously, there is a conflict between the qualities needed to be a good forecaster and leading others. One set of qualities underscores the importance of humility and the ability to change views often, while the other - has strong beliefs, vision and confidence. If I understood correctly, the author suggests placing confidence in an overall strategy and values but remaining humble on methods and tactics. I think Jeff Bezos could be a good example: he has had a big belief that the Internet would change the way we live, but he was also very opportunistic in how to achieve his goals.
I think in investing, it is also important to have strong beliefs that paying a price substantially below what the company is worth is key. Equally important is to find great businesses and stick to them. But an investor should be opportunistic in his approach to finding such opportunities. It could be a low PE or high FCF yield stock, but it could be not. It could be a cyclical company, or it could be a consumer franchise business. It could be a US company or an EM business.
The second interesting piece (and quite philosophical, I should say) was the discussion on how to incorporate the Black Swan idea with Superforecasting. The author knows Taleb personally and has high respect for him. However, his concept that certain events have a higher probability to occur than we think (fat tails instead of normal distribution) and that such events can have a much bigger impact on life than regular events makes forecasting normal events useless. If Black Swans alone determine the history and by their own definition, we cannot properly anticipate them as many have never happened before, then focusing on things that do not matter becomes pointless.
I think Tetlock's point on this is that certain events can be better predicted (not those that would happen 10 or 20 years ahead), and we should still try focusing on them. I want to be completely honest and admit that I probably have not quite understood this section on Black Swans and Superfocasting, so I suggest readers do their own reading.
I think in investing, it is also important to have strong beliefs that paying a price substantially below what the company is worth is key. Equally important is to find great businesses and stick to them. But an investor should be opportunistic in his approach to finding such opportunities. It could be a low PE or high FCF yield stock, but it could be not. It could be a cyclical company, or it could be a consumer franchise business. It could be a US company or an EM business.
The second interesting piece (and quite philosophical, I should say) was the discussion on how to incorporate the Black Swan idea with Superforecasting. The author knows Taleb personally and has high respect for him. However, his concept that certain events have a higher probability to occur than we think (fat tails instead of normal distribution) and that such events can have a much bigger impact on life than regular events makes forecasting normal events useless. If Black Swans alone determine the history and by their own definition, we cannot properly anticipate them as many have never happened before, then focusing on things that do not matter becomes pointless.
I think Tetlock's point on this is that certain events can be better predicted (not those that would happen 10 or 20 years ahead), and we should still try focusing on them. I want to be completely honest and admit that I probably have not quite understood this section on Black Swans and Superfocasting, so I suggest readers do their own reading.
A good summary of 'Superforecasters' method
"Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unknown and leave no assumptions unscrutinised. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also, explore the similarities and differences between your views and those of others - and pay special attention to prediction markets and other methods of extracting wisdom of crowds. Synthesise all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgement as precisely as you can, using a finely grained scale of probability".
Some other good tips
- Strike the right balance between under- and overconfidence, between prudence and decisiveness. Superforecasters understand the risks both of rushing to judgement and of dawning too long near "maybe". They routinely manage the trade-off between the need to take decisive stands (who wants to listen to a waffler?) and the need to qualify their stands (who wants to listen to a blowhard)?
- Look for the errors behind your mistakes but beware of rearview-mirror hindsight bias. Don't try to justify or excuse your failures. Conduct unflinching postmortems: Where exactly did I go wrong? And don't forget to do postmortems on your success too. Not all success imply that your reasoning was right. You may have just lucked out by making offsetting errors.
- Strive to distinguish as many degrees of doubt as the problem permits but no more. Few things are either certain or impossible. And "maybe" isn't all that informative. Nuances matter. The more degrees of uncertainty you can distinguish, the better a forecaster you are likely to be.
- Bring out the best in others and let others bring out the best in you. Master the fine arts of team management, especially perspective taking (understanding the arguments of the other side so well that you can reproduce them to the other's satisfaction), precision questioning (helping others to clarify their arguments, so they are not misunderstood), and constructive confrontation (learning to disagree without being disagreeable). Wise leaders know how fine the line can be between a helpful suggestion and micro managerial meddling or between a rigid group and a decisive one or between a scatterbrained group and an open-minded one.