How Leaders Improve Their Decision-Making with the Brier Score
Verification of weather forecasts has been a controversial subject for more than half a century…. One of the most important difficulties seems to be reaching an agreement on the specification of a scale of goodness for weather forecasts.
Glenn Brier
Monthly Weather Review
February 10, 1950
US Weather Bureau
Australian meteorologist W. Ernest Cooke was the first weatherman to assign probabilities to forecasts in 1906.
It didn’t catch on.
Another 35 years passed before leaders demanded meteorologists assign probabilities to forecasts. They were demanded because the stakes were high.
Americans and the Allied Forces were at war and needed to bomb places more accurately. Unpredictable weather between Britain and Germany required new thinking to drive better results. As generals strategized about how and when to bomb targets, weather and visibility was a top concern.
“Partly cloudy and a chance of rain, sir!” was as unhelpful to military leadership then as it is to you today.
For better decision-making, leaders needed forecasts.
Leaders also needed probabilities—statements of confidence.
While World War II advanced meteorological forecasts, weathermen remained hesitant to cite probabilities. These men of science, finally gaining recognition on national television for their lifelong atmospheric nerdery, felt little need to risk it all by asserting probabilities on something as unpredictable as weather forecasts.
Enter our hero, Glenn Brier, and his mission to identify a “scale of goodness for forecasts.”
Brier Score 101
Brier, a weather statistician and academic, developed an equation to measure the distance between what someone forecasted and what actually happened.
Brier Scores are like golf scores: Lower is better.
A Brier Score of 0.10—assuming you have collected dozens of forecasts—indicates you’re the second coming of Nostradamus. If you maintain a Brier Score of 0.50, it means you share the same accuracy with a dart-throwing chimp, because it’s all a shrug-of-the-shoulders, fifty-fifty probability.
Extreme predictions (“0% chance you’ll close that deal with Lennar by December 1,” or “95% chance Deion Sanders will win 10 games this season at Colorado,”) are rewarded—and punished—accordingly.
How to Use the Brier Score
Brier’s goal was to develop better pattern recognition and improve accuracy of decision making. As a leader in your business, you should share the same goal.
Here’s how you can begin to use the Brier Score.
There are four simple steps—and one tool.
The four steps are:
1. Document your predictions.
2. Assign a confidence interval.
3. Systematically review their accuracy.
4. Adjust your predictions accordingly.
The one tool is the Brier Score template.
We converted it into Excel.
Download it here for free.
(Password: hartmann)
In their book, Superforecasting, by Philip Tetlock and Dan Gardner, they write, “Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs. These habits of thought can be learned and cultivated by any intelligent, thoughtful, determined person.”
Anyone can—and effective leaders must—intentionally develop better decision-making skills.
I’ve been using the Brier Score for the last year to improve the “scale of goodness for forecasts” for myself as well as our executive coaching clients.
What’s been remarkable—and obvious in hindsight—is how often we make mental calculations regarding probabilities and confidence levels, and how rarely we document them.
To improve decision-making, leaders must document a meaningful sample size. My clients and I generally agree to get to fifty forecasts on strategic business efforts within the first month.
History is littered with people armed with ignorance and overconfidence. It’s a dangerous combination.
You can reduce the risk of both by developing your own Brier Score. It will help you make better decisions.
A world war was required to nudge leaders into demanding better forecasts. Don’t wait for such a dramatic event to nudge you to do the same.
Thanks for reading.
Download the Brier Score template here.
(Password: hartmann)
Click this video to hear about how you can use the Brier Score to improve your decision-making.
