How Much Analysis Is Too Much?

Don't Let Your Productivity And Decision Making Fall Victim To Analysis Paralysis

Nov 26, 2019

9 min read

We are generally under the impression that more analysis is a good thing (especially as more and more companies buff up their analytics and data science capabilities). The more time and thought you give a problem, the better and more insightful your solution should be. And that’s true to a degree.

But there are two big caveats. First, the incremental value of more analysis can decline rapidly. While I wouldn’t necessarily call this wasted effort, the bang for buck of your 30th hour of research is probably going to be a lot lower than that of your 3rd hour of research.

Second, it’s possible for people or even entire teams to become frozen in indecision as they gather more and more information. This is commonly known as analysis paralysis and can happen for several reasons:

Too much information (and noise) and too many people attempting to interpret that information can lead to disagreement and massively slow down the decision making process.
The idea that "we can’t leave a single stone unturned" can strand us on an endless treadmill of analysis and checking. This usually is driven by paranoid decision makers who overemphasize the probability and cost of being wrong. This one is pretty hard to break out of because as we do more analysis, it still feels like we are hard at work and making forward progress (which feels good and obscures the indecision) – but the reality is that with each new analysis, we are adding less and less value.

How Can We Avoid Analysis Paralysis?

So how can we avoid being paralyzed by data? While this is by no means an exhaustive checklist, the following bullets have helped a lot whenever I attempt to ingest data, analyze it, and ultimately make a decision based on it.

1. Accept that there will always be some uncertainty

No matter how much analysis we do, there will always be something that we didn’t check. And even if we could check everything, life is not chess – outcomes are random, not deterministic. So the only thing we can do is to become comfortable with randomness and uncertainty. That despite our best efforts, sometimes everything still rides on the flip of a coin.

But it is our job as analysts and data scientists to make sure that we are primarily flipping unfair coins. That is, while there may be uncertainty, we want our opportunity’s (or decision’s) outcome distribution to be shifted in our favor (like a coin with a >50% chance of turning up heads). So our analyses should focus on identifying the set of opportunities where the odds are tilted in our favor (and on avoiding the ones with bad odds).

👁 We want the blue distribution (unfair in our favor)

We want the blue distribution (unfair in our favor)

I want to reiterate this key point –

Our job as analysts, data scientists, and strategists is not to eliminate risk and uncertainty (this is impossible); rather it is to identify and act upon asymmetric risk/reward opportunities (unfair coins).

2. Be cognizant of our mental biases

Prospect theory, a behavioral economics theory developed by Daniel Kahneman and Amos Tversky, tells us that the average person feels the pain of a loss more than the pleasure of a gain. From their theory, Kahneman and Tversky were able to make several interesting observations about our mental biases:

For certain people, it may take a $2,000 gain to make up for the pain of a $1,000 loss. This is an example of how people, on average, feel losses more than gains.
People overestimate small probabilities such as the risk of a market crash or an airplane crash.
People will pay a premium for certainty (the elimination of all variance) – for example, people on average would prefer A: 100% chance to gain $94 over B: 95% chance to gain $100. This is irrational in the traditional economics sense as A has an expected value of $94 while B has a higher expected value of $95 (0.95 * $100 = $95).

So one of the key takeaways from prospect theory is that people will work really hard to avoid mistakes because they:

Overestimate the likelihood of the mistake.
Fear the pain of making a mistake (and incurring a loss) significantly more than they prize the rewards resulting from making a correct decision.
And overvalue the elimination of uncertainty.

It’s probably impossible to eliminate these biases, but knowing that they exist and affect us is a good first step. The next few items on our checklist help us reduce the negative impacts of these mental biases.

3. It’s OK to be thorough; it’s not OK to be afraid to make a decision

Sometimes we find ourselves spinning our wheels and thinking, "Oh maybe I should look at this before I decide what to do. And there was that other thing I wanted to check out too. And I should see what my coworkers think. But they won’t be back for 2 weeks, so I guess I will just hold off on my decision until then."

That’s not being thorough, that’s procrastination. We all know when we’re procrastinating, even if we do our best to rationalize that it’s OK.

What makes us procrastinate and dilly dally? Assuming that we’re not lazy (I’m not, I swear!), then we’re probably putting off making a decision because we’re scared of being wrong (because of all the reasons detailed in Item 2 above). So we need to be really honest with ourselves – if we find that fear of the unknown (and of being wrong) is driving our lack of action, then that’s probably a decent signal that it’s time to make our choice and move on.

4. Understand the key assumptions behind our decision (then focus the checking on the validity of those assumptions)

When we choose something, we should know why we chose it. More specifically, we should be able to identify the key assumptions that need to be true for us to be right. For example, we might think that a stock is cheap because the company owns a fast growing software business that has been overlooked by the rest of the market.

So in this case, our key assumption is that the software business has been mis-valued by other investors, and that it will either grow much faster than everyone expects it to or earn much higher profit margins than expected (or both). In this case, the success of our stock investment primarily hinges on our prediction of the software business’ future. So we should focus the bulk of our checking on the software business (and how we may have mis-analyzed it), and spend relatively less time on the other aspects of the business that we deem to be fairly valued by the market.

5. Quantify the worst case

When left to their own imagination, people routinely overestimate how bad things might get. That’s why insurance companies can routinely turn a profit.

So one way to combat this bias is by quantifying, as best we can, the probability and magnitude of the worst case scenario. If we can make a reasonable estimate of how bad things can get, then we can decide rationally how much we should worry about the worst case. We might find that the worst case is not quite so bad, and that we were worrying for nothing.

And if we find that the magnitude or probability of loss is likely to be more than we can bear, then either we should not go forward at all or we should figure out a way to hedge some of the risk.

I will caution you that estimating worst case scenarios is not easy. For example, worst case scenarios created with purely historical data should be viewed with skepticism – in 2007 and 2008, historical data told us that the housing market would never crash and then it promptly did so.

6. Estimate the incremental value of additional analysis

It’s my own personal belief that the value added of more and more analysis follows the Pareto principle, which states that 80% of the benefit is produced by 20% of the work:

👁 Most of the value is produced by 20% of the work/effort

Most of the value is produced by 20% of the work/effort

Of course that doesn’t mean that we can just do 20% of the work and call it a day – the benefit is rarely if ever accrued linearly. But what it does mean is that we should be attempting to measure the amount of incremental value that each analysis adds.

The key word here is incremental. No matter what we choose for our first analysis, it will probably add a lot of value because we know very little at the beginning. Then as we study up and build more models, they will on average produce less and less incremental value (with some lucky breaks). There’s no right answer as to when we should stop analyzing and start making the decision – but it’s important to realize that each additional analysis is progressively more expensive in terms of value produced per employee hour.

👁 Illustrative example of incremental value produced by additional analyses

Illustrative example of incremental value produced by additional analyses

7. Realize that most decisions are just one step in an ongoing and iterative process (and failures can and should be learned from)

Finally, it helps to keep the big picture in mind. Very few decisions are final (unless you are literally betting the firm). We do some analysis to inform our decision, make a choice based on our read of the data, learn from the result, and then start the process anew again.

Great organizations aren’t the genesis of a single make or break decision.

Rather, they are the result of compounding one good decision after another, all the while weathering and learning from the inevitable bad choices and failures.