Survivorship bias in entrepreneurship

In early April, I helped Mark create this infographic entitled « Entrepreneurs vs. VCs », in which we tried to capture the differences between these two entirely constrasting species that unfortunately cohabitate out of neccessity. While I don't agree with some of the items that fell to a particular side of the infographic, I generally agree and am amused by most of the representations selected in it.

I do feel that there's a very important aspect that's not captured by this series of contrast, which is the understanding about survivorship bias. Generally speaking the VCs operate in a mindset where they try to avoid survivorship bias, while the entrepreneurs almost all exhibit certain syndromes of being under its influence.

Beware of the bias

Beware of the bias


What is survivorship bias?

Survivorship bias is the logical error of concentrating on the people or things that "survived" some process and inadvertently overlooking those that did not because of their lack of visibility. This can lead to false conclusions in several different ways. The survivors may be actual people, as in a medical study, or could be companies or research subjects or applicants for a job, or anything that must make it past some selection process to be considered further.

— Wikepedia

It's well researched and documented that we human beings are born volunerable to various biases. We're not talking about any contextual biases that one acquires from the environment, but rather the things that run in the blood. Throughout the hundreds of thousands of years of evolution, these biases helped our race survive and even prosper despite they never run coherently with the rational side of us. Throughout the history of written languages writers and philosophers touched on these biases that are well categorized only today, but majority of the society had lived and died peacefully (or not) without ever knowing these biases. However, the complexity of our society has multiplied in such a degree for the past two centuries that these biases have become harmful, with financial bubbles being the most prominent negative outcomes of all. 

In general it takes quite some training to unbias oneself and avoid making biased decisions. As far as I know only two modern professions practice this training often enough. One of them is naturally the philosophers. The other is the financiers, or more precisely, the professional investors.

For the philosophers, examining the biases early on in the formation years and constatly throughout their lives is the only way to maintain high levels of rigor in their analyses. For the financiers, the reason they practice unbiasing all the time is simply because the stakes are high: you stand to crash your investment firm — if not the entire world — if your investment decisions involve these biases.

Among all the DNA-coded human biases, survivorship bias is probably the No. 1 bias that financiers try to tackle in their daily lives. While causal investors know that past investment performances don't predict future ones, few outside of the professional investment world also examine survivorship bias regularly when they pick an investment vehicle.

Take mutual funds for example. Non-performing funds usually experience an increasing amount of investor withdraw and eventually go out of business. Once a fund is closed, its historical performances no longer count against the industry. A new fund can be set up with a clean sheet and start reporting its performances anew. And the existing funds that survive, almost by definition, are exactly those that so far perform well enough. As a result, at any given moment the industry-wide performance as reported suffers from survivorship bias — the reported average performance is always better what the comprensive result actually is.

In fact, survivorship bias is such a Lesson 101 in this profession that you can use it as a test for finding out whether your financial advisor is a fake or not. (Warning: chances are he probably IS.)


Survivorship bias and causality fallacy

In the context of entrepreneurship, the survivorship bias usually comes with an extra wrinkle: causality fallacy. A common definition of casuality fallacy is mistaking correlation with causality. For example, middle school students might perform better academically when they're from a family with more than 100 books at home. However it apparently doesn't mean that if a parent buys 100 books the children will perform better academically.

As for in the context of entrepreneurship, I'd like to extend the definition of causality fallacy to seeing the success of survivors, believing they must have done something right and therefore looking into the survivors' stories trying to find the keys to success.

Now you're probably wondering why you're wasting your time reading a terminology-ladden philosophical blog post instead of executing your grand startup plan in the garage. But if you have ever retold any anecdote of Steve Jobs from Walter Isaacson's best-seller to your startup employees in at least 3 different occasions, it's definitely better that you bear with me for a little while.


Entrepreneurial version of sub-prime crisis

In my previous article « Your ingeneous idea is worthless ... », I told a verisimilar story of Jan Koum of Whatsapp where a single random decision changed the outcome of the trajectory entirely, to demonstrate my point about randomness in the life of a startup and its entrepreneur. Now let's expand this model to 1,000 trajectories, each one containing a series of decisions — in finance we call this decision tree analysis — which interact with the stakeholders in the market where you compete for a share. Each trajectory leads to a distinct result on the day of February 19th 2014.

An random example of decision tree analysis

Now you take this 1,000-trajectory data set and you analyze it with the most sophisticated statistical methods, and you believe you've found a formula to optimize the risk-return profile. I then present you, as of today, a starting point that looks exactly like the start of the data set, with the same maturity of market, the same macro condition, the same stakeholders, etc.

Question: can you use your formula to develope a path of strategy execution toward the greatest financial success with the lowest risk?

I think you already know the answer. If you don't, at least you should know that the model I just described is basically what caused the world to crumble almost overnight in 2008.

At the risk of oversimplifying, market participants back then did exactly what you're tryinig to do here: relying solely on historical data to predict the future. The most important variable they fed into their Credit Default Swap pricing models, the volatility, was often estimated only using historical data. The problem is: sub-prime CDS had only became popular 3 or 4 years before. There had never been a systematic meltdown and therefore they kept using the data that did not contain any extremely negative sample. When there's any warning from the economists or the few market prophets, they would take the same data, analyze them and feed the calculated volatility into their models again. This is of course beating the dead horse. As a result, from 2007 to 2008 the CDS market continued to be strong while the fundamental — the default rates of the underlying mortgage assets — deteriorated everyday.

The Black-Scholes Formula is the origin of most modern derivative pricing models

The Black-Scholes Formula is the origin of most modern derivative pricing models

But now you argue — I'm not developing any evil derivative model and I don't have 1,000 samples to analyze. I'm merely quoting paragraphs from Steve Job's biography from time to time, for the sake of inspiration. What's your point?

In fact, that's exactly my point! You have a sample of ONE and you don't even have a model. And you believe that this best-seller book, which famously sold more than 379,000 copies the first week, could help you move your startup in the right direction?


Stop reading biographies of successful people

The fact is, success is a result. In the universe we're living, Whatsapp is a success, Apple is a success. But in other universes, these successes probably had never occured even though you still have the same Jan Koum with the impressive food stamp story and Steve Jobs with his freak nature.

Even if history does repeat itself and (miraculously) success has a formula, you might still have the wrong source data from which you're supposed to develope a formula — it's common that when successful people are asked to tell their stories, they suffer from another grand human bias called confirmation bias. In other words, they tell the anecdotes that are coherent with their success: sleeping in the garage office for 3 straight months, insisting on modifying the design of a certain part of the product one week before its worldwide shipment, etc. They leave out all the incoherent details that might have also contributed to the successes or failures they experienced.

The result is a complete and coherent story, which is anything but a truthful recount of the history.

And even with a truthful recount, you still end up with only ONE sample, while those sharp derivative traders on the Wall Street before 2008 were using tens of thousands of data points trying to predict the future, and still ended up almost destroying the world. What are the odds of you happening to be more right than they were in 2008?


So what's your recommendation, if any?

In fact, I do have one: instead of reading stories of success, try talking to entrepreneurs who have failed completely before.

While success comes rare and is usually the result of many random and non-random factors put together in a certain order, failure can usually be attributed to certain reasons: sticking to the same technology too long without noticing that the world has changed, raising the angel round at too high a valuation thereby scaring all VCs away from Series A, focusing too much on developing the best in-house IT infra while Amazon was lowering its cloud-service prices, etc.

Learn from the losers and befriend the winners would be how I would sum up my recommendation if I absolutely have to. And during the vacation on the beach of Côte d'Azur, if you really truly surely still want to read « Steve Jobs », used hardcover version is currently sold on Amazon at a starting price of $0.38, not including shipping

Never have words of wisdom been so cheap, right?


Chat N' More (contd.)

Your ingeneous idea is worthless ...