24.7.09

Taleb's Problem. Not a problem for "Silent Evidence"

Nassim Taleb, in his book The Black Swan, introduces a concept he calls “silence evidence”—particularly the problem of silence evidence as it weights into the decision-making process. In short, the results and conclusion from whatever pool of data we may be studying can be (and likely is) skewed—possibly contradicted even—by evidence to the contrary which (for one reason or another) did not survive. In order to illustrate this concept in practice, he provides several key examples (there are others to which I will refer, but these are a few):
1. The Adventurer Casanova
2. Restaurants in New York City
3. The Phoenician “Cemetery of Letters”
4. Cicero and the Trouble Worshipers
5. Patients that would have lived (and some who died) from a drug.
In each of these examples, the evidence which failed to survive contradicts or severely affects the evidence which survived to be studied, from which conclusions were drawn.

There are two questions which must be answered in examining the validity of silent evidence, and to what extent it is actually a problem. First, is it a valid factor in our equations? If it is, secondly, how can and should silent evidence be used?

Keep in mind, truly silent evidence (pure) is utterly untraceable and immeasurable. Nothing remains of it to actually study. In this sense, Taleb’s example of New York restaurants is only semi-silent evidence (not pure). While to the casual observer, it seems that restaurants in NY survive and thrive (and so, I conclude, I should open one! Right? That’s how the argument goes), business, property ownership and other records remain to be discovered by the ardent inquirer. This simply goes to illustrate the varying degrees of silent evidence.

My purpose is to explore the purest kind of silent evidence—of which nothing remains: no sign, no marks, no unnatural “void” where something should be and isn’t. The evidence is utterly and completely silent.

So to the first question: If evidence once existed—or yet might remain available for later discovery—to contradict, disprove, or diminish the impact of previous findings, then house (before this discovery) can and should this type of evidence be accounted for?

Silent evidence will either confirm our conclusions (e.g. “people die eventually” and I don’t need all the bodies of the generations to prove it) or else contradict our evidence. In the above example, I’ve drawn an estimation of the number of restaurants that are thriving (visible evidence) and the disproportionate number that have failed (silent evidence). The actual relationship may be much greater—how many really survive? One in one hundred? One in one thousand, or one million? Regardless—the silent evidence (when accounted for) must either confirm the findings of the visible evidence or else contradict it.

It’s worth noting here that Taleb’s main point is to show how prone we are to discount likely outcomes based on errors of interpretation (e.g. we find what we want to find), and so fail to consider a broader scope of possible data. Question: can silent evidence be used to prove both a positive and negative outcome? That is, can it prove some things more unique (less likely) and other things more common (more likely)? How we answer this question will also answer the question of proper use for “silent evidence.”

I submit, and will attempt to prove that silent evidence is helpful only in preventing our estimation of the likelihood of desirable results and our underestimation of negative results; but not vice versa. In this sense, silent evidence is only a problem if we have overestimated the results of our findings (usually positive or desirable) and conclusions, and underestimated negative results and consequences.

Let me illustrate with this example from restaurants in New York. Taleb writes, “Consider the restaurant business in a competitive place like New York. One has indeed to be foolish to open one, owing to the enormous risks involved and the harrying quantity of work to get anywhere in the business, not counting the finicky fashion-minded clients. The cemetery of failed restaurants is very silent: walk around Midtown Manhattan and you will see these warm patron-filled restaurants with limos waiting outside for the dinners to come out with their second, trophy, spouses. The owner is overworked but happy to have all these important people patronize his eatery. Does this mean that it makes sense to open a restaurant in such a competitive neighborhood? Certainly not, yet people do it out of the foolish risk-taking trait that pushes us to jump into such adventures blinded by the outcome” (115). IF we were to chart this “visible evidence” against the “silent evidence” on a Venn diagram, it would look as follows (illustration 1).



We could likewise diagram each of the illustrations Taleb presents:


There is a fundamental different between all of these examples and the further examples Taleb presents—namely about patients who would benefit from a drug that kills a few patients and so is taken off the market (112), and Katrina-related deaths resulting from a re-allocation of resources and funds (110), and the stability of species from the surviving fossil records (108). The diagramed examples have some aspect of “visibility” or what I call soft-spoken evidence. Silent evidence in each case merely shows that inaccurate assumptions were made and applied. And yet, we know that these assumptions (or deductions) are inaccurate because of soft-spoken evidence.

These other examples, on the other hand—the patients, the Katrina survivors, and the stability of species—is pure speculation. There is nothing wrong in asking what species might have existed but left no surviving record, or how many people died because funds were reallocated to Katrina (and away, in his example, from cancer research). Nevertheless, this information remains is beyond the reach of measurement, and as a result (while we can assume and speculate) our Venn diagrams would be little more than the expression of this speculation.



Consider that in each case, there is simply no way to know the relative proportions between the visible and the silent evidence. Take the fossil record, for example. Which of the following two Venn diagrams accurately expresses the real relationship between the types of evidence?

Answer: we could pick either or neither and be right. Until the “silence” is broken—we are left to speculation. On this particular point, the problem of silent evidence is merely a theoretical discussion for scientific observation and empirical data collection.

And so we ask again—is silent evidence helpful? For science, no. Science must continue to record and document what is visible until the silence is broken. I can ask the question, “Could pigs fly?” but it is mere hypothesizing until some evidence presents itself.

It is at this point that we can see how silent evidence is unhelpful, as well as where it is helpful. Some of Taleb’s examples are right on, especially where silent evidence is used to curtail our over-eager risk taking. In the other examples, he simply misleads. Silent evidence can only contradict something, but (so long as it is silence) cannot prove or disprove anything. Whereas, in the realm of risk assessment—Taleb’s main area of study and training—silent evidence is a valid (even significant) aspect and element to consider.

Ask: what are the odds that a restaurant will survive in NY city? What risks must a casino insure again? What is the likelihood that I could lose all my money in the stock market? What factors should I consider when buying a house? These are all types of scenarios where silent evidence should, and must be, considered; or else we risk experiencing Taleb’s “black swan.”

Ask again: How many unregistered ships have been lost as sea? How many once-existing species have vanished without a trace? What is the likelihood a planet will explode? How many children in the 3rd century died of starvation? The truly silent evidence of these questions is unattainable.

Silent evidence serves to protect overestimation, which tends to diminish the possibility of risk. Take two casino owners and their risk management approach. One insures against high losses at the tables and personal injury on the premises. This owner has chosen to ignore the other (outlier) threats to his business (underestimating the negative consequences and risks). Consider these risks:



Keep in mind these are but a few of the many possible risks. If all threat scenarios were within the realm of expectation and possibility, there would never be any black swans. Still, if the casino owner expanded his protection to the range of the dotted line, he would be protected against a range of more possible situations. With each possible consideration, one increases awareness or risk (overestimating a negative possible outcome) and so is better protected by that mitigation (tendency to underestimate risk, overestimate success).

I believe the answer to our first question is: yes. Silent evidence can be used when one is attempting to mitigate risk, contrary to the propensity to assume the best, accept only visible evidence, and dismiss risk. This is the extent to which silent evidence can and should be applied.

Taleb, to the contrary, tries to use his “silent evidence theory” to prove anything. He uses it to argue that the earth isn’t rare in that it exists, but only rare in that it survive. He uses it to argue that people who died of cancer would not have died had monies not been rerouted to Katrina. He uses it to argue that vastly more species existed than we know about, contributing to our erroneous assumption regarding the stability of the species.

Because Taleb makes such a point to stray in this area, I will take him to task on the point of ontology. Taleb writes, “Consider our own fates. Some people reason that the odds of any of being in existence are so low that our being here cannot be attributed to an accident of fate. Think of the odds of the parameters being exactly where they need to be to induce our existen (any deviation from the optimal clibration would have made our world explode, or collapse, or simply not come into existence). It is often said that the world seems to have been built to the specifications that would make our existence possible…it could not come from luck. The problem here with the universe and the human race is that we are the survivain Casanovas. When you start with many adventurous Casanovas, there is bound to be a survivor, and guess what: if you are here talking about it, you are likely to be that particular one… So we can no longer naively compute odds without considering that the condition that we are in existence imposes restrictions on the process that led us here” (117-118).

Let’s graph Taleb’s argument:


How many others didn’t survive, Mr. Taleb? One, two, ten, a billion, a quintillion? The above graph represents a visual ratio of, say, 1:10. So the odds of us being here are really only 10%? Depends what you are graphing. Based on the size of the universe, that there are even 10 places where an “earth” could exist and support life is pretty insignificant. But maybe the ratio should be 1:1,000,000,000,000,000. (At what point does possibility get replaced with probability?)

In either case, we’re just guessing. There’s no proof. Using this reasoning, I can argue just about anything I want: there is no God, there is no soul, frogs used to be able to fly (fossil record simply failed to record it), people used to have tails (ditto), and Jupiter is made out of colored cheese (prove me wrong!).

By attempting to apply silent evidence to scenarios where risk is not involved—but odds and statistics are—the evidence fails. No longer something to consider, it merely becomes something to speculate about. This confusion between risk and odds as mere synonyms is part of the problem. The odds of something aren’t correlative to its risk. They aren’t even the same thing, but Taleb treats them as though they were. The odds of a restaurant or adventurer surviving under certain conditions are unlikely; there are great risks to both. Silent evidence can cause us to dismiss that evidence. But the odds of the earth being here are immense; but there is no risk here. There is no silent evidence that can confirm that we’re the lucky ones. In fact, quite the opposite—silent evidence would prove how incredible the odds of our existence is.

In conclusions, answering our second question, silent evidence can only be used where to counter actions based on visible evidence, which tend to diminish or completely dismiss potential for or risk of failure. Applied to other realms—it become little more than a tautology: proving whatever one believes to be true in the first place.