Selection Bias and the Fallacy of Listing Examples
Advocating a belief by finding supporting examples and listing them is fallacious because of selection bias; examples are selected because they illustrate the belief and contradictory cases are ignored.
Table of Contents
Thought Experiment
Suppose there is a fear that a popular cooking appliance, the Acme Electric Grill, is a fire hazard. Perhaps suspicion of the Acme Electric Grill stems from reports in which it was found at the scene of residential fires.
It might seem an obvious course of action to find and make a list of examples in which there was a fire in a home that had an Acme Electric Grill. Suppose that such an investigation results in a list of 100 such examples. Unfortunately, such a list proves absolutely nothing.
Why the Practice is a Fallacy
In order to see the fallacy of such a practice, one need first consider the implications of the hypothesis being considered. In the Acme Electric Grill thought experiment, the hypothesis being investigated is that possession of an Acme Electric Grill increases a home’s risk of fire.1
If that were the case, at the very least, it would be expected that the proportion of homes with an Acme Electric Grill that had a fire would be greater than the proportion of homes without an Acme Electric Grill that had a fire.
There are four unknowns that need to be found in order to calculate the two proportions. This can be illustrated by the creation of a contingency table, such as Table 1. The body of the table contains counts of homes. The rows separate these counts into homes that have had a fire and homes that have not. The columns separate the counts into homes that have Acme Electric Grills and homes that do not.
Table 1: A 2 by 2 contingency table with all of the information from the list of examples.
| Acme Electric Grill? | |||
|---|---|---|---|
| Yes | No | ||
| Fire? | Yes | ≥ 100 | |
| No | |||
Table 1 is filled in with all the information gleaned from the list of examples generated in the preceding thought experiment. The upper left cell contains “≥ 100” because it is known from the list of examples that there are at least 100 homes with an Acme Electric Grill that had fires, and that is it.
The emptiness of the contingency table illustrates the emptiness of the information that comes from making a list of examples. In this case, it is not known how many homes with an Acme Electric Grill have not had a fire (lower left cell), how many homes without an Acme Electric Grill have had fires (upper right cell), or how many homes without an Acme Electric Grill have not had fires (lower right cell). Therefore, there is no way to check if the proportion of fires is greater among homes with an Acme Electric Grill than among homes without one.
Suppose that the lack of information in the other 3 cells of Table 1 were addressed in the exact same manner, i.e., by finding a list of examples for each case, resulting in Table 2.
Table 2: A 2 by 2 contingency table with all cells filled in by the technique of listing examples.
| Acme Electric Grill? | |||
|---|---|---|---|
| Yes | No | ||
| Fire? | Yes | ≥ 100 | ≥ 100 |
| No | ≥ 100 | ≥ 100 | |
Table 2 is still not informative. All that is known is that there are at least 100 instances of each case. Again, there is no way to check if the proportion of fires is greater among homes with an Acme Electric Grill than among homes without one.
It is not surprising that 100 instances of each of these cases could be found. For instance, in the United States in 2018, there were an estimated 379,600 residential fires. (National Fire Data Center 2020) Also, there were an estimated 138,516,439 homes in the United States. (US Census Bureau n.d.) Given a large enough population, 100 examples for each of the four cases can easily be found.
The fundamental issue causing these lists of examples to be uninformative is that sampling is being done, but the sampling is done with selection bias.
Sampling
Sampling occurs when a smaller number of individuals are observed than the number of individuals in an entire population. This is a common practice. For instance, it is practically impossible to investigate all 138,516,439 homes in the United States and find out how many of them have Acme Electric Grills. Therefore, a smaller sample of homes is typically taken.
Inferential statistics is the academic discipline concerned with how to make conclusions about a population based on a sample from it. In order for statistical inference to work, the sample taken must be representative of the population. Representative samples used in statistical inference are often called “probability samples” because the inference is predicated on every individual in the population having a defined probability of being selected for the sample.2
The simplest kind of probability sample is called a “simple random sample.” In a simple random sample, every possible sample from the population has the same probability of being selected. This implies that all the individuals in the population have the same probability of being selected for the sample.
Simple random samples are the easiest kind of sample to deal with because they do not require further adjustment3 in order to estimate population attributes. A simple random sample will exhibit properties that approximate the properties of the population, given a large enough sample size. If 0.317% of homes in the United States had a fire, and each home has the same probability of being selected in a simple random sample, then around 0.317% of homes in the sample will have had a fire.
Selection Bias
Generally, selection bias occurs when the probability of selecting a sample from the population is different from what is believed to be the case. For a simple random sample, if the probabilities that some individuals are selected for the sample are greater than others, then selection bias has occurred.
Selection bias ruins the estimates made of properties of the population based on the sample. For instance, if what is believed to be a simple random sample of the entire United States is actually taken just from a region of the United States with twice the risk of fire as the national average, then around 0.634% of homes in the sample will have had a fire, while the true proportion of homes that had fires in the population is still 0.317%.
In the fallacy of listing examples, not only is there selection bias, there is nothing but selection bias, since examples are found specifically because they illustrate some phenomenon. Thus, all that is learned from making a list of examples is that one can find what one is looking for.
Arbitrary Belief
A trait of the fallacy of listing examples – and of many fallacies of interpreting evidence generally – is that such a practice can lead to any belief. Whenever a practice leads to any arbitrary belief, it is not useful for separating true belief from false belief.
For instance, there are approximately 7.7 billion people in the world. Thus, whatever is feasible to happen to a human being, has happened to many human beings by virtue of the sheer number of chances of occurring. Thus, the fallacy of listing examples can be used to conclude any arbitrary belief about human beings.
One could easily find thousands of examples of people who smoked tobacco products and died prematurely due to pulmonary disease. However, one could also easily find thousands of examples of people who smoked tobacco products and lived long, healthy lives into their old age. One could just as easily find thousands of examples of people who drank a glass of water and then died shortly thereafter.
If one were to be persuaded by the fallacy of listing examples, then one should have numerous arbitrary beliefs, even contradictory ones or absurd ones. One should simultaneously believe that smoking does and does not cause pulmonary disease. One should believe that drinking a glass of water increases one’s risk of death.
Such arbitrariness is precisely why making a list of examples is fallacious.
Countermeasures to the Fallacy
If one encounters an act of attempted persuasion supported by a list of examples, one ought to reject the conclusion being suggested. However, it is also incorrect to reach the opposite of the conclusion being suggested. Instead, the only correct result is that no conclusion can be drawn.
For instance, in the above thought experiment, it was found that there were at least 100 instances of fires in homes that have an Acme Electric Grill, and it was suggested that Acme Electric Grills cause fires. It is correct to reject such a conclusion as fallacious. However, it would also be fallacious to conclude that Acme Electric Grills do not cause fires. The only correct belief, given the evidence of Table 1 or Table 2, is that it is not known whether or not Acme Electric Grills cause fires.
In the Acme Electric Grill thought experiment, it was alleged that the grills caused fires, but an opposite thought experiment would have proceeded identically. For instance, the Acme company could have suggested that its grills were not causing fires, and its representatives could have produced a list of 100 happy customers who have never had a fire. This alternative thought experiment would have proceeded identically, but with the lower left cell of Table 1 filled in, instead of the upper left. The conclusion suggested by the Acme company would be fallacious for the exact same reasons as the conclusion of the original thought experiment. Again, the only correct belief is that it is not known whether or not Acme Electric Grills cause fires.
Representative Samples
If one wants to draw an actually informative conclusion when presented with a list of examples, the remedy is straightforward: one needs to define the population of interest and take a representative sample from the population.4 The list of examples itself is of no use and can be discarded.
For instance, in the Acme Electric Grill thought experiment, suppose that the population of interest is defined as all of the homes in the United States. The list of examples that led to Table 1 can be discarded. Instead of specifically looking for instances of fires in homes with Acme Electric Grills, some scheme is devised whereby a simple random sample of 10,000 homes in the United States is identified. Of these, all 10,000 are investigated as to whether they had fires in 2018 and whether or not they have Acme Electric Grills. The counts from this are summarized in Table 3.
Table 3: A 2 by 2 contingency table with counts of a sample random sample taken from homes in the United States.
| Acme Electric Grill? | |||
|---|---|---|---|
| Yes | No | ||
| Fire? | Yes | 366 | 867 |
| No | 2651 | 6116 | |
Unlike with Table 1 and with Table 2, the proportions needed to test the hypothesis that Acme Electric Grills are a fire hazard can be calculated from Table 3. The proportion of homes with an Acme Electric Grill that had a fire in 2018 is 366 / (366 + 2651) ≈ 12.1%, and the proportion of homes without an Acme Electric Grill that had a fire in the same time period is 867 / (867 + 6116) ≈ 12.4%.
In this case, the homes with an Acme Electric Grill were actually less prone to fires than the homes that did not. Given this, a conclusion can be reached: Acme Electric Grills do not cause fires.
Need for Statistical Analysis
It might be observed that counts in Table 3 are unrealistic, since as was seen earlier, the proportion of homes that actually had a fire in the United States in 2018 was quite small. A more realistic sample is represented by Table 4.
Table 4: A 2 by 2 contingency table with counts of a more plausible simple random sample taken from homes in the United States.
| Acme Electric Grill? | |||
|---|---|---|---|
| Yes | No | ||
| Fire? | Yes | 11 | 15 |
| No | 3006 | 6968 | |
In Table 4, the proportion of homes with an Acme Electric Grill that had a fire in 2018 is 11 / (11 + 3006) ≈ 0.365%, and the proportion of homes without an Acme Electric Grill that had a fire in the same time period is 15 / (15 + 6968) ≈ 0.215%.
The proportion of homes with an Acme Electric Grill that had a fire is greater than the proportion of homes without an Acme Electric Grill that had a fire. However, this is the case for the proportions among homes in the sample. What is desired is knowledge as to whether this is the case among all homes in the population.
Because this is a thought experiment and Table 4 does not represent real observations, the sample used for the counts in Table 4 was generated by a computer program. The computer program was instructed that, in the population, the proportions of homes with an Acme Electric Grill that had fires is exactly the same as the proportion of homes without an Acme Electric Grill that had fires.
It is plausible that counts like those in Table 4 would be found in a representative sample from a population where the two proportions of interest are identical. To see this, statistical analysis must be done. This is, however, beyond the scope of this article.5
Thus, eliminating the selection bias created by the fallacy of listing examples is necessary to draw a valid conclusion, but a representative sample is not sufficient to reach a valid conclusion.
More Complex Scenarios
The thought experiment used in this article revolves around a hypothesized association between two categorical variables: whether or not a home had a fire, and whether or not a home has an Acme Electric Grill. This was used because it is a very simple scenario that can be illustrated with a 2 by 2 contingency table.
However, what makes the practice of listing examples fallacious applies in more complicated scenarios, as well. All such instances of creating lists of examples are fallacies because, in these instances, the examples are themselves selected because they illustrate some supposed phenomenon. It is this intentional selection bias that makes lists of examples uninformative and conclusions drawn from them fallacious.
This fallacy can occur in scenarios where there are more than two variables being investigated. It can occur with variables that are numerical instead of categorical. Indeed, this fallacy can occur with any sort of analysis, because it is a fallacy in how sampling is done, not in how the analysis is done.
Non-fallacious Use of Lists of Examples
While it is fallacious to attempt to infer a conclusion about a larger population from an unrepresentative sample, sometimes lists of examples can be used in ways that are not fallacious.
There is nothing fallacious about the use of examples for purely illustrative purposes, as long as there is no claim about the prevalence of the phenomenon being illustrated or about the supposed association of other phenomena with it. Indeed, this article uses an example, albeit a fictional one in the form of the Acme Electric Grill thought experiment, in order make discussion of the fallacy of listing examples more concrete, but makes no claim about how frequently this fallacy is occurs.
Furthermore, it is not fallacious for a list of examples to precipitate further investigation. For instance, a first sanity check in investigating whether Acme Electric Grills cause fires could be to find out whether any fires have occurred in homes with an Acme Electric Grill. Thus, lists of examples have a similar status to anecdotes in that, while no conclusion can be drawn from them directly, they can inspire more rigorous collection of empirical evidence.
Conclusion
Someone who is advocating a belief might think it an obvious course of action to accumulate a list of supporting examples. However, as has been seen, these lists of examples are nothing more than extended exercises in selection bias. Whenever such a list of examples is produced, no valid conclusion can be drawn. It thus behooves those who value the separation of truth from falsity to be on guard against this practice, both in others’ acts of attempted persuasion and in their own thinking.
Further Reading
The use of a 2 by 2 contingency table to illustrate bias in interpreting evidence was inspired by the section “The Excessive Impact of Confirmatory Information” in Chapter 3 of How We Know What Isn’t So. (Gilovich 1991)
For more discussion of sampling, selection bias, and how selection bias fits into the tradeoffs between sampling error versus nonsampling error, see Chapter 1 of Sampling: Design and Analysis, (Lohr 2021) which is available as a free “preview” download from the publisher.
Citations
Footnotes
It could also be the case that residential fires and possession of an Acme Electric Grill are associated because of the converse relationship: the risk of residential fire explains an increased probability that a home has an Acme Electric Grill. For instance, this could arise if poorer homes have an increased risk of fire, and the Acme Electric Grill is cheaper than other grills and so popular among poorer homes. However, for the purposes of this thought experiment, any association between the two is framed as possession of the Acme Electric Grill explaining prevalence of residential fires.↩︎
The description used in this article of what makes a sample representative is formulated in the terminology of survey statistics typically found in textbooks on sampling. (Lohr 2021) Outside of contexts specifically concerned with sampling, statistical inference is often studied more abstractly using mathematical models. However, it is not the case that the model-based approach avoids the need for samples to be representative of a larger population in order for inference to be valid. Indeed, the assumptions of the mathematical models of statistical inference are usually very strong and typically include the assumption that the random variables of interest are independent and identically distributed. The assumption that random variables are identically distributed can be construed as a model-based way of stating that a sample is representative of a larger population.↩︎
The adjustment alluded to here is termed “weighting” in statistics. Generally, for more complex sampling, each observed value in the sample is weighted by multiplying it by the inverse of the probability that the individual is selected for the sample. Because this is unnecessary for simple random samples, they are sometimes called “self-weighting” samples.↩︎
Alternatively, for small populations, one could investigate the entirety of the population. This is called a “census.”↩︎
Determining whether two proportions are equal or not is one of the most elementary statistical analyses that can be done. Numerous techniques exist for this case, and any respectable introductory statistics course should cover at least one of them. For an intuition as to why the sample proportions might be different, given that they are identical in the population, it might be noted the proportion of homes that had a fire is quite small. In a simple random sample, there is still randomness as to whether any given individual is selected for the sample, and the inclusion of any home that had a fire changes the proportions in the sample greatly. This randomness can lead to relatively large fluctuations when the population proportions are small.↩︎
