Anecdotes Are Not Evidence

Insufficient Sample Size

Returning to the example of the seasonal flu, suppose one is interested in how effective vaccination was in preventing individuals from contracting the flu in the United States during the 2019-2020 flu season. Suppose there are 10 friends and family that one keeps in touch with regularly. Of these 10, 6 were vaccinated against the flu, and 1 caught the flu this year. The 1 who caught the flu was not vaccinated. What estimate should one make of how effective this year’s vaccine was?

A rough estimate for the probability of catching the flu when not vaccinated could be 1 / 4 = 25%. But what about the probability that someone who was vaccinated caught the flu? None of one’s friends and family that were vaccinated caught the flu this year, so continuing this method of rough estimation leads to an estimate of 0 / 6 = 0%.

Thus, one might be tempted to say that, given these observations, this year’s flu vaccine was absolutely effective. However, this is mistaken. Flu vaccines are effective at decreasing the risk that a vaccinated individual has in catching the flu, but they do not decrease the risk to zero. (“Vaccine Effectiveness: How Well Do the Flu Vaccines Work?” 2020)

The issue this example highlights is that anecdotal evidence rarely consists of enough observations to make an inference to a larger population or to make an inference about cause and effect. Given a sample size of only 10, it is more likely than not that there would not be a single individual in the sample who both was vaccinated and who caught the flu.¹ However, in a larger sample of, for instance, 1,000 individuals, there would be many individuals who were vaccinated and who caught the flu.²

In this case, the anecdotal evidence of one’s friends and family does not consist of a large enough number of observations to calculate even a crude estimate of vaccine efficacy.³

Furthermore, note that any estimate of the probability of catching the flu based on these anecdotes would be prone to wild fluctuations. If another unvaccinated friend happened to catch the flu during the 2019-2020 season, the estimated probability of unvaccinated individuals catching the flu would jump from 25% to 2 / 4 = 50%. This small change in the sample would result in a large change in the estimate about the larger population, which is an indication that the inference is susceptible to being inaccurate due to chance.

The preceding example is generous inasmuch as at least several individuals were observed. Many times, anecdotes consist of a sample size of just 1.

For instance, one might hear from a friend, “I started the Acme Juice Boost program, and ever since then I haven’t caught a cold or flu or anything.” What sort of estimate could be inferred from this? With a sample size of 1, there are two possibilities: that 0 out of 1 individual experiences the outcome or that 1 out of 1 individual experiences the outcome. Given what has been seen about preventative measures against communicable disease, estimating efficacy at 100% is quite reckless. In this example, one cannot infer from a sample size of 1 how effective the Acme Juice Boost is at preventing communicable disease or even if has any effect at all, since many people do not catch cold or flu for finite amounts of time.

Lack of Comparisons

An anecdote that comprises a sample size of 1 brings with it another issue, one that can occur even with larger sample sizes: often times anecdotes do not include any observations to compare against. For instance, suppose that one learns that several of one’s friends use Acme Snake Oil pills whenever they catch the common cold. These friends relate how after taking Acme Snake Oil, one of them recovered from the cold after 3 days, another after 7 days, and a third after 2 days. An estimate of the average time to recover from a cold for someone taking Acme Snake Oil is therefore (3 + 7 + 2) / 3 = 4 days.

The question remains: do Acme Snake Oil pills improve cold recovery times? In order to answer this question, one would need an estimate of how many days it takes for people not taking Acme Snake Oil to recover from the common cold. Only after comparing these two average times can one evaluate the effect, if there is any, of taking Acme Snake Oil. Unfortunately, in this example, the anecdotes of one’s friends do not come with any such information in order to make a comparison.

This issue can be exploited by those trying to sell something. Marketing testimonials often come with success stories of people who have used a product and had a desired outcome. However, such testimonials can easily leave out anecdotes about both those who have had unfavorable outcomes when using the product and those who had the favorable outcome without using the product. Anecdotes used in this way are an instance of the fallacy of listing examples, which is discussed in its own article.

Unrepresentative Samples

Perhaps there is an individual so popular that polling the individual’s friends and family results in a large enough number of observations for inference about a larger population and includes observations to make comparisons. Even if this were the case, another issue remains that prevents such an anecdotal sample from being useful. The sample created from such polling is not representative of a larger population. Instead, the sample is heavily biased towards the idiosyncrasies of the individual in question.

Returning to the example of flu vaccine efficacy in the United States during the 2019-2020 season, suppose that a particularly extroverted individual keeps in regular contact with not just 10, but 200 friends and family. However, suppose the extroverted individual caught the flu during the 2019-2020 season.

In this case, the sample might be large enough for estimation of vaccine efficacy, but this sample will be biased because the extroverted individual was in contact with the 200 other people in the sample and so exposed them to the flu virus. Therefore, the prevalence of flu illness in this sample is expected to be higher than in the general population, which includes individuals who had no direct exposure to the flu virus. Because of this, any estimations regarding the size of the protective effect caused by the vaccine will be incorrect.

This phenomenon is likely to occur whenever sampling is based on personal associations from one individual. There is a similar but opposite issue when starting with an extroverted individual that did not have the flu. The sample created from this individual’s personal associations will likely under-represent, rather than over-represent, the incidence of flu when compared with the general population. This is a result of the fact that the starting individual is known not to have had the flu and to have been in contact with all the other individuals in the sample, and so there is an indication that there is a lack of contagious individuals in the sample.⁴

Fallacy of Celebrated Cases

The situation is even worse for anecdotes that come not from personal associations, but by way of the news media. Much of what qualifies a story as news is that it is rare, unusual, or shocking enough to be of interest. Thus, news stories collectively form an intentionally unrepresentative sample of what is happening in a larger population. This phenomenon was illustrated dramatically by the media frenzy around shark attacks in 2001.

While shark attacks on humans can result in a personal tragedy of injury or death for those involved that should neither be minimized nor made light of, the media representation of shark attacks in 2001 was misleading and irresponsible. A particularly graphic shark attack case was covered on major news media outlets in August of 2001. This was soon followed by story after story of sharks attacking humans. Time magazine ran a cover story labeling the summer of 2001 the “Summer of the Shark.”

The July 30, 2001 cover of Time magazine, which labeled the summer of 2001 the “Summer of the Shark,” labeling an incident which has since become a case study in bad journalism.

This created an impression that shark attacks were increasing at an alarming rate during 2001 and that shark attacks were a particularly prevalent risk. However, it was soon discovered that 2001 had only an average number of shark attacks; indeed, the number of shark attacks and number of fatalities from shark attacks was actually less in 2001 than in 2000. (“‘Summer Of The Shark’ In 2001 More Hype Than Fact, New Numbers Show” 2002)

The so-called “Summer of the Shark” was not a sudden and pronounced increase in shark attacks, but a sudden and pronounced increase in news coverage of shark attacks. However, shark attacks are infrequent compared with other risks humans expose themselves to on a daily basis, so this spike in news coverage encouraged unsubstantiated fears. Nonetheless, the sensational coverage of shark attacks during 2001 did not end until the September 11, 2001 terrorist attacks took over the news cycle. Since then, the so-called “Summer of the Shark” has become an object lesson in bad journalism. (Dempsey 2016)

This alludes to a broader fallacy than just bad journalism. Whenever one is attempting to understand a large population – for instance, in order to make judgments about a society or come up public policy ideas – basing such understanding exclusively on stories that get the most attention is foolish. Such stories form a highly anecdotal, unrepresentative sample that are not informative about the population at large. Indeed, many such stories receive attention precisely because they are so extreme. This can be labeled the “fallacy of celebrated cases.”

Summary

Using anecdotes as evidence is flawed in three ways pertaining to statistical inference: anecdotes constitute sample sizes that are prone to wild fluctuations and are often too small to make precise estimates; anecdotes often come without any observations for comparison; and anecdotes form unrepresentative samples that bias any estimation based on them. For these reasons, anecdotes are not at all useful either for conclusions about a larger population or for conclusions about cause and effect.

In addition to these concerns, there are other flaws in anecdotal evidence pertaining to how human minds work.

Flaws of Memory

The psychological literature around the topic of memory is quite large, and a full review of what has been learned about the fallibility of memory recall is beyond the scope of this article. Instead, this article summarizes a few seminal experiments in the field in order to establish the core point that memory recall, according to the empirical evidence, is quite fallible.

Misinformation Effect

The work of psychologist Elizabeth F. Loftus is a particularly conspicuous part of the body of research on memory fallibility as it relates to anecdotal evidence, in part because much of this work has specifically focused on issues with witness testimony and because of her engagement with the legal system, testifying as an expert witness in numerous high-profile cases.

An early experiment of hers examined 45 students at the University of Washington who were shown the same video of a traffic accident. (Elizabeth F. Loftus and Palmer 1974) The test subjects were then asked to estimate how fast the cars were going at the time of the accident, but the question was phrased using different words such as “smashed,” “collided,” “bumped,” “hit,” or “contacted.” The mean estimated speed, averaged over different test subjects, for those asked with the word “smashed” was 40.8 miles per hour, whereas the mean estimated speed was only 31.8 miles per hour for those asked with the word “contacted.”

More remarkably, in a follow-up, similar experiment with 150 student participants, the word choice of “smashed” versus “hit” appeared to affect whether a detail was falsely remembered. The 150 test subjects were shown a video of a multiple car traffic accident. After watching the video, 50 test subjects were given a questionnaire that included the question “About how fast were the cars going when they smashed into each other?” Another 50 test subjects were given a similar questionnaire that asked how fast the cars were going when they “hit” each other, and a control group of another 50 test subjects were not asked about the speed of the cars.

All the test subjects were given a second questionnaire 1 week later in which they were asked whether they recalled seeing broken glass. There was no broken glass in the original video. Among those who were asked the “smashed” question, 16 of the 50 (32%) replied “yes” to seeing broken glass, whereas only 7 out of 50 (14%) of those asked the “hit” question and 6 out of 50 (12%) of the control group reported seeing broken glass.

This effect of an increased probability of incorrectly remembering broken glass was mediated in part by the greater speed estimates of the “smashed” group, but there was still an effect that averaged to 12 percentage points independent of speed estimate.

Broken glass, which was a falsely remembered detail measured in an experiment by Loftus and Palmer (1974).

This phenomenon in which subjects’ subsequent recall could be modified was further investigated with several experiments that focused on the effect of leading questions. (Elizabeth F. Loftus 1975) In one experiment, 150 students at the University of Washington were shown another video of an automobile accident. The test subjects were given a questionnaire immediately following watching the video, in which half were given the question “How fast was the white sports car going when it passed the barn while traveling down the country road?” while half were given a similar question that did not mention a barn. There was, in fact, no barn in the video.

A week later, the test subjects were given another questionnaire that included the question “Did you see a barn?” Of the ones who had a variant of the earlier questionnaire that mentioned a barn, 13 out of 75 (17.3%) replied that they had seen a barn, whereas only 2 out of 75 (2.7%) whose earlier questionnaire did not mention a barn reported having seen a barn on the later questionnaire.

A barn next to a road, which was a falsely remembered detail measured in an experiment by Loftus (1975).

It is one thing to recall false details such as broken glass or a barn that was not originally there, but this research would go on to show that false memories of an entire experience that did not happen could be inculcated in test subjects.

In a one such experiment, 24 pairs of volunteers ranging in age from 18 to 53 years were recruited by University of Washington students. (Elizabeth F. Loftus and Pickrell 1995) Each pair had an older “relative” volunteer who had knowledge of the younger “subject” volunteer. The subjects were given a booklet that contained 4 stories from their childhood experience. While all of the stories were based on knowledge provided by the relatives, 3 of the stories were true, and 1 was a false story about getting lost while on a shopping trip.

The booklet contained a cover letter that instructed subjects to write what they remembered about the events in the space provided or to indicate they did not remember the event if they did not recall it. Later, the subjects were interviewed twice, with 1 to 2 weeks before each interview, and again asked to provide whatever details they could recall about each event, but during the interviews they were also asked to score the clarity of their memories on a scale from 1 to 10.

Subjects remembered 49 out of 72 (68%) of the total true events throughout the experiment. During the booklet phase of the experiment, 7 out of 24 (29%) of the total false events were remembered as having occurred, but 1 test subject changed her answer during the interview phase, decreasing this total to 6 out of 24 (25%).

The mean clarity rating was higher for the true events (6.3) than for the false events (2.8) during the first interview. However, during the second interview, mean clarity remained roughly the same for true events (6.3), but increased for the false events (3.6).

Even after being told that 1 of the 4 events was fabricated, 5 out of the 24 (21%) identified a true event as the story they believed was invented instead of the actual false event.

The results of this and other similar experiments were timely in the 1990s because of criminal prosecutions based on individuals allegedly recalling repressed memories of childhood abuse, memories that were only discovered after extensive therapy sessions. Such experiments showed how such prosecutions could be based on false memories inculcated by therapists inadvertently using similar methods as in the experiments. (Elizabeth F. Loftus 1993)

Flaws of Serial Reproduction

Anecdotes are not just recalled from memory and told to one other person. Sometimes the person told an anecdote retells it to yet another person. Indeed, by the time one encounters it, an anecdote may have been retold any number of times. This is a situation not unlike a children’s game called “telephone” in the United States and by many other names in other parts of the world.

A group of children sitting in a row playing the “telephone” game.
Pressmaster/Shutterstock.com

In this game, one child is told a story. This child is then tasked with repeating it exclusively to a second child, often by whispering. The second child repeats it only to a third child, and so on, until a final child is told the story. At the end, the story that the final child tells is compared with the original story. The amusement of the game comes in discovering how the final version of the story has been mangled and altered from its original version.

In the psychological literature, this phenomenon is called “serial reproduction.” This is in contrast to what is called “repeated reproduction” in which the same individual recalls a memory again and again. Several experiments throughout the twentieth century confirmed the moral of the telephone game, i.e., that serial reproduction is error prone and distorts information. More recently, a study quantified how much more error prone serial reproduction can be compared to repeated reproduction. (Roediger et al. 2014)

In this experiment, 60 undergraduate students at Washington University were presented with DRM word lists by way of computers, asked to do multiplication problems for 30 seconds, then prompted to recall the DRM word list.⁶ For the repeated reproduction portion of the experiment, a test subject was simply asked to recall the original DRM word list 4 times in this manner. For the serial reproduction portion of the experiment, the list recalled by one test subject was given to another test subject. These serially reproduced lists were passed along 4 different test subjects, with the first given the original DRM list and the next 3 given the previous test subject’s recalled list.

The mean proportion of words correctly recalled from the original DRM word lists, averaged across test subjects, remained relatively constant around 50% during repeated reproduction. However, the mean proportion of words correctly recalled declined at each step in serial reproduction, starting around 50% and ending up at just above 20% after 4 iterations. Furthermore, the mean proportion of recalled words that were actually falsely recalled critical unpresented words remained relatively constant around 5% during repeated reproduction, but increased at each step during serial reproduction, winding up at over 10% after 4 iterations.

What is interesting about the worse performance of serial reproduction in this experiment is that at each step in the serial reproduction, test subjects tended to perform better than in the previous step, inasmuch as they were recalling a greater proportion of the lists that they had been presented. This should not be surprising as each step in serial reproduction tended to start with a shorter list than the previous, and it is generally easier to remember less things.

What caused the overall worse performance of serial reproduction compared with repeated reproduction in this experiment was that each step in serial reproduction carried over the mistakes from the previous steps. Once a word that really did appear on the original DRM world list was forgotten, it would not appear on the input list for subsequent serial steps and thus be lost for the rest of the process. Once a critical unpresented word was falsely recalled, it would be put on the input list for the subsequent serial step and so be indistinguishable from words that actually appeared on the original DRM list.

This highlights a danger in the retelling of anecdotes. Specifically, any errors of recall are compounded as anecdotes are told and retold along a sequence of individuals.

Implications

The implications of these psychological flaws for anecdotal evidence are straightforward: human memory is not like a video camera, accurately recording events. Because memory is demonstrably fallible, no anecdote of any given experience told from memory can be blindly trusted. Instead, all such anecdotes ought to be scrutinized. Furthermore, because of the compounding effects of serial reproduction, anecdotes told secondhand ought to be given even more scrutiny.

However, the evidence does not imply that all anecdotes must be thrown out completely, either. While 29 out of 150 test subjects reported seeing broken glass when there was none, 121 did not. (Elizabeth F. Loftus and Palmer 1974) While 55% of critical unpresented words on DRM lists were falsely recalled, 45% were not. (Roediger and McDermott 1995) Thus, in order to separate truth from falsity in encounters with anecdotes, one can neither accept an anecdote without scrutiny nor arbitrarily reject an anecdote without scrutiny.

Furthermore, such scrutiny does not imply that the person telling the anecdote is “crazy,” “hysterical,” or mentally ill. The test subjects in most of these memory experiments were taken from populations of healthy adults who functioned at a high enough level at least to be attending an institution of higher education. Indeed, even if turns out a given person’s anecdote is false, that does not imply the individual is “crazy.” Rather, it only implies that the person has a normally functioning human brain that, like all other human brains, is imperfect in its memory function.

Such scrutiny also does not necessarily imply suspicion of deception or malicious intent in the person telling the anecdote. The test subjects in these memory experiments had no incentive to lie. Intentional deception is just one possible reason for falsity in anecdotes. The manipulations done in these experiments to induce effects of increased false recall illustrate other reasons, such as the misinformation effect or the effect of leveraging preexisting mental associations.

Therefore, the evidence on psychological flaws is such that all anecdotes ought to be scrutinized and that enlightened individuals ought not to take offense at such scrutiny. Indeed, if one were truly interested in the truth being known and were aware of the evidence on memory fallibility, one would welcome such scrutiny.