The Art of Cherry Picking in Science

Issac Newton lived at a time when his critics had a poor understanding of the real distribution of experimental data. In order to have his work accepted, Newton 'adjusted' the results that supported his calculations for gravity and optics. Image credit: wikipedia.com

Why did Scientists Cherry Pick?

Note: I was inspired to write these articles in part as a rebuttal to the ignorant treatment of astrology and to explore the relationship between science and astrology from a historical and philosophical perspective.

One of the common practices in science is cherry picking. What is cherry picking? It is the practice of removing data that does not fit expectations.

Often data is removed because a researcher is convinced that it is anomalous and represented a glitch, or the error is attributed to malfunctioning equipment, to errors in recording or that contamination has occurred in the apparatus used in the study, etc.

It is a practice that goes back to the beginnings of science. Historically, scientific figures as prominent as Newton and Mendel practiced it. Some of their results no longer pass scrutiny when analyzed by statistical methods today. Why did they do it? For the same reason it is still done, even today: to gain acceptance of their research.

The results of their experiments convinced them that their premise, hypothesis or theory was valid, but in those days they faced considerable ignorance on the part of those that would accept or criticize their results. If they didn’t thin out some of the data by cherry picking, then what they discovered faced being rejected by those that were ignorant of the process and of how data tends to be spread.

Mendel's work with genetics contradicted the beliefs about inheritance that were held by the scientific community of his time. In order for his work to be given any credence, Mendel resorted to cherry picking his data points so that his results would not be dismissed.

Today, when we apply statistical theory to his results, his data does not show enough variance. Image credit: wikipedia.com

Cherry Picking Disguised as Scientific Method

Many of the methods used may have been fairly innocent by intention, but contributed to a distortion of the results, or more significantly masked or hid results that were not artifacts at all but data points that provided information that questioned the assumptions or supported a more complex understanding of the phenomena.

Occam's Razor undoubtedly also became a factor that influenced the removal of unexpected data points. It emphasized the simpler explanation to avoid superfluous complexity. Unfortunately, the simpler explanation was not always the correct choice.

When data is cherry picked, under the pressure to get results, it delays science and creates an obstacle for science to be objective. Research that follows, which ironically could corroborate the un-manipulated results, now faces greater scrutiny, because it contradicts already accepted and published results.

Cherry Picking and the Money Trail

Unfortunately, in modern times, much of medical research is cherry picked. Some estimate that the amount of misrepresentation, in one form or another, is as great as 50%. It is one reason why drugs go on the market that are less effective than claimed or have serious side effects that were not initially revealed, or are revealed too late, or become public knowledge because of the litigation that ensues.

With respect to how researchers may mold or cherry pick data, much is revealed when you follow the money trail. If a researcher is sponsored by a pharmaceutical company that is eager to have their product approved, then there is considerable pressure to get positive results, and as a consequence under report negative results. There are a number of ways that the data or research can be deliberately skewed or manipulated to get the results being sought.

Once simple method is for the researcher to deliberately construct several parallel studies, and only report the positive ones. It is a fairly easy tactic to employ. For example, a researcher may decide to freeze or discontinue the studies that hold negative or not sought after results, and either ignore them, declare them as faulty, contaminated or simply disclose that these studies are still ongoing and it is too early to report results, and only the ‘completed’ studies are being reported.

Cherry Picking in Publication

Aside from the cherry picking that goes on in research, there is also the cherry picking that occurs in publication. Publishers of peer reviewed journals find themselves in the situation that their own authority or reputation as a respected publisher is an important factor in determining both the quality of submissions and what they can charge for publication copies. When journals receive research papers that are controversial or present evidence that questions previously held beliefs or theories, they are often treated very differently, as the example with the publication Nature shows. Instead of the normal peer review process, these are either ignored or when they can not be ignored due to the significance of the research or prestige of the researcher, the path is often chosen to discredit the researcher, or find a catch or loophole in order not to risk the reputation of the journal itself, if it were to publish, or risk that the submission is accepted through peer review.

It is not uncommon for researchers to treat an unexpected result as suspect, and when a possible or convenient explanation is formulated for considering it not a valid observation, to eliminate it.

Unfortunately a bias of expectations, in determining expected outcomes may easily lead to the removal of significant data.

Later researchers face an increased challenge if their results contradict established and published work. This can be even more significant, if the reason that some of the data veered away from expectations, was that it indicated a more complex process, rather than the simple one expected. Examples of where this can become a major problem have occurred in pharmacological studies of drug interaction.

The Consequences of Cherry Picking

Of course, intentional cherry picking can skew the results and promote an incorrect conclusion, or an incorrect confidence level for the data being presented, and this behavior would be rightly considered fraudulent, when it is uncovered is treated as such.

More common has been the practice of removing data points that were not consistent with expected observations. It is not uncommon to re-examine older research and find that in ‘cleaning up’ or removing inconsistent ‘artifacts’, evidence that would have pointed to new paradigms was buried and this delayed progress.

The consequence of cherry picking is that it tends to promulgate itself. If previous publications have cherry picked, then it is more difficult to convince peer reviews that your own observations are accurate or valid without doing the same.

Beyond Cherry Picking

There are other kinds of cherry picking that effect larger scopes of research. Examples abound in archeology. For decades, artifacts have been discovered by reputable investigators, or authenticated with signed statements verifying discoveries, that were ignored because they did not fit the beliefs of the day. Even whole structures were unearthed and although invitations were consistently offered for researchers to investigate the evidence, no one came.

A case in point were the discoveries of pottery and other artifacts in the Andes that were dated to be tens of thousands of years old. They were ignored by mainstream archeology, because of the belief that all of the indigenous populations of North and South America came from the Alaskan-Bering Land Bridge during the last part of the Ice Age, about 12,000 years ago. This theory was supported by the absence of sites older than that date in North America.

More recent excavations in the southern parts of North America and in South America have shown that there were in fact civilizations dating back at least 34,000 years, and that migrations extended from Asia over several ice ages.

Winston Churchill is quoted as having something to say about this. He is quoted as saying: “Men occasionally stumble over the truth, but most of them pick themselves up and hurry off as if nothing ever happened.”

When researchers encounter something that fundamentally contradicts what they believe, they usually ignore it as an anomaly. Those that have the courage to investigate it more often than not, because it goes against the grain of accepted and safe belief, face the ridicule and avoidance of their colleagues.