Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling

*Psychological Science* has scheduled an article for publication in a future issue of the journal: "Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling."

The authors are Leslie K. John of Harvard University, George Loewenstein of Carnegie Mellon University, & Drazen Prelec of the Massachusetts Institute of Technology.

Here is the abstract:

Cases of clear scientific misconduct have received significant media attention recently, but less flagrantly questionable research practices may be more prevalent and, ultimately, more damaging to the academic enterprise. Using an anonymous elicitation format supplemented by incentives for honest reporting, we surveyed over 2,000 psychologists about their involvement in questionable research practices. The impact of truth-telling incentives on self-admissions of questionable research practices was positive, and this impact was greater for practices that respondents judged to be less defensible. Combining three different estimation methods, we found that the percentage of respondents who have engaged in questionable practices was surprisingly high. This finding suggests that some questionable practices may constitute the prevailing research norm.

Here's how the article starts:

Although cases of overt scientific misconduct have received significant media attention recently (Altman, 2006; Deer, 2011; Steneck, 2002, 2006), exploitation of the gray area of acceptable practice is certainly much more prevalent, and may be more damaging to the academic enterprise in the long run, than outright fraud.

Questionable research practices (QRPs), such as excluding data points on the basis of post hoc criteria, can spuriously increase the likelihood of finding evidence in support of a hypothesis.

Just how dramatic these effects can be was demonstrated by Simmons, Nelson, and Simonsohn (2011) in a series of experiments and simulations that showed how greatly QRPs increase the likelihood of finding support for a false hypothesis.

QRPs are the steroids of scientific competition, artificially enhancing performance and producing a kind of arms race in which researchers who strictly play by the rules are at a competitive disadvantage.

QRPs, by nature of the very fact that they are often questionable as opposed to blatantly improper, also offer considerable latitude for rationalization and self-deception.

Concerns over QRPs have been mounting (Crocker, 2011; Lacetera & Zirulia, 2011; Marshall, 2000; Sovacool, 2008; Sterba, 2006; Wicherts, 2011), and several studies--many of which have focused on medical research--have assessed their prevalence (Gardner, Lidz, & Hartwig, 2005; Geggie, 2001; Henry et al., 2005; List, Bailey, Euzent, & Martin, 2001; Martinson, Anderson, & de Vries, 2005; Swazey, Anderson, & Louis, 1993).

In the study reported here, we measured the percentage of psychologists who have engaged in QRPs.

As with any unethical or socially stigmatized behavior, self-reported survey data are likely to underrepresent true prevalence.

(cut)

The study "surveyed over 2,000 psychologists about their involvement in questionable research practices."

The article reports that the findings "point to the same conclusion: A surprisingly high percentage of psychologists admit to having engaged in QRPs."

(cut)

Most of the respondents in our study believed in the integrity of their own research and judged practices they had engaged in to be acceptable.

However, given publication pressures and professional ambitions, the inherent ambiguity of the defensibility of "questionable" research practices, and the well-documented ubiquity of motivated reasoning (Kunda, 1990), researchers may not be in the best position to judge the defensibility of their own behavior.

This could in part explain why the most egregious practices in our survey (e.g., falsifying data) appear to be less common than the relatively less questionable ones (e.g., failing to report all of a study's conditions).

It is easier to generate a post hoc explanation to justify removing nuisance data points than it is to justify outright data falsification, even though both practices produce similar consequences.

(cut)

Another excerpt: "Given the findings of our study, it comes as no surprise that many researchers have expressed concerns over failures to replicate published results (Bower & Mayer, 1985; Crabbe, Wahlsten, & Dudek, 1999; Doyen, Klein, Pichon, & Cleeremans, 2012, Enserink, 1999; Galak, LeBoeuf, Nelson, & Simmons, 2012; Ioannidis, 2005a, 2005b; Palmer, 2000; Steele, Bass, & Crook, 1999)."

(cut)

More generally, the prevalence of QRPs raises questions about the credibility of research findings and threatens research integrity by producing unrealistically elegant results that may be difficult to match without engaging in such practices oneself.

This can lead to a "race to the bottom," with questionable research begetting even more questionable research.

----------------------
Thanks to Ken Pope for this information.

The abstract and article are here.

The Unintended Consequences of Conflict of Interest Disclosure

By George Loewenstein, PhD, Sunita Sah, MD, PhD, & Daylian M. Cain, PhD.

The Journal of the American Medical Association

Conflicts of interest, both financial and nonfinancial, are ubiquitous in medicine, and the most commonly prescribed remedy is disclosure. The Medicare Payment Advisory Commission and the Accountable Care Act impose a range of disclosure requirements for physicians, and almost all medical journals now require authors to disclose conflicts of interest (although these requirements may be imperfectly heeded). Given that some relationships between physicians and industry are fruitful and some conflicts are unavoidable, can disclosure correct the problems that arise when economic interests prevent physicians from putting patients' interests first?

Given that some relationships between physicians and industry are fruitful and some conflicts are unavoidable, can disclosure correct the problems that arise when economic interests prevent physicians from putting patients' interests first?

Disclosure has appeal across the political spectrum because it acknowledges the problem of conflicts but involves minimal regulation and is less expensive to implement than more comprehensive remedies.

More importantly, even if disclosure is rarely seen as providing a complete solution to the problem, it is broadly perceived to have beneficial effects.

There are, however, reasons that disclosure can have adverse effects, exacerbating bias and hurting those it is ostensibly intended to help.

<snip>

ENHANCING THE EFFECTIVENESS OF DISCLOSURE

Despite its potential pitfalls, disclosure is almost certainly a good thing.

It should be a patient's right to know whether his or her physician is receiving financial benefits from prescribing a particular drug or will personally benefit if the patient accepts recommended tests or procedures.

The question for policy should not be whether to disclose but how to ensure that disclosure has its intended effects.

Research has revealed ways of making disclosure more effective.

<snip>

Even if disclosure is crafted in a fashion that increases effectiveness and minimizes potentially adverse consequences, it is no panacea.

Disclosure is simply not applicable to many serious conflicts of interest affecting medicine in the United States.

Although payments from pharmaceutical and device companies have received most of the attention in the literature on disclosure, other conflicts, such as financial arrangements that give physicians (and their institutions) incentives for providing high-cost services of dubious value, may be more consequential.

It is difficult to imagine how disclosure could even be applied to, let alone undo, those problems.

However, perhaps the most significant likely pitfall of disclosure is not its effects on the quality of advice received by individual patients or its inapplicability to many serious conflicts of interest, but the likelihood of a kind of moral licensing on the part of the profession as a whole--the rationalization that, with disclosure, the profession has dispensed with its obligation to deal with conflicts of interest.

Conflicts of interest, including fee-for-service arrangements, are at the heart of the astronomical increases in health care costs in the United States, and transparency is no substitute for more substantive reform.

The entire article is here.

Special thanks to Ken Pope for this information.

I Disclose ... Nothing

The Hard Truths About Disclosure
By Elisabeth Rosenthal
The New York Times Sunday Review
Originally Published January 22, 2012

IN New York and a growing number of American cities, diners are encountering sanitary grades in restaurants’ windows — A, B or C. That system is an example of helpful disclosure, researchers say: information that is simple and comprehensible, important to recipients and easily acted upon. I recently chose between outwardly identical Japanese noodle shops on East Ninth Street in Manhattan based on the system, walking into the A rather than the B.

But as greater disclosure has become the go-to solution for a wide range of problems — from unethical campaign financing to rising corporate carbon emissions — it has often delivered lackluster results, researchers say.

Just last week, the Obama administration announced plans to require drug companies to disclose a wide variety of payments and gifts to doctors, from speaking fees to the purchase of breakfasts for office staffs, in the hope of reducing commercial influence on prescribing practices. President Obama has promised to run the most open, transparent administration in history. But is more disclosure the solution?

If recent history serves as a guide, disclosure laws — meant to elucidate — do not necessarily lead to greater transparency or prevent the things they were meant to deter. Every holder of a subprime mortgage that is now underwater once signed an elaborate disclosure statement required by the Truth in Lending Act describing precisely the risky terms of their loan. Likewise, “super PACs” in the presidential campaign are technically compliant with financial disclosure laws, but have so far proved successful at hiding many of the sources of their money.

Everyone agrees that openness is a virtue in a democracy. So what is going wrong?

One fundamental problem is that disclosure requirements merely get information onto the table, but themselves demand no further action. According to political theory, disclosure is both a citizen’s right and a tool to ensure good government and consumer protection, because it provides information that leads to informed decisions. Instead, disclosure has often become an endpoint in the chain of responsibility, an act of compliance with the letter of the law rather than the spirit of transparency.

“In the beginning, disclosure was a means to an end, and now it’s often an end in itself,” said Kevin P. Weinfurt, professor of psychiatry and behavioral science at Duke University. “People think, ‘If we’ve disclosed we’ve fulfilled our responsibilities.’ ”

The entire story is here.

Psychiatrist faces federal charges in HIPAA case

By Tim McGlone
The Virginian-Pilot

A psychiatrist faces trial in federal court on charges of illegally disclosing medical information of a Virginia state trooper who had been in his care after being held hostage and raped over three nights.

Prosecutors said this could be the first prosecution nationwide of a physician for violating the Health Insurance Portability and Accountability Act, known as HIPAA, which went into effect in 2003. The act prohibits disclosure of health records unless the patient gives consent.

Dr. Kaye

Jury selection and testimony began Tuesday in U.S. District Court, where Dr. Richard Alan Kaye, the former medical director of psychiatry at Sentara Obici Hospital in Suffolk, faces three counts of wrongfully disclosing an individual's health information.

Kaye was working at Obici in 2007 when the female trooper came to him for treatment. Kaye diagnosed her with post-traumatic stress disorder stemming from the attack in her home several months earlier.

But the trooper wasn't happy with the way Kaye was treating her and left after 16 days. She filed a complaint with the hospital and, according to federal prosecutors, he lost his job as a result. The Virginian-Pilot does not disclose the identity of rape victims.

The entire story can be read here.