In the Literature
Apr 2009

Prying Open the File Drawer

Garrett M. Sparks, MD, MS
Virtual Mentor. 2009;11(4):297-300. doi: 10.1001/virtualmentor.2009.11.4.jdsc1-0904.

 

Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252-260.
 

In 1979, Robert Rosenthal coined the term “file drawer problem” to describe the tendency of researchers to publish positive results much more readily than negative results, skewing our ability to discern exactly what an accumulating body of knowledge actually means [1]. He posited the worse-case scenario for experimental trials: journals filled with 5 percent of the studies that show type 1 errors (i.e., find positive results when no positive effect exists), and file drawers filled with 95 percent of the studies that show nonsignificant results. In 1979, sans Watergate-style break-ins, there were few means to estimate how many papers were stuffed into the file drawers. In 2008, former Food and Drug Administration reviewer Erick Turner et al. pried open the file drawer by examining the FDA registry and results database on all phase II and III clinical trials for 12 antidepressant agents approved by the FDA between 1987 and 2004 [2].

To assure transparency in the data submitted for review, drug companies must register with the FDA all trials they intend to use in support of an application for marketing approval or a change in labeling. The registration process requires that drug companies specify the exact methods by which they will collect and analyze data. Raw data must be submitted to prevent biased reporting of favorable trial results. While FDA reviewers have full access to the entire body of data used to make decisions regarding the safety and efficacy of a drug, the clinicians who will be ultimately prescribing these drugs and counseling patients do not.

Turner et al. could not find evidence of publication for 23 out of 74 studies included in their analysis. Thirty-seven out of 38 studies that the FDA deemed “positive” were published. Of the 36 remaining studies classified as “negative” (24) or “questionable” (12), 3 were published as not positive, 11 were published in a way that, in the opinion of Turner et al., conveyed a positive outcome, and 22 were not published at all. By the authors’ estimate, studies judged positively by the FDA were 12 times more likely to be published than studies judged nonpositively. This publication bias leads to an overestimation of total effect size by 32 percent relative to the FDA reviews, ranging from 11 to 69 percent for particular agents. All of the antidepressants still outperform placebo, but not by as much as the published literature would suggest.

The key to understanding the significance of this study lies in the first sentence of the conclusion of the abstract, “We cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, from decisions by journal editors and reviewers not to publish, or both” [3]. The efficacy of the antidepressants studied in these reports is a secondary concern. Turner et al. do not have the means to prove beyond a reasonable doubt that selective dissemination of information regarding the safety and efficacy of these drugs was part of a conscious attempt by researchers to mislead journal readers, but their indictment effectively raises clear suspicion that clinicians should be extremely wary of publication bias when reading clinical-trial results.

Researchers who expend resources on clinical trials to prove drug efficacy no doubt have a personal investment in positive results; rarely are research careers made by demonstrating what does not work, and there is a reason why trials that do not show positive results are deemed failures. If researchers expect positive results, they may be more likely to view their negative results as inherently flawed or lacking much value. Study methods are frequently limited by practical and logistical considerations that may be overlooked in a positive trial but judged the cause of type 2 error in a negative trial. Researchers simply have less incentive to expend effort toward preparing a manuscript of a negative trial. Ninan, Poole, and Stiles defend their unpublished negative trial of low-dose venlafaxine by stating that it established a dose-response relationship, which, they imply, while useful from a regulatory standpoint, did not warrant publication except as supplementary data in another manuscript [4, 5]. Finally, drug companies have responsibilities to shareholders to generate profits by developing and marketing safe, effective drugs. Incentive exists for researchers to publish their data in a way that supports the enterprise of the drug company that funds their work, even if that involves suppressing the data itself.

Similarly, journal editors and reviewers have dual responsibilities to evaluate publications for scientific value and integrity and produce a journal product that justifies its subscription fees. Clinicians treating patients are interested in learning about new treatments that work for the conditions they treat. Drug-company representatives distribute studies that demonstrate what a new drug can do—not what it cannot do. Patients come to physicians looking for answers about how they can be helped, not how they cannot be helped. While negative trials are certainly not absent from published literature, studies with positive results inevitably generate more interest than studies that lack positive results.

Not one of these explanations, however, changes the fact that clinicians who aspire to use treatments that offer the greatest probabilities of fulfilling their patients’ needs find themselves handicapped by publication bias. Popular media readily interpret and package medical literature in ways that often stand to damage the patient-physician relationship. The New York Times review of Turner et al. shows particular restraint in exploring the article’s significance, focusing on the issues surrounding publication bias, but a report by CNNMoney the day before led with the headline “Antidepressants May Not Work: Antidepressant Drugs May Have Little Effect on Patients, Many Unpublished Studies Show” [6, 7].

When physicians appropriately prescribe antidepressants, patients often struggle with the fact that antidepressants work slowly and do not work for everyone. If an antibiotic clears up an infection in a few days, it is not unreasonable for patients to ask why their Prozac doesn’t clear up their depression just as quickly. When CNN tells them that their doctors were either lying to them or unwittingly giving them false information, they understandably question where they can place their trust.

Psychiatry is no stranger to controversy in popular media and serious academic circles. Psychiatric patients suffer stigma nearly unparalleled in other medical specialties, despite improved understanding of the biological contributions to mental illness by the scientific community and culture at large. Much of the popular psychoeducation has unfortunately come in the form of drug-company advertisements. Popular culture myths suggest that psychiatry has worked in conjunction with drug companies to pathologize natural human behavior and emotions in order to make money. Similar criticisms have been heaped upon other medical specialties; consider popular treatment of the increased use of statins, despite extensive evidence supporting their use in the management and prevention of coronary artery disease. Psychiatry, like all fields of medicine, has been working to develop practice models that use principles of evidence-based medicine to optimize patient care. The development of evidence-based practice, however, requires that transparent evidence is easily accessible to clinicians and researchers.

Publication bias is by no means limited to psychiatry. In September 2004, the New England Journal of MedicineThe LancetJAMAAnnals of Internal Medicine, and several other publications announced they would no longer publish the results of pharmaceutical company-sponsored studies that were not registered in a public database prior to the start of the study. Clinicaltrials.gov, the NIH-sponsored registry of federally and privately supported clinical trials conducted worldwide, currently has 68,630 trials with locations in 161 countries [8]. While such registration may not fully force all studies out of the file drawer, it better ensures that those seeking to perform meta-analyses will have the fullest data record possible.

Evidence-based medicine seeks to do much more than simply predict desirable outcomes in populations; it requires that physicians use their knowledge base and clinical experience to collaborate with patients to achieve better health. Published literature informs physicians’ understanding of how to make decisions regarding how they counsel their patients. Less obviously, the unpublished literature must be accounted for as well, as we seek to use the best statistical and experimental methods to treat patients in ways that are worthy of their trust and collaboration.

References

  1. Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull. 1979;86(3):638-641.
  2. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252-260.
  3. Turner et al., 252.

  4. Ninan PT, Poole RM, Stiles GL. Selective publication of antidepressant trials. N Engl J Med. 2008;358(20):2181.

  5. Mendels J, Johnston R, Mattes J, Riesenberg R. Efficacy and safety of b.i.d. doses of venlafaxine in a dose-response study. Psychopharmacol Bull. 1993;29(2):169-174.
  6. Carey B. Antidepressant studies unpublished. The New York Times. January 17, 2008. http://www.nytimes.com/2008/01/17/health/17depress.html?_r=1. Accessed February, 15 2009.

  7. CNNMoney.com. Antidepressants may not work—report. CNNMoney. January 16, 2008. http://money.cnn.com/2008/01/16/news/companies/antidepressants/index.htm. Accessed February 15, 2009.

  8. Clinicaltrials.gov. http://www.clinicaltrials.gov. Accessed February 15, 2009.

Citation

Virtual Mentor. 2009;11(4):297-300.

DOI

10.1001/virtualmentor.2009.11.4.jdsc1-0904.

The viewpoints expressed in this article are those of the author(s) and do not necessarily reflect the views and policies of the AMA.