Virtual Mentor. December 2010, Volume 12, Number 12: 930-933.
Evidence-Based Medicine—Can You Trust the Evidence?
The prevalence in industry-funded research of spin, data dredging, and disagreement between abstract conclusions and the results themselves means physicians must read research articles in their entirety, examine the studies’ methodology, and check for disclosures to gain an accurate understanding of products’ efficacy.
Yvonne M. Buys, MD
Alasbali T, Smith M, Geffen N, et al. Discrepancy between results and abstract conclusions in industry- vs. nonindustry-funded studies comparing topical prostaglandins. Am J Ophthalmol. 2009;147(1):33-38.
At the end of a busy clinic day a drug representative sits you down to discuss her product. She produces a glossy advertisement highlighting the drug’s efficacy with various graphs and references. She then provides a copy of a peer-reviewed article that compares her product to a competitor’s. Without even reading the article you can guess what the conclusions will be—that the rep’s product is better—but to confirm, you skim the abstract and, as suspected, the abstract conclusions state that this company’s product performed better than the competitor’s. You also notice that the study was funded by your visitor’s company. A week earlier, a representative from the competing company made a similar visit, but in that case the peer-reviewed abstract concluded that their product was superior. You look for that article and find that the study was funded by the competitor.
In medicine, we are trained to base decisions on evidence. In deciding which medication to prescribe, for example, there are a variety of considerations: efficacy, side effects, contraindications, ease of use, and cost, among others. Comparing medications from different classes often makes decisions regarding first-line treatment relatively easy; however, when a group of medications produce a given clinical result by means of the same mechanism, the decision may not be as clear. Decisions should be evidence-based, but can we trust the evidence? And which evidence is most trustworthy?
In 1997, Pharmacia introduced a novel medication, latanoprost, for the treatment of glaucoma. As a result of this medicine’s superior efficacy, daily administration, and relatively small number of side effects, latanoprost quickly became the first-choice drug for treating glaucoma—in industry terms, a “blockbuster.” This fact did not go unnoticed by the pharmaceutical competition, which subsequently developed drugs with a similar mechanism of action (“me-too” products), including travoprost (Alcon) and bimatoprost (Allergan). The end result is a variety of products with similar mechanisms of action competing for the same market share. Manufacturers of competing drugs fund studies of their respective products both to create opportunities for clinicians to gain experience with their products and to contribute to the peer-reviewed literature. Their ultimate objective is to influence the decisions of prescribers. Does industry funding of these studies bias results?
In hopes of answering this question, we searched the literature up to November 2007 for peer-reviewed articles on studies that compared the ocular hypotensive efficacy of topical prostaglandins. We reviewed each article to determine whether the findings for the main outcome measures were statistically significant and then compared the statistical findings in the results section to the language in the abstract conclusions. The publications were also divided up into industry-funded and non-industry-funded categories, the results of which were compared.
There were 39 eligible publications, of which roughly 75 percent reported on industry-funded research. In those industry-funded-research articles, there was agreement between the results of the main outcome measure and conclusions in the abstract 38 percent of the time, but in 62 percent of publications, there was disagreement between the results of the main outcome measure and the abstract conclusions. Among the nonindustry-funded studies, the agreement between the results of the main outcome measure and the abstract conclusions was 100 percent (p=0.0006). In publications of the industry-funded research, only 24 percent found a statistically significant main outcome measure, but 90 percent had pro-product conclusions .
You may wonder about the high percentage of pro-manufacturer conclusions in the cases where the main outcome results were not significant. Such conclusions result from authors’ emphasis on surrogate outcomes, minimizing nonsignificant main outcomes and reporting numerical rather than statistically significant differences. Emphasizing secondary outcomes and ignoring the results of the main outcome measure is a process referred to as “spin.” Some studies look at multiple comparisons in attempts to find an outcome of statistical significance. This is referred to as “data dredging.”
These findings are certainly not unique to ophthalmology or publications on prostaglandins. A recent report of published randomized controlled trials with nonsignificant main outcome measures found evidence of spin in 58.3 percent of the abstracts . In addition, there are many studies in the literature outlining the effect of industry bias in publications involving a variety of diseases and therapies [3-19].
Physicians must be aware that it is essential to read literature carefully. One cannot rely solely on the abstract but should read papers in full, focusing on methodology and reading the complete results, because the prevalence of spin is greater in abstracts than in the main text . It is equally important to read the disclosures to determine if a potential for bias exists.
In our study, 4 of the 39 publications did not include published disclosures. By contacting the authors directly, we discovered that 3 of these publications were industry-funded . The International Committee of Medical Journal Editors policy states that “all participants in the peer review and publication process must disclose all relationships that could be viewed as presenting a potential conflict of interest” . In June 2007, when we reviewed the disclosure policies of all indexed English-language ophthalmology journals, we were able to find published web-based policies for authors in 79 percent, for peer reviewers in 7 percent, and for editors in 5 percent of the journals. After we contacted the editors directly, these results changed to 100 percent for authors, 60 percent for peer reviewers, and 33 percent for editors (not counting 5 journals that did not respond to the question regarding authors and 12 journals that did not respond to the question regarding peer reviewers and editors) .
Going forward, measures need to be taken to minimize the possibility of industry bias. Although some have suggested that industry should be prohibited from funding studies, this is unlikely to occur, especially in view of decreased funding by government and nonprofit organizations. Currently, authors and editors have significant responsibility for upholding the integrity of published literature, and perhaps stronger policies are required both for disclosure of potential conflicts and presentation of results. Ultimately, however, it is readers who must take time to read articles in full, and not simply rely on the abstract, before drawing their own conclusions.
Yvonne M. Buys, MD, is a professor in the Department of Ophthalmology and Visual Sciences at the University of Toronto and the co-director of the Glaucoma Unit at the University Health Network. She is the immediate past president of the Canadian Glaucoma Society, glaucoma section editor for the Canadian Journal of Ophthalmology, and treasurer and board member of the Canadian Ophthalmological Society.
Related in VM
The viewpoints expressed on this site are those of the authors and do not necessarily reflect the views and policies of the AMA.
© 2010 American Medical Association. All Rights Reserved.