Limitations of review articles
In the October case study, we briefly discuss the relevance and quality of a small sample of general review articles for the clinical question we were trying to answer.
Review articles are viewed by many clinicians as a quick and easy "surrogate" for the primary data on a given topic, but are subject to tremendous variability in quality and in relevance to a specific clinical question. Often written by one or two "experts" in a field, the review articles are subject to potential bias, including the influence of the authors' personal viewpoints, gaps in literature searching practices that may lead to the omission of relevant research, errors in the translation of data from the primary literature to summarization in the review, misrepresentation or misinterpretation of original source data -- problems that are difficult for a reader to identify without seeking out the primary literature to verify the review's recommendations and conclusions.
In 1999, McAlister et al considered this issue in their study in the Annals of Internal Medicine, "The medical review article revisited: has the science improved?" (PDF). Examining compliance of reviews from 6 major medical journals with 10 quality criteria (e.g. "The methodologic validity or quality of the included studies was assessed"), the authors noted "Of 158 review articles, only 2 satisfied all 10 methodologic criteria (median number of criteria satisfied, 1)."
A JAMA article in 2002 by Rochon et al compared review articles in peer-reviewed journals to those in "throwaway" publications using a quality assessment instrument developed by Barnes and Bero, finding great variability in article quality. The authors note, "mean (SD) quality scores were highest for peer-reviewed articles (0.94 [0.09] for systematic reviews and 0.30 [0.19] for nonsystematic reviews) compared with throwaway journal articles (0.23 [0.03], F2,391 = 280.8, P<.001)" (higher number translates to higher percentage of quality criteria met). Thus, variability in quality measures was revealed not only for trade publications, but also for articles published in major peer-reviewed outlets such as the Annals of Internal Medicine, JAMA, Lancet, and others.
What implications does this have for our searching, selection, and summarization practices? In the current case, we thoroughly consider the primary data first, and then move to an examination of the review items - our thorough understanding of the body of data discussing colistin and its derivates for treatment of antibiotic-resistant Gram-negative pneumonia allows us to critique the review articles and note any omissions or issues in these items.
One of our most basic skills as librarians also can inform this assessment process -- given our comprehensive searching practices, do the authors of the review articles describe how they identified the studies they included? Do they cite references to support key statements and recommendations? Do their reference lists seem to be in line with what we are seeing in our search results, and are they current? Do they include tables or other illustrations that summarize the individual studies in greater detail? Are their inferences in line with our intuitive understanding of the literature?
Consideration of questions such as these can help us assess the quality and relevance of a review article, as well as aid us with identifying instances in which a review article should be accompanied by a few words of caution, supplemental primary data, or omitted completely due to our serious concerns about its methods and/or conclusions. By "filtering" the review literature in this way for clinical questions, we make great use of our skills and strengths as information experts.
Review articles are viewed by many clinicians as a quick and easy "surrogate" for the primary data on a given topic, but are subject to tremendous variability in quality and in relevance to a specific clinical question. Often written by one or two "experts" in a field, the review articles are subject to potential bias, including the influence of the authors' personal viewpoints, gaps in literature searching practices that may lead to the omission of relevant research, errors in the translation of data from the primary literature to summarization in the review, misrepresentation or misinterpretation of original source data -- problems that are difficult for a reader to identify without seeking out the primary literature to verify the review's recommendations and conclusions.
In 1999, McAlister et al considered this issue in their study in the Annals of Internal Medicine, "The medical review article revisited: has the science improved?" (PDF). Examining compliance of reviews from 6 major medical journals with 10 quality criteria (e.g. "The methodologic validity or quality of the included studies was assessed"), the authors noted "Of 158 review articles, only 2 satisfied all 10 methodologic criteria (median number of criteria satisfied, 1)."
A JAMA article in 2002 by Rochon et al compared review articles in peer-reviewed journals to those in "throwaway" publications using a quality assessment instrument developed by Barnes and Bero, finding great variability in article quality. The authors note, "mean (SD) quality scores were highest for peer-reviewed articles (0.94 [0.09] for systematic reviews and 0.30 [0.19] for nonsystematic reviews) compared with throwaway journal articles (0.23 [0.03], F2,391 = 280.8, P<.001)" (higher number translates to higher percentage of quality criteria met). Thus, variability in quality measures was revealed not only for trade publications, but also for articles published in major peer-reviewed outlets such as the Annals of Internal Medicine, JAMA, Lancet, and others.
What implications does this have for our searching, selection, and summarization practices? In the current case, we thoroughly consider the primary data first, and then move to an examination of the review items - our thorough understanding of the body of data discussing colistin and its derivates for treatment of antibiotic-resistant Gram-negative pneumonia allows us to critique the review articles and note any omissions or issues in these items.
One of our most basic skills as librarians also can inform this assessment process -- given our comprehensive searching practices, do the authors of the review articles describe how they identified the studies they included? Do they cite references to support key statements and recommendations? Do their reference lists seem to be in line with what we are seeing in our search results, and are they current? Do they include tables or other illustrations that summarize the individual studies in greater detail? Are their inferences in line with our intuitive understanding of the literature?
Consideration of questions such as these can help us assess the quality and relevance of a review article, as well as aid us with identifying instances in which a review article should be accompanied by a few words of caution, supplemental primary data, or omitted completely due to our serious concerns about its methods and/or conclusions. By "filtering" the review literature in this way for clinical questions, we make great use of our skills and strengths as information experts.
posted by Becky at 4:36 PM
0 Comments:
Post a Comment
<< Home