Research in Focus: Should Low-Quality Evidence Dominate High-Level Evidence?

  •   December 4, 2020

Full Citation: Riley S, Swanson BT, Sawyer SF, Brismée J-M, Staysniak G. Should low-quality evidence dominate high-level evidence? A systematic review and meta-analysis of systematic reviews of musculoskeletal physical therapy interventions. Journal of Manual & Manipulative Therapy 2020 DOI:10.1080/10669817.2020.1839728

Background

Systematic reviews of randomized clinical trials (RCTs) are considered the highest level of evidence available to guide clinical practice decisions. Despite the increasing availability of RCTs relevant to physiotherapy practice and the presence of systematic reviews to guide clinical practice, research has reported that a significant proportion of intervention provided by physiotherapists either has no evidence-based recommendation to support it or is not recommended based on the evidence available.1

The authors of this study highlight that there is some concern with the proliferation of poorly designed clinical trials in the area of musculoskeletal (MSK) physiotherapy, and that when combining poorly designed trials into a systematic review, it enables “low-quality research to masquerade as high-quality research.” The authors also highlight the issue of bias and its potential influence on how a systematic review is designed, including how studies are selected and findings interpreted and reported. Spin, or the “specific reporting strategies used by authors to convince readers that the beneficial effect of the treatment of interest is greater than shown by the results” has also been identified to be a problem, most notably in a recent study which reviewed abstracts of systematic reviews for physiotherapy interventions for low-back pain.2

The purposes of this methodological systematic review were to:

  • Identify systematic reviews of English language RCTs related to MSK physiotherapy
  • Use the PEDro website to determine the quality of studies included in the RCTs
  • Determine the journal impact factor (JIF) of the journals publishing the SRs
  • Assess the quality of the SRs
  • Assess whether the SR authors used spin in reporting their conclusions
  • Determine if a professional librarian was employed in developing the literature search
  • Determine if the journal publishing the SR was financially biased (predatory)
  • Determine if the team conducting the SR had unreported conflicts of interest
  • Determine if there were differences between the PEDro scores reported by SR authors and the official scores published by PEDro
  • Determine if there were relationships between the quality metrics of articles used in the SRs and the quality metrics of the SRs themselves

Methods

  • Methodological Systematic Review
  • Inclusion Criteria:
    • SRs of RCTs published in English related to MSK physiotherapy interventions
    • Publication date 1 January 2016 to 12 December 2019
    • The authors “tried to critically appraise SRs that were high in methodological quality and determine the impact of low quality and top-quality RCTs on this quality”
    • SR had to use the PEDro score to assess the quality of the included RCTs
  • Exclusion:
    • If the SR used multiple tools to assess the RCTs included in the SR (to avoid potential confounding variables in the data analysis)
  • Databases and Search Strategies
    • PubMed, MEDLINE, CINAHL
    • Citation searching of bibliographies of reviews
    • Strategy developed by a professional librarian experienced in systematic literature reviews and peer reviewed by another librarian using the PRESS standard (Peer Review of Electronic Search Strategies)

Findings

  • 24 SRs met the eligibility for inclusion in the study
    • The number of articles included in each SR varied from 3 to 45

Quality of Articles included in SRs

  • No significant correlation between number of articles and quality metrics
    • Implication: SRs that include more studies do not necessarily include better studies.
  • Many SRs included articles published in a journal without a JIF and 7/24 SRs included 25% or more such articles
    • Implication: the JIF relates to the number of times articles published in a journal are cited in other studies. While not a quality metric, many predatory journals are known to have low JIF scores while others do not report a JIF at all. Predatory journals are known for publishing low-quality research and for a lack of reliable peer review
  • The JIF for journals “from which reviewed articles were published varied in a non-normal fashion in many SRs (mostly low JIF values mixed with a few high JIF values)”
    • Implication: one should expect a normal distribution of JIF values clustered around the mean.
  • All but one SR included “low quality” articles (PEDro <6)
  • Collectively, from 24 SRs including 329 articles total, 34% (N=111) were low quality
    • More than half (14/24 SRs) included studies that did not meet Criterion #1 of the PEDro scale, related to external validity and identification of the population of interest (the P of PICO).
      • Implication: this impacts the ability to determine generalizability of the study. “If the generalizability of the RCT cannot be specified, it should not be included in the SR”1
    • 7/24 included pilot studies
      • Implication: By nature, pilot studies are underpowered to determine the effect of an intervention and should not be included in an SR
    • 7/24 included articles that were not peer-reviewed

Quality of SRs

  • 3/24 were “low confidence” and 21/24 were “critically low confidence” as assessed using the AMSTAR 2 tool
  • 16/24 SRs had 3 or 4 “critical validity flaws”
  • 12/24 met the criteria for having “spin”
  • 8/24 made “conclusions about the literature that were based in part on low-quality articles”
  • None of the SRs reported using a librarian to develop the literature search criteria or to perform the literature search
    • Implication: “it is widely accepted that the inclusion of a professional librarian with expertise in SR search methodology decreases errors in search strategies and improves the quality of SRs,”1 the authors also note that including a librarian with “no interest in the outcome of the SR” may also decrease the risk of bias
  • Three instances found where a conflict of interest for an SR author was found which had not been disclosed
  • Meaningful, statistically significant relationships were found between the median JIFs and mean PEDro scores and between median JIFs and the percentage of low methodological quality trials included in the 24 SRs

Limitations

  • Findings are only generalizable to SRs that use the PEDro score as a single instrument for critical appraisal of RCTs during the SR process
  • Limited to SRs related to physiotherapy MSK interventions within the publication date range considered

Author’s Conclusion

“All studies included in this methodological SR included SRs that represented Level 1A evidence related to a Grade A practice recommendation, considered to be strong recommendations that clinicians should follow unless they have a ‘clear and compelling’ reason not to follow them. Based on the findings of this methodological SR, such recommendations may be problematic. The results... suggest that the highest level of evidence may over-represent poor quality evidence based on how the SRs and the RCTs contained within those SRs were reported.”1

Relevance to physiotherapy practice in Alberta

Physiotherapists are expected to employ evidence-informed practice in all aspects of physiotherapy service provision. Critical review of evidence, whether randomized controlled trial, systematic review or other is a critical skill for all clinicians.

Physiotherapists must be aware of the pitfalls and challenges involved in the creation of credible, high-quality evidence. Even systematic reviews, generally considered to be “high-level evidence,” can be subject to bias and other pitfalls when not conducted rigorously. Certainly, other levels of evidence are also subject to bias and when low-quality evidence is the basis of a systematic review, biases and other methodological issues can become amplified, resulting in inaccurate conclusions and practice guidance.

Physiotherapists should be aware of the risks and biases that systematic reviews and other forms of research may present and should use this knowledge to critically review the evidence that they employ to inform their practice.

Disclaimer

The purpose of this summary is to highlight recently published research findings that are not openly accessible. Every effort is made to ensure accuracy and clarity of the summary. Readers are encouraged to review the published article in full for further information.


  1. Riley S, Swanson BT, Sawyer SF, Brismée J-M, Staysniak G. Should low-quality evidence dominate high-level evidence? A systematic review and meta-analysis of systematic reviews of musculoskeletal physical therapy interventions. Journal of Manual & Manipulative Therapy 2020 DOI:10.1080/10669817.2020.1839728
  2. Nascimento DP, Oliveira Pena Costa L, Zoldan Gonzalez G, Maher CG, Moseley AM. Abstracts of low back pain trials are poorly reported, contain spin of information and are inconsistent with the full text: An overview study. Archives of Physical Medicine and Rehabilitation 2019. DOI:10.1016/j.apmr.2019.03.024.