Original Article
The Global Rating Scale complements the AGREE II in advancing the quality of practice guidelines

https://doi.org/10.1016/j.jclinepi.2011.10.008Get rights and content

Abstract

Objective

To explore the role of a four-item Global Rating Scale (GRS) that could be used in place of the Appraisal of Guidelines, Research and Evaluation II (AGREE II).

Study Design and Setting

A mixed four-factor design was used (User Type, Evaluation Type, Clinical Topic, Guideline). Participants were asked to read and evaluate a guideline using both the AGREE II draft and GRS or GRS only and to complete a series of questions regarding overall guideline quality, adoption, utility, and acceptability.

Results

One GRS item varied as a function of User Type. Each item was a significant predictor of participants' outcome measures. All items were rated as useful by stakeholders. The GRS rating scores, outcome measures, and usefulness scores did not vary between the two Evaluation Type conditions. Correlations between the GRS and the outcome measures were stronger compared with those between the AGREE II draft and these measures.

Conclusion

Although the GRS is less sensitive than the AGREE II in detecting differences in guideline quality as a function of User Type, its items did predict important outcome measures related to guideline adoption. The GRS may play a role in guideline evaluation, although further study is warranted.

Introduction

What is new?

  • At this stage, although there may be a role for the Global Rating Scale (GRS) in the practice guideline enterprise, more research is warranted to fully understand its role and place; to that end, the Appraisal of Guidelines, Research and Evaluation II (AGREE II) remains the international standard for the development, reporting, and evaluation of practice guidelines.

  • Overall quality measures and outcomes scores did not vary between participants who used the GRS alone and those who used both the AGREE II and GRS; however, the association between the GRS and outcome measures appeared stronger than that of the AGREE II and outcome measures.

  • Participants' feedback on the GRS focused on providing more detail for the four items of the GRS; those details are already reflected in the AGREE II.

Practice guidelines (guidelines) are systematically developed statements aimed to assist clinicians and patients in making decisions about appropriate health care for specific clinical circumstances and can also be used to guide policy and system-level decisions [1], [2], [3]. In 2003, the Appraisal of Guidelines, Research and Evaluation (AGREE) Instrument, a tool to assess the process of guideline development and reporting [4], was released. Subsequently, a program of research by the AGREE Next Steps Consortium (the Consortium) was established that aimed to refine the original tool, improve its measurement properties, create efficiencies in the process, and create new documentation to better able support its application. This led to the AGREE II, published in 2010, which is designed to guide the development, reporting, and evaluation of practice guidelines [5], [6], [7]. Online training platforms for the AGREE II and translations of the revised tool are now available (see www.agreetrust.org).

In addition to creating the AGREE II, and as part of the Consortium's research program, we explored the role of a brief Global Rating Scale (GRS) that could be used in place of the AGREE II, for example, when time and resources are sparse and application of a comprehensive tool is not feasible. To this end, we explored the performance of the GRS, its link to outcomes related to overall guideline quality and adoption, its utility, and its acceptability by users. This report profiles this component of the research program.

The specific research questions were the following:

  • 1.

    When used to assess a practice guideline, do GRS item scores vary as a function of User Type? Do the secondary overall outcome scores vary as a function of User Type?

  • 2.

    Do quality ratings on the GRS items, User Type, and the interaction of these two factors predict users' endorsement of, intention to use, and overall quality rating of a guideline?

  • 3.

    To what extent are GRS items rated as useful by users? Do these ratings vary by User Type? How can the GRS items be improved to better meet the needs of users?

  • 4.

    Do GRS scores, secondary overall outcome measures, or usefulness scores vary among participants who critically appraised guidelines with the AGREE II draft and the GRS compared with those who critically appraise guidelines with the GRS alone?

  • 5.

    What are the correlations between the AGREE II draft total scores, GRS total scores, and secondary overall outcome scores?

Section snippets

Methods

An overview of the study design and method is presented in Appendix A.

Results

This article reports on findings relevant to the GRS and its interface with the AGREE II draft. The findings related to the performance, utility, and ultimate evolution to the AGREE II are presented elsewhere [5], [6], [7]. A total of 158 individuals participated in the study. Because the sample size calculation for the exploratory three-way interaction was not reached, we focused on our primary research interest, the two-way User Type × Evaluation Type interaction. Table 1 provides the

Discussion

This study explored the performance and utility of the GRS as an alternative and potentially more efficient strategy to the AGREE II to evaluate guidelines. We considered both the performance of the GRS and perceptions of its usefulness by stakeholders and contrasted outcome measures between participants who appraised guidelines using the GRS only with those who appraised guidelines with both the AGREE II draft and the GRS. In contrast to recent findings that demonstrated meaningful significant

Acknowledgments

The AGREE Next Steps Consortium would like to thank the NGC for its assistance in facilitating the identification of eligible practice guidelines used in the research program of the AGREE Next Steps Consortium. The Consortium would also like to thank Ms Ellen Rawski for her support on the project as research assistant from September 2007 to May 2008.

This research was supported by the Canadian Institutes of Health Research (CIHR) (#MOP 77822), who had no role in the design, analysis, or

References (11)

  • J.A. Whitworth

    Best practices in use of research evidence to inform health decisions

    Health Res Policy Syst

    (2006)
  • G.P. Browman et al.

    Population-based cancer control and the role of guidelines—towards a “systems” approach

  • The AGREE Collaboration

    Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project

    Qual Saf Health Care

    (2003)
  • M. Brouwers et al.

    AGREE II: advancing guideline development, reporting and evaluation in healthcare

    CMAJ

    (2010)
There are more references available in the full text version of this article.

Cited by (85)

  • Methodological approaches for developing, reporting, and assessing evidence-based clinical practice guidelines: a systematic survey

    2022, Journal of Clinical Epidemiology
    Citation Excerpt :

    It has 23 items within six domains [47]. In 2012, the same research team developed a shorter version of the AGREE II: the AGREE-Global Rating Scale (GRS), which can be used when time and resources are scarce, and the application of a comprehensive tool is not feasible [48]. The third tool, AGREE-REX, can be used to assess the quality of guideline recommendations.

View all citing articles on Scopus

Conflict of interest: The authors M.C.B., J.S.B., and F.C. are trustees of the AGREE Research Trust.

Members of the AGREE Next Steps Consortium: All the authors of this article and Dr Dave Davis, Association of American Medical Colleges, Washington, DC, USA.

View full text