The National Health Service Cervical Screening Programme (NHSCSP) first published evidence-based guidelines addressing the content of screening letters and leaflets in 1997.¹The NHS Cancer Plan (September 2000) called for honest, comprehensive and understandable screening materials² and a White Paper (November 2004) emphasised the need for up-to-date and factual health information.³ An important NHSCSP priority is the continual improvement of the quality of written information sent to women about cervical screening. As such, a review was commissioned in early 2004 to develop updated NHSCSP patient-information guidelines. The focus of the review was directed and informed by two main questions:

1. What is the existing research evidence base regarding the content of the written information sent to women at all stages of the cervical screening process?

2. What are the information needs of women during cervical screening?

The main purpose of the review was to identify important information for inclusion in the screening programme letters and leaflets (introductory, abnormal result and colposcopy). Important information included facts (length of the screening interval), concepts (purpose of the screening test) and general points (suitable attire). For each key information point, a tabular summary was created of the relevant evidence. The full updated guidelines are described elsewhere.⁴ A selection of the review recommendations is presented in fig 1. The reviewers were experienced in conducting quantitative and qualitative research related to cancer screening and patient and health professional information.

SYSTEMATIC REVIEWS

Conventional systematic reviews aim to bring together and summarise the available research addressing a question of interest so that inferences about findings may be guided by the most appropriate forms of evidence.^5,6 Evidence synthesis systems have, in the past, given preference to quantitative studies (mainly randomised trials) and clinical effectiveness. Accordingly, explicit methods have been established for the systematic review of trial-based evidence including comprehensive literature searching, detailed quality appraisal procedures and standard synthesis techniques.^7,8,9,10 Interest is growing in the development of diverse systematic review methods to incorporate different types of evidence including other quantitative study designs as well as qualitative research.¹¹ Methods for the synthesis of qualitative research are still in the early stages of proposal and evaluation,^12,13 and the suitability of fitting various forms of qualitative research within the framework of conventional review methodology is an active research topic.¹¹ There is, in fact, still debate about whether the synthesis of qualitative evidence is appropriate and whether it is acceptable to combine qualitative studies derived from different traditions.^13–15

LIMITATIONS OF TRADITIONAL HIERARCHIES OF EVIDENCE

Traditional evidence hierarchies were developed specifically to address questions of efficacy and effectiveness⁶ and involved assessing research according to study design with the randomised trial as the premier form of evidence.^7,16 This becomes problematic in areas of research dominated by non-trial quantitative evidence owing to the lower quality scores subsequently assigned to these studies.¹⁷ Also, evidence syntheses concentrating on wider issues such as “How does it work?” or “What are patients’ experiences?” require a different approach because these types of research questions are best answered by evidence from a variety of sources.^15,18–20

QUALITATIVE RESEARCH

Qualitative evidence has a role to play in answering questions not easily evaluated by experimental studies.^18,20,21 Although qualitative studies do not establish probability estimates or effect sizes, they can provide important support for quantitative outcomes and identify patient priorities and concerns.¹² However, the usefulness of qualitative research is not limited to the explanation of process measures. In fact, qualitative research may address many healthcare questions directly and inform evidence-based practice independently of other research methodologies.^12,21,22

IMPORTANCE OF A RANGE OF EVIDENCE

Policy-makers and practitioners assessing complicated healthcare questions draw on diverse sources of evidence during the decision-making process.^13,14,23 The development of methods for the synthesis of different types of research in a single review is an important methodological challenge. A number of approaches (narrative summary, thematic analysis, meta-ethnography and bayesian methods) could be applied to the synthesis of both qualitative and quantitative research.^12,13 However, most have evolved from techniques used for primary data analysis and were initially developed for either qualitative or quantitative data—not both.¹³ Several groups are currently working on methods for the incorporation of qualitative evidence into systematic reviews including the international Cochrane Qualitative Methods Group, the international Campbell Collaboration, the Evidence for Policy and Practice Information and Coordinating Centre in the UK and the Joanna Briggs Institute in Australia.²³ The UK Economic and Social Research Council has also funded a programme of research in this area.^23,24

Although methods of synthesising quantitative and qualitative research are beginning to be developed, few studies have been published that address the problem from a practical point of view.^9,15,25,26 The current lack of specific guidance and accumulated experience in the synthesis of a range of research evidence leaves would-be reviewers in a difficult position. The aim of this paper is to contribute to the debate by describing the review methods developed and difficulties encountered during the process of updating evidence-based guidelines for the content of NHSCSP letters and leaflets.

WHAT IS THE EVIDENCE BASE?

The literature describing screening information content and women’s information needs was dispersed among a variety of disciplines, and numerous research designs potentially addressed the review questions. Consequently, the review evidence base was complex and ill defined. This is not a unique problem, many healthcare questions are not easily evaluated solely by experimental methods (ie randomised trials) or other forms of quantitative investigation.^12,15,17,27 Boxes 1–4 provide examples of the range of data sources included in the review (the four studies described were focused on colposcopy information).

METHODS

Literature searching

Systematic searches of 12 electronic databases (January 1996 to July 2004 inclusive) were conducted. During the search development, a “test” subset of relevant studies was used to assess whether thesaurus or free-text terms could be included in isolation. A combination of both was required to identify all the “test” studies. As discussed by Shaw et al,³² the cost of designing a comprehensive search strategy is the lack of precision. Study design filters may help improve precision, however none were incorporated in this work because of concerns about poor indexing in electronic databases—particularly for qualitative studies.^{9,18,20,32–34} Instead, a core set of subject specific terms (cervical smears, colposcopy, screening and neoplasms) was used in combination with thesaurus and general free-text terms such as “leaflet*”, “knowledge” or “understand*” to identify relevant studies.

Additional references were taken from the table of contents of selected journals, reference sections of relevant studies, and the NHSCSP research literature online database (http://www.nhs.thescienceregistry.com/main.asp). Grey literature was sought from internet resources and contact with subject area specialists. In total, 1063 citations were identified (both published and unpublished, with no language restrictions).

Initial citation and full study report assessments

Titles and abstracts of citations were independently prescreened by two reviewers according to review study selection criteria (box 5). Others have noted that when present, the abstracts of qualitative studies vary in content (ie, study design details may not be reported).^32,34 The inclusion or exclusion criteria were liberally applied and if doubt existed, individual studies were provisionally included for consideration on the basis of full text reports. Only about 3% of the screened citations lacked an abstract.

A total of 233 studies were retrieved for further evaluation. Two reviewers independently assessed the full study reports; any uncertainty was resolved by discussion. Most of the the exclusions fell into three categories: studies reporting interventions aimed at increasing screening uptake (35), non-UK based studies investigating non-generalisable cervical screening issues (41), and studies focused on aspects of cervical screening other than written information materials (51). The remaining 27 excluded studies belonged to several different categories listed in box 5. An important consideration for prospective reviewers is the time required to complete the search strategy development as well as the initial citation and full study report assessments. In this review, around 4 months of full-time work was dedicated to these preliminary steps.

Data extraction

Data extraction was conducted for 79 studies (52 quantitative and 27 qualitative). A data extraction form was used to record full study details and guide decisions about the relevance of individual studies to the review questions. The form was developed using guidelines produced by the UK National Health Service Centre for Reviews and Dissemination (CRD)³ and other publications.^35–37 Similar information was extracted for all studies and included: aims, setting, population, research design, methods, interventions (if appropriate), results and conclusions. Data were extracted from relevant studies by one reviewer and checked by a second reviewer.

Identification of key information points

A core set of key information points for the screening programme letters and leaflets was developed from the 1997 guidelines¹ and the extracted data. All the information points from the original guidelines as well as new points identified from the findings of the data extracted studies were included in the core set. In total, 32 studies reported relevant results (the range of study designs is described in box 6). Although a large number of studies underwent data extraction and ultimately many did not report relevant findings, it was essential to consider a variety of potentially applicable studies to obtain a comprehensive set of key information points. The excluded studies fell into the following categories: studies reporting interventions aimed at increasing screening uptake (n = 3), non-UK based studies investigating non-generalisable cervical screening issues (n = 3), studies focused on aspects of cervical screening other than written information materials (n = 14) and studies containing information-related findings not directly relevant to the agreed key information points (n = 20). A further seven studies (five quantitative and two qualitative) contained insufficient information about study design, methods of analysis and findings for a full assessment.

Study methodology—quality scoring

Study quality appraisal is undertaken in quantitative systematic reviews to assess bias—for example, appropriate randomisation for trials—and to identify other study design specific defects.^11,38 Many quality appraisal checklists are available for different types of quantitative evidence however, for study designs other than randomised trials, the key elements of quality are not as well agreed.³⁸ The issue of how or whether to assess qualitative studies is a matter of some debate.^{13,14,33,39,40} The UK NHS CRD guidance favours structured appraisal of qualitative research but recognises that consensus is lacking on appropriate criteria.⁷

Although the exact function of quality appraisal in reviews of quantitative and qualitative evidence is controversial, it is recognised that reviewers should highlight evidence quality issues—in conduct, reporting, or both for review users.¹³ As such, the quality of each individual study included in the review was assessed by two reviewers using established checklists. Separate checklists were used for different quantitative study designs whereas a single checklist was developed for all qualitative studies (box 6).^7,41–44 All checklists incorporated a coded comments system that allowed reviewers to record an assessment of each checklist component. After consideration of all items on a given checklist, the methodological quality of each study was rated as: ++ (all or most of the criteria have been fulfilled), + (some of the criteria have been fulfilled) or – (few or no criteria have been fulfilled).

The quality appraisal provided an indication of the strengths and weaknesses of each study. Although study-design flaws (eg, no intention-to-treat analysis conducted for a randomised trial) could prompt the exclusion of quantitative evidence, in practice, none was excluded at this stage. No exclusion rules were applied to the qualitative studies as guidance is lacking on how to exclude “weak” qualitative findings. The quality scores assigned to individual studies were taken into account during the research synthesis.

Research synthesis

The first step in the synthesis process was to construct a tabular summary of all studies related to each key piece of information identified as important for inclusion in the NHSCSP letters and leaflets. An example, the evidence profile created for the colposcopy leaflet information point “Indicate that treatment can occur at the first colposcopy clinic visit”, is presented in table 1 (a summary of the study findings is also included).

Many different systems exist for grading quantitative evidence and recommendation strength,⁴⁵ however, there is no agreed hierarchy of evidence in qualitative research¹⁸ or across all research methods.¹³ Recently, a new system of grading quantitative evidence has been proposed by an international group of experts in the field of systematic reviews (the GRADE working group).⁴⁶ For the purposes of this review, a modified GRADE approach was adopted for the quantitative evidence synthesis and a similar but separate system was developed for the qualitative research.

Synthesis of quantitative evidence

The basic principles of the GRADE approach were applied to the synthesis of the quantitative evidence. The quantitative studies were tabulated for each important information point and analysed together in an evidence profile (table 1). An initial level of evidence (high, low or very low) was assigned to the group of quantitative studies reporting findings related to each information point. The initial level of evidence was then increased or decreased after consideration of the following factors: study design, quality, consistency and directness (box 7).^46,49 Data on the size or magnitude of effects or associations were taken into account whenever possible—for example, if a good quality randomised trial of written information provision showed a beneficial effect on outcomes such as knowledge, understanding, acceptability or anxiety, then the kinds of information included in the trial materials were recommended. Data related to population baseline risks and resource utilisation were not available and were omitted from the review evidence profiles.

Synthesis of qualitative evidence

Qualitative studies addressing one particular information point were collated and assigned an initial level of evidence (high ++, low + or very low −) on the basis of the lowest checklist quality score obtained for any study in the group (table 1, box 7). The initial level of evidence was then modified according to the consistency and directness of the evidence across the group of studies (these additional considerations acted cumulatively). As in the GRADE system, four overall qualitative levels of evidence (high, moderate, low and very low) were used during this phase of the evaluation process. Consistency referred to similarities in developed themes and participant experiences across studies whereas directness referred to the similarity between people, interventions and findings compared with the NHSCSP population. It is important to note that individual studies were not directly penalised for reporting contradictory or less direct evidence. Judgements about consistency and directness were made on the basis of the evidence provided by all of the studies, which in turn influenced the guideline recommendation for that specific information point. The existence of contradictory findings across studies was not taken to be evidence of poor quality research instead; uncertainty was reflected in the strength of the given recommendation.

For example, the qualitative studies in table 1 all indicated that women have unanswered questions about the possibility of treatment at the initial colposcopy appointment and that the studies were conducted in directly applicable populations. Therefore the initial level of evidence “high” remained unchanged. If, however, one of the three qualitative studies had received a checklist quality score of “+” instead of “++” then the initial level of evidence assigned to the group of studies would have been “low” instead of “high”. In this situation, the initial level of evidence would have been promoted from “low” to “moderate” owing to the direct and consistent findings reported by the two remaining “++” rated studies. In theory, one study could have found that women prefer to receive treatment information in-person at the colposcopy clinic. At this point, a further decision regarding the importance of the inconsistency between studies would have been necessary. This type of grading system provides increased flexibility for reviewers when making decisions about evidence.

Recommendation system and review findings

Both overall levels of evidence (quantitative and qualitative) were used to determine which information points should be included in the screening letters and leaflets. Two categories of recommendation “definite” and “suggestive” were developed (fig 1). Items for which the quantitative and/or qualitative evidence was graded as “high” and/or “moderate” were given a “definite” recommendation for inclusion in the screening information. Items where quantitative and qualitative evidence were graded as “low” and/or “very low” were designated as “suggestive”. In this way, a final recommendation for each information point was included in the review without obscuring the contribution of each type of evidence or complicating the grading system.

Although different levels of evidence were often assigned to the quantitative and qualitative research, contradictory findings were not reported for any of the information points considered. If the qualitative research had been omitted from the review, many recommendations would not have been incorporated or achieved a “definite” status. This applied to the recommendations presented in fig 1, table 1 and the following:

• The use of statements intended to reassure such as “not to worry” or “no big deal” should be avoided.

• The term “wart virus” should be not be included.

• The staff members participating in colposcopy and treatment appointments should be identified.

DISCUSSION

The development of methods for the synthesis of diverse data sources in a single review is a complex challenge.^14,15,34 It has been said that, “ultimately a subjective and pragmatic judgement must made about methodological issues”.²³ Despite the current lack of guidance, we decided that it was still a valid exercise to attempt to pull together updated evidence-based guidelines for the production of NHSCSP information using both quantitative and qualitative studies. The purpose of this article is to share our experiences with other researchers and contribute to the ongoing discussion about the best way forward.

The GRADE methodology has been developed⁴⁶ and refined by its application to existing systematic reviews composed mainly of randomised trials.⁴⁹ The reliability and sensibility of the approach is currently being assessed.⁴⁹ The attraction of the GRADE system was the ability to modify the level of evidence assigned to a group of studies through consideration of factors other than study design alone. However, even with the improved flexibility of the GRADE system, study design considerations affected the level of evidence assigned to the quantitative research (it was often rated as “low” or “very low”). On the other hand, the qualitative research often obtained evidence level ratings of “high” or “moderate”. This may partly be explained by the checklist quality scores assigned to the qualitative studies and the increase in evidence level allowed by the close conformity of findings between two or more studies. A sensitivity analysis investigating the effect of the grading system on the results of the review was not attempted–further work is required on this subject.

Study quality appraisal is an important unresolved issue in the synthesis of quantitative and qualitative research.¹³ Many different quality criteria for the assessment of qualitative studies have been proposed but no common standards have been agreed.^13,50 Controversy also exists about the defining characteristics of good quality qualitative research and the prescription of structured quality criteria.^9,13,33,50 In contrast, quality checklists and formal hierarchies of evidence are included in procedural manuals for quantitative reviews of effectiveness studies.^7,8

A pragmatic decision was taken to use a quality checklist modified from published sources^41,44 to assess the various qualitative studies included in the review. The checklist quality score was used primarily as a means for highlighting the strengths and weaknesses of each study.⁹ Although checklists are convenient tools, it was often difficult to determine whether the quality of reporting or the design and execution of the study was being assessed. For some checklist criteria, the information required to make a decision about whether an item was “well covered”, “adequately addressed” or “poorly addressed” was not always available. Also, separate reviewers applied and interpreted checklist criteria in different ways (even with a coded comments system), although agreement was always reached by consensus. The quality of each study was often better captured by the detailed study assessment notes compiled during the appraisal process.

The review questions were best answered by evidence from a range of data sources. The inclusion of the qualitative research, which was often highly relevant and specific to many issues covered by the screening information, enabled the production of a set of guidelines that will directly affect policy in the NHSCSP. The updated guidelines for the content of the NHSCSP leaflets and letters appear to be sensible standards that confirm and strengthen the previous recommendations.^1,28

The systematic review and synthesis of diverse research evidence, though rewarding, is challenging and time-consuming. Several aspects of the review process that involve considerable time investment on the part of researchers include searching, study appraisal and analysis. As current methods are further developed the process should become increasingly streamlined, transparent and explicit.^13,26,46 In presenting this research we hope to provide a practical example of how numerous data sources might be brought together to inform guidelines related to the contents of patient information materials.