Clinical Consensus Statement Development Manual

Abstract

Clinical consensus statements reflect opinions drafted by content experts for which consensus is sought using explicit methodology to identify areas of agreement and disagreement. In contrast to clinical practice guidelines, which are based primarily on high-level evidence, clinical consensus statements are more applicable to situations where evidence is limited or lacking, yet there are still opportunities to reduce uncertainty and improve quality of care. This manual describes methodology used by the American Academy of Otolaryngology—Head and Neck Surgery Foundation to promote rapid and consistent development of clinical consensus statements for its membership. Although the manual contains specific practices relevant to the Academy, the principles are also explained and may help other organizations create similar products.

The American Academy of Otolaryngology—Head and Neck Surgery Foundation (AAO-HNSF) defines clinical consensus statements (CCSs) as statements based on expert opinion and the best available research evidence for which consensus is sought using an explicit a priori methodology to identify areas of agreement and disagreement. The resulting statements for which consensus is achieved identify opportunities to improve patient care and clinical outcomes. A CCS is most applicable to situations where the evidence base is insufficient for a clinical practice guideline (CPG)¹ but for which significant practice variations and quality improvement opportunities exist. In contrast to CPGs, which provide explicit recommendations through action statements and evidence profiles, a CCS (Table 1) provides the views of experts whose impact on care requires interpretation and value judgments by clinicians and policy makers.

The core of a CCS is a series of statements for which consensus is sought via a modified Delphi method.² The Delphi method is a systematic, iterative approach to identifying consensus without face-to-face interaction. This manual details the current approach taken by the AAO-HNSF to develop a CCS, recognizing that the approach described herein is one of many that can be used to achieve this goal. Although the manual contains specific practices relevant to the AAO-HNSF, the principles underlying consensus statement development are explained and may help other organizations create similar products.

Table 1. Comparison of Key Characteristics of Consensus Statements vs Guidelines.

Characteristic	Clinical Consensus Statement	Clinical Practice Guideline
Primary output	Statements of fact based on best evidence and expert consensus	Recommendations for action based on best evidence and explicit consideration of benefits, harms, values, and preferences
Level of evidence	Observational studies and expert consensus; higher levels of evidence when available	Systematic reviews and randomized controlled trials; lower-level evidence as need for research gaps
Size of development group	8 to 10; possibly more	15 to 20
Composition of development group	Otolaryngologists; content experts a majority; may include other disciplines as needed	Multidisciplinary, including consumers; content experts a minority; includes all stakeholders in the target audience
Perspective of development group member	Member serves as a content expert based on individual knowledge and experience	Member advocates for the discipline or constituency they were appointed to represent
Time frame	6 to 8 mo	12 to 18 mo
Meeting venues	Conference calls and electronic mail	In-person meetings, conference calls, and electronic mail
External review	Limited review by relevant stakeholders	Extensive review by all stakeholders, including open public comment

Table 2 shows major steps in the development process to produce a CCS. The remainder of the manual provides a more detailed overview of each step. Typically, CCSs are developed within 6 to 8 months over a series of conference calls with a group of content experts, defined as clinicians with special skills or knowledge derived from training or experience in the subject matter of the CCS. To avoid conflicts with work schedules and differences in time zones, conference calls are scheduled in the evening from 8 to 10 pm eastern standard time. Development group members are required to attend all conference calls, as participation in these discussions is imperative to the development and success of the CCS.

Planning

Define Topic and Timeline

Consensus statements should be developed for specific topic areas with significant opportunities for quality improvement despite an insufficient evidence base to support a CPG. For the AAO-HNSF, potential clinical topics for CPGs and CCSs are submitted by AAO-HNSF members or by the otolaryngology specialty societies to the AAO-HNSF Guidelines Task Force. Clinical topics are presented at the fall in-person meeting and assessed by members of the Guidelines Task Force for importance within the otolaryngology specialty, for feasibility of production into a CPG or CCS based on the current evidence base, and for priority of the topic within the field of otolaryngology. Topics selected for development as a CCS are deemed clinically important due to high incidence or prevalence of the condition/procedure, to uncertainty in management and/or significant variations in care, and to risk of procedure or potential complications that warrant additional guidance to reduce risks and complications and to improve patient outcomes.

Table 2. Sample Clinical Consensus Statement Development Timeline.

Month	Activity	Task
0-1	Planning	Identify chair, assistant chair, and methodologist to serve on clinical consensus statement.
		Hold planning call with the chair, assistant chair, methodologists, topic submitters, and staff liaison to identify other stakeholders/organizations that should be represented on the development group.
		Identify/recruit development group members.
		Define PICO(TS) questions for state 1 literature search.
1-2	Stage 1 literature search	Conduct stage 1 literature search for guidelines and systematic reviews; include narrative reviews as needed.
2	Conference call 1	Define scope of the consensus statement.
		Develop and rank topic list questions and corresponding draft consensus statements.
2-3	Stage 2 literature search	Conduct stage 2 literature search for randomized controlled trials and observational studies.
3	Conference call 2	Review ranked questions and statements.
4	Delphi survey 1	Develop and disseminate first Delphi survey.
4	Conference call 3	Discuss results of first Delphi survey and inclusion of topics in second Delphi survey.
5	Delphi survey 2^a	Develop and disseminate second Delphi survey.
5	Conference call 4	Discuss results of second Delphi survey.
		Allocate writing assignments.
6-7	Manuscript development	Writing assignments for each topic area are submitted for consolidation, review, and editing by chair and staff liaison.
7-8	Journal peer review	Submit manuscript for publication.

• Abbreviation: PICO(TS), patient, intervention, comparison, outcome, time frame, setting.
• a Additional iterations of the Delphi survey may be created and distributed to development group members until consensus is attained.

Identify Leadership

A chair is assigned to each CCS to lead the development group through the consensus process. The chair should have knowledge of the topic, experience serving on a CCS panel, and no relevant conflicts of interest. Primary responsibilities of the chair include the following:

• Ensuring that all development group members are comfortable with the scope and content

• Leading and facilitating discussions during each conference call

• Encouraging constructive debate and disagreement when reviewing survey content and results

• Troubleshooting any issues related to participation or adherence to the established CCS methodology

• Expressing their opinion and contributing to the development of the CCS

• Allocating writing assignments to members of the development group

• Writing the Introduction and Conclusion of the CCS manuscript

• Assisting the staff liaison with editing the manuscript

• Assisting the staff liaison in submitting the manuscript to a peer-reviewed journal

• Disclosing any conflicts of interest to the group on each conference call

• Presenting the topic at the closest AAO-HNSF Annual Meeting & OTO EXPO so that the findings can be shared with the membership

• Reviewing literature search results and working with strategists to fine-tune search strategy

An assistant chair is identified who will aid in the responsibilities of the chair and who will be asked to chair a subsequent CCS. An ideal assistant chair should have expertise in the content area and experience with evidence-based medicine but does not need to have prior CCS experience. Specific responsibilities of the cochair include the following:

• Assisting the chair on all conference calls to ensure active participation of all group members and to prevent groupthink and other biases, encouraging constructive debate without forcing agreement

• Assisting with writing a portion of the manuscript

• Reviewing literature search results and working with strategists to fine-tune search strategy

• Disclosing any conflicts of interest to the group on each conference call

• Serving as chair on a subsequent CCS

A methodologist is assigned to the development group. This individual should have experience creating CCSs, including the Delphi method, and should not have any relevant conflicts of interest. The methodologist is a nonvoting member of the CCS development group and does not have to be a content expert. Primary responsibilities of the methodologist include the following:

• Ensuring that the CCS development group adheres to established modified Delphi methodology

• Keeping the development group focused on the scope and purpose of the CCS to avoid drift

• Assisting the chair on all conference calls to ensure active participation of all group members and to prevent groupthink and other biases

• Reviewing and editing survey questions

• Limiting the results to statements of consensus while avoiding language that includes recommendations for action

• Assisting with writing and editing the manuscript

• Approving the final version of the manuscript for journal submission

• Disclosing any conflicts of interest to the group on each conference call

The remaining development group members should understand evidence-based medicine, but they do not need to have prior experience serving on a CCS. They should be comfortable applying their expert opinion to all topic content to avoid nonqualified responses that may skew survey results. If additional stakeholders who are not necessarily content experts express a desire to contribute to the CCS, they can participate as a nonvoting member based on the discretion of the chair and methodologist. Since the scope and content of the CCS will be determined after the group is formed, there will undoubtedly be some statements for which specific members have greater familiarity. An important role is to share expertise and educate peers so that all participants adequately understand the statements they will be rating.

The AAO-HNSF has generally used a small development group with 8 to 10 members plus the chair, but most developers of consensus statements use between 11 and 25 participants in the final voting round.³ Regardless of the group’s size, CCS development group members are responsible for the following:

• Actively participating on all conference calls

• Completing all Delphi surveys and writing assignments in a timely manner

• Responding to emails and other forms of communication efficiently

• Reviewing the current literature disseminated via the staff liaison as related to the topic of interest

• Maintaining confidentiality

• Disclosing any conflicts of interest to the group on each conference call

A staff liaison is assigned to each CCS; his or her responsibilities include the following:

• Recruiting content experts from medical specialty societies, AAO-HNS/F committees, and pertinent organizations

• Assisting, in conjunction with an information specialist, with at least 2 literature searches, organizing the results, and acquiring the full-text versions of all relevant journal articles

• Managing and organizing all manuscript references

• Organizing all conference calls, including scheduling, creating agendas, and sending email reminders to the development group

• Serving as a scribe for conference calls, drafting and distributing call minutes within 7 days, and saving the minutes in a designated electronic folder for future reference

• Assisting the chair with survey development and analyses

• Assisting the development group, specifically the chair, with writing and editing the manuscript

Planning Call

An initial planning call will typically include the individual who originally submitted the topic to the Guidelines Task Force for consideration, the CCS chair, CCS methodologist, and a staff liaison to discuss the proposed topic and all potential stakeholders to be included on the CCS development group. Approximately 1 week before the planning call, the staff liaison should try to condense all conference call files into 1 PDF document for easy dissemination and distribute the following:

1. Call agenda

   Template planning call agenda items:

   • Development group leadership introductions

   • Overview of background, purpose, and scope of CCS

   • Overview of development group member roles and responsibilities

   • Overview of literature searches (CPGs, systematic reviews [SRs], randomized controlled trials [RCTs]) to include the following:

     • Scope

     • Population

     • Target audience (proposed stakeholders)

     • Care settings

     • Search criteria—inclusion and exclusion, year range, publication details (eg, human studies, English only), methodology (SRs, CPGs, RCTs, observational studies, etc)

     • Sources to be searched (pertinent databases) and search filters

     • Data fields to be gathered—author, title, year of publication, journal details (if any), URL, summary/abstract

     • Search results presented in an Excel spreadsheet

   • Identifying development group members

   • Committee members

   • External organizations

2. Development of group roster

3. Proposed CCS timeline

4. Financial disclosure form (if applicable)

Additionally, a proposed timeline, including potential dates for each conference call, should be developed. Following the call, the staff liaison should identify the dates of all 4 conference calls via 4 individual Doodle polls, ensuring that all leaders are available. By identifying the conference call dates in advance of recruiting development group members, the AAO-HNSF is able to make call participation mandatory and accept only development group members who are available for the selected dates.

Recruiting the Development Group

The AAO-HNSF maintains a current contact list of guideline and quality improvement staff at all of the medical societies that the organization has worked with over the years. This ensures that staff are able to make contact with the right person during the first attempt to recruit development group members. The medical society president/CEO, administrator, and/or its AAO-HNS/F Specialty Society Advisory Council representative are copied on this request. By copying the leaders of the organizations, they can authorize and add support to the importance of the organizations’ participation in the development of the CCS.

Members of the CCS development group are representing their fields of expertise, emphasizing their personal insights, knowledge, and clinical experience; they are not official representatives of the society or organization from which they are recruited. An important prerequisite to being a development group member is the availability to participate on all predetermined conference calls. Anyone who cannot commit to participating in all scheduled calls is excused from participation, and an alternate is sought. An example of CCS panel composition is shown in Table 3.

Stage 1 Literature Search: Reviews and Guidelines

The purpose of the stage 1 literature search is to identify evidence that will assist the group in deciding the scope of the CCS and in formulating a list of topic questions that will then serve as a basis for the first Delphi survey. SRs and guidelines may not exist on the specific topic of the CCS, but those published on a broader topic encompassing the chosen CCS topic may assist in identifying specific areas of ambiguity or variations in practice. For example, a review or guideline on the broad topic of “sinusitis” may nonetheless have useful information on the narrower topic of “pediatric chronic sinusitis.”

Table 3. Example of Development Group Composition.⁴

No.	Position	Expertise
1	Chair	Otolaryngologist
2	Assistant chair	Pediatric otolaryngologist
3	Methodologist	Otolaryngologist
4	Staff liaison	Clinical research
5	Content expert	Otolaryngic allergist
6	Content expert	Pediatric otolaryngologist
7	Content expert	Pediatric otolaryngologist
8	Content expert	Pediatric otolaryngologist
9	Content expert	Rhinologist
10	Content expert	Rhinologist

The staff liaison should work with the CCS chair and methodologist to develop the clinical question(s) for the stage 1 literature search in a PICO(TS) format. PICO(TS) is a technique used to develop a clinical research question, and it will help the search strategist in formulating and conducting the literature search. The PICO(TS) developed for the pediatric chronic rhinosinusitis CCS is provided below as an example:

• Population: children (6 months to 18 years)

• Intervention: medical management, surgical management, nasal endoscopy

• Comparator (optional): not applicable

• Outcome(s): subjective and objective

• Time frame (optional): chronic

• Setting (optional): any outpatient setting

Once the PICO(TS) criteria and any other descriptive search parameters have been established,¹ the staff liaison should relay this information to the search strategist, who decides feasibility and the need, if any, for additional refinement or revision. The search strategist is also informed that the group at this stage is interested in SRs and CPGs but that if no relevant evidence is found for these publication types, the search should be extended to include narrative reviews. After implementation, the literature search returned from the search strategist should include the author(s), title, year of publication, journal details, URL, and abstract duplicates should be removed.

Prior to the first conference call, the staff liaison should work with the leadership (chair, assistant chair, and methodologist) to identify all relevant reviews and guidelines from the initial search. Titles and abstracts are reviewed to determine relevancy. All non-English-language papers or those with no available abstract or full-text documents are excluded. A list of all relevant literature should be recorded in an Excel spreadsheet, with all article information included. To assess the quality of the literature, the staff liaison will provide the chair and assistant chair with an Excel spreadsheet, which separates the SRs and CPGs by tabs. The chair and assistant chair should include or exclude studies that do or do not match the search criteria outlined in the planning call.

SRs published by the Cochrane Collaboration or government agencies (Agency for Healthcare Research and Quality) are typically of high methodologic quality and may not require further assessment. Conversely, reviews authored by individuals or other organizations are highly variable in rigor and quality. Minimum-quality criteria for SRs should include the following: (1) clear objective and methods defined explicitly by the reviewers; (2) an explicit search strategy described with full details (dates, databases, search terms, language restrictions, and criteria for including articles); and (c) valid data extraction, usually performed by at least 2 independent investigators, to abstract data and descriptive information from the source articles to minimize bias.

CPGs are highly variable in quality regardless of origin. Minimum quality criteria should include (1) explicit scope and purpose, (2) multidisciplinary stakeholder involvement, (3) systematic literature review, (4) explicit system for ranking evidence, and (5) explicit system for linking evidence to recommendations.

Literature search studies should be excluded if they are duplicate, non-English language, or nonhuman; do not pertain to the CCS topic; or do not meet basic quality inclusion criteria.

The staff liaison should work with the chair to provide a summary of the CPG and SR literature search data to present at conference call 1, as well as acquire the full-text versions of all articles for distribution to the development group. The development group members should have adequate time to review and familiarize themselves with the literature before the first conference call. The AAO-HNSF uses Google Drive as the platform for sharing documentation with its CCS development groups. This platform is available for free and is user friendly.

Prior to the first conference call, the staff liaison should complete the following:

• Compile contact information grid—this should contain the name and title of each development group member, his or her contact information, and conflicts of interest/disclosures.

• Organize the articles in a spreadsheet and identify which are most relevant—this is done with assistance from the CCS leadership once the initial literature search is received.

• Develop a literature summary of the CPG and SR literature search data to present at conference call 1—this may include the literature citation and a brief overview of the study conclusion.

Conference Call 1: Defining the Scope

Purpose of Call 1

The purpose of the first CCS conference call is to (1) disclose and discuss all potential conflicts of interests; (2) review methodology; (3) define scope, target population, and target audience for the CCS; (4) discuss results of the stage 1 literature search; (5) discuss the content for the list of topic questions and (6) review the CCS timeline. The staff liaison will reiterate the roles and responsibilities of development group members and stress the importance of actively participating on each conference call as well as completing each survey in a timeline manner.

Predistribute Electronic Materials

Approximately 1 week before the first conference call, the staff liaison should try to condense all conference call files into 1 PDF document for easy dissemination and distribute the following:

1. An agenda

   Template call 1 agenda items:

   • Introductions, disclosure of any potential conflicts, and experience with consensus statements (staff liaison)

   • Review of roles and responsibilities of development group members (staff liaison/chair)

   • Overview of CCS purpose (chair)

   • Brief overview of Delphi methodology and timeline (methodologist)

   • Discussion (chair and methodologist):

     • Topic definitions

     • Using PICO(TS) to narrow our focus—population, intervention (diagnosis/prognosis), comparator, outcomes, timing, setting

     • Audience

     • Topic list items

   • Assignments and deadlines (staff liaison):

     • RCT literature search

     • Submission of sample statements

     • Next call date and time

2. Contact information grid with disclosure of all potential conflicts of interest

3. Spreadsheet listing all relevant SRs and guidelines (literature search 1 results)

4. Relevant sections of the published CCS manual

5. A published CCS for reference⁴

Conduct of Call 1

It is important to keep in mind that the scope of the CCS should NOT be broad but rather focus on areas in which there is a clear paucity of evidence, opportunities for quality improvement, and variability in care or outcomes. The group should have an explicit discussion to define the scope of the CCS in terms of the following:

• Target condition or procedure

• Target patient or clinical presentation, including exclusion criteria

• Intended audience and practice settings

• Interventions to consider and to exclude

• Outcomes to consider

The chair should review the purpose of the list of topic questions—namely, to identify specific areas within the chosen topic on which to focus, as a basis for the clinical statements that the group will assess for consensus in the iterative Delphi survey. Topic questions should be based on perceived opportunities for quality improvement, broadly summarized as the following⁵:

1. Promoting appropriate care

2. Reducing inappropriate or harmful care

3. Reducing regional variations in delivery of care

4. Improving access to care

5. Educating and empowering clinicians and patients

6. Facilitating coordination and continuity of care

7. Facilitating ethical care

After reviewing the above list with the group, the chair solicits feedback for topic questions. These may be broad (eg, “What is the role of treatment X in managing disease Y?”) or narrow (eg, “Is disease Y better defined as 3 months or 4 months of duration?”). The goal is to develop a preliminary list of questions that will be expanded on after the call, by asking development group members to submit additional suggestions for topic questions. To allow for easier topic classification, the staff liaison should allocate the topic list questions into categories (eg, diagnosis, medical treatment, surgical treatment, outcomes).

Developing the List of Topic Questions

The chair and staff liaison work together to develop a list of topic questions designed to further refine evidence gaps discussed during the first conference call. Each development group member is asked to submit at least 5 questions to the staff liaison based on what she or he perceives are key opportunities to (1) address controversial clinical issues, (2) reduce variability in care, (3) clarify evidence gaps, or (4) improve quality of care through structured expert consensus. The staff liaison collates and organizes the submitted questions (ie, diagnosis, medical therapy) and provides them to the chair for review and revision. The list of topic questions should consist of at least 20 questions.

The final topic question list, based on electronic exchange after the first conference call, is made into a 2-column table. The first column, left blank, has the heading “Rank,” and the second column, containing the topic questions, has the heading “Topic Question.” The staff distributes the list to working group members with the following instructions, replacing the number “30” in this example with the total number of topic questions:

Please rank the 30 topics below in order of importance for inclusion in this clinical consensus summary by placing a number from 1 to 30 under the “Rank” column. Please use each number only once. Assign the number “1” to the most important topic, “2” to the second-most important topic, and number “30” to the least important (in your opinion). The table should not be sorted but should be left in original order. Thank you for your time.

In addition to ranking the topic list questions, each development group member is asked to provide 1 draft consensus statement for each of his or her top 5 ranked topic questions (Table 4). The purpose of this is to assemble a list of draft consensus statements that the chair will use to create the list of consensus statements for the first Delphi survey. The group is reminded that the draft consensus statements are written as expert views, not recommendations for action (Table 5).

The rank tables are collated by the staff liaison to determine the mean rank score for each topic, with lower scores indicating higher priority. When the staff liaison sorts the ranked topic list in order of priority (highest to lowest), the related statements submitted by the development group members under each topic will also be included. This process will facilitate discussion, as well as a more streamlined review of the material, since it will be easier for the chair to proceed down the ranked topic list and readily see related statements when they lead the second conference call.

Table 4. Sample Topic Questions and Associated Draft Consensus Statements.

Topic Question	Draft Consensus Statement^a
Should photodocumentation be required before nasal septoplasty?	Photodocumentation is unnecessary to document septal deviation prior to septoplasty.
What is the role of computed tomography (CT) scanning prior to septoplasty?	A CT scan may not accurately demonstrate the degree of septal deviation and should not be the primary determinant for septoplasty candidacy.
How long should patients try medical therapy prior to septoplasty before concluding that it is ineffective?	A trial of medical therapy of more than 4 weeks duration is unnecessary to assess surgical candidacy for septoplasty.
When should perioperative antibiotics be given to patients undergoing septoplasty?	There is no benefit to routine perioperative antibiotics during septoplasty unless nasal packing or a splint is placed during the procedure.
Is there a role for septoplasty in managing patients who require continuous positive airway pressure (CPAP) for obstructive sleep apnea?	Septoplasty can improve CPAP tolerance for patients with sleep apnea and a deviated septum.
When should septoplasty be performed in conjunction with functional endoscopic sinus surgery?	Septoplasty can improve outcomes of sinus surgery when the septum is contacting the middle turbinate and obstructing the drainage of the ostiomeatal complex.

• a From the American Academy of Otolaryngology—Head and Neck Surgery,“Clinical Consensus Statement: Septoplasty.”⁷

Table 5. Examples of Statements of Fact vs Statements of Action.

Topic	Statement of Fact in a Clinical Consensus Statement	Statement of Action in a Clinical Practice Guideline
Diagnosis	Condition X is defined as…	Clinicians should diagnose condition X when…
Diagnosis	Procedure/test Y is appropriate (or provides important information) in diagnosing patients with…	Clinicians should perform procedure/test Y in diagnosing patient with…
Therapy	Intervention A is beneficial…	Clinicians should prescribe/recommend/offer intervention A…
Therapy	Drug Z may produce a superior clinical response when given for 20 days, instead of 10 days…	Clinicians should prescribe/administer drug Z for 20 days, instead of 10 days…
Therapy	Treatment/procedure Q is not beneficial for…	Clinicians should not perform treatment/procedure Q…
Therapy	The effectiveness/role of procedure Z cannot be determined based on current evidence…	No corresponding recommendation (can potentially be a “no recommendation”)
Prognosis	Patients with factors A, B, and C may have poorer outcomes…	Clinicians should identify factors A, B, and C, and do the following…
Prevention	Doing A, B, and C (or not doing X, Y, and Z) may (or can) prevent…	Clinicians should counsel/advise patients to do A, B, and C (or not do X, Y, and Z) to prevent…

Stage 2 Literature Search: RCTs and Observational Studies

The purpose of the stage 2 literature search is to identify RCTs and observational studies based on the scope agreed on in the first conference call, which should yield more narrow and refined PICO(TS) criteria than those used for the stage 1 literature search. The chair and methodologist should decide on the time frame for the search (eg, past 10 years) and any restrictions on study design or methodology.

Prior to the second conference call, the staff liaison should work with the leadership (chair, assistant chair, and methodologist) to identify all relevant publications from the second literature search and provide a summary of the RCT literature search data to present at conference call 2.

Titles and abstracts are reviewed to determine relevancy. All non-English-language papers or those with no available abstracts or full-text documents are excluded. To assess the quality of the literature, the staff liaison will provide the chair and assistant chair with an Excel spreadsheet. The chair and assistant chair should include or exclude studies that do or do not match the search criteria outlined in the planning call. The staff liaison should work with the chair to summarize the final literature search list. The list should be incorporated under a separate tab within the Excel spreadsheet created for the initial literature search and distributed to development group members. If available, electronic full-text versions of each article should also be included.

RCTs are highly variable in methodology and validity. A simple and efficient scale (Jadad scale) can be used to rank quality from 1 (poor) to 5 (excellent) based on (1) method and adequacy of randomization, (2) method and adequacy of masking, and (3) reporting of withdrawals and dropouts. Literature search studies should be excluded if they are duplicate, non-English language, or nonhuman; do not pertain to the CCS topic; or do not meet basic quality inclusion criteria.

The development group should have at least 2 weeks to review and familiarize themselves with the literature before the second conference call. These publications should be supplemented, as needed, with targeted searches to address specific needs identified in writing the consensus statement.

Conference Call 2: List of Topic Questions and First Delphi Survey

Purpose

The purpose of the second conference call is to (1) update disclosed conflicts of interest if needed, (2) review the second literature search results, and (3) use the prioritized list of topic questions to draft consensus statements for inclusion in the first Delphi survey. The chair should reiterate the purpose of developing a CCS, and the consultant should review the modified Delphi method to ensure that the development group has a clear understanding of what to expect on the initial survey and how to complete it.

Predistribute materials

Approximately 1 week before the second conference call, the staff liaison should try to condense all conference call files into 1 PDF document for easy dissemination and distribute the following:

1. An agenda

   Template call 2 agenda items:

   • Welcome and review of any new conflicts of interest (All)

   • Discuss the state of the evidence base (RCTs) and any other pertinent literature (chair/staff liaison)

   • Review and “finalize” statements for the Delphi survey (chair/methodologist)

   • Assignments and deadlines (staff liaison)

     • Delphi survey 1 development / dissemination

     • Next call date and time

2. Contact information grid with all conflicts of interest/disclosures (highlighting any changes since the first call)

3. Excel spreadsheet with 4 workbooks: SRs, guidelines, RCTs, and observational studies

4. Spreadsheet listing the prioritized topic questions with the associated draft consensus statements submitted by the development group

5. Relevant sections of the published CCS manual

6. RCT literature search summary

7. Categorized development group statements

Conduct of Call 2

The chair reviews the prioritized list of topic questions with the development group to see if any changes are required (eg, moving a specific question higher or lower on the list). Next, the first question is discussed along with the associated draft consensus statements previously submitted by the development group. The goal is to agree on 1 or more draft consensus statements related to the topic question, which will later be assessed with the Delphi process for consensus. The draft statements should facilitate this discussion, but the group is not obligated to retain them and may choose to modify the wording or add entirely new statements. The goal at this time is not linguistic perfection but rather to have a list of draft consensus statements that the chair and methodologist can refine after the call for the first Delphi survey.

After discussing the highest priority topics on the list, the chair continues down the list until an adequate number of consensus statements are drafted. The exact number of statements will vary according to the CCS topic and the degree of controversy regarding specific questions. For example, if the topic is narrow and controversy is limited, about 20 statements would suffice, since many will likely reach consensus. If the topic is broad and very controversial, 40 or 50 statements may be necessary to identify a reasonable number for which consensus can be achieved.

The staff liaison should serve as the scribe for the call; drafting and distributing minutes to the development group within 1 week of the call. The minutes should also be saved in a designated electronic folder for future record.

Delphi Survey 1

Development and Dissemination

The initial survey should be created with web-based software such as SurveyMonkey.⁶ A 9-point Likert scale is used to measure agreement, with the following anchors: strongly disagree (1), disagree (3), neutral (5), agree (7), and strongly agree (9). Statements should be clear and concise, avoiding any leading language that could bias responses. Statements should address all previously determined topic areas. No statement is permitted to prescribe specific action for the clinician to take (eg, should do, may do, must do), since this is not a guideline and should be limited to expert views or opinions, not recommendations for action.

Respondents must provide their names on the survey before rating the statements, or the survey may be set up so that each development group member receives an individualized link to the survey to ensure that all responses are received. Responses by individual development group members are kept anonymous, with only the staff liaison having access to individual results. The chair and group members have access only to de-identified aggregate survey responses. The staff liaison may also want to add comment boxes after each statement for respondents to provide an immediate rationale for their responses, which often leads to enhanced postsurvey discussions. Once the survey is finalized, the staff liaison should notify the development group via email with a link to the survey. Members should have 2 weeks to complete the survey, allowing for an additional 1 week for survey analysis prior to the next conference call.

Analysis

There are many approaches to defining consensus using the Delphi process, many of which rely on thresholds for mean response score, the number of outlier responses, or both.³ The AAO-HNSF methodology categorizes statements (Table 6) as “consensus,”“near consensus,” or “no consensus,” defining outliers as any rating by a development group member that is at least 2 Likert points away from the mean score for that item.

A visual response summary of the responses to each question should be prepared using the format in Table 7.

Table 6. Categorization of Consensus Statements.

Category	Mean Score^a		Outliers^b
Consensus	≥7.00	and	≤1
Near consensus	≥6.50	and	≤2
No consensus	<6.50	or	≥3

• a Nine-point Likert scale from disagree strongly (1) to agree strongly (9).
• b Outliers are defined as any rating ≥2 Likert points from the mean (in either direction).

In addition, a summary spreadsheet is created that contains each statement; its median, mean, range, number of outliers, and number of responses; and the conclusion reached about each statement (ie, no/near consensus or consensus), along with any comments from each statement included in a separate column (Table 8).

Table 7. Example of Delphi Survey Statements and Responses.⁴

	Likert Scale									Mean	Outliers
	Strongly Disagree 1	2	Disagree 3	4	Neutral 5	6	Agree 7	8	Strongly Agree 9
Consensus
Statement 1: Pediatric chronic rhinosinusitis is defined as at least 90 continuous days of ≥2 symptoms of purulent rhinorrhea, nasal obstruction, facial pressure/pain, or cough and either endoscopic signs of mucosal edema, purulent drainage, or nasal polyposis and/or computed tomography scan changes showing mucosal changes within the ostiomeatal complex and/or sinuses in a pediatric patient aged 18 years or younger. (Adapted from EPOS 2012)
Results						1	4	2	2	7.56	0
Near consensus
Statement 13: The middle meatus is the preferred location to obtain a nasal culture in pediatric chronic rhinosinusitis.
Results		1^a			2^a		2	3	1	6.56	2
No consensus
Statement 5: Anterior rhinoscopy alone is insufficient to evaluate the nasal cavity in a child with suspected chronic rhinosinusitis.
Results			2^a	2	1	1	2	1^a		5.22	3

• a Indicates an outlier response, defined as any rating ≥2 Likert points from the mean (in either direction).

Conference Call 3: Reviewing Delphi Survey

Purpose of Call 3

The purpose of the third conference call is to update disclosed conflicts of interest, if needed, and review the first Delphi survey results. Statements achieving consensus should be presented to the development group but not discussed during the call. Statements with a mean score <7.00 (near consensus or no consensus) should be discussed to determine whether the statements should be refined and added to the following Delphi survey or omitted completely.

Table 8. Example of Delphi Survey Results.⁴

No.	Statement	Status	Median	Mean	Range	Outliers	n
21	Adenoidectomy is an effective first-line surgical procedure for children up to 6 years of age with CRS.	Consensus	9	8.33	7-9	0	9
22	Adenoidectomy is an effective first-line surgical procedure for children aged 6 to 12 years with CRS.	Consensus	7	7.11	5-9	1	9
24	Adenoidectomy can have a beneficial effect in patients with PCRS that is independent of endoscopic sinus surgery.	Consensus	7	7.33	4-9	1	9
25	Tonsillectomy (without adenoidectomy) is ineffective treatment for PCRS.	Consensus	9	8.56	7-9	0	9
23	Adenoidectomy is an effective first-line surgical procedure for children aged 13 years and older with CRS.	No consensus	3	3.89	1-6	3	9

• Abbreviations: CRS, chronic rhinosinusitis; PCRS, pediatric chronic rhinosinusitis.

Predistribute Materials

Approximately 1 week before the third conference call, the staff liaison should try to condense all conference call files into 1 PDF document for easy dissemination and distribute the following:

1. An agenda

   Template call 3 agenda items:

   • Welcome and review of development group roster any new conflicts of interest (all)

   • Review of Delphi survey results (chair/methodologist)

   • Assignments and deadlines (staff liaison)

     • Delphi survey 2 development/dissemination

     • Next call date and time

   • Spreadsheet listing results of the Delphi survey

Conduct of Call 3

The call proceeds with a sequential discussion of all statements from the Delphi survey, with the majority of time spent on “near consensus” statements (Table 6). Statements that achieve consensus are reviewed for meaning and importance; statements that are near consensus are reviewed to clarify areas of ambiguity and generate revised language for the next Delphi round; and statements that do not reach consensus are generally discarded unless there is strong evidence or opinion to justify modification and inclusion in the next Delphi round.

Statements that achieved consensus (mean score of 7.00 or higher and no more than 1 outlier) are briefly reviewed to ensure that language is clear and unambiguous and truly presents an opportunity for quality improvement. In other words, does communicating to clinicians that a group of experts reached consensus on this statement improve diagnostic accuracy, promote use of a potentially underutilized therapy, discourage use of ineffective or harmful therapy, or facilitate some other improvement in quality of patient care? If the answer is “no,” the statement should be reworded or discarded, since there is no point publishing a consensus statement that simply reaffirms common knowledge without opportunities to improve care.

Statements that are near consensus (mean score ≥6.50 and≤2 outliers) are reviewed in detail for modification and inclusion in the second Delphi survey. The chair invites discussion on the item by (1) asking individuals who “agree” or “strongly agree” with the statement to explain their rational to the group and (b) asking individuals who gave outlier responses to explain their concerns (assuming that they volunteer to be identified). Some members may not recall their exact responses to certain statements, which can be clarified by the staff liaison if those individuals gave their names when initially responding to the survey. The discussion about near-consensus statements should proceed until the group decides whether to retain the statement in the next Delphi round, with potential rewording or clarification, or reject the statement from further consideration. Group members who have advocated for a specific point of view may provide citations to journal articles for distribution.

Statements that do not reach consensus (mean score <6.50 or≥3 outliers) are dismissed from further discussion unless 1 or more group members feel strongly that clarifying or rewording the statement could yield different results in the next Delphi survey. In general, the chair should limit discussion on these items since the likelihood of reaching future consensus is low and time is more productively spent refining the statements that did reach consensus and discussing those that were near consensus.

The call concludes when all statements in the initial Delphi survey have been adequately discussed and decisions about what will appear in the next survey are agreed on. Statements that reached consensus should not appear in the next survey unless significant changes in wording or context were made. The next Delphi survey will consist primarily of near-consensus statements that the group agreed to pursue further and possibly some no-consensus statements for which there was strong agreement to reword and retain. If the group identified any journal articles that were deemed important, they should be obtained by the staff liaison and distributed prior to the next Delphi round. The staff liaison concludes the call by clarifying the intent and timing of the next Delphi survey.

Delphi Survey 2

After the third conference call, the staff liaison should revise the initial Delphi survey using web-based software, such as SurveyMonkey. All statements achieving consensus and those that the development group chose to omit from the second round survey should be removed. Revisions agreed on to particular statements should be made and a final version of the second round survey distributed and approved by the chair.

New statements should not be added to the second iteration of the Delphi survey—only statements requiring revision should appear. As with the first survey, comment boxes should be provided after every statement to allow respondents to provide immediate rationales for their agreement ratings. Once the survey is developed, it is sent to the chair for review, then to the methodologist for final approval. The staff liaison should then notify the development group via email with a link to the survey. Members should have approximately 2 weeks to complete the survey, allowing for an additional 1 week for survey analysis prior to the next conference call. The responses are analyzed and summarized as previously described for the first round Delphi survey.

Conference Call 4: Review Second Delphi Survey and Discuss Manuscript Development

Purpose of Call 4

The purpose of the final conference call is to (1) review the second-round Delphi survey results and determine if a third round is needed, (2) identify framework for the manuscript, and (3) identify writing assignments for the development group. A third-round Delphi survey should be discouraged because it delays development and is unlikely to result in a substantive change in consensus levels. Consensus developers rarely continue beyond a third Delphi round.³

Predistribute Materials

Approximately 1 week before the fourth conference call, the staff liaison should try to condense all conference call files into 1 PDF document for easy dissemination and distribute the following:

1. An agenda

   Template call 4 agenda items:

   • Welcome and review of any new conflicts of interest (all)

   • Review of Delphi survey (chair/methodologist)

   • Assignments and deadlines (chair/staff liaison)

     • Writing assignments (chair/staff liaison)

     • Next call date and time

2. Contact information grid with all conflicts of interest/disclosures (highlighting any changes since the third call)

3. Spreadsheet listing results of the Delphi survey

Conduct of Call 4

Using the same format as for call 3, the survey questions are discussed sequentially. Emphasis is placed on items that have newly achieved consensus, to ensure that they are clear and unambiguous and that they truly present quality improvement opportunities. Statements with no consensus are discarded.

Results from the second round are typically more defined, as either complete consensus or no consensus—with few, if any, statements achieving near consensus. This is due to the lengthy discussions during the third conference call, which should have swayed members of the development group who were “neutral” to either completely agree or disagree with the revised statement. However, if there are several near-consensus statements still requiring discussion, this should occur during the conference call or via a brief email revote. The development group may then decide to distribute a third-round survey or simply add these statements to the discussion portion of the manuscript. A third-round survey, however, is discouraged and should be planned only when there is strong group agreement about its necessity.

The chair should also assign development group members to assist with writing the manuscript. These individuals will be primarily tasked with writing the Results and Discussion sections, highlighting areas of strong agreement or disagreement. Development group members should be assigned as primary or secondary authors for the Results and Discussion sections for each statement or subgroup of statements based on topic area (Table 9). A timeline for manuscript development should be finalized, and the development group should feel comfortable with all final results of the survey.

At the conclusion of call 4, the staff liaison should distribute the following items:

1. Meeting minutes

2. Writing assignment grid with delegated writing

3. Google Drive information where all ordered articles are stored and organized by topic

Table 9. Sample Writing Assignments Grid.⁴

Topic Area	Primary Author	Secondary Author
Abstract	Chair	Assistant chair
Introduction	Assistant chair	Chair
Methods	Chair	Methodologist
Results/discussion
Definition, diagnosis, spectrum of disease	Member A	Chair
Medical therapy	Member B	Member C
Adenoiditis/adenoidectomy	Member D	Member E
FESS/turbinoplasty	Member F	Member G
Tables/figures	Liaison	Chair
References	Liaison	Assistant chair
Final editing and proofing	Methodologist

• Abbreviation: FESS, functional endoscopic sinus surgery.

Manuscript Development

Writing Assignments

The chair and staff liaison should work with the assigned writers to develop the first draft of the CCS. The staff liaison will distribute tables with the statements reaching consensus to each primary author, along with writing instructions (Table 10). The primary authors are responsible for developing the first draft of their assigned section. Once complete, the draft section should be sent to the assigned secondary authors to ensure that the section is clear and concise. Once both authors are satisfied with the draft, it is forwarded to the staff liaison for collating into the main document. The staff liaison should send reminders to development group members of all deadlines to ensure that they stick to the timeline approved by the group.

Collation of Writing Assignments into First Draft

The staff liaison receives the completed writing assignments via email, collates it into a first draft of the full manuscript, and ensures that all appropriate references have been included. The collated first draft should be reviewed by the chair and methodologists to align ideas and concepts provided by the writers. In addition, some editing may be necessary to standardize the writing style. Authors of the original writing assignment will have ample opportunity to comment on any changes made by the chair in style or format, for consistency with the remainder of the consensus statement.

Table 10. Instructions for Composing Supporting Text for Consensus Statements.

1.	Your contribution should be a clear, logical, focused explanation of why the statement is important and how it may lead to improved quality of care. These quality improvement opportunities should also be added to the statement tables.
2.	Length should be modest—about 2 medium-sized paragraphs for each statement is appropriate.^a
3.	If certain statements cannot be supported with much evidence, include any discussions with panel members used to reach consensus.
4.	Literature citations should be listed in the text as follows: first author’s name and publication year (eg, Smith 2008). If >1 citation exists for that year, distinguish with an “a, b, c, etc” after the year (eg, Smith 2008b). At the end of the document, list complete reference citations alphabetically by author last name. The staff liaison will format the references for journal submission using reference citation software (eg, EndNote) once the manuscript has been finalized.

• a Each section should be concentrated, with approximately 1 paragraph for results (5 sentences per paragraph) and 2 or 3 paragraphs for discussion.

The manuscript should follow a traditional structure: abstract, background, methods, results, discussion, a summary of research needs, and conclusion (Table 11). A table listing all statements that did not meet criteria for consensus should be created for inclusion in the manuscript (Table 12). In addition, a table listing all statements reaching consensus broken down by topic area should be included (Table 13). This table should also include a column for the quality improvement opportunity, categorized as 1 or more of the following:

1. Promoting appropriate care

2. Reducing inappropriate or harmful care

3. Reducing regional variations in delivery of care

4. Improving access to care

5. Educating and empowering clinicians and patients

6. Facilitating coordination and continuity of care

7. Facilitating ethical care

Once the draft manuscript is complete, having been edited by the chair and methodologist, the staff liaison should add continuous line numbers to the document and forward the manuscript to the entire development group as a PDF for edits and feedback. Having the line numbers on the draft will help with submission of edits by the group members. The development group should be given at least 2 weeks to review the manuscript and submit proposed changes by line number to facilitate collating, not by adding comments or changes directly onto the manuscript. The staff liaison and chair should work together to incorporate all edits from the development group into the manuscript and distribute a “final” version for approval. The staff liaison should assist the chair with submitting the manuscript for publication.

Table 11. Suggested Outline for Each Section of the Clinical Consensus Summary.

Manuscript Section	Content
Abstract	As per journal instructions.
Introduction/purpose	Define clinical consensus statement and its purpose.
Background	This section should focus on the topic of interest, the rationale for a consensus statement on the topic, and the literature that currently exists within the topic area.
Methods	Briefly describe the methodology behind the literature search (or searches), the selection of experts to serve on the development group, and the development of qualitative and Delphi surveys.
Results	Briefly highlight results of each literature search. Discuss key statements, followed by those that achieved consensus by topic area.
Discussion	The section should focus on an overview of which topic areas achieved consensus, statements that were controversial and were either revised or dropped, and statements that did not achieve consensus.
Research needs	Research needs should focus specifically on those topics and statements that did not achieve consensus
Summary	Summarize the consensus statements and implications for clinicians.

Table 12. Sample Table for Statements That Did Not Meet Criteria for Consensus.⁴

No.	Statement	Subgroup	Status	Mean	Outliers
8	Gastroesophageal reflux disease can contribute to PCRS.	Definition and diagnosis of PCRS	No consensus	6.11	1
16	Current evidence supports a role for topical antibiotic therapy in managing selected children with CRS.	Medical management of PCRS	No consensus	4.67	2
29	Balloon sinuplasty is safe for treating children with PCRS.	Endoscopic sinus surgery / turbinoplasty	Near consensus	6.56	2
30	Balloon sinuplasty is effective for treating patients with PCRS.	Endoscopic sinus surgery / turbinoplasty	No consensus	5.33	0
31	Inferior turbinate reduction can benefit children with CRS by reducing nasal congestion and by improving penetration of topical medications.	Endoscopic sinus surgery / turbinoplasty	No consensus	6.22	1

• Abbreviations: CRS, chronic rhinosinusitis; PCRS, pediatric chronic rhinosinusitis.

Table 13. Sample Table for Statements That Reached Consensus for Medical Management.⁴

No.	Statement	Mean	Outliers	QI Opportunity
9	Twenty consecutive days of antibiotic therapy may produce a superior clinical response in PCRS patients compared with 10 days of antibiotic therapy.	7.44	0	Promoting appropriate care
10	Culture-directed antibiotic therapy may improve outcomes for PCRS patients who have not responded to empiric antibiotic therapy.	8	0	Promoting appropriate care
11	Daily topical nasal steroids are a beneficial adjunctive medical therapy for PCRS.	7.44	0	Promoting appropriate care
12	Daily topical nasal saline irrigations are a beneficial adjunctive medical therapy for PCRS.	7.78	0	Promoting appropriate care
13	Empiric treatment for gastroesophageal reflux disease is not a beneficial adjunctive medical therapy for PCRS.	7	0	Reducing inappropriate or harmful care

• Abbreviations: PCRS, pediatric chronic rhinosinusitis; QI, quality improvement.

Disclaimer

CCSs should be published with an accompanying disclaimer, created by the organization’s legal counsel, to set clear bounds on the intended use of the document. The AAO-HNSF adds this disclaimer text to the end of the manuscript:

Clinical consensus statements are based on the opinions of carefully chosen content experts and provide for informational and educational purposes only. The purpose of the development group is to synthesize information, along with possible conflicting interpretations of the data, into clear and accurate answers to the question of interest. Clinical consensus statements may reflect uncertainties, gaps in knowledge, opinions, or minority view points, but through a consensus development process, many of the uncertainties are overcome, a consensual opinion is reached, and statements are formed. Clinical consensus statements are not clinical practice guidelines and do not follow the same procedures as clinical practice guidelines. Clinical consensus statements do not purport to be a legal standard of care. The responsible physician, in light of all the circumstances presented by the individual patient, must determine the appropriate treatment, diagnosis, and management. Consideration of clinical consensus statements will not ensure successful patient outcomes in every situation. The AAO-HNSF emphasizes that these clinical consensus statements should not be deemed to include all proper diagnosis/management/treatment decisions or methods of care, or to exclude other treatment decisions or methods of care reasonably directed to obtaining the same results.

Internal and External Review of CCS

Since a CCS is based on expert opinion and an iterative survey process, statements and the supporting text do not undergo external peer review or an open public comment period like that of the CPGs. It would not be possible to incorporate others’ feedback into the document without altering the opinions of the development group or including an additional iteration of the Delphi survey process to address each comment and further rank each statement based on the revised language.

Dissemination and Implementation

Each CCS should be submitted for publication to a peer-reviewed journal. To increase dissemination of information, each CCS is also submitted for presentation at the AAO-HNSF Annual Meeting & OTO EXPO, and a journal podcast should be scheduled with the chair and one other member of the development group to accompany the publication. Once published, the full-text version of the CCS should be made freely available to all members of the public.

Author Contributions

Richard M. Rosenfeld, chair, writer; Lorraine C. Nnacheta, writer; Maureen D. Corrigan, writer.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: American Academy of Otolaryngology—Head and Neck Surgery Foundation. Each clinical consensus statement is funded solely by the Foundation. Development group members are not compensated, since travel and lodging are not required.