Consensus in CBME: It’s Not Just What You Did, It’s What You Report

By: Susan Humphrey-Murto, MD, MEd Competency-Based Medical Education (CBME) has taken off over the past few decades and continues to gain momentum in many parts of the world. For some regions, CBME remains a relatively new concept—one that continues to evolve in both theory and implementation. A particularly distinctive feature of CBME is the development of Entrustable Professional Activities (EPAs), which often rely on formal consensus methods such as the Delphi technique and the Nominal Group Technique (NGT). Their importance is well-recognized—for example, the 2023 Medical Teacher article by Marije Hennus1 identifies the use of formal consensus methods as “Tip 8” in the successful development of EPAs. As someone who’s spent a lot of time thinking about consensus methods, I’ll admit: one thing continues to frustrate me—the inconsistent and often superficial reporting of how these consensus processes are actually conducted. The Problem with Poor Reporting There is no universally ‘correct’ way to conduct a Delphi or NGT study. Variation is in fact expected in: The number and backgrounds of expert participants, Geographic or institutional representation, The number of rounds conducted, The criteria used to define consensus. What is non-negotiable, however, is transparency. Without clear and thorough reporting, readers are unable to evaluate the rigor, trustworthiness, or applicability of a study’s conclusions. A national EPA framework developed by a small, homogenous panel may still be valuable—but only if readers are given sufficient information to make that judgment themselves. Why Reporting Guidelines Matter In medical education, we’ve historically been ambivalent—if not outright resistant—toward formal reporting checklists. They can seem too rigid, too positivist, or simply not relevant to our context-sensitive work. But when it comes to consensus methods, reporting guidelines serve not to restrict, but to clarify. They help ensure that: The rationale behind methodological decisions is made explicit, Readers can assess study quality and relevance of the work to their context, Scholarly work contributes meaningfully to a growing and cumulative evidence base. In short, reporting guidelines raise the bar for everyone—and help protect the integrity of the field. It’s worth noting that reporting guidelines are not “how-to” manuals. They are designed to promote transparency in what was done—not prescribe how to do it. For those seeking guidance on how to design and implement consensus studies, there are valuable resources available, including our 2017 publication offering practical advice on key design decisions for Delphi and NGT methods. 2 That paper outlines key considerations such as participant selection, number of rounds, and consensus thresholds. Two Recent Reporting Guidelines You Should Know ACCORD (ACcurate COnsensus Reporting Document) Developed through a modified Delphi process, ACCORD outlines 24 essential items for planning, conducting, and reporting consensus studies in biomedical research. It emphasizes transparency in panel composition, methodological choices, consensus thresholds, and data analysis. 3 DELPHISTAR Created by an interdisciplinary team in the social and health sciences, DELPHISTAR provides comprehensive guidance on reporting Delphi studies. It emphasizes clarity in study objectives, round structure, participant selection, and feedback procedures. 4 Both are freely available and directly applicable to medical education research. Where to Find Them Visit the EQUATOR Network (https://www.equator-network.org/) and search “education”—you’ll find over 20 reporting guidelines relevant to our field, including those for: Delphi studies, Qualitative research, Surveys, Mixed methods studies, Educational interventions. If you’re conducting a study that involves consensus—whether to develop EPAs, assessment criteria, or curricular frameworks—EQUATOR is a great starting point to ensure adequate reporting. What Should Be Reported? The following provides examples of what elements should be reported for a Delphi study in CBME (this is not an exhaustive list, but a useful starting point): Purpose: Aim of the Delphi or consensus method Panel composition: Who participated, and how were they selected and recruited? Round structure: How many rounds? What feedback was provided to participants? Questionnaire development: What resources were used to develop? Consensus definition: Was it set a priori? What threshold was used? Item evolution: Were items added, removed, or modified during the process? Who was involved in these decisions? Attrition: How many participants dropped out, and at what stage? These are not overly technical requirements—they are simply good research practices. Yet, they are frequently missing from published work. A Call to Action As educators and researchers, we hold our learners to high standards of clarity, professionalism, and evidence-based reasoning. We should hold ourselves—and our methods—to the same standards. So, the next time you undertake a consensus study, take a few moments to consult ACCORD or DELPHISTAR. You may find that the checklist doesn’t constrain your work at all—it just makes it stronger. About the Author: Susan Humphrey-Murto, MD, MEd is a Professor of Medicine at the University of Ottawa, in the Division of Rheumatology and Associate Director of Education Scholarship at CiMED. She holds a Tier 2 Research Chair in medical education; her research focuses on learner education handover and consensus group methods. References Hennus MP, Ten Cate O. Ten tips for implementing entrustable professional activities in medical training. Med Teach. 2023;45(6):653–658. doi:10.1080/0142159X.2023.2192020. Humphrey-Murto S, Varpio L, Wood TJ, et al. The use of the Delphi and other consensus group methods in medical education research: A review. Acad Med. 2017;92(10):1491–1498. doi:10.1097/ACM.0000000000001815. Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med. 2024;21(1):e1004326. doi:10.1371/journal.pmed.1004326. Niederberger M, Schifano J, Deckert S, et al. Delphi studies in social and health sciences—Recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study. PLoS ONE. 2024;19(8):e0304651. doi:10.1371/journal.pone.0304651.