The role of previously undocumented data in the assessment of medical trainees in clinical competency committees.

Publisher: Ubiquity Press Country of Publication: Netherlands NLM ID: 101590643 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 2212-277X (Electronic) Linking ISSN: 22122761 NLM ISO Abbreviation: Perspect Med Educ Subsets: MEDLINE

Publication: 2023- : [London] : Ubiquity Press
Original Publication: [Houten] : Bohn Stafleu van Loghum

Clinical Competence/*standards ; Students, Medical/*statistics & numerical data; Clinical Competence/statistics & numerical data ; Educational Measurement/methods ; Educational Measurement/statistics & numerical data ; Humans ; Interviews as Topic/methods ; Ontario ; Qualitative Research

Introduction: The clinical competency committee (CCC) comprises a group of clinical faculty tasked with assessing a medical trainee's progress from multiple data sources. The use of previously undocumented data, or PUD, during CCC deliberations remains controversial. This study explored the use of previously undocumented data in conjunction with documented data in creating a meaningful assessment in a CCC.
Methods: An instrumental case study of a CCC that uses previously undocumented data was conducted. A single CCC meeting was observed, followed by semi-structured individual interviews with all CCC members (n = 7). Meeting and interview transcripts were analyzed iteratively.
Results: Documented data were perceived as limited by inaccurate or superficial data, but sometimes served as a starting point for invoking previously undocumented data. Previously undocumented data were introduced as summary impressions, contextualizing factors, personal anecdotes and, rarely, hearsay. The purpose was to raise a potential issue for discussion, enhance and elaborate an impression, or counter an impression. Various mechanisms allowed for the responsible use of previously undocumented data: embedding these data within a structured format; sharing relevant information without commenting beyond one's scope of experience; clarifying allowable disclosure of personal contextual factors with the trainee pre-meeting; excluding previously undocumented data not widely agreed upon in decision-making; and expecting these data to have been provided as direct feedback to trainees pre-meeting.
Discussion: Previously undocumented data appear to play a vital part of the group conversation in a CCC to create meaningful, developmentally focused trainee assessments that cannot be achieved by documented data alone. Consideration should be given to ensuring the thoughtful incorporation of previously undocumented data as an essential part of the CCC assessment process.

Comment in: Perspect Med Educ. 2020 Oct;9(5):269-271. (PMID: 33001415)

Hauer KE, Ten Cate O, Holmboe E, et al. Ensuring resident competence: A narrative review of the literature on group decision-making to inform the work of clinical competency committees. J Grad Med Educ. 2016;8:156–64. (PMID: 10.4300/JGME-D-15-00144.1)
Holmboe ES, Yamazaki K, Edgar L, et al. Reflections on the first 2 years of milestone implementation. J Grad Med Educ. 2015;7:506–12. (PMID: 10.4300/JGME-07-03-43)
French JC, Dannefer EF, Colbert CY. A systematic approach toward building a fully operational clinical competency committee. J Surg Educ. 2014;71:e22–e7. (PMID: 10.1016/j.jsurg.2014.04.005)
Hauer KE, Chesluk B, Iobst W, et al. Reviewing residents’ competence: a qualitative study of the role of clinical competency committees in performance assessment. Acad Med. 2015;90:1084–92. (PMID: 10.1097/ACM.0000000000000736)
Andolsek K, Padmore J, Hauer KE, Holmboe E. Clinical competency committees: a guidebook for programs [Internet]. 2nd ed. Accreditation council for graduate medical education; 2017. 1–73 p. http://www.acgme.org/acgmeweb/Portals/0/ACGMEClinicalCompetencyCommitteeGuidebook.pdf . Accessed April 22, 2019.
Royal College of Physicians & Surgeons of Canada. Competence committee guideline: process and procedures in decision making. Ottawa: Royal College of Physicians & Surgeons of Canada; 2017.
Rose SH, Burkle CM. Accreditation Council for Graduate Medical Education competencies and the American Board of Anesthesiology clinical competence committee: a comparison. Anesth Analg. 2006;102:212–6. (PMID: 10.1213/01.ane.0000189099.13286.97)
Colbert CY, French JC, Herring ME, Dannefer EF. Fairness: the hidden challenge for competency-based postgraduate medical education programs. Perspect Med Educ. 2017;6:347–55. (PMID: 10.1007/s40037-017-0359-8)
Dickey CC, Thomas C, Feroze U, Nakshabandi F, Cannon B. Cognitive demands and bias: challenges facing clinical competency committees. J Grad Med Educ. 2017;9:162–4. (PMID: 10.4300/JGME-D-16-00411.1)
Regehr G, Eva K, Ginsburg S, Halwani Y, Sidhu R. Assessment in postgraduate medical education: Trends and issues in assessment in the workplace. Members FMEC PG Consort. 2011. http://www.afmc.ca/future-of-medical-education-in-canada/postgraduate-project/activities-environmental.php . Accessed April 23, 2019.
Ginsburg S, McIlroy J, Oulanova O, Eva K, Regehr G. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med. 2010;85:780–6. (PMID: 10.1097/ACM.0b013e3181d73fb6)
Ginsburg S, van der Vleuten CPM, Eva KW. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Acad Med. 2017;92:1–5. (PMID: 10.1097/ACM.0000000000001669)
Ginsburg S, van der Vleuten CPM, Eva KW, Lingard L. Cracking the code: residents’ interpretations of written assessment comments. Med Educ. 2017;51:401–10. (PMID: 10.1111/medu.13158)
Regehr G, Ginsburg S, Herold J, Hatala R, Eva K, Oulanova O. Using “standardized narratives” to explore new ways to represent faculty opinions of resident performance. Acad Med. 2012;87:419–27. (PMID: 10.1097/ACM.0b013e31824858a9)
Lefebvre C, Hiestand B, Glass C, et al. Examining the effects of narrative commentary on evaluators’ Summative assessments of resident performance. Eval Health Prof. 2020;43:159–61. (PMID: 10.1177/0163278718820415)
Ginsburg S, van der Vleuten C, Eva KW, Lingard L. Hedging to save face: a linguistic analysis of written comments on in-training evaluation reports. Adv in Health Sci Educ. 2016;21:175–88. (PMID: 10.1007/s10459-015-9622-0)
Ginsburg S, Regehr G, Lingard L, Eva KW. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Educ. 2015;49:296–306. (PMID: 10.1111/medu.12637)
Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med. 2005;80(Supplement):S84–S7. (PMID: 10.1097/00001888-200510001-00023)
Cleland JA, Knight LV, Rees CE, Tracey S, Bond CM. Is it me or is it them? Factors that influence the passing of underperforming students. Med Educ. 2008;42:800–9. (PMID: 10.1111/j.1365-2923.2008.03113.x)
Friedman KA, Raimo J, Spielmann K, Chaudhry S. Resident dashboards: helping your clinical competency committee visualize trainees’ key performance indicators. Med Educ Online. 2016;21:29838. (PMID: 10.3402/meo.v21.29838)
Thomas MR, Beckman TJ, Mauck KF, Cha SS, Thomas KG. Group assessments of resident physicians improve reliability and decrease halo error. J Gen Intern Med. 2011;26:759–64. (PMID: 10.1007/s11606-011-1670-4)
Baxter P, Jack S. Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep. 2008;13:544–59.
Pack R, Lingard L, Watling CJ, Chahine S, Cristancho SM. Some assembly required: tracing the interpretative work of Clinical Competency Committees. Med Educ. 2019;53:723–34. (PMID: 10.1111/medu.13884)
Chahine S, Cristancho S, Padgett J, Lingard L. How do small groups make decisions? A theoretical framework to inform the implementation and study of clinical competency committees. Perspect Med Educ. 2017;6:192–8. (PMID: 10.1007/s40037-017-0357-x)
Odorizzi S, Cheung W, Sherbino J, Lee AC, Thurgur L, Frank J. A signal through the noise: do professionalism concerns impact the decision-making of competence committees? Acad Med. 2020;95:896–901. (PMID: 10.1097/ACM.0000000000003005)
Schumacher DJ, Martini A, Bartlett KW, King B, Calaman S, Garfunkel LC, et al. Key factors in clinical competency committee members’ decisions regarding residents’ readiness to serve as supervisors: a national study. Acad Med. 2019;94:251–8. (PMID: 10.1097/ACM.0000000000002469)

Keywords: Clinical competence committees; Competency-based medical education; Data management; Group decision-making; Undocumented data

Date Created: 20201007 Date Completed: 20210707 Latest Revision: 20230207

20240829

PMC7550499

10.1007/s40037-020-00624-x

33025382