Publication/Presentation Date
7-25-2014
Abstract
Abstract
Background
The ABMS requires that physicians in 18 out of 24 represented medical specialties undergo an oral examination for board certification. The ACGME evaluates mock-oral examination programs during annual review of accredited residency programs, as this experience is educationally valuable for residents. Within the Lehigh Valley Health Network, residents in the department of OB/GYN take a mock-oral examination annually.
Objective
We evaluated our mock oral examination program in regards to its consistency with other methods of assessment, as well as the level of resident satisfaction with the program. Resident mock-oral performance was compared with CREOG scores, self-evaluations, and satisfaction survey results in a retrospective analysis to determine the respective correlations.
Methods
Our statistical analysis included Spearman’s rho correlations between mock-oral performance and predicted performance, CREOG score, level of preparedness, and reported level of educational value using SPSS 16.0
Results
Mock-oral performance had moderately strong correlations with CREOG scores and self-predicted scores. This demonstrates the objectivity and validity of mock-oral examinations and shows that LVHN OB/GYN residents have a high level of self-awareness. Mock-oral performance had weak correlations with self-assessed level of preparedness and reported level of educational value. This indicates that oral examinations assess more than knowledge, as they are more difficult to prepare for, and nearly all residents considered the administration of mock-oral examinations to be educationally valuable.
Conclusion
Oral examinations are a valid assessment of clinical competence. We hope that the future brings standardized cases for use in all mock-oral programs, as well as multi-institutional mock-oral programs with raters from different residency programs.
Mock-Oral Exams: A vital and effective tool in training tomorrow’s physicians
Effective clinical communication is crucial to patient care. It is the basis of a successful doctor/patient relationship, and better communication increases a patient’s likelihood of compliance and satisfaction.1 Medical training programs have emphasized the importance of effective communication in patient care and eliminating medical errors.2 The American Board of Medical Specialties, ABMS, recognizes the crucial role of communication and requires oral examinations for board certification in eighteen of the twenty-four represented medical specialties.3 Oral examinations are used as supplements to the written exams and are able to assess what a written exam cannot.4 They provide a distinct method for evaluating students’ problem-solving abilities, communication skills, and clinical competence.5 Oral examinations have previously been criticized, though, for lack of inter-rater reliability among evaluators of oral exams, subjectivity, and inefficacy in truly determining resident competence.6 However, recent studies have demonstrated that oral examination results are consistent and correlate with other methods of evaluation.4
Within the Lehigh Valley Health Network, the obstetrics and gynecology residency program has employed annual mock oral examinations since 2008 to evaluate the residents’ progress and help prepare them for board certification. Canadian psychiatry residents reported in a 2002 study that preparation for oral examination is “of paramount importance”.7 Mock oral examinations have proven beneficial to many residents in a variety of residency programs, and these mock exams help determine strengths and weaknesses of residents as they progress through their respective programs.8 In fact, The Accreditation Council for Graduate Medical Education evaluates resident mock-oral performance during annual review of ACGME-accredited residency programs.9 The purpose of this study is to evaluate our mock oral examination program in regards to its consistency with other methods of assessment, as well as to evaluate the level of resident satisfaction with the program. Resident mock-oral performance will be compared with CREOG scores, self-evaluations, and satisfaction survey results in a retrospective analysis to determine the respective correlations.
Methods
Obstetrics and gynecology residents in the Lehigh Valley Health Network (LVHN) annually undergo two hour mock oral examinations and are assigned a grade of “excellent,” “pass,” “marginal,” or “fail” in regards to their performance in each of four subjects: general knowledge, ambulatory care, obstetrics, and gynecology. Each subject takes half an hour and has two raters, for a total of eight raters.
Additionally, residents take the CREOG, or Council on Resident Education in Obstetrics and Gynecology, each year, which is a national subspecialty exam administered to all OB/GYN residents. The exam has 180 questions covering general knowledge, ambulatory care, obstetrics, gynecology, reproductive endocrinology, gynecologic oncology, and genetics and genomics. Scores for each section are based on a Gaussian curve, where 200 is the national mean, and scores for each year are determined based on each resident’s percentile and PGY level.
During academic years 2009 through 2014, OB/GYN residents in the LVHN were asked to predict how they would perform on each section of the mock oral examination, and immediately following administration, they were provided with a survey to evaluate their satisfaction with the mock oral program. Of particular interest for this study is how well residents felt they prepared, and to what degree they felt the mock-oral examination was educationally valuable. Residents were asked to judge these factors on a 10-point Likert scale (1 = not valuable, 10 = extremely valuable).
The data considered in this study were the mock-oral and CREOG scores in the subjects of ambulatory care, obstetrics, and gynecology for residents from 2009 through 2014. Each data point for this study consisted of a mock-oral examination score, which will be correlated to the respective CREOG score, resident predicted score, self-assessed level of preparedness, and resident determined level of educational value. Forty-nine residents were included in the study, 32 of which have more than one set of mock-oral and CREOG scores to consider, and each set consists of mock-oral and CREOG exams in three subjects, for a total of 318 individual data points to consider.
Statistical analysis of the data included Spearman’s rho correlations between mock-oral performance and predicted performance, CREOG score, level of preparedness, and reported level of educational value. This was done using SPSS version 16.0.
Results
For the purposes of analysis, a grade of “pass” was denoted as a 1, “marginal” as a 2, and “fail” as a 3.
Table 1: Correlation Coefficients and p-values
Correlation Coefficient p-value N
Mock Oral Performance v. Predicted Performance 0.577
Mock Oral Performance v. CREOG Score -0.578
Mock Oral Performance v. Level of Preparedness -0.202
Mock Oral Performance v. Educational Value -0.196
For the purpose of clarification, the negative correlation coefficient for Mock Oral Performance v. CREOG Score indicates that as mock-oral performance goes down (1 to 3), so does CREOG score (about 250 to 150).
Figure 1: Actual v. Predicted Mock Oral Performance (N=318)
Figure 2: Mock Oral Performance v. CREOG Score Range
Figure 3: Reported Level of Preparedness (N=106)
Figure 4: Reported Level of Educational Value (N=106)
Discussion
All of our findings were statistically significant. There is a moderately strong correlation between mock-oral performance and CREOG scores. This both attests to the validity of oral examinations in assessing clinical competence and demonstrates a high level of oral examination objectivity. A weakness with this portion of the study is that, although we analyzed 318 sets of mock-oral and CREOG scores, there were only 49 residents, so many sets of data belonged to the same resident, which may have created redundancy. Another weakness is that the study only analyzed resident performance in the LVHN OB/GYN residency program. It speaks to the efficacy of our program, but may not apply to programs that are less well developed.
Our findings also show a moderately strong correlation between mock-oral performance and predicted performance. This indicates that the residents in our program have a high level of self-awareness and recognize their strengths, and that the oral exams are capable of assessing these abilities. A weakness with this portion of the study is that about half of the residents received their CREOG scores before taking the mock-oral examination, which may have led to a bias in their predictions. However, we believe there was minimal to no bias because the residents do not see their full CREOG score breakdown by section.
Our findings show a weak correlation between mock-oral performance and self-assessed level of preparedness. The direction of this correlation was as expected: as performance goes down, so does assessed level of preparedness. However, the weakness of this correlation speaks to the criticality of oral examinations in determining clinical competence. They are able to assess skills a written exam cannot, including self-confidence, logical presentation, and interpersonal skills. This makes mock-oral preparation challenging: it is much easier to remember facts and answer multiple choice questions than to hone one’s critical thinking and self-confidence. The fact that self-assessed level of preparedness does not correlate strongly to performance indicates the importance of utilizing oral examinations. The moderately strong correlation between mock-oral performance and CREOG scores indicates that mock-orals are a valid assessment of medical knowledge. But the weak correlation between preparedness and performance demonstrates that mock-orals assess more than knowledge: they assess competency.
Overall, the residents have a high level of satisfaction with the mock-oral program and rate it highly in regards to educational value. This was demonstrated in a weak correlation between mock-oral performance and reported level of educational value: almost all ratings were high regardless of performance. However, there is a very slight correlation showing that residents who perform poorly on their mock-oral examinations will consider it a more educationally valuable experience. We believe this is important to note, as it ascertains that mock-oral examination is a beneficial program and is important to those who need to practice their communication skills. It challenges residents in regards to effective clinical communication and prepares them for board certification.
We believe the implications of this study are numerous, as more residency programs are adapting mock-oral examinations in their resident education. As mock-oral popularity grows, there may be a demand for standardized cases for use in mock-oral exams at all institutions. Moreover, we hope the future will bring multi-institutional mock-oral programs, where raters from different residency programs work together to administer the mock-oral exams. This synthesis would provide greater standardization and efficacy in preparing residents for board certification.
Conclusion
Oral examinations are a valid method of assessing clinical competence and communication skills in future physicians. They are able to assess the interpersonal and critical thinking skills that a written exam cannot. Resident mock-oral examination performance in the Lehigh Valley Health Network Department of Obstetrics and Gynecology has a moderately strong correlation with objective CREOG scores, which indicates their validity. There is also a moderately strong correlation between mock-oral exam performance and predicted performance, which demonstrates a high level of resident self-awareness. The weak correlations between mock-oral exam performance, level of preparedness, and reported level of educational value are also of note. Oral examinations can be difficult to prepare for because they evaluate more than knowledge, and proper preparation for oral exams is beneficial to residents. Employing mock-oral examinations in residency programs is educationally valuable, and the future will certainly bring standardized mock-oral examinations and multi-institutional programs.
1. Kourkouta, L., Barsamidis, K., & Lavdaniti, M. (2013). Communication skills during the clinical examination of patients. Progress in Health Sciences, 3(1), 120-122.
A study evaluating the importance of good communication between doctor or therapist and patient.
2. Marshall, S., Harrison, J., & Flanagan, B. (2009). The Teaching Of A Structured Tool Improves The Clarity And Content Of Interprofessional Clinical Communication. Quality and Safety in Health Care, 18(2), 137-140.
3. Welcome to ABMS: Improving healthcare quality through board certification. (n.d.). American Board of Medical Specialties. Retrieved June 2, 2014, from http://www.abms.org/
The web page for the American Board of Medical Specialties. Discusses the board certification process for twenty-four medical specialties.
4. Schubert, A., Tetzlaff, J. E., Tan, M., Ryckman, V. J., & Mascha, E. (1999). Consistency, Inter-rater Reliability, and Validity of 441 Consecutive Mock Oral Examinations in Anesthesiology. Anesthesiology, 91(1), 288-298.
A study of mock oral examinations in anesthesiology residents and their role in resident evaluation. The study concludes that mock oral examinations can be consistent and reliable methods of assessment.
5. Rahman, G. (2011). Appropriateness of using oral examination as an assessment method in medical or dental education. Journal of Education and Ethics in Dentistry, 1(2), 46.
A paper evaluating the appropriateness of using oral examinations in the dentist board certification process.
6. Mcguire, C. H. (1966). The Oral Examination as a Measure of Professional Competence. Academic Medicine, 41(3), 267-74.
An older study analyzing oral examinations and finding fault with consistency and inter-rater reliability.
7. Crockford, D., Holt-Seitz, A., & Adams, B. (2004). Preparing Psychiatry Residents for the Certification Exam: A Survey of Residency and Exam Experiences. Canadian Journal of Psychiatry, 49(10), 690- 695.
A survey to evaluate what methods of study were most useful to Canadian psychiatry residents in preparation for board certification. The study concluded that mock oral examinations would have been beneficial in preparing the residents.
8. Subhas, G., Yoo, S., Yeon-Jeen, C., Peiper, D., Frikker, M. J., Bouwman, D. L., et al. (2009). Benefits of Mock Oral Examinations in a Multi-institutional Consortium for Board Certification in General Surgery Training. American Surgeon, 75(9), 817-821.
A study looking at the benefits of mock oral examinations for surgery residents.
9. Rose, S. H., & Long, T. R. (2010). Accreditation council for graduate medical education (ACGME) annual anesthesiology residency and fellowship program review: a "report card" model for continuous improvement. BMC Medical Education, 10(1), 13.
An article discussing the anesthesiology residency evaluation program of the ACGME.
Published In/Presented At
Fulllington, M., Patruno, J., Huang, H., Friel, K., Flicker, A.,(2014, July, 25) Mock-Oral Exams: A Vital and effective tool in training tomorrow's physicians. Poster presented at LVHN Research Scholar Program Poster Session, Lehigh Valley Health Network, Allentown, PA.
Department(s)
Research Scholars, Research Scholars - Posters
Document Type
Poster
Comments
Mentor: Craig Koller