Publication/Presentation Date
7-31-2015
Abstract
The Educational Value of the Use of a Simcapture-feedback Process in the Training of Obstetrics and Gynecology Residents
Joseph Patruno, MD, Timothy Pellini, MD, Martin Martino, MD, Grace Bova
Lehigh Valley Health Network, Allentown, Pennsylvania
Abstract:
Background: The basis of renowned training hospitals like LVHN is a well-defined and effectively applied method for optimal teaching, especially in the operating room. Unfortunately, many physicians don’t provide the immediate and detailed feedback most valuable to residents. Thus, video feedback and detailed surveys shown to surgeons upon procedure completion are tools used to potentially optimize the educational experience for Obstetrics and Gynecology residents in the operating room.
Method: For each chosen OB/Gyn procedure the surgical field was recorded. After the surgery, detailed surveys were given to each physician to assess the resident’s performance on each particular part of the operation. A debrief was also stimulated to discuss the specific weaknesses and strengths that the resident displayed during the procedure. Later that day, an Educational Value Survey was sent to both physicians to obtain feedback on how educationally effective the video and debriefing process were. Simultaneously, the same survey was sent to a control resident and attending who had done a similar procedure without the video and survey aspect present.
Results: Attending surgeons’ responses in the subject cases were significantly higher with respect to the timeliness and effectiveness of the debrief than the responses from the control cases. Generally, both attendings and residents found the surgeries with the SimCapture video and evaluation tools more educationally valuable than those without the stimulated feedback resources.
Conclusions: SimCapture video feedback as well as detailed evaluation forms can enhance the teaching and learning for residents doing OB/Gyn procedures. Additionally, with this foundation in place attending surgeons are able to engage the resident in a structured debrief after a surgery.
Introduction: In a teaching hospital, attending surgeons are expected to provide immediate and comprehensive feedback to residents in order to improve surgery techniques and patient care. Unfortunately this expected debrief does not always occur or often the feedback given isn’t direct & clear enough to be effective. However the SimCapture technology used for video feedback as well as a series of surveys filled out by both surgeons are possible resources that can be utilized in order to optimize the educational value of a procedure.
According to the BID model for teaching in the operating room created by Dr. Nicole K. Roberts (2009) the learner must be “actively involved in creating learning objective & proposing how instruction might be extended to future practice” (p.300). This model is easy to use because it solely requires brief conversation between attending and resident while scrubbing, operating, & closing. By utilizing these events that occur at every operation, very little additional time is required from either surgeon.
Furthermore, the educational value of written assessments regarding resident’s surgical performance has also been rendered valid and useful. Such assessments “encourage the faculty and the resident to focus on areas of demonstrated competence and those areas needing improvement” (Dougherty et all, 2013, p.333). Detailed evaluation forms allow for the surgeons to reflect on each step of the procedure. These forms are most useful when completed directly after the surgery, allowing for more comprehensive recollection of the operation. Likewise, the resident benefits more from the stimulated debrief soon after he finishes the procedure.
The stimulated debrief in this project utilizes the “Operative Performance Rating System “ (OPRS). Using this form, both surgeons can assess the resident’s performance for each part of the procedure. Although these evaluations have not get been validated for Ob/Gyn procedures, they have proven to be especially effective in general surgery. Unlike more broad evaluations of technical skill, the OPRS offers a “controlled, systematic, observation/evaluation and appropriate sampling of performance across the entire performance domain” (Larson et al, 2005, p. 646). Thus, the OPRS allows evaluators to rate the residents on a likert scale for each step of the procedure as well as more general surgical flow and knowledge. Detailed feedback like this provides the residents with specific surgical skills and methods that need improvement.
The objective of our study is to determine the educational value of SimCapture video feedback as well as post-procedure evaluation forms with respect to Obstetrical and Gynecological procedures. We anticipate that these tools will enhance both the learning and teaching process for the residents and attendings respectively.
Materials and Methods: In order to evaluate whether the SimCapture video and survey forms combined made a positive effect on the educational value of an operation we filmed about 40 OR cases. Among these cases included Cesarean sections, Robotically-assisted Laparoscopic Hysterectomies, Hysteroscopies, Bilateral Tubal Ligations, and Laparoscopically-Assisted Vaginal Hysterectomies.
After choosing several cases to record from the OR list, the team found both the attending and resident on the case and informed them on the SimCapture project and asked for permission to film their upcoming procedure in the OR. Upon consent, the team set up the video system and oriented the OR staff to the project.
During the procedure, only the surgeon’s hands were recorded. The residents were all assigned a number; thus, a resident number was associated with each video instead of individual’s names. Annotations were made within the video to indicate when the resident is operating.
After the procedure was complete, both attending and resident were asked to fill out both a milestone form and an OPRS evaluation. The OPRS having questions specific to each step of the particular type of case. Having completed the surveys, the resident was asked to reflect on what he thought his weaknesses and strengths were in that particular case. This ignited the debriefing conversation, often resulting in meaningful discussion between both surgeons regarding the resident’s technique and skill level.
Later that day, a survey was sent to both the attending and resident to assess the educational significance of their procedure that day. Among these questions were ones about the questions asked, the effectiveness of the debrief, and the overall educational value of the case. Each questions required a response between 1-5, usually 1 relating to least educational and 5 being most educationally valuable.
Concurrently, another set of the same educational value surveys was sent to a control attending and resident that had participated in a similar case that day. Thus, the control data would allow us to evaluate the overall effect of the SimCapture and feedback process on the educational process. Using these cases for controls allows us to determine how well residents are regularly being effectively taught, especially via a debrief after the procedure.
The questions in this survey asked responders to answer the following questions on a likert scale 1 being “strongly disagree” and 5 being “strongly agree”.
1. I/the resident came adequately prepared
2. Optimal training provided?
3. The questions asked by the attending were fair & appropriately challenging
4. There was an appropriate debrief before, during & after the case?
5. My opinion of my performance correlates with the attending’s perception of my performance.
6. Specific, timely, & valuable feedback was given.
7. I would rate the overall educational value of this case as… (1-Poor and 5-Excellent)
The responses were then compiled into four separate groups: control residents, subject residents, control attendings, and subject attendings. The mean response for each question for each group was calculated. Then the controls’ and subjects’ mean responses were compared to assess the overall educational effect of the video and feedback process. Statistical analysis was also done. A t-test was performed to obtain p-values for the data set. Using the results of the t-test, we were able to determine the statistical significance and validity of each question for the residents (controls and subjects) and attendings (controls and subjects).
Results: By the end of our six-week collection period, forty total cases were recorded: seventeen Cesarean Sections, eight Robotic Hysterectomies, twelve Hysteroscopies, and three Bilateral Tubal Ligations. We also collected fifteen control cases: four Cesarean Sections, one Robotic Hysterectomy, eight Hysteroscopies, and two Bilateral Tubal Ligations. Twelve out of twenty-three OB/Gyn residents participated in the study: two interns, five in their second year of residency, three in their third, and two in their fourth. Twenty-two different attendings participated as well.
A brief comparison of the controls and subjects was done with the mean responses to each question. These results are as follows. The more statistically significant differences will be further discussed. Refer to Table 1 in the Appendix to see the average values for each question.
1. I/the resident came adequately prepared
Residents: Similar responses regardless of process
Attendings: Similar responses regardless of process
2. Optimal training provided?
Residents: Interestingly, the controls felt more optimally trained in the OR than the subject did
Attendings: Similar responses regardless of process
3. The questions asked by the attending were fair & appropriately challenging
Residents: Controls’ responses were higher; felt they had been asked more appropriate questions
Attendings: Controls’ responses were slightly higher
4. There was an appropriate debrief before, during & after the case?
Residents: Subjects’ responses were higher. Not surprisingly, as there was better communication during the debrief developed by the process
Attendings: Subjects’ responses were generally much higher. The attendings were alert to the stimulated debriefing process
5. My opinion of my performance correlates with the attending’s perception of my performance.
Residents: Similar responses regardless of process
Attendings: Similar responses regardless of process
6. Specific, timely, & valuable feedback was given
Residents: Similar responses; feel they are getting feedback in either process
Attendings: Subjects’ responses were higher; attendings felt much more confident regarding their feedback with this process guiding them
7. I would rate the overall educational value of this case as… (1-Poor and 5-Excellent)
Residents: Subjects’ responses were higher. The educational value ties together importance of preparation, supervision, education, debriefing, and feedback, several of which the SimCapture/feedback process aimed to enhance.
Attendings: Subjects’ responses were higher as well.
The following is a chart (Figure 1) of the average response for residents (both control and subject) and the p values from the t-test. The highlighted columns are the values that are most statistically significant.
(Figure 1)
The following is a chart (Figure 2) of the average response for attendings (both control and subject) and the p values from the t-test. The highlighted columns are the values that are most statistically significant.
(Figure 2)
Furthermore, in the subject’s surveys there was a question about the value of using SimCapture video as a form of feedback. 87% of subject cases had residents who saw value in using video as a method of feedback. 75% of the subject cases had attendings who felt the video feedback could be useful for optimal teaching and learning
Discussion: The residents in the subject cases found the surgeries with the SimCapture and feedback process generally more educationally valuable. Many of the residents enjoyed the opportunity to receive direct and detailed feedback from their mentors. The residents also benefitted from reflecting on their own performance, considering both their strengths and weaknesses.
However, the residents’ responses that were particularly statistically significant were about the questions asked during surgery; interestingly, the controls felt that the questions asked were more fair and appropriate than the subjects. This difference may be due to the presence of the SimCapture equipment and team. This addition to the OR may have made the attendings feel as though the resident was in the spotlight and, therefore, did not need as many questions.
The attendings who participated in the SimCapture and feedback process were able to recognize the educational importance of the debrief after a procedure. The attendings in subject cases felt significantly more confident about the timeliness and value of the feedback they gave to the resident after the procedure. By stimulating a debrief session between both surgeons directly after the procedure, attendings were able to recall specific challenges and strengths the resident displayed in the surgery. Thus, they felt they were able to deliver complete and procedure-specific criticism to the residents.
Additionally the attendings also became more aware of the debriefing process. Many attendings further discovered the value and effectiveness of giving immediate, detailed feedback. Some of the anonymous comments left on the educational value survey were, “I need to debrief more. We have to go after cases to meet the next patient and sometime don't take the time that we should be”… “I may ask the resident more questions during the case to assess their understanding”. Thus, this process increased personal teaching awareness, hopefully motivating the attendings to provide more comprehensive and timely feedback to the residents after a procedure.
Regarding the use of SimCapture video as feedback, most of the faculty and residents who participated in the cases felt the video technology could be an effective and useful form of feedback. Several surgeons requested to view their videos upon completion and were able to easily visualize their weaknesses and strengths with the objective point of view on the camera.
One of the limits of this research was the lack of a large cohort of control cases. Due to our time constraint, we were unable to get the widespread responses we had hoped for both control and subject cases. Additionally many residents and attednings did not fill out the educational value surveys at all. Others completed them days after the surgery, which could possibly limit their ability to recall the specifics of the residents’ performances in the particular cases. Lastly, in some of the subject cases we were not able to stimulate a debrief due the surgeon’s schedules. In these cases both surgeons filled out the OPRS evaluations but were unable to discuss the resident’s performance in the procedure. The lack of this debrief may have affected our final response data.
Conclusion: Overall the SimCapture video and the OPRS evaluations feedback method was educationally valuable. Attendings especially found the guided feedback process specifically helpful and effective. We expect that attendings will try to incorporate more timely and detailed feedback to their residents in their everyday practice. The detailed OPRS evaluations provided residents with specific areas of technique that need improvements.
By enhancing the teaching and learning process in the OR, residents are able to learn more advanced skills faster and more efficiently. Attendings are able to assess the resident’s direct areas of weakness or strength with a simple feedback mechanism. Essentially, the SimCapture with feedback process can potentially advance the way residents are trained, thus improving patient care.
Acknowledgements: Thank you to my project mentor Dr. Joseph Patruno, as well as Dr. Timothy Pellini, Dr. Martin Martino, Dr. Hubert Huang, Diane Leuthardt, the library services, and all the residents, physicians, nurses, and faculty in the OB/GYN department. This project would not have been possible without your support. Finally, thank you to Dr. Joseph Napolitano and the Dorothy Rider Pool Health Care Trust for giving me this opportunity and incredible experience.
References:
Dougherty, P, Kasten, S.J., Reynolds, R.K., Prince, M.E., Lypson, M.L. (2013, June). Intraoperative Assessment of Residents. Jornal of Graduate Medicine, 333-334. http://dx.doi.org/10.4300/JGME-D-13-00074.1
Larson, J.L., Williams, R. G., Ketchum, J, Boehler, M.L., Dunnington, G.L. (2005, October). Feasibility, reliability, and validity of operative performance rating system for evaluating surgery residents.Surgery, 138 (4), 640-649. http://dx.doi.org/10.1016/j.surg.2005.07.017
Levy, B. (2012, April). Experience Counts. Obstetrics & Gynecology, 119(4),
693-694.
Roberts, N.K., Williams, R.G., Kim, M.J., Dunnington, G.L. (2009, February). The Briefing, Intraoperative Teaching, Debriefing Model for Teaching in the Operating Room. Journal of the American College of Surgeons, 208(2), 299-303. http://dx.doi.org/10.1016/j/jamcollsurg.2008.10.024
Swing, S.R. (2007).The ACGME outcome project: retrospective and prospective. Medical Teacher, 29, 648-654. http://dx.doi.org/10.1080/01421590701392903
Published In/Presented At
Bova, G. (2015, July 31). The Educational Value of the Use of a Simcapture-Feedback Process in the Training of Obstetrics and Gynecology Residents. Poster presented at LVHN Research Scholar Program Poster Session, Lehigh Valley Health Network, Allentown, PA.
Department(s)
Research Scholars (Acknowledgements and Co-authored Publications), Research Scholars - Posters
Document Type
Poster
Comments
Mentor: Joseph Patruno, MD
Mentor: Timoth Pellini, MD
Mentor: Martin Martino, MD