Predictive Validity of Simulation Training for Robotic Surgery and Other Modalities

The big question that many doctors and hospitals continually ask themselves, “will this time I spend on a simulator actually end up having an impact on my patients?”, plays a big part in how training is structured. In terms of validation, correlating time spent on a simulator to improving patient outcomes is referred to as predictive validity.

The first research study to look into this for robotic surgery was published by Dr. Patrick Culligan in 2014, which developed the “Morristown Protocol”.  Using the Morristown Protocol curriculum, Dr. Culligan, et al., demonstrated predictive validity by setting an expert-based proficiency benchmark and asking 14 attending surgeons to complete a curriculum of 10 exercises to this same level of proficiency as the experts that were benchmarked. After completing the simulation training, the attending surgeons were able to complete their first robotic hysterectomy case within the same or better parameters as the experts. These parameters included things such as operative time, blood loss, and technique as assessed by experts. While this study was done using Mimic software, the simulation training was carried out using the Intuitive Surgical Skills Simulator hardware.

The table below gives the data for the experts as well as a control group of surgeons who had privileges at the institution but had not spent any time on a simulator for training:


More recently, Dr. Gokhan Sami Kilic of the University of Texas Medical branch at Galveston carried out a study that looked at the impact of simulation training on surgical outcomes for Hysterectomies that not only looked at robotic surgery but also included open surgery, laparoscopic, and vaginal approaches to Hysterectomy.

Unlike Dr. Culligan’s study, Dr. Kilic also focused on residents as opposed to already trained surgeons. The average age of the surgeons who went through the Morristown protocol was just under 50 years old. Dr. Kilic’s study, however, was focused on surgeons who were in their residency and were grouped in PGY2, PGY3, and PGY4, typically under the age of 40.

This study looked at patient outcomes such as estimated blood loss, postoperative hospital stay, intraoperative adverse events, and mean operative time. The study was retrospective and covered a period from 2009 to 2014.

Simulation was introduced in 2010 at the institution for all modalities except robotics. Robotics was introduced in 2011 with the acquisition of Mimic’s dV-Trainer. The simulators were from a wide range of manufacturers in addition to Mimic including, 3-Dmed and Limbs&Things. Residents followed a structured simulation-based training program for Total Abdominal Hysterectomy (TAH), Vaginal Hysterectomy (VH), Total Laparoscopic Hysterectomy (TLH) and Robot Assisted Hysterectomy (RAH).

In total, 1,397 patients were included in the study and 41% (n = 576) underwent TAH, 22% (n = 305) underwent VH, 20% (n = 272) underwent TLH and 17% (n = 244) underwent RAH.

The patient populations did demonstrate some variations between the modalities and there were no statistically significant variations in relation to age, BMI, parity, or the number of previous surgeries.

The results can be seen in the table below


As you can see in the table, the average estimated blood loss before and after simulation-based training was significantly different in TAH and RAH groups, but no significant difference was found for VH and TLH. The mean of length of hospital stay was also significantly different before and after simulation-based training for each technique.

It is interesting that there was no statistical impact for OR time, though perhaps understandable as OR time is more related to overall team performance and thus requires team simulation as opposed to surgical skill.  Intraoperative complications did not seem to be impacted either by simulation, though they did trend downward in the robotic cohort.

Although the study was not intended to look at this specifically, it does seem to indicate that while robotic surgery might have marginally longer operative times it does seem to have a lower level of intraoperative complications, lower blood loss, and the lowest length of stay along with Vaginal Hysterectomies.

At Mimic, the dV-Trainer was developed with the objective of helping surgeons master the da Vinci ® robotic system allowing them to improve outcomes for patients. It is great to see a research study that validates in this general direction not only for robotics but for other modalities that incorporate simulation training as well.