Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure
Lisette Hvid Hovgaard, Steven Arild Wuyts Andersen, Lars Konge, Torur Dalsgaard, Christian Rifbjerg Larsen
Published in Surgical Endoscopy, March 2018
OBJECTIVE: The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study’s objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.
METHODS: Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module ‘Guided Vaginal Cuff Closure’ six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach’s alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups’ method.
RESULTS: The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach’s alpha). The experienced surgeons’ mean composite score for all six repetitions were significantly better than the novice surgeons’ (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives.)
CONCLUSION: Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.
The Validation of a Novel Robot-Assisted Radical Prostatectomy Virtual Reality Module
Patrick Harrison, BSc, Nicholas Raison, MRCS, Takashige Abe, PhD, MD, William Watkinson, BSc, Faizan Dar, MBBS, Ben Challacombe, MS, FRCS(Urol), Henk Van Der Poel, MD, PhD, Muhammad Shamim Khan, FRCS(Urol), Prokar Dasgupa, MD, FEBU, FRCS(Urol), Kamran Ahmed, FRCS(Urol), Ph
Published online:September 30, 2017
OBJECTIVE: To perform the first validation of a full procedural virtual reality robotic training module and analysis of novice surgeon’s learning curves.
DESIGN: Participants completed the bladder neck dissection task and urethrovesical anastomosis task (UVA) as part of the prostatectomy module. Surgeons completed feedback questionnaires assessing the realism, content, acceptability and feasibility of the module. Novice surgeons completed a 5.5-hour training programme using both tasks.
SETTING: King’s College London, London.
PARTICIPANTS: 13 novice, 24 intermediate and 8 expert surgeons completed the validation study.
RESULTS: Realism was scored highly for BDN (mean 3.4/5) and UVA (3.74/5), as was importance of BDN (4.32/5) and UVA (4.6/5) for training. It was rated as a feasible (3.95/5) and acceptable (4/5) tool for training. Experts performed significantly better than novice group in 6 metrics in the UVA including time (p = 0.0005), distance by camera (p = 0.0010) and instrument collisions (p = 0.0033), as well as task-specific metrics such as number of unnecessary needle piercing points (p = 0.0463). In novice surgeons, a significant improvement in performance after training was seen in many metrics for both tasks. For bladder neck dissection task, this included time (p < 0.0001), instrument collisions (p = 0.0013) and total time instruments are out of view (p = 0.0251). For UVA, this included time (p = 0.0135), instrument collisions (p = 0.0066) and task-specific metrics such as injury to the urethra (p = 0.0032) and bladder (p = 0.0189).
CONCLUSIONS: Surgeons found this full procedural VR training module to be a realistic, feasible and acceptable component for a robotic surgical training programme. Construct validity was proven between expert and novice surgeons. Novice surgeons have shown a significant learning curve over 5.5 hours of training, suggesting this module could be used in a surgical curriculum for acquisition of technical skills. Further implementation of this module into the curriculum and continued analysis would be beneficial to gauge how it can be fully utilised.
The Robotix Simulator: Face And Content Validation Using The Fundamentals Of Robotic Surgery(Frs)Curriculum
Ismail Omar*, James Dilley, Philip Pucher, Philip Pratt, Torath Ameen,
Justin Vale, Ara Darzi, Erik Mayer, London, United Kingdom
The Journal of Urology
Volume 197, Issue 4, Pages e700-e701 (April 2017)
INTRODUCTION AND OBJECTIVES: Robotic surgical training is poorly delivered with limited skills-based training offered to trainees. To improve this, validated high quality robotic simulators need to be developed and made available. The RobotiX simulator is a new platform which this study sought to validate in the context of a recognised robotic surgery curriculum.
METHODs: Surgeons(n¼29) with ranging robotic experience and experience with other simulators(da Vinci Backpack and Mimic) were invited to complete all 6 FRS curriculum exercises. Participant performance was scored using the Global Evaluation Assessment of Robotic skills(GEARS). Participants completed a Likert scale based face and content validity questionnaire graded as negative(1-2/5), neutral(3), or positive(4-5).
RESULTS: Overall, analysis included 27 participants. There was good concurrent GEARS score reliability (Cronbach’s Alpha 0.801) between participants performing exercises on both the RobotiX and an alternative robotic simulator.
CONCLUSIONS: The RobotiX simulator demonstrated excellent Face and Construct validity evidence both in terms of general usage and in the specific context of the FRS curriculum.(Table 2 Q1,2)
The RobotiX performed at least as well as other simulators(da Vinci Backpack/Mimic) and was found to simulate a robotic platform such as the da Vinci Robot. (Table 2 Q4,5) Furthermore 81% of surgical trainees would recommend using the RobotiX(Table 2 Q3).
Validation of the RobotiX Mentor Robotic Surgery Simulator
Whittaker, A. Aydin, N. Raison, F. Kum, B. Challacombe, M.S. Khan,P. Dasgupta, K. Ahmed
MRC Centre for Transplantation, King’s College London; Department of Urology, Guy’s
and St. Thomas’ NHS Foundation Trust, London, United Kingdom
Proceedings of the Hamlyn Symposium on Medical Robotics
20-23 June 2015, Imperial College London London, UK
ISBN: 978-0-9563776-6-1 p 69-70
INTRODUCTION: With robotic-assisted surgery becoming more common practice in urology, effective training remains a challenge. There is a considerable learning curve associated with robotic training, though this has been reported as gentler than for laparoscopic training1.
Simulation has gained wide acceptance as a method of reducing the initial phase of the learning curve. The RobotiX MentorTM is a new virtual reality simulator which mimics the interface of the da Vinci® Surgical System, with integrated training modules including Fundamentals of Robotic Surgery (FRS) and Robotic Suturing. Face and content validity of the suturing module have been confirmed in a previous study2, though this was performed on a different platform. This study aims to assess face, content and construct validity of the RobotiX MentorTM. It also aims to assess its acceptability as a training tool and feasibility of its use in training.
MATERIALS AND METHODS: This prospective, observational and comparative study recruited novice (n=20), intermediate (n=15), and expert (n=11) robotic surgeons as participants from institutions
across the United Kingdom and at the 30th European Association of Urology Annual Meeting. Each participant completed nine surgical tasks across two modules on the simulator, followed by a questionnaire to evaluate subjective realism (face validity), task importance (content validity), feasibility, and acceptability. Outcome measures of novice, intermediate, and expert groups were compared using
Mann-Whitney U-tests to assess construct validity. Outcome measures of novice, intermediate, and expert groups were compared using Mann-Whitney U-tests to assess construct validity.
RESULTS: Construct validity was demonstrated in a total of 17/25 performance evaluation metrics (p<0.001). Experts performed better than intermediates in regard to time taken to complete the first (p=0.002) and second (p=0.043) module, number of instrument collisions (p=0.040), path length (p=0.049), number of cuts >2mm deep (p=0.033), average distance from suture target (p=0.015), and number of suture breakages (p=0.038). Participants determined both the simulator console and psychomotor tasks as highly realistic (mean: 3.7/5) and very important for surgical training (4.5/5), with system pedals (4.2/5) and knot tying task (4.6/5) scoring highest respectively. The simulator was also rated as an acceptable (4.3/5) tool for training and its use highly feasible (4.3/5).
CONCLUSIONS: Construct, face, and content validity were established for the RobotiX Mentor and feasibility and acceptability of incorporation into surgical training was ascertained. The RobotiX Mentor shows potential as a valuable tool for training and assessment of trainees in robotic skills and may reduce the initial learning curve if utilised as an adjunct to operating-room training. Investigation of
concurrent and predictive validity is necessary to complete validation and evaluation of learning curves
would provide insight into its value for training.
REFERENCES:  Goldstraw MA, Challacombe BJ, Patil K, Amoroso P, Dasgupta P, Kirby RS. Overcoming the challenges of robot-assisted radical prostatectomy. Prostate cancer and prostatic diseases. Mar 2012;15(1):1-7.
 Abboudi H, Khan MS, Aboumarzouk O, et al. Current status of validation for robotic surgery simulators-a systematic review. BJU international. Feb 2013;111(2):194-205. 69
From Design to Conception: An Assessment Device for Robotic Surgeons
Alyssa Tanaka, M.S.; Manuela Perez, M.D. Mireille Truong M.D., Khara Simpson M.D. Gareth Hearn, Roger Smith, Ph.D.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2014
ABSTRACT: The daVinci Surgical System offers surgeons improved capabilities for performing complex minimally invasive procedures; however, there is no standardized assessment of robotic surgeons and a need exists to ensure that a minimal standard of care is provided to all patients. The Department of Defense and governing surgical societies convened consensus conferences to develop a national initiative, resulting in a curriculum called the Fundamentals of Robotic Surgery (FRS). FRS is comprised of an online curriculum and a psychomotor skills dome.This paper describes the production process used to create a psychomotor skills assessment device – the FRS Dome. The device was designed to measure the essential skills that are required of any robotic surgeon and to provide a basis upon which to grant or deny privileging with the robot. It was constructed to test seven tasks of manual dexterity: Docking, Ring Tower Transfer, Knot Tying, Suturing, 4th Arm Cutting, Puzzle Piece Dissection, and Energy Dissection.
The initial design of the device was created by a committee of experienced minimally invasive surgeons, with a background in testing protocols and materials. The design was rendered in computer animation, which kickstarted a prototyping effort with physical materials. These included platinum cure silicone approximating human tissue and a 3D polyjet printer for the structural framework. Usability testing was conducted and iterative modifications were made to improve ergonomics, standardization, and cost requirements. Final CAD diagrams and specifications were created and distributed to medical and simulation companies for both physical and digital manufacturing. This development process demonstrates the evolution of a simulation and a physical testing device based on international expert consensus. The specifications are open source, allowing competitive production and future iterations. The goal of this paper is to discuss how this device evolved from an idea to a manufactured product and a digital simulation.
Current status of robotic simulators in acquisition of robotic surgical skills.
Kumar A, Smith R, Patel VR.
Department of Urology, Global Robotics Institute, Florida Hospital-Celebration Health Chief Technology Officer, Florida Hospital-Nicholson Center, University of Central Florida College of Medicine, Orlando, Florida, USA.
Curr Opin Urol. 2015 Mar;25(2):168-74.
PURPOSE OF REVIEW: This article provides an overview of the current status of simulator systems in robotic surgery training curriculum, focusing on available simulators for training, their comparison, new technologies introduced in simulation focusing on concepts of training along with existing challenges and future perspectives of simulator training in robotic surgery.
RECENT FINDINGS: The different virtual reality simulators available in the market like dVSS, dVT, RoSS, ProMIS and SEP have shown face, content and construct validity in robotic skills training for novices outside the operating room. Recently, augmented reality simulators like HoST, Maestro AR and RobotiX Mentor have been introduced in robotic training providing a more realistic operating environment, emphasizing more on procedure-specific robotic training . Further, the Xperience Team Trainer, which provides training to console surgeon and bed-side assistant simultaneously, has been recently introduced to emphasize the importance of teamwork and proper coordination.
SUMMARY: Simulator training holds an important place in current robotic training curriculum of future robotic surgeons. There is a need for more procedure-specific augmented reality simulator training, utilizing advancements in computing and graphical capabilities for new innovations in simulator technology. Further studies are required to establish its cost-benefit ratio along with concurrent and predictive validity.
Further studies are required to establish its cost-benefit ratio along with concurrent and predictive validity.
Surgical suturing training with virtual reality simulation versus dry lab practice: an evaluation of performance improvement, content, and face validity
Michael J. Amirian, Samuel M. Lindner, Edouard J. Trabulsi, Costas D. Lallas
Department of Urology, Thomas Jefferson University Hospital, 1015 Walnut Street, Suite 1100, Philadelphia, PA 19107, USA
J Robotic Surg (2014) 8:329–335
The purpose of this study is to evaluate the effectiveness of virtual reality (VR) simulation versus dry lab suturing practice at improving suturing performance in robotic surgery. Nineteen novice participants with no prior robotic suturing experience were randomized to two groups, VR simulation and dry lab, which consisted of inanimate training on a da Vinci Si surgical system. Each group underwent baseline suturing evaluation, then trained on the Simbionix™ Suturing Module (SSM) or undertook suturing practice using the da Vinci Surgical System in a dry lab. Final suturing performance was evaluated using the objective suture scoring method. Participants in the VR simulation group were surveyed to assess the face and content validity of the SSM. Both groups experienced significant improvement after training (VR simulation group p = 0.0078; dry lab group p = 0.0039). There was no significant difference in improvement between the two groups after undergoing training with either SSM or in the dry lab. Improvements in composite timing scores were 123 and 172 in the VR simulation and dry lab test groups, respectively (p = 0.36). Face validation varied with respect to the category assessed, but participants confirmed content validity of the SSM in all categories. In this sample of novice operators, there was no significant advantage in training with VR simulation using the SSM over dry lab training in improving suturing performance. Users of the SSM found it useful and relevant as a training tool for improving suturing performance.