Over 100,000 abdominal surgeries are performed annually in the Netherlands, often robot-assisted. To enhance surgeon performance, intraoperative videos will be augmented with preoperative imaging and AI-based digital twins. This improves orientation, reduces collateral damage, increases tumor resection rates, and shortens surgery and training time, leading to better outcomes and broader access to robot-assisted surgery.

Data preparation

It is important to mark organs and cancerous tissue in medical images (like CT and MRI scans) before surgery. It is also important to mark certain key points (called landmarks). This helps check if the digital twin (a digital copy of the patient) is accurate. These marked images are also needed to train the artificial intelligence (AI) part of the digital twin. This results in complete datasets with marked CT and MRI scans before surgery, and marked key points in videos taken during surgery.

Development of digital twin for localizing malignant tissue.

To enhance the surgeons’ performance, we will augment the intraoperative video with information obtained from preoperative images. This will facilitate orientation and identification of anatomic structures. To align the preoperative images with the inter-operative video, a digital twin, which uses artificial intelligence and classical modelling to create a virtual copy of the patient.

Validation of real-time overlay of the location of malignant tissue

We hypothesize that additional information may reduce intraoperative collateral damage, enable higher rates of radical tumor resection, and reduce operation time, thus improving patient outcomes and survival. Visual feedback might reduce the duration of surgical training resulting in increased access to robot-assisted surgery.