In 2024, the Johns Hopkins Data Science and AI Institute awarded two Malone Center members the first of its Trusted Dataset Awards, providing them the opportunity to further DSAI’s mission to establish Johns Hopkins as a premier source of trusted datasets for research, scholarship, and translation.

“A principal goal of the Trusted Datasets Awards was to begin building a body of tools and datasets broadly useful for AI researchers with a particular focus on the data,” says Jim Bellingham, DSAI’s research director for safety and assurance. “We are excited by the results of this first round of research, the first of what we hope to be many contributions to safer, more trustworthy AI.”

Mathias Unberath, a John C. Malone Associate Professor of Computer Science, received one of the awards to support his project, “A Trusted Living Dataset and Compute Infrastructure for Ambient Intelligence in JHM Operating Rooms.” Through this work, Unberath and collaborators in the Johns Hopkins Whiting School of Engineering, School of Medicine, Applied Physics Laboratory, and Armstrong Institute for Patient Safety and Quality have created a trusted living dataset and developed sophisticated algorithms to advance ambient intelligence within the Johns Hopkins community.

The research team generated more than 50 videos that capture simulated operating room workflows, including OR staff reenacting surgical procedures. Using these, the team is developing a suite of tools and algorithms that support the detection of humans and objects in the OR. These toolsets, videos, and annotations comprise a dataset and infrastructure that can advance research in ambient AI for surgical environments. In compiling this dataset, Unberath’s team developed a new protocol for annotating surgical videos, addressing limitations in previous related work. The new protocol highlights the need for a pre-specified annotation vocabulary and dictionary, quality assurance, and control measures.

Additionally, Unberath and his team are working with Information Technology at Johns Hopkins to establish a multi-tiered storage and compute access framework that is HIPAA-compliant to protect sensitive patient health information. The goal is to make the dataset securely accessible through Johns Hopkins platforms to enable compliant and efficient model development and training.

Together, the team’s videos, annotations, and toolsets comprise a one-of-a-kind trusted dataset and infrastructure poised to significantly advance research in ambient AI for surgical environments.

“The Trusted Dataset Award allowed us to advance our work on ambient AI for operating room analytics, building our platforms for the secure handling of OR video data captured from both room-level and egocentric camera video feeds, as well as algorithms that handle de-identification among other processing steps,” says Unberath. “We will continue this line of work beyond the Trusted Dataset performance period, further enhancing and expanding the platform and building algorithms for automated data analytics to enhance OR efficiency.”

The second Trusted Dataset Award was presented to Swaroop Vedula, an associate research professor in the Malone Center, for his work, “Datasets to Enable Trusted and Generalizable Artificial Intelligence for High-Stakes Assessment of Surgeons and Autonomous Feedback for Surgeons.” With support from the award, Vedula and his collaborators developed prototypes of resources that allow surgeons and engineers to create datasets for trustworthy surgical AI.

Vedula shares that while using AI for surgery is predicted to rapidly expand and have a significant impact on the field, the current paradigm is to create large datasets within silos defined by investigators or institutions.

“These data silos are necessary to protect patient privacy and confidentiality, but they stifle innovation using large datasets needed to train AI models. Our work creates the building blocks that make the silos transparent while protecting patient privacy and confidentiality. This transparency enables investigators to use all relevant data to develop AI models, which would otherwise not be possible,” explains Vedula.

His team established protocols for creating and sharing trusted annotations of surgical videos, created annotations for a public video dataset focused on cataract surgery, and developed a metadata model for datasets used in AI for surgery. Vedula is now leading the creation of a multi-institutional, multidisciplinary collaboration to complete this work and build a lasting community to sustain the effort and to assure its impact on AI innovation for surgery.

“Rapid advances in AI and sensing technologies promise to make surgery more safe, effective, and personalized to each patient. We must know what data exists so we can build the best AI using all available data,” Vedula says. “Our work is important because it allows us to know what data is available to develop AI models and to make the data interoperable. It also allows us to accelerate innovation, trusted evaluation, and translation of AI for surgery.”

Learn more about the Data Science and AI Institute Trusted Dataset Awards here.

Image Caption: Mathias Unberath and Swaroop Vedula