Monitoring the safety and effectiveness of medical technologies is a key challenge--and only increases as treatments become more complex. Laura Hatfield, along with Christine Baugh, Vanessa Azzone, and Sharon-Lise Normand, recently developed new methods for improving the safety decision-making process. The research was published in Medical Decision Making.

Officials charged with monitoring the safety of medical devices need actionable data on safety issues that arise in clinical care. For example, hospitals report on rates of wrong-site surgeries and hospital-acquired infections. When these data begin to show signs of a safety problem, decision-makers face a choice: are the costs of taking action in response to a possible safety issue worth the potential benefits to patients?

Most statistical methods only address the strength and reliability of evidence for a safety problem. This research formalizes the process of choosing an action by trading off risks and benefits in response to a possible safety problem.

In this study, the authors used a nationwide database of pediatric hospital admissions. For each hospital, they counted how many admissions involved a problem with a medical device, then computed the rate of these problematic admissions out of all pediatric admissions in that hospital in that year.

Using a statistical model fit to the hospitals’ rates from 2000 to 2010, they produced a target expected rate of problems for each hospital in 2011 and compared the actual performance in 2011 to the target; the difference is called a “safety signal.” If a hospital is performing much worse than its target, it may be evidence of a safety problem at the hospital.

Next the researchers specified a set of actions that a hypothetical safety official might take, ranging from simply collecting more data all the way to shutting down the hospital. For each action, they quantified the costs and benefits of taking that action on a hospital with a given safety signal. A mathematical function called a “loss function” described these trade-offs numerically.

Finally, using a statistical framework called Bayesian decision theory, the authors combined the safety signals with the loss functions to determine the optimal action for each hospital. For most hospitals, the optimal action was doing nothing at all, because their rates of problematic admissions were close to the target. For a few hospitals, the model indicated that the best action is to collect more data or send in safety remediation teams.

The researchers show that their method produces better decisions than approaches that do not account for the costs and benefits of each action. Hatfield explained, “This paper provides a proof of concept for these methods. The really exciting promise of the work is to ask safety officials to tell us how they weigh various factors when making decisions in the real world.”

This work was a project of the MDEpiNet Methodology Center. 

Enjoy content like this? Sign up for our newsletter and follow HCP on Twitter.