Mind the Gap: Methods to Evaluate the Quality of Clinical Care

The evaluation of clinical care is fundamentally constrained by the need for large, labeled datasets—resources that range from expensive to impossible to obtain. This talk introduces methods to evaluate the quality of clinical care by making use of unlabeled data to evaluate decision-making in two directions: looking backward at human clinical judgment, and forward toward algorithmic systems. In the first part, I focus on retrospective evaluation of past diagnostic decisions, and present a framework to measure the rate of missed diagnoses in the health record. In the second part, I turn to prospective evaluation of algorithmic decision-making in clinical care, and introduce a method to assess the performance of clinical predictive models in the absence of abundant labeled data. Together, these projects highlight a shared challenge in both human and algorithmic decision-making – evaluating decision quality under data constraints – and provide steps towards robust evaluation of both human and algorithmic care.

Zoom Link

Image

Divya Shanmugam is a postdoctoral researcher at Cornell Tech, where she works on developing data-efficient methods for reliable machine learning. Her work is often inspired by the real-world challenges involved in balancing the potential benefits of machine learning against the potential harms of automated systems in high-stakes settings. Divya earned her Ph.D. and B.S. in Computer Science from MIT.