In this talk, Dr. Momin Malik will connect two sets of topics: first, they discuss precedents for quantitative practice that reject the mythologies of quantification as objective, neutral, or universal. If we understand quantitative methods as subjective, contextual, and contingent, is their use ever justifiable over more humanistic qualitative methods, and if so, when and how? This draws on Malik's work on revisiting Phil Agre’s idea of “critical technical practice” as a form of critical consciousness, and applying this to ideas of “socio-technical integrations”.

Second, Malik shows how applying some of these perspectives can help guide how to quantitatively investigate proposed healthcare AI classifiers. By theorizing the nature of machine learning models in terms of statistical abstractions and correlations, we can better understand how and why a model that performs well in retrospective data may fail to generalize; or, a model may fail to generalize to other settings; or, models that actually do perform well may fail to positively impact practice or improve patient outcomes; or, models with an initial positive impact might start to drift (or seem to drift) over time.

Their work here involves study designs and setups for monitoring models after deployment to properly capture such potential failures. This includes silent evaluation and confirmatory testing for random subsets of predicted negatives, porting lessons about forms of sampling bias from diagnostic testing, and uncertainty quantification for power calculations and hypothesis testing for machine learning “fairness”. Malik will also present preliminary work on modeling the proximal causes of potential unfairness. While such quantitative work might ultimately not go beyond “strategic positivism”, Malik nevertheless argues for possibilities and potential in hybridizing critical analysis and quantitative action.

Malik is an assistant professor of biomedical informatics and an associate consultant in pediatric research at the Mayo Clinic in Rochester, MN. He also serves as an instructor at the University of Pennsylvania’s School of Social Policy & Practice and a senior investigator at the Institute in Critical Quantitative, Computational, & Mixed Methodologies at Johns Hopkins University. He holds a Ph.D. in Societal Computing and a Master’s in Machine Learning from Carnegie Mellon University, as well as a master’s from the Oxford Internet Institute and an undergraduate degree in history of science from Harvard University.

Filter by

Search Results

Anti-positivist data science, and applications to healthcare AI

Get the latest news