Measuring Effectiveness to Inform Decision-Makers

To achieve our goals effectively, we often turn to science for advice.  And scientists across many fields—from social science to medicine—test the effectiveness of interventions at achieving our goals, for example, the effectiveness of treatments at curing diseases. When doing so, scientists measure and report how effective these interventions are using effect size measures. While there are many different effect size measures, they all quantify how much an intervention changed an outcome of interest when tested. The resulting effect sizes are often used in evidence-based decision-making, ranging from clinical decisions to policymaking. However, such usage faces a serious and underappreciated problem. Even if accurate, effect sizes omit information that can be vital for rational decision-makers—information about how a tested intervention changed an outcome of interest. I suggest a path forward: using normative decision theory, we can show precisely in which circumstances effect size measures do, and in which they do not, provide all information relevant to rational decision-making. In this talk, I illustrate this approach with one widely used effect size measure: the mean difference. Based on my results, I offer advice for how researchers and decision-makers should use mean differences when aiming to inform and make rational decisions between interventions. More generally, findings on when different effect size measures do and do not provide all decision-relevant information allow us to recognize new avenues for improving evidence-based decision-making. Such findings also suggest that not all effect size measures inform rational decision-making equally well for all people—an observation that may help uncover hitherto hidden forms of social injustices in evidence-based decision-making.

Dr Ina Jäntgen is a postdoctoral fellow at the Munich Center for Mathematical Philosophy (Chair of Philosophy of Science). She specialises in philosophy of science, with a focus on philosophy of the biomedical and social sciences, and in formal epistemology and decision theory. Her research focuses on various topics relevant to how science can best produce knowledge helpful for decision-makers. She recently completed her PhD in Philosophy at the University of Cambridge. Her thesis discusses how scientists could use effect sizes of tested interventions to inform rational decision-making.