Bay Area Tech Economics Seminar with Nathan Kallus, Cornell/Netflix

Speaker: Nathan Kallus, Associate Professor, Cornell Tech, Cornell University; Research Director, Machine Learning & Inference, Netflix

Presentation Title: Learning about Tradeoffs and Proxies from Historical A/B Tests

Abstract: Experimentation on digital platforms often faces a dilemma: we want to experiment rapidly at scale, but we also want to make decisions based on long-term metrics that are both insensitive and take too long to observe. Usually, one resorts to looking at short-term outcomes like engagement. However, with increasingly complex platforms, engagement can have many modalities (e.g., video vs. games) and qualities (e.g., total time vs title completions). So a key question is how these different objectives trade off and how to combine them into the best proxy metric for one's north star. There are many pitfalls in doing this using observational data on users. We will discuss how to use meta-analysis of past experiments to answer this question in ways that are robust to confounding between short- and long-term and to unmediated effects and the impact of doing this at Netflix.

The logistics details for each session will be provided to registered participants.  Sign up here to be included on the mailing list and learn about future events.

This talk is co-sponsored by USF's Master's in Applied Economics and the Stanford Causal Science Center. For additional information and abstracts from past talks, please click here.