“Black Box Measures? The Challenges for Studying Users’ Algorithm Skills”

Considerable scholarship has established that algorithms are an increasingly important part of what information people encounter in everyday life. Much less work has focused on studying users’ experiences with, understandings of, and attitudes toward how algorithms may influence what they see and do. The dearth of research on this topic may be in part due to the difficulty of studying a subject about which there is no ground truth given that details about algorithms are proprietary and rarely made public. In this talk, I will discuss the methodological challenges of studying people’s algorithm skills to shed light on the special considerations required when researching a topic about which even scholars possess limited know-how. I will draw on our interviews with 83 Internet users in five countries and will advocate for more such scholarship to accompany existing system-level analyses of algorithms’ social implications.