Explanation, prediction, replication, application

This is a comment on applied psychology. It is a comment inspired on the one hand by a few very optimistic statements about the good that psychology is doing in the world, that appeared recently in Perspectives on Psychological Science, and on the other hand by a decidedly more pessimistic view to be published soon in the same journal, as well as by a problem in what you could call the ‘conceptual replication’ position in the current crisis debate in psychology.

In July, Perspectives published a ‘Special Symposium on the Future Direction of Psychological Science’, and naturally most of the papers began by reflecting on the current state of the discipline. What struck me was that quite a few of the authors were very positive, almost glowing about psychology’s contributions to society. Susan Fiske wrote that”(a)s a field, we are everywhere in policy advice” (2017, p. 652) producing reports on topics ranging from safe sunbathing to national security and counterfeiting. Likewise, Carol Dweck rejoiced that psychology’s passion for addressing social issues “has earned our field a seat at decision-making tables around the world, advising world leaders, economic councils, and ministries of health and education.” (Dweck, 2017, p. 657) Diane Halpern (2017) and Ethan Kross (2017) presented similar optimistic assessments of psychology’s social impact.

Sitting at the decision-making table is one thing, but how did those psychologically improved decisions work out? There is little to nothing in these four papers about the effects of psychological interventions. There is a lot of ‘working towards …’, ‘beginning to …’, and similar hedging of the difference that psychology makes in practice, but no concrete success stories. It is good to know that government is becoming more evidence-based, but one would also like to see the evidence for the efficacy of psychological interventions. Especially since there are reasons to be more pessimistic of psychology’s technological prowess as well.

For instance, Tal Yarkoni and Jacob Westfall have a paper in Perspectives (online first) in which they argue that “psychology’s emphasis on explaining the causes of behavior has led to a near-exclusive focus on developing mechanistic models of cognition that hold theoretical appeal but rarely display a meaningful capacity to predict future behavior.” (Yarkoni & Westfall, 2017, p. 2) Models in psychology are typically overfitted: they predict the behaviour of the sample that they’re based on very well, but fail badly for new samples. The problem is compounded by p-hacking.

As a result, we now have in psychology “(e)laborate theories seemingly supported by statistical analysis in one dataset [that] routinely fail to generalize to slightly different situations or even to new samples putatively drawn from the same underlying population.” (Yarkoni & Westfall, 2017, p. 8) And those different situations of course include applied contexts. The focus on underlying mechanisms of behaviour – psychology as a basic science – leads to statistical strategies that produce overfitted models with little predictive success, that are of little use for developing interventions. Yarkoni and Westfall only briefly touch on application, but this is what their analysis implies.

Explanation and prediction are also at issue in the replication debate. There are a number of (social) psychologists who argue that in their field the focus should be on conceptual rather than direct replications, firstly because their main aim is to develop explanatory theories of social behaviour, and secondly because social behaviour is inherently difficult to predict. To develop and refine explanatory theories you must do conceptual replications, testing the same theory in different ways. Direct replication is no use, because social behaviour is far too sensitive to context. What worked with one sample may not work with another sample, at another time, in another country. In social psychology, “one can never step in the same river twice” (Crandall & Sherman, 2016, p. 94).

Further on in the same article, Crandall and Sherman add a brief aside: in the context of practical application, “careful attention to exact replication is essential” (2016, p. 97), because then we want to know whether a particular intervention reliably produces the same effect. But then, half a page further down, without noting any problem, Crandall and Sherman repeat that “direct replication in social psychology is practically impossible” (2016, p. 97).

A similarly puzzling position was defended a few years ago by Wolfgang Stroebe and Fritz Strack (2014), who also argued that direct replications are of little use in basic research, because it’s about the theory, not about the specific operations, and because the context sensitivity of social behaviour makes failed direct replications uninformative. They too added that it is different in applied research: “A scientist who wants to establish the efficiency of a specific treatment or intervention is well advised to repeatedly apply exactly the same procedure.” (Stroebe & Strack, 2014, p. 60)

And here is the thing: if direct replication is so difficult in social psychology (practically impossible!), yet essential in an applied context, isn’t that reason to be very cautious about the application of social psychological theories? And isn’t it cause for skepticism about the celebrated contributions of all those psychologists sitting at the decision making tables? If we cannot and should not expect the same experimental procedure to produce the same result under controlled laboratory conditions, why should we trust a psychological intervention to be effective in the hustle and bustle of daily life? What’s more, precisely psychology’s focus on basic research, on explanation, should make us wary about its practical use, as Yarkoni and Westfall showed.

Finally, here is another possible conclusion from the same arguments: to the extent that psychological interventions are effective, helpful contributions to the solution of social and individual problems, their succes may not primarily be due to basic, explanatory theory, but to other factors. To the people performing and delivering those interventions, would be my guess.