1  Background & Motivation for Causal Inference in Service Evaluations

When performing evaluations of healthcare services or interventions in the NHS, our key evaluation question of interest is almost always causal in nature:

Yet many of the methods currently used to evaluate interventions within the NHS are often descriptive in nature. Dashboards are frequently used in this manner, where any changes in an outcome of interest are fully attributed to a single change in implementation in services, without considering whether there are alternative explanations and/or other confounding factors are involved that could instead explain the change instead.

To better illustrate this point, Tyler Vigen publishes great examples of spurious correlations on his webpage (https://tylervigen.com/spurious-correlations), from which I’ve attached an example of below (Vigen 2024):

As you can see, butter consumption is positively correlated with the economic output of the Washington metro area. Needless to say, I highly doubt butter consumption was actually “greasing the wheels” of the local economy in the Washington metro area! It is therefore imperative that we don’t conflate association with causation when we’re conducting service evaluation as well.

Many books, papers, and reports have been published providing guidance on how one can estimate the causal effect of different interventions have on specific outcomes, in the absence of a randomised trial. Some resources have even been developed to help narrow down which types of methods would be most appropriate depending on the context. However, much of the guidance has been limited to methods originating from a specific field (e.g., econometrics, social sciences, epidemiology), rather than encompassing all approaches relevant to service and intervention evaluation within the realm of health and social care.

This handbook aims to bridge this gap in the literature, and establish new unifying guidance for selecting appropriate study designs and analytical methodologies for conducting causal inference in service evaluations. You’ll find:

  1. A decision tree diagram to aid with method selection
  2. Concise overviews of each approach
  3. Pointers to more in-depth resources for technical details about each method (including technical explanations in textbooks, applied examples in the literature, and practical coding exercises)

Do note that this handbook itself will not be going into too much technical detail about each analytical method, but instead serves as a resource to guide analysts in choosing the appropriate to use for their question.

I would also like to note that this handbook will primarily focus largely on established statistical methods for causal inference. More novel and experimental machine learning based methods do exist for causal inference. However, most of these methods require additional computational power, have components that are more opaque (akin to a black box), and many of these still haven’t been extensively used in the literature yet. For the purposes of keeping this guide as approachable as possible, and our methods and assumptions as transparent as we can, this is the focus we have chosen for this handbook. An extensions section will be included at the end for those who may be interested in further delving into novel machine methods as well.