Appropriate Research Methods

10. Understanding What Works

Quite often, egregious methodologic errors result from confusing an upstream unit of random assignment (such as a community or school) with a downstream unit of analysis (such as an individual student).

Generally speaking, quantitative methods tend to be employed exclusively to measure outcomes of downstream interventions, where individuals are the unit of analysis. These individual-level experiments could almost certainly benefit from judicious integration of appropriate qualitative methods (e.g., focus groups to optimize the intervention). As one moves upstream, the utility of quantitative methods becomes problematic, not because they are intrinsically defective or flawed, but because the phenomena to which they are applied (the units of investigation) are of a qualitatively different type. This is demonstrated by the typology presented in Figure 2. Rigorous experimental control and manipulation are not always possible at the level of sociopolitical intervention, especially when change is unexpected or unplanned. Thus, different design approaches, measurements, and data collection techniques must be employed.

When an intervention program is applied to an aggregate unit (community, school, worksite) and the analysis is based on individual level observations, the residual error is deflated by intracluster correlation and leads to overstatement of the statistical significance, not to mention the more important problem of measuring the wrong outcome. Downstream approaches to assess the effectiveness of interventions for individuals (usually patients) required the randomization of individuals. Individual-level randomized controlled trials (RCTs) remain the principal means to determine the effectiveness and safety of therapeutic interventions and they are discussed in Clinical Trials by Duolao Wang (Senior Lecturer in Medical Statistics, London School of Hygiene and Tropical medicine and Ameet Bakhai (consultant cardiologist and physician at Barnet General & Royal Free Hospitals, London, UK).

Moving upstream to organizations (like schools and factories) or to neighborhoods or communities requires cluster or group randomized trials, where social entities are randomly assigned to receive or not receive some intervention. The challenges and potential of these cluster trials are discussed in the chapter Cluster Unit Randomized Trials by Allan Donner (Professor, University of Western Ontario). Renewed conceptual clarity on different levels of analysis (individual biophysiologic processes, life style influences, environmental factors and the role of geographic location (and the different interventions required at these different levels) has ushered in renewed interest in multi-level modeling and this is described by Dr. SV Subramanian (Professor, Harvard School of Public Health) in the Multilevel Modeling chapter.