Share

Observational Studies

10. Statistical Conclusion Validity

Most data are generated in part by chance processes. It is impossible to know in advance exactly what the data will show, and whether, if the data were generated a second time, they will differ at least a bit in ways that can not be anticipated. This uncertainty in the data means that any summary statistics computed and any conclusions that follow will be subject to uncertainty as well.

Example 6

Conducting survey interviews can be especially problematic in inner city areas. Residents may be suspicious of outsiders and even hostile if they suspect that they are just being used to advanced some purely academic agenda. Holbrook and her colleagues (2006) studied the use of indigenous interviewers in such circumstances. A key issue was the quality of the data compared to data collected by professional interviewers. To this end, they compared summary statistics from survey interviews conducted by local residents to those from survey interviews conducted by professional interviewers. They also built regression models to characterize how the backgrounds of interviewers might affect respondent answers to sensitive questions. The authors do not discuss how they conceptualized the sources of uncertainty. Well over 70 hypothesis tests were conducted. No discounting of p-values is mentioned. Moreover, p-values are typically not reported. Rather one is given the popular "starsystem:" one star for p < .10, two stars for p < .05, and three stars for p < .01.