Share

Observational Studies

3. Descriptive Validity

Summaries of the Data

Any evaluation requires summaries of the data, with some summaries more useful than others. Indeed, some summaries can be misleading. Perhaps the most well known example is the manner in which outliers can affect measures of central tendency: the mean, median, or mode. The mean is especially vulnerable. But outliers can dramatically affect many other summary statistics such as the standard deviation, the Pearson correlation coefficient, and the regression coefficient.

Example 1

The research of Galster and Temkin (2004):502 asks, how can one show whether "efforts by government, community development corporations (CDCs), and for-profit developers to revitalize distressed, inner-city neighborhoods make any demonstrable difference? Put differently, can a method be devised for persuasively quantifying the degree to which significant, place-based investments causally contributed to neighborhoods' trajectories, compared to what would have occurred in the absence of interventions."

A central conceptual concern is how to best define the counterfactual. But they also address a number of statistical issues. They favor a pooled cross-section time series design with neighborhoods as the observational units. An important issue is how best to take spatial dependence into account. Other things equal, neighborhoods close by one another will tend to be more alike than neighborhoods farther away. When the authors regress median neighborhood housing prices on a set of predictors, including a binary variable for an intervention, they use the inverse Euclidian distance between neighborhoods to weight their regressions (Galster and Temkin, 2004:516). But is this a good statistical summary of spatial proximity? It assumes that dependence declines linearly with distance and that the decline is smooth. Yet, the quality of neighborhoods can change sharply in just a few blocks, and breaks in neighborhood continuity caused by freeways, parks, and bodies of water can introduce abrupt changes in the degree of dependence.