Share

Observational Studies

4. External Validity

Credible Generalizations

The next topic to be considered is where the data come from. Answers to that question will indicate the kinds of generalizations that can be made and the obstacles to be faced making any inferences beyond the data on hand.

Example 2

When troublesome adolescents are sent to special alternative schools where strict discipline is enforced, does the experience increase or decrease the likelihood of subsequent success in regular public schools? Wolf and Wolf (2008) report the results of a program meant to break the "school to prison pipeline" for a number of school-aged children from the Syracuse (New York) city school district. The program had seven components:

  1. Systematic support for students anticipating a transition from special alternative schools to mainstream schools;
  2. Out-of-school activities meant to improve social and academic skills;
  3. Promotion of "bonding" between the youth and "caring" adults.
  4. Counseling and referrals;
  5. Support groups for students with incarcerated loved ones.
  6. Regular contact with parents; and
  7. Collaborative training for teachers, administrators, and other school staff.

The students in the study were essentially a convenience sample of all students in the Syracuse city school district, and Syracuse itself is a convenience sample of large cities in the United States. Yet, the observational study was motivated by the desire to learn about the efficacy of such interventions in general. What would be the point of learning whether the intervention worked for the several hundred students in the study unless that information could inform future policy decisions? In this instance, the intervention did not seem to produce beneficial effects overall and may even have produced some undesirable effects. But what kinds of generalizations can be properly justified?