5. Designing a Sample
Scientific surveys should be based on probability sampling from clearly specified study populations. Probability sampling provides the statistical framework for making credible inferences about the study population.
In a probabilty sample, all units in the population have a positive probability of being selected. That is, no part of the population is left out of the sampling process. In addition, the probability of selecting a sampled unit is known, which is necessary for making valid estimates from survey data.
Although many people think of random sampling as haphazard, it involves well-defined rules for selecting individuals in the population. The random aspect of probability samples has to do with the units that are chosen to be included in the sample.
In contrast, other sampling methods involve personal judgment or a tendency to recruit readily available units. Selection biases associated with non-probabilty approaches lead to biased estimates, and there is no valid method for calculating standard errors. Designs that should be avoided include:
- Purposive samples, in which the investigator personally selects specific sample units without a chance mechanism;
- Convenience samples such as intercept surveys, in which persons walking by are asked to complete a survey;
- Quota samples, in which data collection within a group is halted once the number of responses for the group reaches a specific quota level; and
- Other non-probability sampling methods such as respondent-driven sampling that do not rely on a formal probability sampling mechanism.