Sample Surveys

4. Total Survey Error

Total survey error is a conceptual framework used to systematically consider types of survey error during the design process and in describing its quality when completed.

Sampling Error

The most familiar form of error is sampling error. Sampling error represents the uncertainty in estimates that occurs because we observe data on a sample of individuals in the population rather than on every individual. We minimize sampling error by designing samples to provide the most precise estimates given resources. Sampling error is often expressed as standard errors or margins of error (e.g., a 95% confidence interval) for estimates, although these measures can be affected by other sources of error such as measurement error. Other forms of survey error are called nonsampling error, which can lead to bias or reduced precision in estimates.

Coverage Error

A common problem in studies of human populations is coverage error. Coverage error is the bias that occurs when we are unable to select the sample from the whole study population. For example, a telephone survey of households selected from white pages listings will not include households with no phone, households with unlisted land line numbers, or cell phone-only households. It is difficult to quantify coverage error without special studies of the unsampled portion of the population.

Nonresponse Error

Nonresponse error is the bias that occurs when some of the sampled individuals cannot be contacted or decline to provide a response to the survey and they differ in important ways from the respondents. Response rates have traditionally been used to indicate the potential for bias. There is, however, increasing evidence that response rates do not necessarily provide an accurate indicator of bias. In particular, response rates by themselves do not indicate how respondents and nonrespondents are different from each other as groups. Other methods for evaluating the potential effect of nonresponse include comparing sample composition to known population demographics, conducting follow-up studies of nonrespondents, and studying how mean responses change in relation to the effort required to obtain the response.