Observational Studies

6. Measurement Validity

Measurement validity begins with an assessment of how the target of the measurement is conceptualized. One problem can be "over-coverage." Another problem can be "under-coverage."

In the case of IQ, if the concept of general cognitive ability does not include a full range of cognitive skills (e.g., musical), there is under-coverage. If the concept of cognitive ability includes attributes that are really culturally based (e.g., vocabulary), there is over-coverage.

In the case study by Rush and his colleagues (2008), the questionnaire items seem intuitively sensible, but no definition of the need for treatment is provided. Consequently, both under-coverage and over-coverage are possible.  For example, all but one of the questionnaire items provide a reference time period (e.g., 90 days). There are good reasons for this from the point of view of item construction. One needs to specify a suitable interval for recall. But are there people in need who will be missed without a longer recall interval? And are there features of need that are not addressed with items (a)-(e), or items that are do not really reflect what might be meant by "need?" If yes, the measure is likely to be systematically inaccurate. Some refer to this as measurement bias.