| Bulletin 40 - February 2004 |
| Survey Methodology, Sample Representativeness, and Accurate Reporting of Population Health Statistics |
| Abstract |
R. Elsbett-Koeppen, RRFSS Project Manager, The Institute for Social Research, York University, Canada
D. Northrup, Associate Director, The Institute for Social Research, York University, Canada
A. Noack, Ph.D. Candidate, Graduate Programme in Sociology, York University, Canada
K. Moran, Epidemiologist, Durham Region Health Department, Ontario, Canada
|
|
Rationale
|
| The collection of survey data is the core of all risk factor surveillance systems. Given the preeminent role of survey methods in conducting these studies, it is critical to assess the integrity of the data that are collected. Declining response rates have been a bane to researchers who rely on survey data to measure behaviour, attitudes and awareness of health-related issues. In order to mitigate against declining response rates as well as increase sample representativeness and the accuracy of population health statistics, better survey research organizations make a larger number of call attempts in order to reach people. This contrasts with other survey providers (mostly commercial) who routinely make only four call attempts.
|
|
Methods
|
Data Sources
The Rapid Risk Factor Surveillance System (RRFSS) Survey is carried out by the Institute for Social Research (ISR) at York University, Toronto, Canada, for a consortium of 21 Health Units and Health Departments (HUs) in the Province of Ontario. Survey respondents are selected using a two-stage probability selection process. Random digit dialing (RDD) procedures are utilized to produce a sample of households in each HU region. Within each household, the interview is completed with a randomly selected adult (18 years of age or older). For the 2002 RRFSS, 22,853 interviews were completed. A sample of the telephone numbers is developed for each wave of data collection and within each wave the sample is subdivided into replicates, each of which is a random sample of the whole.
Data Analysis
The 2002 RRFSS data were used to identify 'easy to reach' (1 to 4 calls), 'not so easy to reach' (5 to 10 call attempts) and 'hard to reach' (11 or more call attempts) respondents. The distributions of the data were analyzed, as well as some associations. Frequencies and cross-tabulations were used to determine if selected health indicators varied by ease of finding the respondent at home. A Negative Binomial Regression model was used to predict the cumulative probability of survey completion for a specific respondent profile (characteristics of an ideal respondent).
|
| Results |
| Cross-tabulations showed no significant differences between the released replicates. No significant differences were found in a comparison of the age distributions of the Ontario population (Statistics Canada, 2001) and the 2002 RRFSS age groups. The distribution for the fully employed respondents indicates the value of increasing the number of call attempts for the fully employed. A base model for the Negative Binomial Regression model analysis of the number of call attempts included gender, age, education, income and being fully employed. All independent variables, except education, were significant. The results indicate that respondents that are hard to reach tend to be male, younger, have a higher household income and are fully employed. Selected health indicators, representing health status as well as some health behaviours, health risks and knowledge of cancer risk factors demonstrate the need of increasing the number of call attempts in order to accurately represent the population.
|
| Discussion |
| Our results stress the importance of making large number of call attempts to insure sample representativeness as this affects our understanding of health issues. To collect data from 'easy to reach' respondents will underestimate, for example, good health. Good survey practice translates into more accurate data and better understanding of the health issues that are critical to health planning and policy development.
Cost implications and speed of data collection are the main reasons why most data collection agencies limit the number of call attempts.
|
| Limitation |
| The 2002 RRFSS data are subject to all of the biases associated with self-report data. The data were not weighted since we did not attempt to calculate population estimates.
|
| Conclusion |
| Data collection agencies that limit call attempts are more likely to ignore a part of the population that is different from the 'easy to reach' respondents in certain health risk behaviours. Consequently, users of surveillance system data must be cognizant of the efforts made by data collection agencies in order to obtain representative samples.
|
| Related Articles or References |
|
Cottler, L., Zipp, J., Robins, L. & Spitznagel, E. (1987). Difficult-to-recruit respondents and their effect on prevalence estimates in an epidemiologic survey. American Journal of Epidemiology, 125:2:329-339.
Kristal, A., White, E., Davis, J., Corycell, G., Raghunathan, T., Kinne, S. & Lin, T. (1993). Effects of enhanced calling efforts on response rates, estimates of health behavior, and costs in a telephone health survey using random-digit dialing. Public Health Reports, 108:3:372-379.
Mishra, S., Dooley, D., Catalano, R. & Serxner, S. (1993). Telephone health surveys: potential bias from noncompletion. American Journal of Public Health, 83:1:94-99.
Northrup, David A, David Bates and Freda Marsden. (1985). "Number of Call Attempts and Sample Representativeness." A paper presented at: The Association of Applied Social Research at the Learned Societies Conference. Guelph, Ontario. May 14-17.
Triplett, T. (1998). Respondents: Snap Choices, By People Who Care.
Turner, H. (1999). Participation Bias in AIDS-Related Telephone Surveys: Results From the National AIDS Behavioral Survey. The Journal of Sex Research, 36:1:52-62.
Voigt, L., Koepsell, T. & Daling, J. (2003). Characteristics of Telephone Survey Respondents According to Willingness to Participate. American Journal of Epidemiology, 157:1:66-73.
|
View the Population Health Statistics poster, or
download (2.5mb, right click to save) a PDF version for printing.
|
|
