From: Siegel, A.F. (1997). Practical Business Statistics, 3rd Edition. Irwin/McGrawHill.

Case - Can This Survey Be Saved?

"What's troubling me is that you can't just pick a new random sample just because somebody didn't like the results of the first survey. Please tell me more about what's been done." Your voice is clear and steady, trying to discover what actually happened and, hopefully, to identify some useful information without the additional expense of a new survey.

"It's not that we didn't like the results of the first survey," responded Steegmans, "it's that only 54% of the membership responded. We hadn't even looked at their planned spending when the decision [to sample again] was made. Since we had (naively) planned on receiving answers from nearly all of the 400 people initially selected, we chose 200 more at random and surveyed them also. That's the second sample." At this point, sensing that there's more to the story, you simply respond "Uh huh . . ." Sure enough, more follows: "Then Eldredge had this great idea of following up on those who didn't respond. We sent them another whole questionnaire, together with a crisp dollar and a letter telling them how important their responses are to the planning of the industry. Worked pretty well. Then, of course, we had to follow up the second sample as well."

"Let me see if I understand," you reply. "You have two samples: one of 400 people and one of 200. For each, you have the initial responses and followup responses. Is that it?"

"Well, yes, but there was also the pilot study - 12 people in offices downstairs and across the street. We'd like to include them with the rest because we worked so hard on that at the start, and it seems a shame to throw them away. But all we really want is to know average spending to within about a hundred dollars."
At this point, you feel that you have enough of the background information to evaluate the situation and to either recommend an estimate or an additional survey. Additional details for the survey of the 8,391 overall membership in order to determine planned spending over the next quarter are provided on the following page.

Discussion Questions

1.Do you agree that drawing a second sample was a good idea?
2.Do you agree that the followup mailings were a good idea?
3.How might you explain differences among averages in the results?
4.Should the pilot data be included? Why or why not?
5.Are any or all of the results here useful? How would you treat the data that has been collected? What additional data collection, if any, would you do?

Pilot
Study First Sample Second Sample Both Samples All Combined

Initial Mailing
Mailed 12 400 200 600 612
Responses 12 216 120 336 348
Average $39,274.89 $3,949.40 $3,795.55 $3,894.45 $5,114.47
Std. Dev. $9,061.91 $849.26 $868.39 $858.02 $6,716.42

Followup Mailing
Mailed 0 184 80 264 264
Responses 0 64 18 82 82
Average $1,238.34 $1,262.34 $1,243.60 $1,243.60
Std. Dev. $153.19 $156.59 $153.29 $153.29

Initial and Followup Mailings Combined
Mailed 12 400 200 600 612
Responses 12 280 138 418 430
Average $39,274.89 $3,329.73 $3,465.13 $3,374.43 $4,376.30
Std. Dev. $9,061.91 $1,364.45 $1,179.50 $1,306.42 $6,229.77

To answer the questions and evaluate the situation, we need to understand the information provided in the case and the data collected from the survey. Let's go through the questions one by one, keeping in mind the available data.

1. Do you agree that drawing a second sample was a good idea?
Drawing a second sample can be a good idea in certain situations. In this case, the reason for drawing a second sample was the low response rate of only 54% in the first survey. However, it is important to note that the second sample was selected without considering the results of the first survey. This means that the second sample may not be representative of the population. In order to assess whether drawing a second sample was a good idea, we need to analyze the characteristics of the two samples and compare them with the overall population.

2. Do you agree that the follow-up mailings were a good idea?
The follow-up mailings were conducted to improve the response rate among those who did not respond to the initial survey. It is generally a good idea to follow up with non-respondents to increase the response rate. However, we need to consider the response rate and how the follow-up mailings affected the composition of the sample. If the response rate improved significantly and the characteristics of the respondents remained representative of the population, then the follow-up mailings may have been effective. Analyzing the response rates and comparing the characteristics of respondents from the initial and follow-up mailings can help assess the effectiveness of the follow-up strategy.

3. How might you explain differences among averages in the results?
Differences among averages can arise due to various factors. In this case, we have different averages for planned spending obtained from different samples and mailings. These differences may be due to sampling error, which means that different samples from the same population can yield slightly different results simply due to chance. Moreover, the pilot study conducted on 12 individuals may not be representative of the larger population, and therefore, the average spending from the pilot study may differ significantly from the averages obtained from larger samples. Additionally, response rate and non-response bias could also contribute to differences in the averages. To explain the differences among averages, it is important to compare the characteristics of the different samples and assess their representativeness.

4. Should the pilot data be included? Why or why not?
The decision of whether to include the pilot data depends on its representativeness and its usefulness in estimating the average spending. Since the pilot study included only 12 individuals from nearby offices, it may not be representative of the overall membership. However, if the pilot data shows consistent patterns with the larger samples, it could be included to provide additional information. It is necessary to compare the characteristics of the pilot group with the larger samples and assess whether their inclusion would impact the accuracy of the estimates.

5. Are any or all of the results here useful? How would you treat the data that has been collected? What additional data collection, if any, would you do?
The usefulness of the results depends on the research objectives and the representativeness of the samples. To treat the data that has been collected, we can calculate the averages and standard deviations for each sample and mailing combination, as shown in the case. We can also compare the response rates and characteristics of respondents across the different samples and mailings. This analysis will help us assess the reliability and representativeness of the results obtained so far.

To determine whether additional data collection is necessary, we need to consider the research objectives and the quality of the existing data. If the existing data is deemed unreliable or unrepresentative, another survey may be required. However, if the existing data is considered reasonably representative and meets the objectives, additional data collection may not be necessary. Analyzing the current data and assessing its suitability for the research objectives will provide insights into whether further data collection is required.