Posted by charles on .
You think that the best sampling strategy is stratified sampling. Youd like to list the characteristics of schools in this district, then randomly select 80 schools who roughly match the demographic characteristics of the entire population of schools in this district. 40 of these would come from schools that had the experimental curriculum and 40 would come from schools that kept the old curriculum.
However, people from the school board have made the following statements regarding sampling. One well meaning school board member has argued that ""because we are interested in how successful the curriculum is, we should randomly sample 80 schools with the experimental curriculum."" The school board member who represents a middle class neighborhood has said that ""These sorts of exercises always focus on improving schools in poor areas, but what is most important is how these reforms affect the typical middle class kid at the typical middle class school. You should only sample schools whose average family income is close to the overall district-wide average."" Finally, one school board member represents a district where a number of textbook publishers are located. This member has told you ""if the new curriculum is adopted then the textbook producers in the city will benefit by getting tons of orders for new books. You need to think about the greater good here; adopting the new curriculum would be a boon to the city in these hard financial times. You should sample the best performing schools with the new curriculum and the worst performing schools with the old curriculum in the sample...the bigger we can make the impact look, the greater the support will be to adopt the new curriculum district wide.""
Describe what is wrong methodologically with each of the three suggestions you received from the various school board members. You will want to focus on issues such as sampling error, the sorts of biases that will be introduced from such sampling methods, how you might expect to see those biases manifested in the data/data analysis, and how these issues will affect your ability to comment on the district's original question as to whether or not it would be a good idea to roll out the new curriculum to all schools in the district.