I have a quick question for you all. I have done a repeated measures ANOVA for hypothetical data in my Stats class. There were 5 subjects.. all subjects rated their mood on a cloudy day, then they all rated their mood on a sunny day. In sum.. they all underwent the same order of reporting mood.
I found a significant effect of weather. Where I'm stuck is where my prof asked me "how sure can i be of my results?" I know drawbacks to repeated measures designs, but I'm not sure what to say beyond the usual "There could be spillover" problem. Thanks so much for any help you can offer!
My knee jerk reaction is to ask, "at what level of significance?" At the 10% level? at the 5% level? at the 1% level.
In other words, with respect to sampling "error" only, how sure are you of your results.
Your "spillover" comment suggest that your prof may be asking about non-sampling error; a very legitimate question, but hard to answer.
Thanks so much for responding. I think it was actually at the 2% level of significance.
The only thing my professor is askig is just generally "how sure can I be of my results", and he said just talk about drawbacks of the design... so I guess (hopefully) I'm on the right track.
Five subjects is a very small sample. How many measurements did you have on each subject? How did you control other factors that might effect mood (e.g., restrictions that might occur on cluody/stormy days)?
I'm not sure what you mean by "spillover," but do you know if the order of sunny and cloudy days had any effect?
Did you actually calculate the level of significance or is that just a personal estimate? ("I think it was....")
Possibly answering some of these questions will help you to estimate your sureness of your results and the conclusions they might indicate.
I hope this helps a little more. Thanks for asking.
To determine the level of confidence or how sure you can be of your results, you need to consider both sampling error and non-sampling error.
Sampling error refers to the variability in your sample that arises due to chance. In your case, with a repeated measures design and 5 subjects, the sample size is quite small. The smaller the sample size, the larger the sampling error and the less reliable your results may be. To get an idea of the sampling error, you can calculate the standard error of the mean (SEM) for your data. The SEM represents the average amount of variability you would expect across repeated samples of the same size. The smaller the SEM, the more precise your estimate of the population mean.
Non-sampling error, on the other hand, refers to other factors that can affect your results beyond chance. In a repeated measures design, one potential source of non-sampling error is the order of conditions (in this case, sunny and cloudy days) and its potential effect on mood ratings. If the order of conditions had an effect on mood, it could confound your results, making it difficult to attribute the observed differences solely to the weather conditions. To evaluate the potential impact of non-sampling error, you can consider conducting a counterbalancing procedure, where you randomize the order of conditions across participants to minimize any order effects.
Additionally, you should consider other factors that could influence mood ratings, such as time of day, personal factors, or any restrictions on certain weather conditions. Controlling for these factors can help increase the internal validity of your study.
Finally, it's important to calculate the actual level of significance based on your data rather than relying on personal estimates. This can be done by conducting a statistical test, such as a repeated measures analysis of variance (ANOVA), and calculating the p-value associated with the observed effect. A p-value represents the probability of obtaining the observed effect (or a more extreme effect) under the assumption that the null hypothesis (no effect of weather) is true. The smaller the p-value, the stronger the evidence against the null hypothesis and the more confident you can be in your results.
In summary, to determine how sure you can be of your results, you should consider the sampling error by calculating the standard error of the mean, address potential non-sampling error by controlling for confounding factors and order effects, and calculate the actual level of significance using a statistical test.