Posted by Sam on .
I'm confused about significance tests. I've tried to look up what it means for the p to be less than the alpha but none of the explanations made sense. I understand that the alpha is the probability of wrongly rejecting the null hypothesis but I'm not sure about the t and p that must also be calculated. I don't understand why the alpha must be picked arbitrarily and why the calculated probability must be less than the value of alpha in order for the null hypothesis to be rejected. Also, why are significance levels complements of confidence levels? Please help me understand. Thank you!

Statistics 
PsyDAG,
The p ≤ .01 (or some other value) is the criteria (cutting line) that you use to reject your null hypothesis. This means that the results you found would happen by chance only or less than 1 time in 100, and you are assuming that this is NOT that one time. However, the probability is still there, and you could be wrong (alpha error).
This is why experiments are replicated. Let's say that you have done the experiment three times and rejected the null hypothesis each time at P ≤. 01. What is the probability that you would have gotten the same results at that level all three times? It is .01^3 = .000001 = one time in a million. Although there might be one chance in a million of alpha error, I would be VERY confident in rejecting the null hypothesis.
Confidence levels indicate your confidence in an estimate (e.g., mean). 99% confidence level from a sample indicates that you are 99% sure from the sample that the true mean lies within that interval. (The larger the sample, the smaller — more specific — your estimate of the population mean will be.) However, there is still a 1% chance that the true mean will be outside that interval, but we assume that this is NOT the case.
I hope this helps you understand. 
Statistics 
Sam,
Let me see if I understand this:

null hypothesis: an observed difference is only due to chance  not because a particular variable affects another one
alternate hypothesis: an observed difference is due to one variable affecting another one, not because of chance
p value: the probability of getting a particular value assuming that the null hypothesis is true (if it's due to chance, then what is that chance)
If the p value is sufficiently low, that means we can reject the null hypothesis because the likely hood of getting a certain value merely by chance is so small that we can assume the null hypothesis was simply false to begin with.

^ Is all of that correct? 
Statistics 
PsyDAG,
Sounds good!