I'm confused about significance tests. I've tried to look up what it means for the p to be less than the alpha but none of the explanations made sense. I understand that the alpha is the probability of wrongly rejecting the null hypothesis but I'm not sure about the t and p that must also be calculated. I don't understand why the alpha must be picked arbitrarily and why the calculated probability must be less than the value of alpha in order for the null hypothesis to be rejected. Also, why are significance levels complements of confidence levels? Please help me understand. Thank you!

The p ≤ .01 (or some other value) is the criteria (cutting line) that you use to reject your null hypothesis. This means that the results you found would happen by chance only or less than 1 time in 100, and you are assuming that this is NOT that one time. However, the probability is still there, and you could be wrong (alpha error).

This is why experiments are replicated. Let's say that you have done the experiment three times and rejected the null hypothesis each time at P ≤. 01. What is the probability that you would have gotten the same results at that level all three times? It is .01^3 = .000001 = one time in a million. Although there might be one chance in a million of alpha error, I would be VERY confident in rejecting the null hypothesis.

Confidence levels indicate your confidence in an estimate (e.g., mean). 99% confidence level from a sample indicates that you are 99% sure from the sample that the true mean lies within that interval. (The larger the sample, the smaller — more specific — your estimate of the population mean will be.) However, there is still a 1% chance that the true mean will be outside that interval, but we assume that this is NOT the case.

I hope this helps you understand.

Let me see if I understand this:

----------------------------------------

null hypothesis: an observed difference is only due to chance - not because a particular variable affects another one

alternate hypothesis: an observed difference is due to one variable affecting another one, not because of chance

p value: the probability of getting a particular value assuming that the null hypothesis is true (if it's due to chance, then what is that chance)

If the p value is sufficiently low, that means we can reject the null hypothesis because the likely hood of getting a certain value merely by chance is so small that we can assume the null hypothesis was simply false to begin with.

----------------------------------------

^ Is all of that correct?

Sounds good!

Significance tests are statistical procedures used to determine if there is enough evidence to reject a null hypothesis and support an alternative hypothesis. The p-value is a key concept in significance tests. Let's break down your questions one by one and explain them.

1. What does it mean for the p to be less than the alpha?
The p-value is the probability of obtaining the observed data (or more extreme) under the assumption that the null hypothesis is true. Alpha (α) is the significance level, which is a threshold set by the researcher to determine the level of evidence needed to reject the null hypothesis. If the p-value is less than the chosen alpha level, it suggests that the observed data is unlikely to have occurred by chance alone, and we have enough evidence to reject the null hypothesis.

2. Why must alpha be chosen arbitrarily?
Alpha is selected by the researcher based on the desired level of confidence in the results. It is a subjective decision influenced by factors such as the consequences of making a Type I error (rejecting a true null hypothesis) and the need for statistical power. Commonly used alpha values are 0.05 and 0.01, but other values can be chosen depending on the study's requirements.

3. Why do we need to calculate t and p-values?
In significance tests, different statistical tests are used depending on the nature of the data and the research question. The t-value is commonly used in tests involving small samples or when parameters (such as population means) are unknown. The t-value measures the difference between the observed sample statistic and what is expected under the null hypothesis, taking into account the variability of the data.

The p-value is the probability of obtaining a test statistic as extreme as the observed value, assuming the null hypothesis is true. It quantifies the strength of evidence against the null hypothesis. By comparison, if the observed data is highly unlikely when the null hypothesis is true (i.e., small p-value), it suggests that the alternative hypothesis is more plausible.

4. Why are significance levels complements of confidence levels?
Significance levels and confidence levels are related but conceptually opposite. The significance level (α) measures the researcher's willingness to make a Type I error (rejecting a true null hypothesis). It is the probability of rejecting the null hypothesis when it is true.

On the other hand, confidence levels are used in estimating population parameters from a sample. A 95% confidence level implies that if we were to replicate the study many times and construct confidence intervals, about 95% of these intervals would contain the true population parameter.

Thus, a common choice for alpha is 0.05, which corresponds to a 95% confidence level. By rejecting the null hypothesis at the 0.05 significance level, we are saying that we have 95% confidence in the alternative hypothesis being true.

In summary, significance tests help researchers make decisions based on evidence from data. The p-value indicates the strength of evidence against the null hypothesis, and the alpha level sets the threshold for accepting or rejecting the null hypothesis.