Significance tests

Think of a significance test like being a judge in a courtroom. Someone (let's call them the 'new idea' or 'alternative') comes in and says, 'Hey, I think something different is happening!' For example, maybe a company claims their new super-duper battery lasts longer than the old one.

The judge (that's you, doing the significance test) starts by assuming the null hypothesis (pronounced 'nuhl hy-POTH-uh-sis') is true. This is like assuming the person is innocent until proven guilty. In our battery example, the null hypothesis would be: 'The new battery lasts the same amount of time as the old one.' It's the 'nothing new is happening' idea.

Then, you look at the evidence (your data from testing the batteries). If the evidence is really strong and very unlikely to happen if the null hypothesis were true, then you might say, 'Okay, I have enough evidence to reject the null hypothesis!' This means you're concluding that the new battery probably does last longer. If the evidence isn't strong enough, you say, 'I don't have enough evidence to say the new battery is better,' which is like saying 'not guilty' – it doesn't mean the old battery is definitely better, just that there wasn't enough proof for the new one.

Let's say you're a video game developer, and you've just released a new update for your game. You're hoping this update makes more players want to buy extra items in the game. Before the update, about 20% of players would buy extra items.

After the update, you track 100 players, and you find that 28% of them buy extra items. You're super excited! But wait, is this 28% really a sign that your update worked, or could it just be a lucky group of 100 players, and the true percentage of all players buying items is still 20%?

This is where a significance test comes in! You'd set up your judge's mindset:

• Null Hypothesis (H₀): The update had no effect. The true percentage of players buying items is still 20%. (The 'nothing new is happening' idea).
• Alternative Hypothesis (Hₐ): The update did have an effect. The true percentage of players buying items is greater than 20%. (Your 'new idea').

You would then do some calculations (which we'll learn about!) to see how likely it is to get 28% (or even higher) in a sample of 100 players, if the true percentage was still 20%. If it's very, very unlikely, you'd say, 'Aha! I have strong evidence that the update did increase purchases!' If it's fairly likely to happen by chance, you'd say, 'Hmm, not enough evidence yet. Maybe it was just a fluke.'

Performing a significance test is like following a recipe. Here are the main ingredients and steps:

1. State Hypotheses: Clearly write down your null hypothesis (H₀) (the 'nothing new is happening' statement) and your alternative hypothesis (Hₐ) (what you're trying to prove). These are always about the true population parameter (like the true percentage), not your sample.
2. Check Conditions: Before you cook, you need to make sure you have the right ingredients and equipment. For proportions, you need to check if your sample is random, if the sample size is large enough (successes and failures are at least 10), and if the population is much larger than your sample.
3. Calculate Test Statistic: This is a special number that measures how far your sample result is from what you'd expect if the null hypothesis were true. Think of it like a 'weirdness score' for your data.
4. Find the P-value: The P-value is the probability of getting a sample result as extreme as, or more extreme than, what you observed, assuming the null hypothesis is true. It's like asking, 'If the old battery truly lasted the same, how likely is it that I'd see my new battery last this much longer just by chance?'
5. Make a Decision: Compare your P-value to a pre-set significance level (α) (pronounced 'alpha'), which is usually 0.05 (or 5%).
   • If P-value < α: You have strong evidence against H₀. You reject the null hypothesis. This means you believe your alternative hypothesis is true.
   • If P-value ≥ α: You do not have enough evidence against H₀. You fail to reject the null hypothesis. This means you don't have enough proof to say something new is happening.
6. Write a Conclusion in Context: Always explain your decision in plain language, relating it back to the original problem. Don't just say 'reject H₀'; say 'We have convincing evidence that the new battery lasts longer.'

The P-value is probably the trickiest part, but it's super important! Imagine you're playing a game of 'Is this coin fair?' You flip it 10 times and get 8 heads. That seems like a lot of heads, right?

Your null hypothesis (H₀) is: 'The coin is fair' (meaning 50% heads).

The P-value would answer this question: 'If the coin really is fair, how likely is it to get 8 or more heads out of 10 flips just by random chance?' If that probability (the P-value) is very, very small (like less than 5%), you'd say, 'Wow, that's incredibly unlikely if the coin is fair! I don't think it's fair.' You'd reject the null hypothesis.

But if the P-value was, say, 20% (meaning getting 8 or more heads happens 20% of the time with a fair coin), you'd say, 'Okay, it's a bit unusual, but it's not so unlikely that I can say the coin is definitely unfair.' You'd fail to reject the null hypothesis. A small P-value means your observed data is unlikely under the null hypothesis, giving you reason to doubt the null.

Here are some common traps students fall into and how to steer clear of them:

• Mistake 1: Confusing P-value with probability of H₀ being true.

  • ❌ Thinking: 'My P-value is 0.03, so there's a 3% chance the null hypothesis is true.'
  • ✅ How to avoid: Remember, the P-value is the probability of observing your data (or more extreme) given that H₀ is true. It doesn't tell you the probability of H₀ itself. It's evidence against H₀, not a direct probability of H₀.

• Mistake 2: Accepting the null hypothesis.

  • ❌ Saying: 'Since my P-value was high, I accept the null hypothesis.'
  • ✅ How to avoid: You can only fail to reject the null hypothesis. Think of the courtroom again: if there's not enough evidence to convict, you say 'not guilty,' not 'innocent.' You just don't have enough proof to say something new is happening.

• Mistake 3: Forgetting to check conditions.

  • ❌ Jumping straight to calculations without checking randomness, sample size, or population size.
  • ✅ How to avoid: Always make checking conditions your second step! If conditions aren't met, your test results might not be reliable, like trying to bake a cake without flour. You need a random sample (to avoid bias), enough successes and failures (usually at least 10 of each, so the distribution is bell-shaped), and the 10% condition (sample size less than 10% of the population, so sampling without replacement doesn't mess things up too much).

• Mistake 4: Not writing a conclusion in context.

  • ❌ Ending with: 'Reject H₀ because P < 0.05.'
  • ✅ How to avoid: Your conclusion needs to tell the story! Always state what you found in terms of the original problem. For example: 'We have convincing evidence that the new medicine does reduce blood pressure, unlike the claim that it has no effect.'