← Back to Master Index

Statistical Inference

Understanding:

Statistical Inference is the process of using data analysis to deduce properties of an underlying distribution of probability. Essentially, it's about making conclusions about a population based on a sample.

ADHD friendly explanation:
| We have a probability distribution | ➔ We do data analysis on it | ➔ That is the process | ➔ This process is Statistical Inference | ➔ Output = Conclusion |

1. Null Hypothesis ($H_{0}$) & P-Value

In statistical hypothesis testing, we use the Null Hypothesis ($H_{0}$) and the P-Value together to determine if a result is a real effect or just a lucky coincidence.

Step 1: The Null Hypothesis ($H_{0}$)

The null hypothesis is NOT a proven fact. It is just our starting stance of skepticism. Think of it like "Innocent until proven guilty" in a courtroom.

We assume the defendant is innocent ($H_0$: No Crime/Effect) not because we have proven they are innocent, but because we need a neutral starting point before looking at the evidence.

Example: If you are testing a new drug, we start by assuming "This drug does nothing" ($H_0$). We hold onto this assumption until the data (evidence) becomes so strong that we are forced to abandon it.

Step 2: The P-Value

The p-value is a probability (ranging from 0 to 1) that represents how likely it is to observe your specific results if the null hypothesis were actually true. In simple words it is the probability that the null hypothesis is true.

  • 🔹 Low P-Value (≤ 0.05): Indicates the results are "strange" or unlikely under the null hypothesis. This gives us evidence to Reject $H_{0}$. (Statistically Significant).
  • 🔹 High P-Value (> 0.05): Indicates the results are consistent with chance. We Fail to Reject $H_{0}$.

Step 3: Decision Making

We compare the p-value to a pre-set threshold called the Significance Level ($\alpha$), which is most commonly 0.05. If our P-Value is smaller than $\alpha$, we conclude the effect is real!

Visualizing the "Luck" Probability

The curve represents the Null Hypothesis (e.g., the drug does nothing). Drag the slider to see how likely your experimental result would be if the drug actually had 0 effect.

t = 0.00 P-Value = 0.50

How to read the graph?

Drag the slider to the right (Effect Strength > 2.0).
The Red Area is the P-Value. It shows how much of the "Zero Effect" distribution matches your result.
If the P-Value is tiny (Red area disappears), it means your result is too big to be a coincidence!

Deep Dive: The Calculation & Logic

1. The "Formula"

There isn't one single formula, but the general logic for finding P is:

Step 1: Calculate Test Statistic (Z) = (Observed - Expected) / Error
Step 2: P-Value = Area under the curve beyond Z

Why Low P = Reject Null?
Think of P as the "Probability of Coincidence".
• If P = 0.03 (3%), it means there is only a 3% chance this happened by luck. That's too rare. So we assume it wasn't luck (Reject Null).
• If P = 0.40 (40%), it means there's a 40% chance this was just noise. That happens all the time. So we assume it was just noise (Keep Null).

ADHD Friendly Example: The Cat Detector

Scenario: You have Model A (Untrained/Random) and Model B (New/Trained).
Null Hypothesis ($H_0$): "Model B is just as dumb as Model A (Random guessing)."

The Test: Both models classify 100 images.
• Model A gets 50 right (50%).
• Model B gets 65 right (65%).

Calculation: "What are the odds of a random guesser getting 65%?"
Result (P-Value): 0.03 (3%).

Conclusion:
• P = 0.03: Only a 3% chance a dummy model gets this score. That's super rare!
• Decision: Model B is NOT guessing. It actually learned! (Reject Null).

(Contrast: If Model B got 52 right, P-value might be 0.40. "40% chance of luck? Yeah, it's probably just guessing.")

2. Confidence Intervals (CI)

Instead of a single number estimate, a Confidence Interval gives us a range that is likely to contain the true population parameter.

"95% Confidence" means: If we repeated this experiment 100 times, 95 of the calculated intervals would capture the true value.

Simulation: Catching the True Mean

The vertical white line is the True Population Mean (Hidden from the observer).
Click "Sample" to run an experiment and generate an interval.

Success Rate: 0% (Target: ~95%)

3. Common Hypothesis Tests

While the P-Value is the universal "score," we calculate it using different tests depending on the type of data we have.

T-Test (Student's t-test)

Use when: Comparing the averages (means) of two groups.

Example:
Does the new website design lead to longer visit times than the old one?
  • Group A: Old Design (Avg: 45s)
  • Group B: New Design (Avg: 52s)

Result: P = 0.04 (Significant difference!)

ANOVA (Analysis of Variance)

Use when: Comparing the averages of three or more groups.

Example:
Comparing the effectiveness of three different diets.
  • Diet A vs. Diet B vs. Diet C

Result: F-Statistic checks if at least one differs from the others.

Chi-Square ($\chi^2$) Test

Use when: Comparing categories / counts (not averages).

Example:
Is ice cream flavor preference related to age group?
  • Count of Chocolate vs. Vanilla fans in Kids vs. Adults.

Result: Tests if the distribution of preferences matches expectations.