Go to main content
Back to Courses
MODULE 17

Statistical Significance, A/B Testing, and Experimentation Science

Take experimentation to a serious level with causal design, power planning, variance reduction, and governance.

Editorial cover of the module Statistical Significance, A/B Testing, and Experimentation Science
Foundations of Statistical Significance - official lesson image on GinnyTech, created by AD
1
Advanced 18 min

Foundations of statistical significance

Null and alternative hypotheses, sampling distribution, test statistic, and p-value. From Ronald Fisher's tea lesson (1925) to numerical A/A testing, including limitations of frequentist testing and operational validation templates.

Prerequisites: 1

Go to lesson
Business Hypotheses and Experimental Questions - official lesson image on GinnyTech, created by AD
2
Advanced 18 min

Well-Formulated Causal Questions and Business Hypotheses

Formulare domande causali e ipotesi business falsificabili: meccanismo, metrica primaria, unità di analisi e direzione attesa, con esito scritto prima del lancio e guardrail dichiarati.

Prerequisites: 1

Go to lesson
Experimental Design, Randomization, and Metrics - official lesson image on GinnyTech, created by AD
3
Advanced 18 min

Experimental design, randomization, and unit of analysis

Allineare unità di randomizzazione, unità di analisi e metrica primaria, con randomizzazione bilanciata e guardrail che intercettano i danni collaterali.

Prerequisites: 1

Go to lesson
A/A Test and System Reliability - official lesson image on GinnyTech, created by AD
4
Advanced 18 min

A/A testing and measurement system validation

Il test A/A come collaudo del sistema di misura: confrontare la quota di falsi positivi con la soglia attesa e indagare randomizzazione, logging e pipeline quando i falsi allarmi superano l'atteso.

Prerequisites: 1

Go to lesson
A/B Test: From Variant to Decision - official lesson image on GinnyTech, created by AD
5
Advanced 18 min

Complete flow of a professional A/B test

Il flusso completo di un A/B test professionale: domanda, ipotesi, disegno e guardrail fissati prima del lancio, monitoraggio senza decisioni intermedie e raccomandazione netta tra rollout, iterazione e nuovo test.

Prerequisites: 1

Go to lesson
P-Value, Errors, and Correct Interpretation - official lesson image on GinnyTech, created by AD
6
Intermediate 18 min

P-value, Errors, and Correct Interpretation

What the p-value really measures and how to read it together with effect size, intervals, power, and business decision.

Prerequisites: 1

Go to lesson
Sample Size, Power, MDE, and Duration - official lesson image on GinnyTech, created by AD
7
Advanced 18 min

CUPED and variance reduction

CUPED e variance reduction: scegliere una covariata precedente al trattamento e bilanciata, costruire la metrica corretta Y_adj e verificare la riduzione di varianza a parità di campione.

Prerequisites: 1

Go to lesson
Peeking, Multiple Testing, and Experiment Governance - official lesson image on GinnyTech, created by AD
8
Advanced 18 min

Peeking, multiple testing and sequential testing

Peeking, multiple testing e sequential testing: durata minima e regola di lettura fissate prima del lancio, correzione della soglia con α/K e stop deciso solo dalla regola scritta.

Prerequisites: 1

Go to lesson
Case Study: Redoing Checkout Without Being Misled - official lesson image on GinnyTech, created by AD
9
Advanced 22 min

Bayesian A/B, switchback test, and geo-test

Quando la randomizzazione per utente non è possibile: switchback test per interferenza temporale, geo-test per interventi locali e modello bayesiano per decidere in fretta con poco traffico.

Prerequisites: 1

Go to lesson
Cheat Sheet: Statistical Significance and A/B Testing - official lesson image on GinnyTech, created by AD
10
Advanced 28 min

Experimental governance + end-to-end case study

Governance sperimentale e caso studio end-to-end: registrare ipotesi, metrica primaria, MDE e durata prima del lancio, regole di stop comuni e decision memo che separa effetto, rischio e prossima azione.

Prerequisites: 1

Go to lesson

Module connections

Natural prerequisiteAnalytics Engineering with dbt and Semantic Layer13 lessons

Transform raw data into reliable, reusable, and governed models with layering, tests, contracts, and semantic layer.

Same stage of the pathMathematics for Data Analysis10 lessons

Introduces the mathematical rigor that makes models, inference, forecasting, and applied regressions more readable.

Same stage of the pathAdvanced SQL for Analytical Systems14 lessons

Use SQL as a reasoning language, not just extraction, with patterns for cohorts, funnels, experiments, and performance.

Previous moduleCourse Overview and Study Method for Data Work8 lessons

Transform the course from catalog to cognitive system, with study method, technical memory, and professional output plan.

Previous moduleData Collection & Tracking Systems8 lessons

Move from isolated tags and tools to reliable, governed, and observable measurement systems.

Previous moduleMetrics, KPI Trees, and Analytical Foundations6 lessons

Learn to read numbers as systems, with KPI trees, unit economics, correct baselines, and causal signal interpretation.