Resources / 7: Research Methods & Statistics / Inferential Statistical Tests

Maya draws a bell curve on a whiteboard while the bunny sits above the peak

Inferential Statistical Tests

7: Research Methods & Statistics

Study guide by Anders Chan, PsyD · Updated

Why Statistical Tests Matter More Than You Think

You're sitting at your desk, analyzing data from a study comparing three therapy approaches for depression. You have numbers, spreadsheets, and a nagging question: "Do these differences actually mean something, or could they have happened by chance?" This is where inferential statistics comes in, the decision-making tool that helps you distinguish between real patterns and statistical noise.

For the EPPP, understanding which statistical test to use isn't about memorizing formulas. It's about developing a mental map that guides you from research question to appropriate analysis. Let's build that map together.

The Two Main Families of Statistical Tests

Statistical tests fall into two camps: parametric and nonparametric. Understanding the difference will immediately cut your options in half when choosing the right test.

In plain words: A parametric test works with the actual scores and asks about a population value, usually the mean. (A "parameter" is a number that describes a whole population, like its true average.) These tests assume the scores behave in a certain way, for example roughly bell-shaped within each group. A nonparametric test works with ranks (1st, 2nd, 3rd) or with category counts instead. It makes different assumptions, not zero assumptions. You will also see the word residual. A residual is how far one person's score sits from their own group's mean. If a client scores 18 and her group's mean is 15, her residual is 3.

Parametric tests include t-tests and ANOVA for continuous outcomes. Check assumptions for the specific analysis. A paired t-test concerns the distribution of within-pair differences; an independent-groups analysis concerns the group distributions or residuals. For ordinary independent-groups one-way ANOVA, check approximate normality within groups, similar variances, and independent observations. Do not judge normality from pooled scores alone. (Whitley & Ball, 2002; Bewick et al., 2004)

Nonparametric tests include rank procedures for ordered or continuous outcomes and chi-square procedures for counts. They still have assumptions. A small sample or unequal group sizes alone does not require switching tests. Choose the procedure for the design, outcome, and question. (Whitley & Ball, 2002; Fay & Proschan, 2010; McHugh, 2013)

Here's the decision path:

Data TypeAssumptions Met?Sample SizeTest Family
Interval/RatioYesAdequateParametric
Interval/RatioNoAnyExamine the design and target; consider an appropriate rank procedure
Nominal/OrdinalCheck the chosen test's own assumptionsAnyNonparametric

The Selection Process: Your Decision Framework

Choosing the right statistical test requires answering specific questions in order:

  1. What's your measurement scale? (Nominal, ordinal, interval, ratio)
  2. How many independent variables? (One, two, three or more)
  3. How many levels does each independent variable have? (Two groups, three groups, etc.)
  4. Are groups related or unrelated? (Same people measured twice vs. different people)
  5. How many dependent variables? (One outcome vs. multiple outcomes)
  6. Do you need to control for an extraneous variable?

Each answer narrows your choices until you land on the right test. Let's explore the major tests you'll encounter.

The Chi-Square Test: Analyzing Categories

The chi-square test works with nominal data. Frequencies and categories. There are two versions, and a simple trick makes them easy to distinguish: substitute the word "variable" for "sample."

Single-Sample Chi-Square (Goodness-of-Fit Test): Use this when you have one variable with multiple categories. Imagine surveying your therapy clients about their preferred session format: in-person, telehealth, or hybrid. You have one variable (preference) with three categories. The test tells you whether the distribution of preferences differs from what you'd expect by chance.

Multiple-Sample Chi-Square (Test for Contingency Tables): Use this when you have two or more variables. This includes both descriptive studies with multiple variables and experimental studies with independent and dependent variables.

Here's a crucial point: Count ALL variables when deciding which chi-square test to use.

Example: You want to know if college students prefer hard-copy or digital textbooks. That's one variable (preference) with two categories. Use the single-sample chi-square.

Now expand this: You also want to see if preference varies by course type (online vs. face-to-face). Now you have two variables: textbook preference and course type. This creates four possible combinations:

  • Prefers hard-copy/face-to-face course
  • Prefers hard-copy/online course
  • Prefers digital/face-to-face course
  • Prefers digital/online course

You'd use the multiple-sample chi-square to analyze these frequencies.

Expected Frequencies

In plain words: Chi-square compares the counts you actually got with the counts you would expect if nothing were going on. The "expected count" is that nothing-going-on number. The bigger the gaps between observed and expected counts, the bigger the chi-square value.

For a chi-square test of independence in a contingency table, the expected frequency for each cell = (row total × column total) / grand total. For example, a row total of 40 and a column total of 30 in a sample of 100 give an expected count of 12. For a goodness-of-fit test, each expected count instead equals the sample size multiplied by that category’s hypothesized probability.

The chi-square test of independence requires independent observations. Its approximation also depends on expected cell counts, calculated under the null hypothesis. A common rule is that at least 80% of cells have expected counts of 5 or more and no cell has an expected count below 1. An observed count of 3 alone does not show that this rule is broken (McHugh, 2013).

Worked example: Back to textbooks. You survey 100 students. 60 take face-to-face courses and 40 take online courses. 50 prefer hard copy and 50 prefer digital.

  1. Pick one cell: hard copy and online.
  2. Find its row total (hard copy = 50) and its column total (online = 40).
  3. Multiply, then divide by the grand total: (50 × 40) / 100 = 20. If preference has nothing to do with course type, you expect about 20 students in that cell.
  4. Do the same for the other cells. Hard copy/face-to-face = (50 × 60) / 100 = 30. Digital/online = 20. Digital/face-to-face = 30. The four expected counts add back up to 100.
  5. Check the rule. Every expected count is 5 or more, so the rule is met.
  6. Degrees of freedom for a contingency table = (rows - 1) × (columns - 1) = (2 - 1) × (2 - 1) = 1.
  7. Goodness of fit is simpler. If 90 clients choose among three session formats and your null says each format is equally likely, each expected count is 90 × 1/3 = 30, and df = categories - 1 = 2.

So on the exam: expected count = (row total × column total) / grand total for a contingency table, and N × the hypothesized proportion for goodness of fit. The "5 or more" rule is about expected counts, not observed counts.

The Student's t-Test: Comparing Two Means

When you have one independent variable with exactly two levels and one dependent variable measured on an interval or ratio scale, you need a t-test. The t-test compares two means to determine if they're significantly different.

Picture comparing average depression scores between clients receiving cognitive-behavioral therapy versus interpersonal therapy. You have two groups (two levels of your independent variable) and one outcome measured on a continuous scale.

There are three types of t-tests, each for a specific situation:

t-Test for a Single Sample: Compares a sample mean to a known population mean. Say you want to know if psychologists who attended your EPPP workshop scored differently than the national average. You compare your group's mean score to the known population mean.

t-Test for Unrelated Samples (Independent Samples): Compares means from two separate, unrelated groups. Typically, this means participants were randomly assigned to groups. These people have no connection to each other before the study.

Random assignment is the classic case, but it is not required. Groups that are separate by nature, such as clients at two different clinics, also count as unrelated, as long as no one in one group is paired with anyone in the other.

t-Test for Related Samples (Paired Samples): Compares means when there's a meaningful relationship between groups. This happens in three scenarios:

  1. Natural pairs (identical twins, spouses)
  2. Matched pairs (you pair participants based on similar characteristics, then assign one to each group)
  3. Repeated measures (same people measured at two time points. They're "paired with themselves")
t-Test TypeWhen to UseExample
Single sampleCompare sample to known populationWorkshop attendees vs. national average
Unrelated samplesTwo separate groups, randomly assignedCBT group vs. IPT group
Related samplesPaired participants or repeated measuresSame clients at pretest and posttest

Worked example: Which t-test, and how many degrees of freedom? (Degrees of freedom, or df, is a count based on your sample size. Roughly, it is how many scores are free to vary once the mean is fixed. It tells the test which cutoff value to use.)

  1. 25 clients take an anxiety measure with a known population mean of 100. Their mean is 107 and their SD is 15. One group against a known population mean = single-sample t-test, with df = n - 1 = 24.
  2. To get t, first find the standard error. It is how much a sample mean usually wobbles from sample to sample. SE = SD / √n = 15 / √25 = 15 / 5 = 3.
  3. Then t = (sample mean - population mean) / SE = (107 - 100) / 3 ≈ 2.33. The two-tailed cutoff at alpha .05 with df = 24 is about 2.06. Since 2.33 is larger, the difference is significant.
  4. 40 clients are randomly assigned to CBT (20) or IPT (20). Two unrelated groups = independent-samples t-test, with df = n₁ + n₂ - 2 = 20 + 20 - 2 = 38.
  5. 30 clients are measured before and after treatment. Same people twice = related-samples t-test. It runs on the 30 difference scores, so df = number of pairs - 1 = 29.

So on the exam: first ask "one group or two, and is anyone paired?" Then df is n - 1 (single sample), n₁ + n₂ - 2 (independent), or pairs - 1 (related).

One-Way ANOVA: Comparing Multiple Groups

When you have one independent variable with more than two levels and one dependent variable on an interval or ratio scale, you need a one-way analysis of variance (ANOVA).

Imagine you're comparing four different meditation apps to see which reduces anxiety most effectively. You randomly assign clients to use one of the four apps for eight weeks, then measure their anxiety levels. You have one independent variable (app type) with four levels and one dependent variable (anxiety score).

Why not just run multiple t-tests? You could compare App 1 vs. App 2, then App 1 vs. App 3, then App 1 vs. App 4, and so on. But here's the problem: Each statistical test carries a risk of Type I error (falsely concluding there's an effect when there isn't one). When you set alpha at .05, you're accepting a 5% chance of this error per test.

Running multiple tests inflates this risk, the experimentwise error rate. With four groups, you'd need six separate t-tests to compare all pairs. If six tests were independent, a 5% error rate per test would give about a 26% chance of at least one false positive. Pairwise tests that share groups are not independent, so that calculation is only an illustration here (Bewick et al., 2004). The one-way ANOVA solves this by comparing all groups simultaneously while maintaining your chosen alpha level.

Worked example: Where does "about 26%" come from?

  1. Count the pairs. With k groups, the number of pairs is k(k - 1) / 2. For 4 groups: (4 × 3) / 2 = 6.
  2. On one test at alpha .05, when there is truly no effect, the chance of no false alarm is 1 - .05 = .95.
  3. If the six tests were independent, the chance of no false alarm on all six is .95 multiplied by itself six times: .95⁶ ≈ .735.
  4. So the chance of at least one false alarm is 1 - .735 ≈ .265, about 26%.
  5. A Bonferroni correction (covered below) would test each of the six pairs at .05 / 6 ≈ .0083.

So on the exam: more comparisons means a higher experimentwise error rate. Running one ANOVA first, and correcting alpha for any follow-up comparisons, keeps it in check.

Understanding the F-Ratio

In plain words: F asks one question. Are the group averages farther apart than you would expect from the normal spread of people inside each group? A mean square is a variance estimate: the sum of squared distances from a mean, divided by its df. "Between" means the spread among the group means. "Within" means the spread of people around their own group's mean.

The ANOVA produces an F-ratio, which has two components:

Numerator (Mean Square Between. MSB): Measures variability between groups. This includes both treatment effects (real differences caused by your independent variable) plus random error.

Denominator (Mean Square Within (MSW): Measures variability within groups. This is error only) individual differences, measurement inconsistency, and random fluctuation.

When the F-ratio is larger than 1.0, the between-group differences exceed within-group variability, suggesting your independent variable had an effect. Whether this is statistically significant depends on your alpha level and degrees of freedom.

For a one-way ANOVA:

  • Degrees of freedom for MSB = C - 1 (where C = number of groups) (Bewick et al., 2004).
  • Degrees of freedom for MSW = N - C (where N = total participants) (Bewick et al., 2004).

If you're testing three therapy approaches with 45 total clients (15 per group), you'd have df = 2 for MSB (3-1) and df = 42 for MSW (45-3).

Worked example: Three therapy approaches, 15 clients in each group, 45 clients total.

  1. df between = C - 1 = 3 - 1 = 2.
  2. df within = N - C = 45 - 3 = 42.
  3. Say MSB = 40 and MSW = 10. Then F = MSB / MSW = 40 / 10 = 4.0. The between-groups estimate is four times the size of the within-groups (error) estimate.
  4. If therapy type had no effect, both mean squares would estimate the same error, so F would land near 1.
  5. The cutoff for F with df of 2 and 42 at alpha .05 is about 3.2. Since 4.0 is larger, the result is significant.

So on the exam: F = MSB / MSW. An F near 1 is what you expect when there is no effect. A significant F says at least one group mean differs, not which one.

Advanced ANOVA Variations You Need to Know

Factorial ANOVA: For studies with multiple independent variables. A two-way ANOVA has two IVs, a three-way ANOVA has three IVs, and so on. This design produces separate F-ratios for each main effect and all possible interactions. For instance, you might study how both therapy type (CBT, ACT, IPT) and session frequency (weekly, biweekly) affect depression. That's a 3 × 2 factorial design.

Mixed ANOVA (Split-Plot ANOVA): Used when you have both between-subjects and within-subjects variables in the same study. Imagine comparing three therapy types (between-subjects) while measuring clients at baseline, midpoint, and endpoint (within-subjects). This design combines elements of independent and repeated measures.

Randomized Block ANOVA: Controls for an extraneous variable by including it as a factor in your analysis. The extraneous variable becomes the "blocking variable." If you suspect age affects treatment response, you could block participants into age groups (20-30, 31-40, 41-50) and analyze both treatment effects and age effects.

ANCOVA (Analysis of Covariance): Another way to control for an extraneous variable, but instead of including it as a factor, ANCOVA statistically removes its effects from the dependent variable. This variable is called the "covariate." You might use baseline depression scores as a covariate when analyzing posttreatment scores. Essentially asking "Do treatments differ after accounting for where people started?"

MANOVA (Multivariate Analysis of Variance): For studies with multiple dependent variables, each measured on an interval or ratio scale. Rather than running separate ANOVAs for depression scores, anxiety scores, and quality of life scores, MANOVA analyzes all three outcomes simultaneously.

Trend Analysis: Used with quantitative independent variables to determine if there's a linear or nonlinear relationship with the dependent variable. You might examine how therapy dose (4, 8, 12, or 16 sessions) relates to outcome, testing whether the relationship is linear (steady improvement) or quadratic (rapid initial gains that plateau).

Digging Deeper: Planned Comparisons and Post Hoc Tests

A significant F-ratio tells you that at least one group differs from others. But not which specific groups differ. This is where planned comparisons and post hoc tests come in.

Planned Comparisons (A Priori Tests): Specified before data collection based on theory or hypotheses. Suppose you're testing four teaching methods: lectures only (L), lectures with peer study groups (LP), lectures with instructor-led in-person sessions (LIP), and lectures with instructor-led Zoom sessions (LIZ). If your hypothesis specifically predicts that instructor-led sessions outperform lectures alone, you'd plan to compare only L vs. LIP and L vs. LIZ. Not all possible combinations.

Post Hoc Tests (A Posteriori Tests): Conducted after obtaining a significant F-ratio when you want to examine all possible group differences. These tests compare every pair of means.

Both approaches face the experimentwise error rate problem. Solutions include:

Bonferroni Correction: Divide your alpha by the number of comparisons. If you're making four comparisons with alpha = .05, each comparison uses alpha = .0125 (.05/4). Simple but conservative. It reduces power to detect real effects.

Specialized Post Hoc Tests: Different tests for different situations:

  • Tukey's HSD (Honestly Significant Difference): Good general-purpose test; maintains error rate well
  • Scheffé Test: Protects the error rate across all contrasts among means, including complex contrasts suggested by the data. This broad protection can be conservative, but it is not a universal ranking against every other method (Hall & Bird, 1986).
  • Newman-Keuls Test: Less conservative; more likely to detect real differences but slightly higher error risk

Beyond Statistical Significance: What Really Matters

In plain words: In statistics, "significant" means "results this extreme would be unusual if there were truly no effect." It does not mean big, important, or helpful to clients.

A statistically significant result means the p-value falls below the chosen alpha level. It does not tell you the probability that chance caused the result, or whether the effect matters in practice (Greenland et al., 2016). This distinction matters enormously in clinical practice.

Practical Significance (Effect Size): Measures the magnitude of an intervention's effect. Cohen's d is the most common measure, expressing the difference between groups in standard deviation units.

For two independent groups, calculate Cohen's d as (Mean₁ - Mean₂) / Pooled SD (Lakens, 2013).

In plain words: d tells you how many standard deviations apart the two group means are. Say the treated group averages 20 on a depression scale, the control group averages 26, and the pooled SD is 10. Then d = (20 - 26) / 10 = -0.6. The minus sign only shows direction (the treated group scored lower). The size, 0.6, falls between Cohen's medium and large benchmarks.

Cohen's guidelines:

  • d around 0.2: Small effect
  • d around 0.5: Medium effect
  • d around 0.8 or higher: Large effect

Imagine two therapy approaches both show statistically significant improvement over a control group. Therapy A has d = 0.3 (small effect), while Therapy B has d = 1.2 (large effect). Both "work," but Therapy B produces much more substantial change.

For designs with more than two groups, use Cohen's f instead of Cohen's d.

Clinical Significance: Even practical significance doesn't tell you if clients move from dysfunctional to functional. A therapy might produce statistically significant and practically meaningful reductions in depression symptoms, but do clients actually recover? Do they move from clinical depression to normal mood functioning?

The Jacobson-Truax Method assesses clinical significance through two steps:

Step 1: Calculate Reliable Change Index (RCI)

RCI = (Posttest score - Pretest score) / Standard error of difference (Ferrer-Urbina et al., 2023).

When RCI exceeds ±1.96 in the expected direction, the change is considered reliable (not due to measurement error).

In plain words: The standard error of the difference is how much a score could change from measurement error alone, given how reliable the test is. The RCI asks whether the real change is big enough to beat that noise.

Step 2: Determine Cutoff Score

Identify the score that separates dysfunctional from functional populations. Often calculated as the midpoint between clinical and non-clinical population means. In Jacobson and Truax's version (cutoff c), that midpoint is weighted by each group's standard deviation, so it sits the same number of SDs from each mean. With equal SDs, it is the exact midpoint (Jacobson & Truax, 1991).

Then classify each individual:

  • Recovered: Passed both RCI and cutoff criteria (reliable change into functional range)
  • Improved: Passed RCI but not cutoff (reliable change, still in dysfunctional range)
  • Unchanged: No reliable change, whether or not the cutoff was crossed
  • Deteriorated: Passed RCI in wrong direction (reliable worsening)

This approach is like evaluating a weight loss program not just by average pounds lost, but by how many participants moved from obese to healthy BMI ranges and maintained that change reliably.

Worked example: A depression scale where lower scores are better. The scale's SD is 10 and its reliability is .91. Clinical clients average 30, nonclinical people average 14, and both groups have the same SD. Client A scores 30 at pretest and 20 at posttest.

  1. Standard error of measurement (SEM) = SD × √(1 - reliability) = 10 × √.09 = 10 × .3 = 3.
  2. Standard error of the difference = √(2 × SEM²) = √(2 × 9) = √18 ≈ 4.24.
  3. RCI = (20 - 30) / 4.24 ≈ -2.36. That is past -1.96 and in the expected direction (scores went down), so the change is reliable.
  4. Cutoff: with equal SDs, it is the midpoint, (30 + 14) / 2 = 22.
  5. Client A's posttest of 20 is on the functional side of 22. Reliable change plus crossing the cutoff = Recovered.
  6. Client B goes from 38 to 28. RCI = (28 - 38) / 4.24 ≈ -2.36, so the change is reliable. But 28 is still above 22. Reliable change without crossing the cutoff = Improved.

So on the exam: the RCI answers "is this change bigger than measurement noise?" The cutoff answers "did the person land in the functional range?" Recovered needs both.

Common Mistakes to Avoid

Confusing chi-square tests: Remember the "variable" substitution trick. Count all variables in your study. One variable = single-sample chi-square. Two or more variables = multiple-sample chi-square.

Using multiple t-tests instead of ANOVA: This inflates your Type I error rate. When comparing more than two groups, start with ANOVA.

Forgetting the "related" in related t-tests: Participants must be meaningfully paired. Through matching, natural relationships, or repeated measurement of the same people.

Ignoring assumptions: Check independence and the assumptions of the specific test. For paired t-tests, examine the differences. Rank tests also depend on pairing and distribution assumptions. Unequal group sizes alone do not decide the test. (Whitley & Ball, 2002; Rietveld & van Hout, 2017)

Stopping at statistical significance: Always consider effect size. A finding can be statistically significant but practically trivial, especially with large sample sizes.

Overlooking clinical significance: In applied settings, ask whether interventions produce meaningful improvement in people's lives, not just significant p-values.

Memory Aids for Test Selection

The Scale Narrows Your Choices:

  • Nominal data (category counts) → Chi-square
  • Ordinal data (ranks) → Rank tests, such as Mann-Whitney or Kruskal-Wallis
  • Interval/Ratio data → t-test or ANOVA family

Treat this as a first filter, not the final answer. Interval or ratio data can still call for a rank test when the design or the shape of the scores makes a test of means a poor fit (see "Rank tests still depend on the design" below).

Count Your IVs and Their Levels:

  • One IV, two levels → t-test
  • One IV, 3+ levels → One-way ANOVA
  • Multiple IVs → Factorial ANOVA

Check the Relationship:

  • Unrelated groups → Between-subjects design
  • Related groups (paired, matched, repeated) → Within-subjects design
  • Both → Mixed ANOVA

Special Circumstances:

  • Need to control extraneous variable → ANCOVA or Randomized Block ANOVA
  • Multiple DVs → MANOVA
  • Quantitative IV, examining shape of relationship → Trend Analysis

Quick Reference Table:

Research QuestionTest
One categorical variable, frequency countsSingle-sample chi-square
Two+ categorical variables, frequency countsMultiple-sample chi-square
Compare sample mean to population meanOne-sample t-test
Compare two unrelated group meansIndependent samples t-test
Compare two related group meansPaired samples t-test
Compare 3+ unrelated group meansOne-way ANOVA
Compare groups across 2+ IVsFactorial ANOVA
Compare groups controlling for covariateANCOVA
Compare groups on multiple DVsMANOVA

Worked example: Walk one vignette through the six questions. A researcher randomly assigns 60 adults with insomnia to CBT for insomnia, sleep medication, or a waitlist (20 each). After 6 weeks she measures minutes to fall asleep. She also has each person's baseline minutes and wants to account for where people started.

  1. Measurement scale: minutes is a ratio scale, so start with the t-test and ANOVA family.
  2. Number of IVs: one (treatment).
  3. Levels: three (CBT, medication, waitlist). Three levels rules out a t-test.
  4. Related or unrelated: different people in each group, so unrelated.
  5. Number of DVs: one (minutes to fall asleep).
  6. Control for an extraneous variable: yes, baseline minutes. A measured variable you remove statistically is a covariate, so the answer is ANCOVA.

Now change one detail at a time:

  • No baseline adjustment: one-way ANOVA, with df = 3 - 1 = 2 between and 60 - 3 = 57 within.
  • Outcome is "sleeping well, yes or no" counted per group: two categorical variables (treatment and outcome), so multiple-sample chi-square.
  • Outcome is minutes to fall asleep plus a sleep-quality score: two DVs, so MANOVA.
  • Each group is measured at baseline and at week 6, and both time points are analyzed: one between-subjects IV plus one within-subjects IV, so mixed ANOVA.

So on the exam: answer the six questions in order. When two vignettes look alike, the one detail that differs (outcome type, number of DVs, pairing, a covariate) usually decides the test.

Key Takeaways

  • Choose the test for the outcome, design, and target question. Both parametric and nonparametric procedures have assumptions. (Whitley & Ball, 2002; Fay & Proschan, 2010)

  • Chi-square analyzes categorical data: single-sample for one variable, multiple-sample for two or more variables

  • t-tests compare two means: single-sample (vs. population), independent samples (unrelated groups), or paired samples (related groups)

  • One-way ANOVA compares 3+ group means while controlling experimentwise error rate

  • Factorial ANOVA handles multiple independent variables; mixed ANOVA combines between and within-subjects factors

  • ANCOVA controls extraneous variables statistically; randomized block ANOVA includes them as factors

  • MANOVA analyzes multiple dependent variables simultaneously

  • Significant F-ratios require follow-up: planned comparisons (theory-driven) or post hoc tests (exploratory)

  • Effect size (Cohen's d) measures practical significance beyond statistical significance

  • Clinical significance (Jacobson-Truax method) determines whether individuals move from dysfunctional to functional

  • Bonferroni correction and specialized post hoc tests control experimentwise error rate

Understanding these tests isn't about memorization. It's about recognizing patterns. Each research design creates a unique signature that points to a specific test. With practice, you'll develop intuition for matching questions to analyses, a skill that serves you well beyond the EPPP into actual research and practice.

Match the analysis to the observations

In plain words: Before you pick a test, ask two things. What kind of score is the outcome? And did each score come from a different person, or do some scores belong together (the same person twice, twins, matched pairs)? The word omnibus, used below, means "overall." A one-way ANOVA tells you whether the groups differ somewhere, not where.

Start with the outcome and who supplied each observation. An independent-samples t-test compares two unrelated group means. A paired t-test analyzes within-pair differences, such as the same people's scores before and after treatment. For small samples, check approximate normality of the relevant group distributions or paired differences, not just all scores pooled together. One-way ANOVA tests equality of means across independent groups; its omnibus result does not identify which pairs differ. Chi-square tests use category counts and need independent observations and adequate expected counts. A large sample does not make repeated observations independent. (Whitley & Ball, 2002; Bewick et al., 2004; McHugh, 2013)

Rank tests still depend on the design

In plain words: Rank tests swap raw scores for their order (lowest = 1, next = 2, and so on). That makes them less thrown off by a few extreme scores, but they still have rules, and each one fits one design. Mann-Whitney is for two separate groups. Wilcoxon signed-rank and the sign test are for paired scores. A location difference means one set of scores sits higher or lower overall. Symmetric means the differences spread out about the same amount on both sides of their middle.

The Mann-Whitney test compares two independent samples using ranks. Wilcoxon signed-rank uses paired differences and their ranked magnitudes; its usual location interpretation needs a symmetric difference distribution. The sign test uses the directions of nonzero differences, discarding their magnitudes. These tests answer different questions from a test of group means. Nonparametric does not mean assumption-free. Pairing, independence, distribution shape, and the target question still matter. Unequal group sizes alone do not require a rank test. (Whitley & Ball, 2002; Fay & Proschan, 2010; Rietveld & van Hout, 2017)

Choosing a model for a curve

In plain words: If the dots on a scatterplot bend, a straight line will fit badly. A residual plot shows each person's miss (actual score minus the line's predicted score). If the misses form a curve instead of a random cloud, the straight line is the wrong model.

A straight-line regression describes a linear relationship between a quantitative predictor and outcome. A curved pattern is a warning that this simple model may be inadequate. A scatterplot and residual plot help check model form; naming the variables alone does not establish a suitable model (Bewick et al., 2003).

Power comparisons require assumptions

In plain words: No single test wins every time. Say scores are bell-shaped within each group, both groups have similar spread, and the treatment simply moves everyone up or down by the same amount (a mean shift). Then the ordinary pooled t-test is a strong choice. Efficient means it gets a lot out of your sample size. When scores are skewed or have extreme values, a rank test can have more power than the t-test.

For two independent groups with normal outcomes, equal variances, and a mean-shift alternative, the pooled t test is a suitable efficient test of the mean difference. A rank test addresses a different set of assumptions and can perform better in other distributions. There is no universally most powerful test across all designs (Fay & Proschan, 2010).

Significance magnitude and recovery

In plain words: Three questions, three tools. The p value asks "how surprising are these results if there were no effect?" Cohen's d asks "how big is the difference?" Clinical significance asks "did this person get better enough to function well?" A yes on one does not give you a yes on the others.

A p value describes how unusual results at least as extreme would be under the null model and its assumptions. It is not the probability that chance caused the result. Cohen's d describes a standardized mean difference; statistical significance does not establish a large effect or individual recovery. Compare effect sizes in context and assess clinical functioning separately (Greenland et al., 2016; Lakens, 2013).

Flag-resolution sources

  • W Hall, K D Bird. Simultaneous multiple comparison procedures in psychiatric research. The Australian and New Zealand journal of psychiatry. 1986 Mar;20(1):46-54. doi:10.3109/00048678609158864. PMID:3460585. https://pubmed.ncbi.nlm.nih.gov/3460585/

Ready to practice?

Get started in the app