Resources / 7: Research Methods & Statistics / Research – Internal/External Validity

Maya draws a bell curve on a whiteboard while the bunny sits above the peak

Research – Internal/External Validity

7: Research Methods & Statistics

Study guide by Anders Chan, PsyD · Updated

Why Internal and External Validity Actually Matter

You're going to spend years working with clients, and you need to know which treatments actually work. Should you use that new therapy everyone's talking about? Does the research showing it works actually hold up? This is where internal and external validity come in. These concepts help you figure out which studies you can trust and which ones have flaws that make their findings questionable.

Internal validity answers one question: Did the treatment actually cause the change we saw? External validity asks: Will these results work in my office with my real clients? Both are essential, but they serve different purposes. Let's break down how to evaluate research so you can make informed decisions in your practice.

The Foundation: What These Terms Really Mean

Internal validity tells you whether a study's conclusions about cause-and-effect are accurate. If you started a new diet and lost weight, but also started a new job with longer hours and more stress during that same period, you couldn't be sure which factor caused the weight loss. Research studies face the same problem. Did the therapy help, or was it something else that happened during the study?

External validity tells you how far you can extend those findings. A study might have solid internal validity (the treatment definitely caused the improvement) but if it only worked with college students in a lab setting, will it work with your 45-year-old client dealing with job loss and divorce? That's external validity.

Here's what makes this tricky: these two types of validity often conflict. It's like trying to test whether a new kitchen gadget works. You could test it in a perfectly controlled test kitchen with ideal conditions (high internal validity), but that doesn't tell you if it'll work in your actual messy kitchen with your habits and constraints (external validity). Researchers constantly balance these priorities.

Breaking Down External Validity Into Five Parts

External validity isn't just one thing. It has five distinct components:

Type of External ValidityWhat It MeasuresExample Question
Population ValidityCan we generalize to other people?Will this treatment for anxiety work with people beyond just the college students studied?
Ecological ValidityCan we generalize to real-world settings?Will this therapy work in a clinic, not just in a controlled lab?
Temporal ValidityCan we generalize across time?Will this treatment still work five years from now?
Treatment Variation ValidityCan we generalize to modified versions?If we shorten the therapy from 16 to 12 sessions, will it still work?
Outcome ValidityCan we generalize to related measures?If this reduced depression symptoms, will it also improve work performance?

Understanding these distinctions helps you evaluate studies more critically. A study might have great population validity but poor ecological validity, meaning the results apply to different groups of people but might not translate to real-world clinical settings.

The Seven Threats to Internal Validity

Think of these as the ways a study can go wrong when trying to establish cause-and-effect. You need to spot these when reading research.

1. History

History occurs when outside events during the study affect the results. Imagine you're testing a new stress-reduction app with participants over three months, and halfway through the study, a major company in your city announces massive layoffs. Suddenly, everyone's stress levels spike. Not because of anything in your study, but because of this external event.

The solution? Use multiple groups with random assignment. If you randomly assign participants to a treatment group (gets the app) and a control group (doesn't get the app), both groups experience the same historical events. Any difference between them at the end isn't due to history.

There's a trickier version: when you're testing groups separately and something happens to just one group. Say you're running therapy groups, and during one session, the fire alarm goes off and everyone has to evacuate. That disruption only affects that particular group. This is harder to control and you need to acknowledge it when interpreting results.

2. Maturation

Maturation refers to natural changes that happen over time. Physical growth, mental development, getting tired, or just getting older. If you're testing whether a reading program improves children's skills over a school year, kids naturally get better at reading as they age anyway. How do you know the program caused the improvement versus just natural development?

Random assignment to multiple groups solves this too. If you have a treatment group and a control group, both mature at roughly the same rate. The difference between them shows the treatment effect beyond maturation. The longer your study runs, the more maturation threatens your results.

3. Differential Selection

Despite its name, this actually refers to how you assign people to groups. If groups start out different, you can't trust your conclusions. Imagine comparing two therapy groups where one consists of volunteers who really want help, and another consists of people court-ordered to attend. Any differences in outcomes might reflect their initial motivation, not the therapy itself.

Random assignment is your defense here. When you randomly assign participants to groups, you create similar groups at the start. No random assignment? You've got a problem.

4. Statistical Regression

This one's subtle but important. Statistical regression happens when you select people specifically because they scored extremely high or low on something, and their scores naturally drift toward average over time.

Think about performance reviews at work. If you had your worst quarter ever, your next quarter will probably be better. Not necessarily because you improved, but because that awful quarter was an outlier influenced by random bad luck. Similarly, if you had your best quarter ever, the next one probably won't be quite as good.

In research, if you only study people with the most severe depression (extreme scorers), their scores might improve somewhat over time simply because they're unlikely to maintain that extreme level. Even without treatment. You might wrongly conclude your treatment worked when it was really just regression to the mean.

Control this by either avoiding selecting only extreme scorers or making sure all your groups have similar numbers of extreme scorers.

Worked example: a clinic picks only extreme scorers (made-up numbers).

  1. A clinic gives a depression scale to 200 adults. The group average is 20.
  2. It picks the 20 people with the highest scores. Their average is 40.
  3. Nobody gets any treatment. Four weeks later, the same 20 people retake the scale. Their average is now 33.
  4. Why the drop? Part of each extreme score came from a bad week or a little measurement error. That part does not repeat on the retest, so the scores slide part of the way back toward the group average of 20. The less reliable the scale, the bigger this slide.
  5. Had the clinic given a treatment, that 7-point drop (40 minus 33) would look like a treatment effect.
  6. The fix: randomly assign the extreme scorers to treatment or control. Both groups slide back by about the same amount, so the gap between the groups shows the real treatment effect.

So on the exam: people chosen because of extreme scores, then retested, with no comparison group, points to statistical regression.

5. Testing

Taking a test can change how you perform on that test later. If you take a practice EPPP exam, you'll likely do better on the next one. Not just because you studied, but because you remember some questions, you're less anxious, or you've learned test-taking strategies specific to that format.

In research, if you pretest participants on depression, then provide treatment, then posttest them, some improvement might come from taking the pretest rather than from the treatment. They might have thought more about their symptoms after the pretest, or they learned how to answer the questions in a way that looks better.

Solutions? Skip the pretest entirely, or use the Solomon four-group design (more on that later).

6. Instrumentation

Instrumentation problems occur when your measuring tool changes over time. Say you're having multiple coders watch therapy sessions and rate the quality. As they code more sessions, they might get better at spotting subtle behaviors, changing their ratings even though the actual behaviors haven't changed. Or a scale might get worn out, bathroom scales might become less accurate, or interviewers might get fatigued and less thorough.

The only real solution is keeping your measurement procedures completely consistent. If that's impossible, you need to acknowledge this limitation when reporting results.

7. Differential Attrition

People drop out of studies. That's normal. But differential attrition occurs when people drop out of different groups for different reasons, changing the composition of those groups in ways that affect results.

Suppose you're comparing two treatments for social anxiety. Treatment A is more demanding and time-consuming. The people who stick with Treatment A might be more motivated or have more flexible schedules than those who stick with Treatment B. If Treatment A shows better results, is it because the treatment is superior, or because only highly motivated people completed it?

This is tough to control because you often don't know why people drop out or how dropouts differ from those who stay. Your best bet is tracking dropout rates and characteristics, comparing completers to non-completers when possible, and being transparent about this limitation.

Worked example: spotting differential attrition (made-up numbers).

  1. 100 adults with social anxiety are randomly assigned: 50 to Treatment A and 50 to Treatment B.
  2. In Treatment A, 30 people drop out. Dropout rate = 30 / 50 = 60%.
  3. In Treatment B, 10 people drop out. Dropout rate = 10 / 50 = 20%.
  4. Now 20 people finish A and 40 finish B.
  5. If the 20 who finished A were the most motivated, A's better posttest scores could come from who stayed, not from the treatment.
  6. Check it: compare dropouts with completers on things measured at the start, like severity and motivation.
  7. Many trials also run an intention-to-treat analysis. That means everyone is analyzed in the group they were randomly assigned to, even people who dropped out. This helps keep the fair comparison that random assignment created.

So on the exam: groups that lose people at different rates, or for different reasons, after random assignment point to differential attrition.

The Four Threats to External Validity

Now we shift from "Did the treatment cause the effect?" to "Will this work outside the study?"

1. Reactivity

Reactivity means people act differently because they know they're being studied. This includes two specific problems:

Demand characteristics are cues that tell participants what you expect. If you're in a study testing a new relaxation technique and the researcher keeps asking "Do you feel more relaxed now? Notice how calm you feel?" you'll probably report feeling relaxed whether you actually do or not. You've picked up on what's expected.

Experimenter expectancy occurs when researchers inadvertently influence results. This could be as subtle as a researcher nodding and smiling when a participant gives the "right" answer, or as blatant as recording data in a biased way that supports the hypothesis.

Control reactivity through:

  • Unobtrusive measures: Observe behavior without people knowing
  • Deception: Don't reveal the study's true purpose (with proper ethical safeguards)
  • Single-blind technique: Participants don't know which group they're in (treatment vs. control)
  • Double-blind technique: Neither participants nor researchers interacting with them know who's in which group

Double-blind is the main fix for experimenter expectancy, since a researcher who doesn't know who got the treatment can't nudge that group. Some textbooks also list experimenter expectancy as a threat to internal validity. On a question, follow the clue it gives.

2. Multiple Treatment Interference

This occurs in within-subjects designs where each person receives multiple treatments or conditions. The order matters.

If you're testing three different meditation apps by having each person try all three over several weeks, the benefits you see with the third app might partly result from having already used the first two. Maybe participants just got generally better at meditating, or they're comparing each app to the previous ones rather than evaluating each on its own merits.

Counterbalancing solves this. You have different groups receive the treatments in different orders. If Group 1 tries App A, then B, then C, Group 2 might try B, C, A, and Group 3 tries C, A, B. The Latin square design is a formal approach where each treatment appears equally often in each position across groups.

In plain words, counterbalancing spreads order effects evenly, so no treatment always gets the first or last spot. Trying every possible order of 3 treatments (complete counterbalancing) takes 3 × 2 × 1 = 6 orders, and 4 treatments take 24. A Latin square needs only as many orders as treatments: 3 orders for 3 treatments, with A, B, and C each landing once in each position. One limit: counterbalancing balances order effects across groups. It does not stop carryover inside any one person.

3. Selection-Treatment Interaction

Your sample might differ from the general population in ways that affect how they respond to treatment. People who volunteer for psychology studies tend to be more educated, more psychologically minded, and more motivated than average. A therapy that works great with motivated volunteers might flop with reluctant clients who only came because their partner insisted.

The best control is random selection from the population. But this is often impractical or impossible. Be aware of how your sample might differ from those you want to generalize to, and be cautious about overextending your conclusions.

4. Pretest-Treatment Interaction

Also called pretest sensitization, this occurs when taking a pretest changes how people respond to the treatment. If you survey people about their attitudes toward mental health before providing educational materials, completing that survey might make them pay extra attention to the materials and think more deeply about them than they otherwise would. The treatment might appear more effective than it would be without the pretest.

The Solomon four-group design handles this elegantly:

GroupPretest?Treatment?Posttest?
Group 1YesYesYes
Group 2YesNoYes
Group 3NoYesYes
Group 4NoNoYes

In plain words, the Solomon design asks one question: is the treatment's boost the same size whether or not people took a pretest? If the boost is bigger in the pretested groups, the pretest is helping the treatment work, and the result may not hold for people who never take one.

Compare the treatment-control difference in pretested groups (Group 1 minus Group 2) with the difference in unpretested groups (Group 3 minus Group 4). This contrast tests whether the treatment effect depends on pretesting. Comparing Groups 1 and 3 alone mixes a pretest effect with a possible pretest-by-treatment interaction. Groups 2 and 4 show the pretest effect without treatment. (El Karkri et al., 2025)

Worked example: reading a Solomon four-group study (made-up posttest averages, higher = more mental health knowledge).

  1. Group 1 (pretest, treatment) = 30. Group 2 (pretest, no treatment) = 20. Group 3 (no pretest, treatment) = 26. Group 4 (no pretest, no treatment) = 20.
  2. Treatment effect with a pretest: Group 1 minus Group 2 = 30 minus 20 = 10 points.
  3. Treatment effect without a pretest: Group 3 minus Group 4 = 26 minus 20 = 6 points.
  4. Compare the two: 10 versus 6. The treatment helps more when people were pretested. That gap is a pretest-by-treatment interaction, a threat to external validity. For clients who never take a pretest, the 6-point effect is the better guess.
  5. Pretest effect alone: Group 2 versus Group 4 = 20 versus 20. The pretest by itself changed nothing, so there is no simple testing effect here.

So on the exam: unequal treatment-control gaps in pretested versus unpretested groups mean pretest-treatment interaction. Group 2 versus Group 4 shows the testing effect.

Step by step: name the threat in a vignette

Exam vignettes often describe a study and ask what went wrong.

Worked example: A researcher tests a new group therapy for teen depression at one school. She picks the 30 students with the highest depression scores. All 30 get the therapy. There is no control group. Scores improve a lot by the end of the school year. Halfway through, the school also hires two new counselors.

  1. Ask what the worry is: did the therapy cause the change (internal), or would it hold elsewhere (external)? Here it is cause.
  2. Look for a comparison group. There is none, so outside events and natural change can't be ruled out.
  3. Match each clue to a threat. "The 30 highest scores" points to statistical regression. "A whole school year" with teens points to maturation. "New counselors halfway through" points to history.
  4. Answer the threat the question points at. If it points at choosing only the highest scorers, the answer is statistical regression, not differential selection. Differential selection needs two groups that started out different.
  5. Name the fix: randomly assign the high scorers to therapy or control. Both groups then share regression, maturation, and history.

So on the exam: first decide cause (internal) or generalizing (external), then match the key clue to the threat.

Common Mistakes Students Make

Mistake #1: Confusing internal and external validity

Students often mix these up or think they're the same thing. Remember: Internal validity = cause-and-effect accuracy. External validity = generalizability. A study can have one without the other.

Mistake #2: Thinking random selection and random assignment are the same

Random selection means choosing participants randomly from a population (helps external validity). Random assignment means assigning chosen participants randomly to groups (helps internal validity). These serve different purposes.

Quick example: a clinic network has 5,000 clients with insomnia. You draw 100 names at random. That is random selection, and it helps your results generalize to all 5,000. Then you flip a coin for each of the 100 to decide therapy or waitlist. That is random assignment, and it helps you say the therapy caused the change. Many clinical trials use random assignment with volunteers, so they have random assignment but no random selection.

Mistake #3: Believing that having threats means the study is worthless

Nearly every study has some threats to validity. The question is whether they're controlled or acknowledged. A study might have differential attrition but still provide valuable information if researchers measured and reported the characteristics of dropouts.

Mistake #4: Not recognizing that internal and external validity trade-offs exist

Highly controlled lab studies might nail down cause-and-effect (internal validity) but feel artificial and not generalize well (poor external validity). Naturalistic studies in real clinics might be highly generalizable (external validity) but have many uncontrolled factors (weaker internal validity). Researchers make strategic choices about which to prioritize.

Mistake #5: Forgetting that maturation and history are controlled the same way

Both are controlled through multiple groups with random assignment. Students sometimes think they need different solutions for each threat.

Memory Strategies for the EPPP

For Internal Validity Threats, use the acronym "HI MISS DID" plus a T:

  • History
  • Instrumentation
  • Maturation
  • I (skip this one. Just helps the acronym work)
  • Statistical regression
  • Selection (differential selection)
  • Differential attrition
  • I (skip)
  • D (skip)
  • Testing

Or try "HIST DIMS" (History, Instrumentation, Selection, Testing, Differential attrition, Maturation, Statistical regression; the second I is just filler). Use whichever works for you.

If the skips bother you, try "THIS RAM": Testing, History, Instrumentation, Selection (differential), Regression (statistical), Attrition (differential), Maturation. Seven letters, seven threats, no skips.

For External Validity Threats, remember "RAMPS":

  • Reactivity
  • A (skip)
  • Multiple treatment interference
  • Pretest-treatment interaction
  • Selection-treatment interaction

Key design solutions to remember:

ProblemSolution
History, maturation, differential selectionMultiple groups + random assignment
Statistical regressionAvoid only extreme scorers, or balance extreme scorers across groups
Testing effectsNo pretest, or Solomon four-group design
ReactivityBlind procedures, unobtrusive measures, deception
Multiple treatment interferenceCounterbalancing (Latin square)
Pretest-treatment interactionSolomon four-group design

Visual connection strategy:

For internal validity threats, think "INTERNAL = IN the study went wrong." These are problems with what happened during the research.

For external validity threats, think "EXTERNAL = OUT there in the world." These are problems with applying findings outside the study context.

Key Takeaways

  • Internal validity determines whether you can trust a study's cause-and-effect conclusions. External validity determines whether those conclusions apply beyond the study's specific conditions.

  • External validity has five types: population (other people), ecological (other settings), temporal (other times), treatment variation (modified treatments), and outcome (related measures).

  • The seven internal validity threats are history, maturation, differential selection, statistical regression, testing, instrumentation, and differential attrition. Most are controlled through random assignment to multiple groups.

  • The four external validity threats are reactivity, multiple treatment interference, selection-treatment interaction, and pretest-treatment interaction. Control methods vary but include blind procedures, counterbalancing, and specialized designs.

  • Random assignment helps internal validity by creating equivalent groups. Random selection helps external validity by ensuring your sample represents the population.

  • The Solomon four-group design simultaneously addresses testing as a threat to internal validity and pretest-treatment interaction as a threat to external validity.

  • Studies almost always have some threats to validity. The question is whether they're adequately controlled, measured, or acknowledged. Perfect studies are rare.

  • When reading research, always ask: "Did the treatment truly cause this effect?" (internal validity) and "Will this work with my clients in my setting?" (external validity). These questions guide evidence-based practice.

Understanding validity helps you become a critical consumer of research, which directly impacts the quality of care you provide. You'll know which studies to trust, which to question, and which to apply cautiously with specific populations or settings.

Compare the right groups and time periods

In plain words: to show a treatment caused a change, compare the treated people with similar people over the same stretch of time, and watch for anything else that changed at the same moment. A confound is any other factor that could explain the result. An interrupted time series measures the same thing many times before and many times after a program starts, then looks for a sudden jump (a level change) or a change in direction (a trend change) right when the program began.

Internal validity asks whether the intervention explains the observed difference. History is an outside event during the study; maturation is change within participants over time. Instrumentation is a change in measurement, testing is an effect of taking the measure, and selection is a preexisting group difference. Unequal dropout can leave different kinds of people in the groups. A comparison group measured over the same period helps address shared events. An interrupted time series uses many observations before and after an intervention to examine level or trend changes. A simultaneous outside event can still confound it. Random assignment reduces systematic allocation differences but does not guarantee identical groups or a representative sample. External validity concerns other people, settings, and times. In a Solomon design, compare the treatment-control difference among pretested groups with that difference among groups without a pretest to examine pretest-by-treatment interaction. (Slater & Hasson, 2025; Handley et al., 2018; El Karkri et al., 2025)

Worked example: reading an interrupted time series (made-up numbers).

  1. A clinic counts missed appointments every month for 12 months before it starts text reminders. Misses hold steady at about 100 a month.
  2. It keeps counting for 12 months after the reminders start.
  3. Level change: misses drop to about 80 in the first month and stay there. That is a sudden jump down of about 20.
  4. Trend change: misses fall by about 2 each month, ending near 76 after 12 months (100 minus 2 × 12 = 76). That is a change in direction, not a sudden jump.
  5. Now check for threats. In the same month the reminders started, a new bus line opened near the clinic. That outside event (history) could also explain fewer misses.
  6. The fix: count misses over the same months at a comparison clinic on the same bus line with no reminders. If its misses stay near 100, the bus line is a less likely explanation.

So on the exam: many measurements before and after a program means interrupted time series, and an outside event at the same time means history, which a comparison series helps rule out.

Ready to practice?

Get started in the app