Research – Internal/External Validity

Why Internal and External Validity Actually Matter

You're going to spend years working with clients, and you need to know which treatments actually work. Should you use that new therapy everyone's talking about? Does the research showing it works actually hold up? This is where internal and external validity come in. These concepts help you figure out which studies you can trust and which ones have flaws that make their findings questionable.

Internal validity answers one question: Did the treatment actually cause the change we saw? External validity asks: Will these results work in my office with my real clients? Both are essential, but they serve different purposes. Let's break down how to evaluate research so you can make informed decisions in your practice.

The Foundation: What These Terms Really Mean

Internal validity tells you whether a study's conclusions about cause-and-effect are accurate. {{M}}If you started a new diet and lost weight, but also started a new job with longer hours and more stress during that same period, you couldn't be sure which factor caused the weight loss.{{/M}} Research studies face the same problem. Did the therapy help, or was it something else that happened during the study?

External validity tells you how far you can extend those findings. A study might have solid internal validity (the treatment definitely caused the improvement) but if it only worked with college students in a lab setting, will it work with your 45-year-old client dealing with job loss and divorce? That's external validity.

Here's what makes this tricky: these two types of validity often conflict. {{M}}It's like trying to test whether a new kitchen gadget works. You could test it in a perfectly controlled test kitchen with ideal conditions (high internal validity), but that doesn't tell you if it'll work in your actual messy kitchen with your habits and constraints (external validity).{{/M}} Researchers constantly balance these priorities.

Breaking Down External Validity Into Five Parts

External validity isn't just one thing. It has five distinct components:

Understanding these distinctions helps you evaluate studies more critically. A study might have great population validity but poor ecological validity, meaning the results apply to different groups of people but might not translate to real-world clinical settings.

The Seven Threats to Internal Validity

Think of these as the ways a study can go wrong when trying to establish cause-and-effect. You need to spot these when reading research.

1. History

History occurs when outside events during the study affect the results. {{M}}Imagine you're testing a new stress-reduction app with participants over three months, and halfway through the study, a major company in your city announces massive layoffs. Suddenly, everyone's stress levels spike. Not because of anything in your study, but because of this external event.{{/M}}

The solution? Use multiple groups with random assignment. If you randomly assign participants to a treatment group (gets the app) and a control group (doesn't get the app), both groups experience the same historical events. Any difference between them at the end isn't due to history.

There's a trickier version: when you're testing groups separately and something happens to just one group. {{M}}Say you're running therapy groups, and during one session, the fire alarm goes off and everyone has to evacuate. That disruption only affects that particular group.{{/M}} This is harder to control and you need to acknowledge it when interpreting results.

2. Maturation

Maturation refers to natural changes that happen over time. Physical growth, mental development, getting tired, or just getting older. {{M}}If you're testing whether a reading program improves children's skills over a school year, kids naturally get better at reading as they age anyway.{{/M}} How do you know the program caused the improvement versus just natural development?

Random assignment to multiple groups solves this too. If you have a treatment group and a control group, both mature at roughly the same rate. The difference between them shows the treatment effect beyond maturation. The longer your study runs, the more maturation threatens your results.

3. Differential Selection

Despite its name, this actually refers to how you assign people to groups. If groups start out different, you can't trust your conclusions. {{M}}Imagine comparing two therapy groups where one consists of volunteers who really want help, and another consists of people court-ordered to attend. Any differences in outcomes might reflect their initial motivation, not the therapy itself.{{/M}}

Random assignment is your defense here. When you randomly assign participants to groups, you create similar groups at the start. No random assignment? You've got a problem.

4. Statistical Regression

This one's subtle but important. Statistical regression happens when you select people specifically because they scored extremely high or low on something, and their scores naturally drift toward average over time.

{{M}}Think about performance reviews at work. If you had your worst quarter ever, your next quarter will probably be better. Not necessarily because you improved, but because that awful quarter was an outlier influenced by random bad luck. Similarly, if you had your best quarter ever, the next one probably won't be quite as good.{{/M}}

In research, if you only study people with the most severe depression (extreme scorers), their scores might improve somewhat over time simply because they're unlikely to maintain that extreme level. Even without treatment. You might wrongly conclude your treatment worked when it was really just regression to the mean.

Control this by either avoiding selecting only extreme scorers or making sure all your groups have similar numbers of extreme scorers.

5. Testing

Taking a test can change how you perform on that test later. {{M}}If you take a practice EPPP exam, you'll likely do better on the next one. Not just because you studied, but because you remember some questions, you're less anxious, or you've learned test-taking strategies specific to that format.{{/M}}

In research, if you pretest participants on depression, then provide treatment, then posttest them, some improvement might come from taking the pretest rather than from the treatment. They might have thought more about their symptoms after the pretest, or they learned how to answer the questions in a way that looks better.

Solutions? Skip the pretest entirely, or use the Solomon four-group design (more on that later).

6. Instrumentation

Instrumentation problems occur when your measuring tool changes over time. {{M}}Say you're having multiple coders watch therapy sessions and rate the quality. As they code more sessions, they might get better at spotting subtle behaviors, changing their ratings even though the actual behaviors haven't changed.{{/M}} Or a scale might get worn out, bathroom scales might become less accurate, or interviewers might get fatigued and less thorough.

The only real solution is keeping your measurement procedures completely consistent. If that's impossible, you need to acknowledge this limitation when reporting results.

7. Differential Attrition

People drop out of studies. That's normal. But differential attrition occurs when people drop out of different groups for different reasons, changing the composition of those groups in ways that affect results.

{{M}}Suppose you're comparing two treatments for social anxiety. Treatment A is more demanding and time-consuming. The people who stick with Treatment A might be more motivated or have more flexible schedules than those who stick with Treatment B. If Treatment A shows better results, is it because the treatment is superior, or because only highly motivated people completed it?{{/M}}

This is tough to control because you often don't know why people drop out or how dropouts differ from those who stay. Your best bet is tracking dropout rates and characteristics, comparing completers to non-completers when possible, and being transparent about this limitation.

The Four Threats to External Validity

Now we shift from "Did the treatment cause the effect?" to "Will this work outside the study?"

1. Reactivity

Reactivity means people act differently because they know they're being studied. This includes two specific problems:

Demand characteristics are cues that tell participants what you expect. {{M}}If you're in a study testing a new relaxation technique and the researcher keeps asking "Do you feel more relaxed now? Notice how calm you feel?" you'll probably report feeling relaxed whether you actually do or not. You've picked up on what's expected.{{/M}}

Experimenter expectancy occurs when researchers inadvertently influence results. {{M}}This could be as subtle as a researcher nodding and smiling when a participant gives the "right" answer, or as blatant as recording data in a biased way that supports the hypothesis.{{/M}}

Control reactivity through:

• Unobtrusive measures: Observe behavior without people knowing
• Deception: Don't reveal the study's true purpose (with proper ethical safeguards)
• Single-blind technique: Participants don't know which group they're in (treatment vs. control)
• Double-blind technique: Neither participants nor researchers interacting with them know who's in which group

2. Multiple Treatment Interference

This occurs in within-subjects designs where each person receives multiple treatments or conditions. The order matters.

{{M}}If you're testing three different meditation apps by having each person try all three over several weeks, the benefits you see with the third app might partly result from having already used the first two. Maybe participants just got generally better at meditating, or they're comparing each app to the previous ones rather than evaluating each on its own merits.{{/M}}

Counterbalancing solves this. You have different groups receive the treatments in different orders. If Group 1 tries App A, then B, then C, Group 2 might try B, C, A, and Group 3 tries C, A, B. The Latin square design is a formal approach where each treatment appears equally often in each position across groups.

3. Selection-Treatment Interaction

Your sample might differ from the general population in ways that affect how they respond to treatment. {{M}}People who volunteer for psychology studies tend to be more educated, more psychologically minded, and more motivated than average. A therapy that works great with motivated volunteers might flop with reluctant clients who only came because their partner insisted.{{/M}}

The best control is random selection from the population. But this is often impractical or impossible. Be aware of how your sample might differ from those you want to generalize to, and be cautious about overextending your conclusions.

4. Pretest-Treatment Interaction

Also called pretest sensitization, this occurs when taking a pretest changes how people respond to the treatment. {{M}}If you survey people about their attitudes toward mental health before providing educational materials, completing that survey might make them pay extra attention to the materials and think more deeply about them than they otherwise would.{{/M}} The treatment might appear more effective than it would be without the pretest.

The Solomon four-group design handles this elegantly:

By comparing Groups 1 and 3 (both got treatment, only 1 got pretested), you see whether pretesting affected the treatment's effectiveness. By comparing Groups 2 and 4 (neither got treatment, only 2 got pretested), you see whether pretesting alone affected posttest scores. This design is powerful but requires more participants.

Common Mistakes Students Make

Mistake #1: Confusing internal and external validity

Students often mix these up or think they're the same thing. Remember: Internal validity = cause-and-effect accuracy. External validity = generalizability. A study can have one without the other.

Mistake #2: Thinking random selection and random assignment are the same

Random selection means choosing participants randomly from a population (helps external validity). Random assignment means assigning chosen participants randomly to groups (helps internal validity). These serve different purposes.

Mistake #3: Believing that having threats means the study is worthless

Nearly every study has some threats to validity. The question is whether they're controlled or acknowledged. A study might have differential attrition but still provide valuable information if researchers measured and reported the characteristics of dropouts.

Mistake #4: Not recognizing that internal and external validity trade-offs exist

{{M}}Highly controlled lab studies might nail down cause-and-effect (internal validity) but feel artificial and not generalize well (poor external validity). Naturalistic studies in real clinics might be highly generalizable (external validity) but have many uncontrolled factors (weaker internal validity).{{/M}} Researchers make strategic choices about which to prioritize.

Mistake #5: Forgetting that maturation and history are controlled the same way

Both are controlled through multiple groups with random assignment. Students sometimes think they need different solutions for each threat.

Memory Strategies for the EPPP

For Internal Validity Threats, use the acronym "HI MISS DID":

• History
• Instrumentation
• Maturation
• I (skip this one. Just helps the acronym work)
• Statistical regression
• Selection (differential selection)
• Differential attrition
• I (skip)
• D (skip)
• Testing

Or try "HIST DIMS" (History, Instrumentation, Selection, Testing, Differential attrition, Instrumentation, Maturation, Statistical regression). Use whichever works for you.

For External Validity Threats, remember "RAMPS":

• Reactivity
• A (skip)
• Multiple treatment interference
• Pretest-treatment interaction
• Selection-treatment interaction

Key design solutions to remember:

Visual connection strategy:

For internal validity threats, think "INTERNAL = IN the study went wrong." These are problems with what happened during the research.

For external validity threats, think "EXTERNAL = OUT there in the world." These are problems with applying findings outside the study context.

Key Takeaways

• Internal validity determines whether you can trust a study's cause-and-effect conclusions. External validity determines whether those conclusions apply beyond the study's specific conditions.

• External validity has five types: population (other people), ecological (other settings), temporal (other times), treatment variation (modified treatments), and outcome (related measures).

• The seven internal validity threats are history, maturation, differential selection, statistical regression, testing, instrumentation, and differential attrition. Most are controlled through random assignment to multiple groups.

• The four external validity threats are reactivity, multiple treatment interference, selection-treatment interaction, and pretest-treatment interaction. Control methods vary but include blind procedures, counterbalancing, and specialized designs.

• Random assignment helps internal validity by creating equivalent groups. Random selection helps external validity by ensuring your sample represents the population.

• The Solomon four-group design simultaneously addresses testing as a threat to internal validity and pretest-treatment interaction as a threat to external validity.

• Studies almost always have some threats to validity. The question is whether they're adequately controlled, measured, or acknowledged. Perfect studies are rare.

• When reading research, always ask: "Did the treatment truly cause this effect?" (internal validity) and "Will this work with my clients in my setting?" (external validity). These questions guide evidence-based practice.

Understanding validity helps you become a critical consumer of research, which directly impacts the quality of care you provide. You'll know which studies to trust, which to question, and which to apply cautiously with specific populations or settings.