Null Hypothesis

no effect, no difference, no relationship; always includes an equality sign

Alternative Hypothesis

an effect, difference, or relationship exists, uses > or <, never an equality

P-value

calculated after you collect your data. A small p-value suggests your result is unlikely under the null, so it’s evidence against the null

Alpha-level

fixed threshold you chose before collecting data. If alpha= 0.05, 5% of tests will wrongly reject a true hypothesis

A-priori

planned comparisons made before data collection, based on theory, prior research, or specific research questions. It’s purpose is to test specific, pre-specified hypotheses rather than explore patterns in the data

Post-hoc

comparisons made after data collection, often to explore which groups differ once the overall null hypothesis is rejected. It’s purpose is to identify specific pairwise differences among groups, often in exploratory analysis

Null distribution

the probability distribution of a test statistic when the null hypothesis is true

Sampling distribution

the distribution of a statistic (sample mean, sample proportion) across all possible samples of size n from the same population

P-values is

the probability, under the null distribution, of getting a result as extreme or more extreme than the one observed

Independent variable

factor you intentionally change to see how it affects something else

Dependent variable

what you measure or observe in response to the change you made

Control variable

the factors you keep the same so they do not influence the dependent variable

Causal inference in hypothesis testing

The process of using statistical tests to determine whether a treatment or intervention causes a change in an outcome, rather than merely being associated with it Did X cause Y?

Type 1 error

rejecting a true null hypothesis → you conclude an effect exists when it actually does not

Type 2 error

failing to reject a false null hypothesis → you fail to detect an effect that truly exists

Test-retest reliability

measures stability over time; a test is given twice to the same group; scores are correlated

Inter-rater reliability

measures consistency across observers or raters, used when human judgement is involved (grading essays, coding behavior)

Internal consistency reliability

consistency among items within a test, assesses whether items measure the same underlying construct

Content validity

whether the test fully represents the construct

Construct Validity

asks whether the test truly measures the theoretical construct

Criterion Validity

how well a measure predicts or correlates with an external criterion

Face validity

whether a test appears to measure what it claims to measure

Relationship between reliability and validity

Reliability= consistency Validity= accuracy

Types of measures/observations

The Four Classical Measurement Scales (Nominal, Ordinal, Interval, Ration) Qualitative vs. Quantitative Discrete vs. Continuous

Nominal

labels or categories with no inherent order (gender, blood type, ZIP code)

Ordinal

categories with a meaningful order, but unequal spacing between levels (class rank, pain levels)

Interval

numerical values with equal intervals, but zero is arbitrary ( temperature is Celsius or Fahrenheit, calendar years)

Ratio

numerical values with equal intervals and a meaningful zero

Simple Random Sampling

every individual has an equal chance of being selected

Systematic Sampling

every _-th individual after a random start

Stratified Sampling

divide the population into strata (age groups, income levels) and sample from each

Cluster Sampling

randomly select groups (clusters), then sample all or some individuals within them

Multistage Sampling

a combination of methods (randomly select counties → schools → students)

Convenience Sampling

select individuals who are easiest to reach

Voluntary Response Sampling

participants choose themselves (online polls)

Purposive (judgment) Sampling

researcher selects individuals based on specific characteristics

Quota Sampling

fills quotas for certain groups (50 men, 50 women) without random selection

Snowball Sampling

participants recruit others, often used for hidden or hard-to-reach populations

Generalizability concerns

Whether results derived from a specific sample, model, or context can reliably extend to new populations, settings, or time periods

The relationship between Type 1/Type 2 error and sample size

1. Increasing sample size increases power 2. Lowering alpha reduces power 3. effect size matters

Manipulations

allow researchers to establish cause-and-effect relationships

Random assignment

The process of placing participants into experimental groups using chance so that each person has an equal probability of being in any condition

Control groups vs. Control variables

Control group: baseline group in an experiment that does not receive the independent variable. Purpose is to show what happens without the treatment so the researcher can compare it to the experimental group Control variables: factors that the researcher intentionally keeps constant across all groups so they do not influence the outcome

Within-participant

exposes each participant to every level of the independent variable

Between-participant

assigns different participants to different conditions

Mixed designs

includes at least one within-participant factor and one between-participant factor

Relationship between causal inference and experimentation

Causal inference is the goal-figuring out what causes what Experimentation is the most reliable way to do it, but not the only way

Confounds

Influences both a treatment and an outcome Example: studying whether exercise reduces stress → people with high workloads may exercise less; workload also directly increases stress → workload is a confound: it influences both exercise (treatment) and stress (outcome)

Factorial Design

An experiment that tests two or more treatments at the same time so you can see each treatment’s effect and whether they work differently when combined You can learn interaction effects Used when you want to test multiple ideas efficiently and when you think treatments might interact

Interactions

When the effect of one independent variable depends on the level of another, meaning the variables do not operate independently

Internal Validity

the accuracy of the causal claim. Asks whether the observed effect in an experiment was truly caused by the independent variable rather than by confounding factors

External Validity

about the generalizability of the findings. Asks whether the results would hold for other people, settings, times, or variations of the procedure

Longitudinal Design

Research approach in which the participants are studied repeatedly over an extended period to track changes, development, or causal effects over time

Cross-sectional design

Measures a population or sample at a single point in time to examine the prevalence, relationships, or differences among variables

The 'cohort' problem in cross-sectional research

The risk that differences between age groups in a cross-sectional study reflect generational (cohort) differences, not true developmental or age-related changes For example: If 20-year-olds score higher on a technology-use test than 60-year-olds, is it because younger adults are at a different developmental stage or because they grew up with digital technology and belong to a different cohort?

Reasons for Quasi-Experimental Design

Used when researchers want to study causal relationships but cannot randomly assign participants due to practical, ethical, or logistical constraints

Strengths and Weaknesses of Quasi-Experimental Design

Strengths: useful when you can’t randomly assign people (school programs, policies), results often apply well to everyday settings, often use existing groups or data, great for studying big problems or policies - especially when they roll out unevenly Weaknesses: Weaker proof of causation because there’s no random assignment, groups may differ in important ways before the study even starts, people may choose their group which complicates results, other factors may explain the outcome (timing, environment)

Mixing quasi-experimental design with experimental variables

When you put them together, you get a design where one factor is fully controlled and the other is naturally occurring Example: Two schools → within each school you randomly assign students to method a or b → school (quasi experimental variable) → teaching method (experimental variable)

Correlational Research

Examines how strongly two or more variables are related without manipulating them

What correlation measures as a concept

Measures strength and direction of a relationship between two variables, show how they change together but not whether one causes the other

Naturalistic Observational Research

Researchers study behavior in its real-world environment without manipulating variables or interfering with what is happening Focuses on observing people, animals, or social settings exactly as they occur in everyday life

Participant Observation

Researcher actively engages in the setting or group being studied while simultaneously observing its behaviors, interactions, and norms Involves active involvement in the setting, simultaneous observation, extended engagement, fieldnotes, and reflexivity

Structured Observation

Used when a researcher wants to capture naturally occurring behavior but still maintain a high level of consistency across observers, settings, or time. Relies on a predetermined framework that specifies what will be observed, how it will be recorded, and under what conditions

Case Studies

In-depth, detailed investigation of a single case- such as a person, group, organization, event, or community- used to understand real-world phenomena

Archival Research

Method of studying original, historical, or primary-source records preserved in archives to answer research questions about the past

Practical relationship between internal and external validity

Internal validity refers to how confidently a study can claim that X caused Y, free from confounding factors. External validity refers to how well the findings generalize beyond the study- across people, settings, times, and contexts. The more tightly a study controls conditions to establish cause-and-effect, the less its findings may generalize to real-world settings, and vice versa

Exclusion Items

Questions or prompts used to screen out respondents who do not meet study criteria or who fail to provide trustworthy answers Researchers use exclusion items for (1) Ineligible participants (e.g., a study meant for parents excludes non-parents) (2) speeders who rush through surveys without reading (3) random responders who choose answers arbitrarily (4) bots or fraudulent participants in online panels (5) participants who misrepresent themselves to qualify for incentives

Counterbalancing

Helps ensure responses are not influenced by the sequence in which items appear. In surveys and experiments, the order of questions or conditions can unintentionally shape how people think, recall information, or interpret later items

True Experimental Designs

Strengths: strongest evidence for causality, high internal validity. Replicable Weaknesses: artificial settings may reduce external validity, many variables cannot be manipulated (trauma, socioeconomic status), resource intensive (time, money, etc.)

Quasi-Experimental Designs

Strengths: useful when randomization is impossible or unethical, more naturalistic than true experiments Weaknesses: weaker causal claims, lower internal validity

Correlational Designs

Strengths: efficient for studying large samples and multiple variables, allows study of variables that cannot be manipulated, useful for prediction Weaknesses: cannot establish causality, vulnerable to third-variable problems and directionality issues, measurement error can distort relationships

Observational Designs

Strengths: high ecological validity, useful for studying rare, sensitive, or complex phenomena, can generate hypotheses for later experimental testing Weaknesses: no causal inference possible, observer bias and reactivity (people change behavior when watched), hard to control extraneous variables

Qualitative Designs

Strengths: provides rich, detailed understanding of complex human experiences, flexible and responsive to context, useful for theory development and exploring new phenomena Weaknesses: not generalizable to large populations, subjective interpretation may introduce bias, cannot test causal hypotheses

The validity and reliability of measures

Validity refers to whether a measure actually captures the concept it is intended to measure, and reliability refers to the consistency or stability of that measure across time, items, or raters

Why scales of measurement matter

It determines what analyses are valid, what conclusions can be drawn, and how accurately your data represents the phenomenon you are studying

How p-values are derived, and what they really indicate

They come from probability models and quantify how incompatible your data is with the null hypothesis It tells you how surprising your data would be if the null hypothesis were actually true, not the probability that the null is true

The relationship between power, effect size, and sample size

Power increases when effect size or sample size increases because larger effects and larger samples make true differences easier to detect Each influences the other

The relationship between a-level and type 1/2 error

The alpha level sets the probability of a Type 1 error, and lowering alpha reduces Type 1 errors but increases the chance of Type 2 errors Lower alpha = fewer Type 1 errors, more Type 2 errors, lower power Higher alpha = more Type 1 errors, fewer Type 2 errors, higher power

Scientific Criteria for causal inference

Scientists decide whether A causes B by checking whether A truly makes a difference, ruling out other explanations, and looking for consistent, believable patterns in data They look for a real difference when A changes, experiments when possible, and when experiments aren’t possible, patterns that support causation

Between- Participant Designs

Pros: no order effects (experience only one condition), participants cannot compare conditions, shorter, simpler sessions, better for irreversible or lasting treatments Cons: requires larger samples, more vulnerable to individual differences (pre-existing differences like mood and ability add noise and threaten internal validity), needs careful random assignment to balance groups and avoid confounds

Within-Participant Designs

Pros: each participant serves as their own control, reducing error variance, smaller sample sizes (far fewer participants needed to detect differences), controls individual differences (stable traits like IQ, motivation, cancel out because the same person appears in all conditions), useful for detecting subtle effects Cons: practice, fatigue, and carryover can distort results; participants see multiple conditions and may infer the study’s purpose (risk of hypothesis guessing), longer sessions, not suitable for irreversible manipulations (learning a skill or receiving a permanent treatment)

Interactions in Between-Participant Designs

Different groups experience each condition Compare group means across all combinations of the IVs Look for the difference between groups under one condition not being the same as the difference under another Interactions rely on larger samples to detect reliably Between-subjects ANOVA

Interactions in Within-Participant Designs

The same participants experience all conditions Compare each participant’s pattern of change across the levels of both IVs Repeated-measures ANOVA

Mixed Designs

Test whether the within-participant effect differs across between-participant groups Group x condition interaction

Conditions in factorial design

Each condition is just one combination of the factor levels and the total number of conditions is the product of the number of levels in each factor Example: 2x2 design = 4 conditions, 2x3x4 design = 24 conditions

Write and read factorial notation (e.g. A study had a 2x3x2 between-participants design)

Contains → Factor 1: 2 levels, Factor 2: 3 levels, Factor 3: 2 levels = three independent variables 2x3x2= 12 experimental conditions Factor A (2 levels): Low vs. High temperature; Factor B (3 levels): short, medium, long exposure time, Factor C (2 levels): drug vs. placebo A factorial design allows you to study main effects, two-way interactions, and three-wau interactions 2x2= two factors, each with two levels, 3x2=one factor with 3 levels, one with 2, 4x3x2= three factors with 4, 3, and 2 levels