How to test consistency of responses?

I am analyzing data from several trials where individuals were submitted to different stimuli and a biological response was measured. Some individuals participated in more than one trial where the same stimulus was used.

Here is some dummy data that mimics what I have:

Subject Stimulus    Response
A       1           44.8
A       1           49.1
A       1           46.5
A       2           10.3
A       2           19.1
B       1           58.1
B       1           79.4
B       3           79.1
B       3           69.2
...

My aim is to determine if patients react differently to different stimuli, but before that, I want to know if their response to the same stimulus is consistent. If so, I am thinking of averaging the multiple observations that individuals have for the same stimulus and then find out if in general the response to different stimuli is different.

My question is: With a structure like this, how do I test if patients have a similar response to the same stimulus?

I read about test-retest reliability, but I am not sure how to do it or if it is applicable to this case. I use R for all my statistical analysis. Any suggestions are welcome.

Cross Validated Asked by user4451922 on January 7, 2022

One approach here would be a components of variance model, where you partition the total variance in the data in parts due to Subjects, Stimulus, and Error (Response). This can be estimated as a random effects model, for example in R with the package lme4 something like:

mod <- lme4::lmer(Response ~ (1 | Subject / Stimulus), data=your_data_frame)

Then you must judge if the component of variance due to stimulus is small enough.

Here is an example with simulated data:

mod <- lme4::lmer(Response ~ (1 | Subject/Stimulus), data=df)
mod
Linear mixed model fit by REML ['lmerMod']
Formula: Response ~ (1 | Subject/Stimulus)
Data: df
REML criterion at convergence: 559.3188
Random effects:
Groups           Name        Std.Dev.
Stimulus:Subject (Intercept)  2.2841
Subject          (Intercept) 12.1752
Residual                      0.4322
Number of obs: 180, groups:  Stimulus:Subject, 60; Subject, 20
Fixed Effects:
(Intercept)
40.09

Code for simulated data:

N_subj  <- 20
N_stimu <- 3
N_repl  <- 3

sd_subj <- 10
sd_stimu <- 2
sd_repl <- 0.5

set.seed(7*11*13) # My public seed
N_total <- N_subj * N_stimu * N_repl
Subject <- as.factor(rep(1:N_subj, rep(N_stimu*N_repl, N_subj)))
Stimulus <- as.factor(rep(rep(1:N_stimu, rep(N_repl, N_stimu)), N_subj))

Response <- 40+rep(rnorm(N_subj, 0, sd_subj), rep(N_stimu*N_repl, N_subj)) +
rep(rnorm(N_stimu*N_subj, 0, sd_stimu), rep(N_repl, N_stimu*N_subj)) +
rnorm(N_total, 0, sd_repl)

df <- data.frame(Response, Subject, Stimulus)

Answered by kjetil b halvorsen on January 7, 2022

A textbook approach here would be to use a linear model with a term for each subject and a term for each stimulus. A classical analysis of variance (ANOVA) using such a model can compare the variation within the same subject or stimulus to the variation between different subjects or stimuli.

my_lm = lm(Response ~ Subject + Stimulus, data = your_dataframe)
aov(my_lm)

If you get a significant result for the coefficients describing the effect of the stimulus, that's the smoking gun. You should make sure the size of the effect is scientifically meaningful, though. It's good practice to also check the ANOVA assumptions (constant variance, independent errors, and less importantly, rough normality).

Answered by eric_kernfeld on January 7, 2022

Related Questions

How to determine relationship categorical and numerical data

1  Asked on January 9, 2021 by onhalu

Multiple Poisson regression (?) in R

2  Asked on January 9, 2021 by jonas8

Propose a model for this time series

1  Asked on January 8, 2021 by le-anh-dung

Would a 3D CNN require less training samples than a corresponding 2D CNN?

0  Asked on January 8, 2021 by alexander-soare

Can regression to the mean be corrected by linear mixed effects?

0  Asked on January 8, 2021 by lili

T value vs T-stat

1  Asked on January 8, 2021 by student010101

How can I perform a two-sample multivariate t-test where one group is a subset of the other?

0  Asked on January 7, 2021 by grint

Minimize the limit of K-L (Kullback Leibler) divergence for a given conditional probability $p(y|x)$ distribution?

0  Asked on January 7, 2021

Can I use coefficients of one set of regressions as dependent variable in a new regression?

1  Asked on January 7, 2021 by jeremy

What’s a word meaning “drawn from the same distribution”?

0  Asked on January 6, 2021 by gkhagb

What Statistical principles are being violated by comparing specific Trainer Fatality Rates to Race Track Fatality rates?

0  Asked on January 6, 2021 by pseudoego

How to automatically choose the number of components for PCA?

1  Asked on January 6, 2021 by foobar

Cosine Similarity Intuition

3  Asked on January 6, 2021 by ccb

Is there a way to get the optimal cutoff points based on probability of topic models and the outcomes?

1  Asked on January 6, 2021 by kuni

How can I use the box plot to explain the Empirical Rule for a normal distribution?

1  Asked on January 6, 2021 by storymay

PCA: Dimension Reduction

0  Asked on January 5, 2021 by shank

How to choose a good operation point from precision recall curves?

4  Asked on January 5, 2021 by amelio-vazquez-reina

How to develop a likelihood based prediction model to predict chance of rain in a particular hour of a year?

0  Asked on January 5, 2021 by nahid

How well does GAN (generative adversarial network) perform for small samples?

1  Asked on January 4, 2021

Using the Hotelling package in R

1  Asked on January 4, 2021 by pitchounet