How to test consistency of responses?

Cross Validated Asked by user4451922 on January 7, 2022

I am analyzing data from several trials where individuals were submitted to different stimuli and a biological response was measured. Some individuals participated in more than one trial where the same stimulus was used.

Here is some dummy data that mimics what I have:

Subject Stimulus    Response
A       1           44.8
A       1           49.1
A       1           46.5
A       2           10.3
A       2           19.1
B       1           58.1
B       1           79.4
B       3           79.1
B       3           69.2
...


My aim is to determine if patients react differently to different stimuli, but before that, I want to know if their response to the same stimulus is consistent. If so, I am thinking of averaging the multiple observations that individuals have for the same stimulus and then find out if in general the response to different stimuli is different.

My question is: With a structure like this, how do I test if patients have a similar response to the same stimulus?

I read about test-retest reliability, but I am not sure how to do it or if it is applicable to this case. I use R for all my statistical analysis. Any suggestions are welcome.

One approach here would be a components of variance model, where you partition the total variance in the data in parts due to Subjects, Stimulus, and Error (Response). This can be estimated as a random effects model, for example in R with the package lme4 something like:

mod <- lme4::lmer(Response ~ (1 | Subject / Stimulus), data=your_data_frame)


Then you must judge if the component of variance due to stimulus is small enough.

Here is an example with simulated data:

mod <- lme4::lmer(Response ~ (1 | Subject/Stimulus), data=df)
mod
Linear mixed model fit by REML ['lmerMod']
Formula: Response ~ (1 | Subject/Stimulus)
Data: df
REML criterion at convergence: 559.3188
Random effects:
Groups           Name        Std.Dev.
Stimulus:Subject (Intercept)  2.2841
Subject          (Intercept) 12.1752
Residual                      0.4322
Number of obs: 180, groups:  Stimulus:Subject, 60; Subject, 20
Fixed Effects:
(Intercept)
40.09


Code for simulated data:

N_subj  <- 20
N_stimu <- 3
N_repl  <- 3

sd_subj <- 10
sd_stimu <- 2
sd_repl <- 0.5

set.seed(7*11*13) # My public seed
N_total <- N_subj * N_stimu * N_repl
Subject <- as.factor(rep(1:N_subj, rep(N_stimu*N_repl, N_subj)))
Stimulus <- as.factor(rep(rep(1:N_stimu, rep(N_repl, N_stimu)), N_subj))

Response <- 40+rep(rnorm(N_subj, 0, sd_subj), rep(N_stimu*N_repl, N_subj)) +
rep(rnorm(N_stimu*N_subj, 0, sd_stimu), rep(N_repl, N_stimu*N_subj)) +
rnorm(N_total, 0, sd_repl)

df <- data.frame(Response, Subject, Stimulus)


Answered by kjetil b halvorsen on January 7, 2022

A textbook approach here would be to use a linear model with a term for each subject and a term for each stimulus. A classical analysis of variance (ANOVA) using such a model can compare the variation within the same subject or stimulus to the variation between different subjects or stimuli.

my_lm = lm(Response ~ Subject + Stimulus, data = your_dataframe)
aov(my_lm)


If you get a significant result for the coefficients describing the effect of the stimulus, that's the smoking gun. You should make sure the size of the effect is scientifically meaningful, though. It's good practice to also check the ANOVA assumptions (constant variance, independent errors, and less importantly, rough normality).

Answered by eric_kernfeld on January 7, 2022

Related Questions

Error metric to compare ratios derived from a binary prediction task

0  Asked on November 9, 2021

Machine learning to select features to create predictive model

0  Asked on November 9, 2021 by atrag

Summary statistics to collect for data that is too large in volume

0  Asked on November 9, 2021 by ryu576

Endogeneity testing using correlation test

3  Asked on November 9, 2021 by sabiste

find upper and lower bounds on average causal effect $theta$

1  Asked on November 9, 2021 by moneyball

How to correct for unequal observation time in poisson regression

1  Asked on November 9, 2021 by thomas-richardson

On gradient boosting and types of encodings

0  Asked on November 9, 2021

RFE: Pre-define a specificity threshold

1  Asked on November 9, 2021

Best software for image segmentation for time-series images?

1  Asked on November 9, 2021 by tshast2

semantic value of a convolution neural network

1  Asked on November 9, 2021

Getting Different R-squared for Same Model Equation

0  Asked on November 9, 2021

How to maximize the steady state transition probability for a state in a Markov chain by altering that state’s outgoing transition probabilities?

0  Asked on November 9, 2021 by rob-goldstone

Find out if increase in a series of values is statistically significant

1  Asked on November 9, 2021

Covariance between sample mean of two simple random samples

0  Asked on November 6, 2021 by ana-pau-de-la-fuente

Distribution of the maximum of two correlated normal variables

2  Asked on November 6, 2021

How to formulate a classification problem with time series element

1  Asked on November 6, 2021 by rishu

Asymmetric error measure for forecasts

1  Asked on November 6, 2021 by user235039

How to model variance of a heteroskedastic dataset

1  Asked on November 6, 2021 by yilei-huang

What analysis do I do? (Wage, education, gender)

2  Asked on November 6, 2021 by marvdste

Is random seed a hyper-parameter to tune in training deep neural network?

1  Asked on November 6, 2021