Quantify whether a set of binary segmentation models (experts) have diversity on a fixed dataset?

I have a set of models for binary segmentation task (M=10), and a set of images (N=1000). I also can collect the prediction map for all these N=1000 images, resulting in 10,000 grayscale maps. Now, I want to quantitatively check whether these models have strong diversity in their predictions or not.

Diversity holds when the rank of the models in terms of their performance (e.g. Jaccard Index or Dice similarity) differs for each single query image.

What do you suggest? We also can reduce the set of models to N=3 to simplify the problem.

Cross Validated Asked by Saeed on December 30, 2020

1 Answers

One Answer

If you are interested in measure how heterogeneous the ranks are, then you can look at rank correlations statistics like $rho$ or $tau$. Otherwise, you could think of the problem as being an inter-rater reliability problem and ask the "dual" question of how much homogeneity there is between your models. This means you could look at statistics like intraclass correlation or models like ANOVA to assess the diversity.

Answered by MachineEpsilon on December 30, 2020

Add your own answers!

Related Questions

How to calculate ARIMA(1,0,0)(1,0,1)12 prediction by hand

1  Asked on December 10, 2021 by code_diy


ANOVA determining percentage of variation

2  Asked on December 8, 2021 by unistudent87


what is Multivariate Data

2  Asked on December 8, 2021


Ask a Question

Get help from others!

© 2022 All rights reserved.