So I’ve been looking into the idea of looking into conformal predictions, or to obtain prediction intervals instead of just single point predictions. Basically my take is that I would have my deep learning model (NN) that predicts the residual for this to be useful. However by using some distance function for this it will only be as meaningful as it is correlated to the error, which begs the question, wouldn’t it better to predict the error directly with a separate model?

I came across this post Prediction Intervals for ML Models and thought it would be interesting to apply to my own models. In my case I am dealing with a ML regression model to predict the stability of my proteins and wish to define at the end in 5 steps or less.

1. Data Splitting
2. Training the Model
3. Calibrating the model or measuring the skewness of the predictions (original labels vs predicted labels).

I was thinking of using a secondary model such as the KNN to measure the differences between labels.

4. Obtaining prediction intervals
5. Evaluating prediction intervals

Basically my question at the end of this is that would this be the correct way to do this or is this even smart to do? If so what would be the best approach for this?

Thanks for any input.

Possible Output

Sequence  Prediction Interval

AAADD     [0.34, 0.67]
AABBD     [0.24, 0.77]
AABCD     [0.36, 0.75]

Conformal Predictions Overview