Optimizing decision threshold on model with oversampled/imbalanced data

I am not sure if in the last point, you meant the validation set instead of the testing set.

Here is my advice: 1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need to handle the data imbalance problem.

2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.

3- use the training set for training, the validation set for tuning the hyper parameters , BUT do not touch the testing set

4- use the testing set for testing only

5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.