TransWikia.com

When one model is superior in real world use?

Data Science Asked by I_Play_With_Data on August 1, 2020

I have an NLP neural network that I have developed with Keras for multi-label classification.

I have fit the model several times and save the best results (via best validation accuracy score) after each set of epochs completes. All of my saved models are in the 96%+ validation accuracy score (according to Keras).

However, when I run these models against real-world data where I also know the result (e.g. effectively a second round of validation) one model in particular outperforms the rest. I can take the champion model (96.29% validation accuracy) and put it up against another model (with something like 96.18% validation accuracy) and the champion model can achieve 90%+ accuracy in the second round of validation while the other model – or any other model – will do nowhere near that. This one model will achieve a minimum 8% accuracy above all other models.

I have double-checked my methodology and I’m nearly positive that all models are being created with the same code and process.

Should I be concerned that this one particular model outperforms the rest? Does it indicate anything in particular in my overall methodology?

One Answer

Maybe I did not get the question but all looks fine. This is how you do model selection. You have several models (either same algorithm with different parameters or different algorithms. does not matter) and then you perform cross validation to get the best model according to empirical errors coming from validation set. The best model wins the game and is chosen. Everything seems to be right.

Answered by Kasra Manshaei on August 1, 2020

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP