AnswerBun.com

Training accuracy is ~97% but validation accuracy is stuck at ~40%

Data Science Asked by Punit Singh on October 30, 2020

I am trying to classify images into 27 classes using a Conv2D network. The training accuracy rises through epochs as expected but the val_accuracy and val_loss values fluctuate severely and are not good enough.

I am using separate datasets for training and validation. The images are 256 x 256 in size and are binary threshold images.

Sample of the Training and validation image

There are 22127 images in the training set (~800 for each class) and
11346 images in the validation set (~400 for each class) so I believe there is no class imbalance.

Here is my architecture:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 126, 126, 32)      320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 63, 63, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 61, 61, 32)        9248      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 30, 30, 32)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 28800)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 128)               3686528   
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 96)                12384     
_________________________________________________________________
dropout_2 (Dropout)          (None, 96)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 64)                6208      
_________________________________________________________________
dense_4 (Dense)              (None, 27)                1755      
=================================================================



Total params: 3,716,443
Trainable params: 3,716,443
Non-trainable params: 0

Found 22127 images belonging to 27 classes.
Found 11346 images belonging to 27 classes.

here is the code I used to make this model:

classifier = Sequential()

classifier.add(Convolution2D(32, (3, 3), input_shape=(sz, sz, 1), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Convolution2D(32, (3, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))

classifier.add(Flatten())

classifier.add(Dense(units=128,
                     activation='relu'))
classifier.add(Dropout(0.40))


classifier.add(Dense(units=96,
                     activation='relu'))
classifier.add(Dropout(0.40))


classifier.add(Dense(units=64,
                     activation='relu'))

classifier.add(Dense(units=27, activation='softmax'))

optimizer used is adam and loss function I used is categorical_crossentropy.

from keras import optimizers

opt = optimizers.Adam(learning_rate = 0.001)

classifier.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

classifier.summary()

This is the code for flowing the data from the directories to the model

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory('Dataset2/Training Data',
                                                 target_size=(sz, sz),
                                                 batch_size=10,
                                                 color_mode='grayscale',
                                                 class_mode='categorical',
                                                 shuffle = True)

test_set = test_datagen.flow_from_directory('Dataset2/Test Data',
                                            target_size=(sz , sz),
                                            batch_size=10,
                                            color_mode='grayscale',
                                            class_mode='categorical',
                                            #shuffle = True
                                            )

classifier_obj = classifier.fit_generator(
        training_set,
        #steps_per_epoch=22127,
        epochs=10,
        validation_data=test_set,
        #validation_steps=11346
    verbose = 1)

the output this code produced was:

Epoch 1/10
2213/2213 [==============================] - 302s 136ms/step - loss: 0.0130 - accuracy: 0.7108 - val_loss: 0.0604 - val_accuracy: 0.3293
Epoch 2/10
2213/2213 [==============================] - 312s 141ms/step - loss: 0.0035 - accuracy: 0.9371 - val_loss: 0.0279 - val_accuracy: 0.4069
Epoch 3/10
2213/2213 [==============================] - 305s 138ms/step - loss: 0.0024 - accuracy: 0.9558 - val_loss: 0.0260 - val_accuracy: 0.4266
Epoch 4/10
2213/2213 [==============================] - 336s 152ms/step - loss: 0.0018 - accuracy: 0.9680 - val_loss: 0.0452 - val_accuracy: 0.4323
Epoch 5/10
2213/2213 [==============================] - 310s 140ms/step - loss: 0.0015 - accuracy: 0.9731 - val_loss: 0.0339 - val_accuracy: 0.3659
Epoch 6/10
2213/2213 [==============================] - 317s 143ms/step - loss: 0.0014 - accuracy: 0.9746 - val_loss: 0.0415 - val_accuracy: 0.4496
Epoch 7/10
2213/2213 [==============================] - 297s 134ms/step - loss: 0.0012 - accuracy: 0.9777 - val_loss: 0.0379 - val_accuracy: 0.4512
Epoch 8/10
2213/2213 [==============================] - 285s 129ms/step - loss: 0.0012 - accuracy: 0.9782 - val_loss: 0.0157 - val_accuracy: 0.4603
Epoch 9/10
2213/2213 [==============================] - 274s 124ms/step - loss: 0.0012 - accuracy: 0.9795 - val_loss: 0.0289 - val_accuracy: 0.4430
Epoch 10/10
2213/2213 [==============================] - 274s 124ms/step - loss: 9.1970e-04 - accuracy: 0.9837 - val_loss: 0.0459 - val_accuracy: 0.4800
Model Saved
Weights saved

Seeing promising results on this, I increased the number of epochs in hopes of getting a higher val_Accuracy but the result was that the val_accuracy was lower than the previous one.

I changed some parameters and ran the code with 100 epochs overnight. This was the output:

Accuracy over 100 epochs

Loss over 100 epochs

Some of the things that I found in other answers and have tried but failed:

  1. used SGD optimizer
  2. used regularizers
  3. played around with the number of units in Dense layers.

Based on other answers, I think that either my model is overfitting or the data is bad. Please help me as I am completely clueless.

One Answer

One of the major issue is the amount of records per categories. Definitely using data augmentation is always helpful to get good accuracy but much better approach is to use transfer learning techniques with data augmentation.

In your approach model is started overfitting because of that it's started having good training accuracy but bad validation accuracy. Let's have look into implementation of transfer learning. To understand more on transfer learning check this link Article Link

import tensorflow.keras as keras

model = keras.models.Sequential()

model.add(keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=(128, 128, 3)))

model.add(keras.layers.Flatten())

model.add(keras.layers.Dense(27, activation='softmax')) 

model.layers[0].trainable=False

model.compile(keras.optimizers.Adam(0.001), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()

model_obj = model.fit_generator(training_set, epochs=1, validation_data=test_set, verbose = 1)

Don't forgot to pass data augmented generator as a input data into fit_generator. There is no effect if you are using data augmentation on validation/test. You can use cross validation techniques or cnn based ensemble techniques for more good results.

Answered by Swapnil Pote on October 30, 2020

Add your own answers!

Related Questions

Clustering data with a constraint

0  Asked on February 2, 2021 by aibek

       

Suspiciously good accuracy using neural network

1  Asked on February 1, 2021 by po-chen-liu

   

Question about reshaping array size for KNN Classifiers

1  Asked on January 31, 2021 by leeann-capistran

     

NASDAQ Trade Data

3  Asked on January 31, 2021 by marin

   

Supported GPU for Pytorch

1  Asked on January 31, 2021

   

Ask a Question

Get help from others!

© 2022 AnswerBun.com. All rights reserved. Sites we Love: PCI Database, MenuIva, UKBizDB, Menu Kuliner, Sharing RPP