ROBERT A, SLOAN
The context: Why is this problem important to solve?
The objectives: What is the intended goal?
The key questions: What are the key questions that need to be answered?
The problem formulation: What are we trying to solve using data science?
The data set consists of 3 folders, i.e., 'test', 'train', and 'validation'. Each of these folders has four subfolders:
‘happy’: Images of people who have happy facial expressions.
‘sad’: Images of people with sad or upset facial expressions.
‘surprise’: Images of people who have shocked or surprised facial expressions.
‘neutral’: Images of people showing no prominent emotion in their facial expression at all.
PROBLEM STATEMENT:
CONTEXT: Computer Vision has use cases across almost all domains, from self-driving and evaluating the driver's experience in more automated and self-driving vehicles to the customer service industry across many types of jobs. The objective here is to build a range of different CNNs and test them to see what architecture works best for the problem of taking in an image and being able to recognize that A) there is a person in the image, but then also determine the emotional state of the person in the image.
Key Questions:
CNNs: Are they the right type of model for this task?
If CNNs are the right type of model, how are they constructed, and what kind of depth to the convolution layers is necessary and useful?
What is the best level of accuracy and overall performance that we can achieve before the model starts to become overfitted?
Transfer learning: How much can be gained by using pretrained models that are open-sourced? These models have been trained at no small expense on a large amount of labeled data that's time-intensive to collect and label. To reproduce them would be outside of what we are currently able to afford for the task, so can we gain from the hard work done by others, and if so, to what level?
We are trying to take in pixels from an image, value the pixel intensity, and then construct a neural network that can learn to see the world in terms of pixel values but be able to recognize the emergent features beyond that of just the pixel value. To see the human face and identify that one face has ears, and where there are ears, there is probably a face when looking at other images. Building on that to recognize the mouth and the facial features that change during different emotional states.
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
if gpu_info.find('failed') >= 0:
print('Not connected to a GPU')
else:
print(gpu_info)
Fri Feb 23 04:20:06 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 | | N/A 54C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+
!pip install tensorflow
!pip install keras-tuner
!pip install pandas
!pip install numpy
!pip install matplotlib
!pip install scikit-learn
!pip install imbalanced-learn
Requirement already satisfied: tensorflow in /usr/local/lib/python3.10/dist-packages (2.15.0) Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.4.0) Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.6.3) Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.5.26) Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.5.4) Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0) Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.9.0) Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (16.0.6) Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0) Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.25.2) Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.3.0) Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.2) Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.20.3) Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow) (67.7.2) Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0) Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.4.0) Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (4.9.0) Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.14.1) Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.36.0) Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.60.1) Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.2) Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0) Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0) Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow) (0.42.0) Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.27.0) Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (1.2.0) Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.5.2) Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.31.0) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.0.1) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (5.3.2) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.3.0) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (4.9) Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (1.3.1) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.6) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2024.2.2) Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow) (2.1.5) Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.5.1) Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (3.2.2) Collecting keras-tuner Downloading keras_tuner-1.4.6-py3-none-any.whl (128 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.9/128.9 kB 3.3 MB/s eta 0:00:00 Requirement already satisfied: keras in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (2.15.0) Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (23.2) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (2.31.0) Collecting kt-legacy (from keras-tuner) Downloading kt_legacy-1.0.5-py3-none-any.whl (9.6 kB) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (3.6) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (2024.2.2) Installing collected packages: kt-legacy, keras-tuner Successfully installed keras-tuner-1.4.6 kt-legacy-1.0.5 Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3) Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.4) Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.25.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0) Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.25.2) Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.7.1) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.2.0) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.49.0) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.5) Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.25.2) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (23.2) Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (9.4.0) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.1.1) Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0) Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.2.2) Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.25.2) Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.11.4) Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.3.2) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.3.0) Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.10/dist-packages (0.10.1) Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.25.2) Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.11.4) Requirement already satisfied: scikit-learn>=1.0.2 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.2.2) Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.3.2) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (3.3.0)
NOTE: Please use Google Colab from your browser for this notebook. Google.colab is NOT a library that can be downloaded locally on your device.
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
import tensorflow as tf
import zipfile
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import PIL
# Library for creating data paths
import os
from IPython.display import HTML
# Library for randomly selecting data points
import random
# Library for creating and showing plots
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
# Importing Deep Learning Libraries
from tensorflow.keras.utils import load_img
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Dense, Input, Dropout, GlobalAveragePooling2D, Flatten, Conv2D, BatchNormalization, Activation, MaxPooling2D, LeakyReLU
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.optimizers import Adam, SGD, RMSprop, Adamax, AdamW
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
from tensorflow.keras.utils import to_categorical
# Importing all the required sub-modules from Keras
from keras.applications.vgg16 import VGG16
from tensorflow.keras.callbacks import LearningRateScheduler
from sklearn.utils import resample
from imblearn.over_sampling import SMOTE
path = '/content/drive/My Drive/Deep learning/CapstoneProject/Facial_emotion_images.zip'
with zipfile.ZipFile(path, 'r') as zip_ref:
zip_ref.extractall()
picture_size = 48
folder_path = "Facial_emotion_images/"
Note:
Let's look at our classes.
Write down your observation for each class. What do you think can be a unique feature of each emotion, that separates it from the remaining classes?
folder_path = 'Facial_emotion_images/'
num_happy = len(os.listdir(folder_path + "train/" + 'happy'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'happy'</b> is: <b>{num_happy}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
expression1 = 'happy'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression1 + "/" +
os.listdir(folder_path + "train/" + expression1)[i], target_size = (picture_size, picture_size))
# Seperate the two expressions with a horizontal line
if i == 8:
plt.axhline()
plt.imshow(img)
Number of training images for class 'happy' is: 3976
Observations and Insights:__
folder_path = 'Facial_emotion_images/'
num_surprise = len(os.listdir(folder_path + "train/" + 'surprise'))
num_happy= len(os.listdir(folder_path + "train/" + 'happy'))
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; padding: 10px;text-align: center; font-size: 25px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'surprise'</b> is: <b>{num_surprise}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
expression = 'surprise'
plt.figure(figsize= (30
,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression + "/" +
os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
plt.imshow(img)
plt.show()
Number of training images for class 'surprise' is: 3173
Observations and Insights:__
from IPython.display import HTML
import os
folder_path = 'Facial_emotion_images/'
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'sad'</b> is: <b>{num_sad}</b></p>" \
f"</div>"
display(HTML(output_html))
expression = 'sad'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression + "/" +
os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
plt.imshow(img)
plt.show()
Number of training images for class 'sad' is: 3982
Observations and Insights:__
folder_path = 'Facial_emotion_images/'
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'neutral'</b> is: <b>{num_neutral}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
expression = 'neutral'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
plt.subplot(2, 10, i)
img = load_img(folder_path + "train/" + expression + "/" +
os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
plt.imshow(img)
plt.show()
Number of training images for class 'neutral' is: 3978
Observations and Insights:__
import os
from keras.preprocessing.image import load_img
import matplotlib.pyplot as plt
# Assuming 'base_dir' is correctly set to your dataset directory
base_dir = 'Facial_emotion_images/train/'
categories = ['happy', 'sad', 'neutral', 'surprise']
fig, ax = plt.subplots(nrows=len(categories), ncols=4, figsize=(30, 8)) # Adjusted figsize for better visibility
for i, category in enumerate(categories):
category_dir = os.path.join(base_dir, category)
sample_images = os.listdir(category_dir)[:4] # Get first 4 images of each category
for j, image in enumerate(sample_images):
img_path = os.path.join(category_dir, image)
img = load_img(img_path, target_size=(48, 48), color_mode='grayscale')
ax[i, j].imshow(img, cmap='gray')
ax[i, j].axis('off')
if j == 0: # Mark the category name more prominently
ax[i, j].set_ylabel(category.capitalize(), fontsize=14, labelpad=10, color='blue')
# Enhance layout and visibility
plt.tight_layout(pad=3.0)
plt.show()
folder_path = 'Facial_emotion_images/'
num_surprise = len(os.listdir(folder_path + "train/" + 'surprise'))
num_happy= len(os.listdir(folder_path + "train/" + 'happy'))
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; padding: 10px;text-align: center; font-size: 25px; border-radius: 10px;'>" \
f"<h3 style='color: #333;'>Training Data Information:</h3>" \
f"<p style='color: blue;'>Number of training images for class <b>'surprise'</b> is: <b>{num_surprise}</b></p>" \
f"<p style='color: green;'>Number of training images for class <b>'neutral'</b> is: <b>{num_neutral}</b></p>" \
f"<p style='color: orange;'>Number of training images for class <b>'happy'</b> is: <b>{num_happy}</b></p>" \
f"<p style='color: red;'>Number of training images for class <b>'sad'</b> is: <b>{num_sad}</b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
# Code to plot histogram
plt.figure(figsize = (30, 7))
# Data
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
%matplotlib inline
plt.subplot(2, 1, 1),
plt.bar(classes, num_images, color=['blue', 'orange', 'green', 'red'])
plt.xlabel('Class')
plt.ylabel('Number of Images')
plt.title('Number of Training Images per Class')
plt.show()
Number of training images for class 'surprise' is: 3173
Number of training images for class 'neutral' is: 3978
Number of training images for class 'happy' is: 3976
Number of training images for class 'sad' is: 3982
Think About It:
OBSERVATION: The categorys sad neutral and happy all have the same ammount of images however the suprise category has less.
Observations and Insights:__
The data set is skewed and needs to have some data augmentation methods applied to equalize the data set.
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
plt.figure(figsize=(5, 5))
plt.pie(num_images, labels=classes, autopct='%1.1f%%', startangle=140, colors=['blue', 'orange', 'green', 'red'])
plt.title('Proportion of Each Class in the Dataset')
plt.show()
# Check the Data Set for imbalance
min_class_size = min(num_images)
max_class_size = max(num_images)
imbalance_ratio = max_class_size / min_class_size
import matplotlib.pyplot as plt
# Data
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
%matplotlib inline
# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<h3 style='color: red'>Imbalance Ratio: Categorical representation in the data set (Max Class Size / Min Class Size): 1.25 :</h3>" \
f"<p style='color: #666;'><b>''</b>Imbalance Ratio between classes is: <b>{imbalance_ratio:.2f}</b></p>" \
f"<p style='color: red;'><b>''</b> Sad image representation is: <b> 26.4% </b></p>" \
f"<p style='color: green;'><b>''</b>Neutral image representation is: <b> 26.3% </b></p>" \
f"<p style='color: orange;'><b>''</b> Happy image representation is: <b> 26.3% </b></p>" \
f"<p style='color: blue;'><b>''</b>Surprise image representation is: <b> 21.0% </b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
''Imbalance Ratio between classes is: 1.25
'' Sad image representation is: 26.4%
''Neutral image representation is: 26.3%
'' Happy image representation is: 26.3%
''Surprise image representation is: 21.0%
In the initial EDA of the dataset, the categories of emotions appeared to be balanced except for 'surprise'. Digging into the statistical significance of the disparity, I have concluded the following
An imbalance ratio of 1.25 (Max Class Size / Min Class Size) suggests a relatively mild imbalance among the classes in the dataset.
Impact of Mild Imbalance Model Performance: Model Performance Modern neural networks, particularly those used for image classification tasks (like CNNs), are quite robust to mild class imbalances. An imbalance ratio close to 1 indicates that the classes are fairly well represented.
Learning Dynamics: The slight imbalance might not significantly skew the learning process, meaning the model can still learn to generalize well across all classes without heavy reliance on data balancing techniques.
To make sure that the mild inbalance is not going to impact the training process the following steps have been employed in the following models traing. Monitor Class-specific Metrics: Even with a mild imbalance, monitoring metrics like precision, recall, and F1-score for each class is good practice. This ensures that all classes are correctly learned by the model, and no class is systematically favored or ignored.
Mild Data Augmentation: While aggressive oversampling or synthetic data generation might not be necessary, employing data augmentation techniques rotation, flipping, scaling on the training set, can further help handle the slight imbalance's impact.
In this section, we are creating data loaders that we will use as inputs to our Neural Network.
You have two options for the color_mode. You can set it to color_mode = 'rgb' or color_mode = 'grayscale'. You will need to try out both and see for yourself which one gives better performance.
# Re-importing nbformat as the execution state was reset and redefining the necessary function
import nbformat
# Function to extract F1 scores from the notebook cells
def extract_f1_scores(notebook_cells):
f1_scores = {}
current_model = None
for cell in notebook_cells:
if cell.cell_type == 'markdown':
# Look for model names in markdown cells
if 'Model' in cell.source:
lines = cell.source.split('\n')
for line in lines:
if 'Model' in line:
current_model = line.strip('# ').strip()
elif cell.cell_type == 'code':
# Extract F1 scores from code cells outputs
for output in cell.get('outputs', []):
if output.output_type == 'execute_result':
if 'text/plain' in output.data:
text_output = output.data['text/plain']
if 'f1-score' in text_output:
# Assuming that the F1 score is the last float number in the text_output
f1_score = float(text_output.strip().split()[-1])
if current_model:
f1_scores[current_model] = f1_score
return f1_scores
# Read the notebook
with open('/mnt/data/Capstone_Facial_emotion_detection.ipynb', 'r', encoding='utf-8') as f:
nb = nbformat.read(f, as_version=4)
# Extract the F1 scores
f1_scores = extract_f1_scores(nb.cells)
f1_scores
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-44-107aa6ccca3e> in <cell line: 31>() 29 30 # Read the notebook ---> 31 with open('/mnt/data/Capstone_Facial_emotion_detection.ipynb', 'r', encoding='utf-8') as f: 32 nb = nbformat.read(f, as_version=4) 33 FileNotFoundError: [Errno 2] No such file or directory: '/mnt/data/Capstone_Facial_emotion_detection.ipynb'
from keras.preprocessing.image import ImageDataGenerator
# Define image size and paths
picture_size = 48
base_path = "Facial_emotion_images/"
train_path = base_path + "train/"
validation_path = base_path + "validation/"
test_path = base_path + "test/"
# Initialize the data generator for training data with data augmentation
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# For validation and test sets, usually only rescaling is applied
test_val_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
validation_generator = test_val_datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
test_generator = test_val_datagen.flow_from_directory(
test_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical',
shuffle=False # Keep data in order for evaluation
)
# Getting the total Training images, Testing images and Validation images.
traingen = train_generator.n
testgen = test_generator.n
validationgen = validation_generator.n
# Using inline HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
f"<p style='color: red;'><b>''</b>Total Training Images <b> {traingen}</b></p>" \
f"<p style='color: black;'><b>''</b>Total Testing Images <b> {validationgen} </b></p>" \
f"<p style='color: blue;'><b>'Data Set'</b>Total Validation Images <b>{testgen} </b></p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
''Total Training Images 15109
''Total Testing Images 4977
'Data Set'Total Validation Images 128
#Defining the function for creating the custom classification report
def metrics_score(actual, predicted):
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(actual, predicted))
cm = confusion_matrix(actual, predicted)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f', xticklabels = class_names_list, yticklabels = class_names_list)
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report
def plot_model_performance_and_confusion_matrix(model, history, test_generator, metrics_function=None):
# Plot training and validation accuracy
plt.figure(figsize=(25, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('Model Accuracy')
plt.legend()
# Plot training and validation loss
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Model Loss')
plt.legend()
# Predict labels for the test set
predicted_labels = np.argmax(model.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Optionally print class names
class_names_list = list(test_generator.class_indices.keys())
print("Class names in the dataset:", class_names_list)
# Optionally calculate and display metrics
if metrics_function is not None:
metrics_function(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", xticklabels=class_names_list, yticklabels=class_names_list)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix")
plt.show()
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report
def plot_model_stats(model, history, test_generator, model_name):
# Compute confusion matrix
cm = confusion_matrix(test_generator.classes, np.argmax(model.predict(test_generator), axis=1))
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title(f'{model_name} Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title(f'{model_name} Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot confusion matrix
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title(f"Confusion Matrix for {model_name}")
plt.tight_layout() # Adjust layout to not overlap figures
plt.show()
def metrics_score(actual, predicted):
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(actual, predicted))
cm = confusion_matrix(actual, predicted)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f', xticklabels = class_names_list, yticklabels = class_names_list)
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
import os
from IPython.display import HTML, display
base_path = "Facial_emotion_images/"
subsets = ['train', 'test', 'validation']
classes = ['happy', 'sad', 'neutral', 'surprise']
subset_counts = {subset: 0 for subset in subsets}
for subset in subsets:
for emotion in classes:
# Dir construction
dir_path = os.path.join(base_path, subset, emotion)
# Count files in directories and add to subset count
if os.path.exists(dir_path):
files_count = len([name for name in os.listdir(dir_path) if os.path.isfile(os.path.join(dir_path, name))])
subset_counts[subset] += files_count
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 10px; border-radius: 10px;'>" \
f"<p style='color: red;'><b>Total Training Images:</b> {subset_counts['train']}</p>" \
f"<p style='color: black;'><b>Total Testing Images:</b> {subset_counts['test']} </p>" \
f"<p style='color: blue;'><b>Total Validation Images:</b> {subset_counts['validation']} </p>" \
f"</div>"
# Display the styled output
display(HTML(output_html))
Total Training Images: 0
Total Testing Images: 0
Total Validation Images: 0
# prompt: create a plot to display the total data set count, the total amount of training images and the total amount of validation images
# Create a bar plot with three bars
x_values = [0, 1, 2]
bar_heights = [subset_counts['train'], subset_counts['validation'], subset_counts['test']]
bar_labels = ['Training', 'Validation', 'Testing']
plt.figure(figsize=(10, 6))
plt.bar(x_values, bar_heights, color=['blue', 'orange', 'green'])
plt.xlabel('Dataset Subsets', fontweight='bold', fontsize=16)
plt.ylabel('Number of Images', fontweight='bold', fontsize=16)
plt.xticks(x_values, bar_labels, fontsize=14)
plt.title('Total Images in Each Subset', fontweight='bold', fontsize=20)
plt.show()
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-11-1523434e0c57> in <cell line: 5>() 3 # Create a bar plot with three bars 4 x_values = [0, 1, 2] ----> 5 bar_heights = [subset_counts['train'], subset_counts['validation'], subset_counts['test']] 6 bar_labels = ['Training', 'Validation', 'Testing'] 7 NameError: name 'subset_counts' is not defined
import os
from IPython.display import HTML, display
base_path = "Facial_emotion_images/"
subsets = ['train', 'test', 'validation']
classes = ['happy', 'sad', 'neutral', 'surprise']
class_counts = {emotion: 0 for emotion in classes}
for subset in subsets:
for emotion in classes:
dir_path = os.path.join(base_path, subset, emotion)
if os.path.exists(dir_path):
class_counts[emotion] += len([name for name in os.listdir(dir_path) if os.path.isfile(os.path.join(dir_path, name))])
# Generate HTML content
output_html = "<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>"
output_html += "<h3 style='color: #333;'>Class Distribution</h3>"
for emotion, count in class_counts.items():
output_html += f"<p style='color: #333;'><b>{emotion.capitalize()}:</b> {count}</p>"
output_html += "</div>"
# Display the styled output
display(HTML(output_html))
Happy: 0
Sad: 0
Neutral: 0
Surprise: 0
from sklearn.utils.class_weight import compute_class_weight
import numpy as np
from IPython.display import HTML, display
total_samples = sum(class_counts.values())
n_classes = len(classes)
# Compute class weights using sklearn's compute_class_weight function for better accuracy
y_integers = np.array([classes.index(emotion) for emotion in class_counts.keys() for _ in range(class_counts[emotion])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(y_integers), y=y_integers)
class_weights_dict = {classes[i]: weight for i, weight in enumerate(class_weights)}
# List the class weights in styled HTML
output_html = "<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>"
output_html += "<h3 style='color: #333;'>Computed Class Weights</h3>"
for emotion, weight in class_weights_dict.items():
output_html += f"<p style='color: #333;'><b>{emotion.capitalize()}:</b> {weight:.2f}</p>"
output_html += "</div>"
# Display the styled output
display(HTML(output_html))
Happy: 0.87
Sad: 0.98
Neutral: 0.97
Surprise: 1.26
from sklearn.utils.class_weight import compute_class_weight
import numpy as np
class_labels = np.array([classes.index(cls) for cls in classes for _ in range(class_counts[cls])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(class_labels), y=class_labels)
class_weights_dict = {i: weight for i, weight in enumerate(class_weights)}
class_labels = np.array([classes.index(cls) for cls in classes for _ in range(class_counts[cls])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(class_labels), y=class_labels)
class_weights_dict = {i: weight for i, weight in enumerate(class_weights)}
class_weights_dict = {classes[i]: weight for i, weight in enumerate(class_weights)}
print("Class labels:", class_labels)
print("Class weights dictionary:", class_weights_dict)
print("Class weights:", class_weights)
# Adjust class weights to sum to 1
total_weight = sum(class_weights)
class_weights_adjusted = {cls: weight / total_weight for cls, weight in class_weights_dict.items()}
print("Class weights adjusted:", class_weights_adjusted)
Class labels: [0 0 0 ... 3 3 3] Class weights dictionary: {'surprise': 1.262743628185907, 'happy': 0.8663637922166981, 'neutral': 0.96699196326062, 'sad': 0.9806908596933825} Class weights: [1.26274363 0.86636379 0.96699196 0.98069086] Class weights adjusted: {'surprise': 0.30973966105899836, 'happy': 0.21251125039569885, 'neutral': 0.23719443619558198, 'sad': 0.24055465234972084}
classes_sorted = sorted(classes)
class_weights_dict = {cls: weight for cls, weight in zip(classes_sorted, class_weights)}
# Adjusting ClassWeights to ensure proper indicies assignment
class_weights_adjusted = {train_generator.class_indices[cls]: weight for cls, weight in class_weights_dict.items()}
Think About It:
# from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam
cnn_model_1 = Sequential([
# First Convolutional Block
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.25),
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(128, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.5),
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.5),
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(1024, (3, 3), activation='relu', padding='same'),
GlobalAveragePooling2D(),
(Flatten()),
# Dense Layers
Dense(1024, activation='relu'),
Dense(256, activation='relu'),
Dropout(0.3),
# Output
Dense(128, activation='relu'),
Dropout(0.1),
Dense(64, activation='relu'),
Dropout(0.25),
Dense(4, activation='softmax')
])
cnn_model_summary = cnn_model_1.summary()
# prompt: install from keras.utils.vis_utils import plot_model
!pip install keras.utils
Collecting keras.utils Downloading keras-utils-1.0.13.tar.gz (2.4 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: Keras>=2.1.5 in /usr/local/lib/python3.10/dist-packages (from keras.utils) (2.15.0) Building wheels for collected packages: keras.utils Building wheel for keras.utils (setup.py) ... done Created wheel for keras.utils: filename=keras_utils-1.0.13-py3-none-any.whl size=2631 sha256=ae276b6142d6b7d183236c5e7b08af909764c1ba8cbeb67aada1cdd8ee3c2811 Stored in directory: /root/.cache/pip/wheels/5c/c0/b3/0c332de4fd71f3733ea6d61697464b7ae4b2b5ff0300e6ca7a Successfully built keras.utils Installing collected packages: keras.utils Successfully installed keras.utils-1.0.13
# Import the required functions
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, Flatten
from tensorflow.keras.utils import plot_model
# Define your model
model = Sequential([
Conv2D(32, (3, 3), activation='relu', input_shape=(64, 64, 3)),
MaxPooling2D((2, 2)),
Flatten(),
Dense(64, activation='relu'),
Dense(4, activation='softmax')
])
# Use plot_model to create a diagram of the model
plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)
# Display the image in the notebook
from IPython.display import Image
Image(filename='model_architecture.png')
# prompt: install any tools for visualizing the model archiceture from keras.utils.vis_utils import plot_model from keras.models import Sequential from keras.layers import Dense # Define Keras model here model = Sequential() model.add(Dense(units=64, activation='relu', input_dim=100)) model.add(Dense(units=10, activation='softmax')) # Generate a plot of the model plot_model(model, to_file='model_plot.png'
!pip install pydot
!pip install graphviz
from keras.utils.vis_utils import plot_model
from keras.models import Sequential
from keras.layers import Dense
# Define Keras model here
model = Sequential()
model.add(Dense(units=64, activation='relu', input_dim=100))
model.add(Dense(units=10, activation='softmax'))
# Generate a plot of the model
plot_model(model, to_file='model_plot.png')
Requirement already satisfied: pydot in /usr/local/lib/python3.10/dist-packages (1.4.2) Requirement already satisfied: pyparsing>=2.1.4 in /usr/local/lib/python3.10/dist-packages (from pydot) (3.1.1) Requirement already satisfied: graphviz in /usr/local/lib/python3.10/dist-packages (0.20.1)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-33-ffdc8e4907dd> in <cell line: 5>() 3 get_ipython().system('pip install pydot') 4 get_ipython().system('pip install graphviz') ----> 5 from keras.utils.vis_utils import plot_model 6 from keras.models import Sequential 7 from keras.layers import Dense ModuleNotFoundError: No module named 'keras.utils.vis_utils' --------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt. To view examples of installing some common dependencies, click the "Open Examples" button below. ---------------------------------------------------------------------------
import keras.utils.vis_utils
from keras.utils.vis_utils import plot_model
from keras.models import Sequential
from keras.layers import Dense
# Define Keras model here
model = Sequential()
model.add(Dense(units=64, activation='relu', input_dim=100))
model.add(Dense(units=10, activation='softmax'))
# Generate a plot of the model
plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-29-b5c4535a7d57> in <cell line: 1>() ----> 1 import keras.utils.vis_utils 2 from keras.utils.vis_utils import plot_model 3 from keras.models import Sequential 4 from keras.layers import Dense 5 ModuleNotFoundError: No module named 'keras.utils.vis_utils' --------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt. To view examples of installing some common dependencies, click the "Open Examples" button below. ---------------------------------------------------------------------------
import tensorflow as tf
# Fixing the random seed generator to ensure random initilization numbers
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam
cnn_model_1 = Sequential([
# First Convolutional Block
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.25),
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(128, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.5),
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.5),
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(1024, (3, 3), activation='relu', padding='same'),
GlobalAveragePooling2D(),
(Flatten()),
# Dense Layers
Dense(1024, activation='relu'),
Dense(256, activation='relu'),
Dropout(0.3),
# Output
Dense(128, activation='relu'),
Dropout(0.1),
Dense(64, activation='relu'),
Dropout(0.25),
Dense(4, activation='softmax')
])
cnn_model_summary = cnn_model_1.summary()
# Use plot_model to create a diagram of the model
plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)
# Display the image in the notebook
from IPython.display import Image
Image(filename='model_architecture.png')
Model: "sequential_2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_10 (Conv2D) (None, 48, 48, 64) 1792 batch_normalization_5 (Bat (None, 48, 48, 64) 256 chNormalization) conv2d_11 (Conv2D) (None, 48, 48, 64) 36928 max_pooling2d_4 (MaxPoolin (None, 24, 24, 64) 0 g2D) dropout_6 (Dropout) (None, 24, 24, 64) 0 conv2d_12 (Conv2D) (None, 24, 24, 128) 73856 batch_normalization_6 (Bat (None, 24, 24, 128) 512 chNormalization) conv2d_13 (Conv2D) (None, 24, 24, 128) 147584 max_pooling2d_5 (MaxPoolin (None, 12, 12, 128) 0 g2D) dropout_7 (Dropout) (None, 12, 12, 128) 0 conv2d_14 (Conv2D) (None, 12, 12, 256) 295168 batch_normalization_7 (Bat (None, 12, 12, 256) 1024 chNormalization) conv2d_15 (Conv2D) (None, 12, 12, 256) 590080 batch_normalization_8 (Bat (None, 12, 12, 256) 1024 chNormalization) conv2d_16 (Conv2D) (None, 12, 12, 256) 590080 max_pooling2d_6 (MaxPoolin (None, 6, 6, 256) 0 g2D) dropout_8 (Dropout) (None, 6, 6, 256) 0 conv2d_17 (Conv2D) (None, 6, 6, 1024) 2360320 batch_normalization_9 (Bat (None, 6, 6, 1024) 4096 chNormalization) conv2d_18 (Conv2D) (None, 6, 6, 1024) 9438208 global_average_pooling2d_1 (None, 1024) 0 (GlobalAveragePooling2D) flatten_2 (Flatten) (None, 1024) 0 dense_7 (Dense) (None, 1024) 1049600 dense_8 (Dense) (None, 256) 262400 dropout_9 (Dropout) (None, 256) 0 dense_9 (Dense) (None, 128) 32896 dropout_10 (Dropout) (None, 128) 0 dense_10 (Dense) (None, 64) 8256 dropout_11 (Dropout) (None, 64) 0 dense_11 (Dense) (None, 4) 260 ================================================================= Total params: 14894340 (56.82 MB) Trainable params: 14890884 (56.80 MB) Non-trainable params: 3456 (13.50 KB) _________________________________________________________________
cnn_model_1.compile(optimizer=AdamW(learning_rate=0.0005), # Adjust learning rate as needed
loss='categorical_crossentropy',
metrics=['accuracy'])
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
early_stopping = EarlyStopping(monitor='accuracy', patience=5, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=1, min_lr=0.000002)
history = cnn_model_1.fit(
train_generator,
epochs=25, # Adjust based on early stopping
validation_data=validation_generator,
callbacks=[early_stopping, reduce_lr],
class_weight=class_weights_adjusted
)
Epoch 1/25 473/473 [==============================] - 33s 54ms/step - loss: 1.2893 - accuracy: 0.3542 - val_loss: 1.1666 - val_accuracy: 0.4744 - lr: 5.0000e-04 Epoch 2/25 473/473 [==============================] - 25s 52ms/step - loss: 1.0293 - accuracy: 0.5333 - val_loss: 0.9968 - val_accuracy: 0.5445 - lr: 5.0000e-04 Epoch 3/25 473/473 [==============================] - 25s 53ms/step - loss: 0.8731 - accuracy: 0.6343 - val_loss: 0.8473 - val_accuracy: 0.6383 - lr: 5.0000e-04 Epoch 4/25 473/473 [==============================] - 25s 53ms/step - loss: 0.7914 - accuracy: 0.6766 - val_loss: 0.9087 - val_accuracy: 0.6365 - lr: 5.0000e-04 Epoch 5/25 473/473 [==============================] - 25s 54ms/step - loss: 0.7443 - accuracy: 0.6993 - val_loss: 0.7194 - val_accuracy: 0.7079 - lr: 5.0000e-04 Epoch 6/25 473/473 [==============================] - 25s 54ms/step - loss: 0.7043 - accuracy: 0.7152 - val_loss: 0.7093 - val_accuracy: 0.7229 - lr: 5.0000e-04 Epoch 7/25 473/473 [==============================] - 25s 53ms/step - loss: 0.6734 - accuracy: 0.7295 - val_loss: 0.7230 - val_accuracy: 0.7163 - lr: 5.0000e-04 Epoch 8/25 473/473 [==============================] - 25s 53ms/step - loss: 0.6391 - accuracy: 0.7433 - val_loss: 0.7180 - val_accuracy: 0.7203 - lr: 5.0000e-04 Epoch 9/25 473/473 [==============================] - 25s 53ms/step - loss: 0.6195 - accuracy: 0.7541 - val_loss: 0.6271 - val_accuracy: 0.7511 - lr: 5.0000e-04 Epoch 10/25 473/473 [==============================] - 25s 53ms/step - loss: 0.5876 - accuracy: 0.7668 - val_loss: 0.6473 - val_accuracy: 0.7396 - lr: 5.0000e-04 Epoch 11/25 473/473 [==============================] - 25s 54ms/step - loss: 0.5608 - accuracy: 0.7767 - val_loss: 0.6750 - val_accuracy: 0.7326 - lr: 5.0000e-04 Epoch 12/25 472/473 [============================>.] - ETA: 0s - loss: 0.5481 - accuracy: 0.7880 Epoch 12: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513. 473/473 [==============================] - 25s 54ms/step - loss: 0.5475 - accuracy: 0.7883 - val_loss: 0.6758 - val_accuracy: 0.7324 - lr: 5.0000e-04 Epoch 13/25 473/473 [==============================] - 25s 53ms/step - loss: 0.4534 - accuracy: 0.8245 - val_loss: 0.6064 - val_accuracy: 0.7816 - lr: 1.0000e-04 Epoch 14/25 473/473 [==============================] - 25s 53ms/step - loss: 0.4201 - accuracy: 0.8327 - val_loss: 0.6151 - val_accuracy: 0.7824 - lr: 1.0000e-04 Epoch 15/25 473/473 [==============================] - 25s 54ms/step - loss: 0.3974 - accuracy: 0.8439 - val_loss: 0.6472 - val_accuracy: 0.7746 - lr: 1.0000e-04 Epoch 16/25 472/473 [============================>.] - ETA: 0s - loss: 0.3793 - accuracy: 0.8480 Epoch 16: ReduceLROnPlateau reducing learning rate to 2.0000000949949027e-05. 473/473 [==============================] - 25s 54ms/step - loss: 0.3794 - accuracy: 0.8479 - val_loss: 0.6692 - val_accuracy: 0.7752 - lr: 1.0000e-04 Epoch 17/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3476 - accuracy: 0.8654 - val_loss: 0.6500 - val_accuracy: 0.7808 - lr: 2.0000e-05 Epoch 18/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3418 - accuracy: 0.8661 - val_loss: 0.6576 - val_accuracy: 0.7830 - lr: 2.0000e-05 Epoch 19/25 473/473 [==============================] - ETA: 0s - loss: 0.3340 - accuracy: 0.8690 Epoch 19: ReduceLROnPlateau reducing learning rate to 4.000000262749381e-06. 473/473 [==============================] - 25s 54ms/step - loss: 0.3340 - accuracy: 0.8690 - val_loss: 0.6585 - val_accuracy: 0.7840 - lr: 2.0000e-05 Epoch 20/25 473/473 [==============================] - 25s 54ms/step - loss: 0.3277 - accuracy: 0.8707 - val_loss: 0.6671 - val_accuracy: 0.7822 - lr: 4.0000e-06 Epoch 21/25 473/473 [==============================] - 25s 54ms/step - loss: 0.3277 - accuracy: 0.8656 - val_loss: 0.6694 - val_accuracy: 0.7824 - lr: 4.0000e-06 Epoch 22/25 472/473 [============================>.] - ETA: 0s - loss: 0.3257 - accuracy: 0.8731 Epoch 22: ReduceLROnPlateau reducing learning rate to 2e-06. 473/473 [==============================] - 25s 54ms/step - loss: 0.3255 - accuracy: 0.8732 - val_loss: 0.6722 - val_accuracy: 0.7820 - lr: 4.0000e-06 Epoch 23/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3227 - accuracy: 0.8726 - val_loss: 0.6737 - val_accuracy: 0.7816 - lr: 2.0000e-06 Epoch 24/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3273 - accuracy: 0.8723 - val_loss: 0.6743 - val_accuracy: 0.7820 - lr: 2.0000e-06 Epoch 25/25 473/473 [==============================] - 25s 53ms/step - loss: 0.3227 - accuracy: 0.8707 - val_loss: 0.6747 - val_accuracy: 0.7820 - lr: 2.0000e-06
history = cnn_model_1.fit(
train_generator,
epochs=25, # Adjust based on early stopping
validation_data=validation_generator,
callbacks=[early_stopping, reduce_lr],
class_weight=class_weights_adjusted
)
Epoch 1/25 473/473 [==============================] - 27s 46ms/step - loss: 1.3672 - accuracy: 0.3026 - val_loss: 1.3214 - val_accuracy: 0.2648 - lr: 5.0000e-04 Epoch 2/25 473/473 [==============================] - 22s 46ms/step - loss: 1.2208 - accuracy: 0.3796 - val_loss: 1.2417 - val_accuracy: 0.3122 - lr: 5.0000e-04 Epoch 3/25 473/473 [==============================] - 22s 45ms/step - loss: 1.1498 - accuracy: 0.4201 - val_loss: 1.2578 - val_accuracy: 0.3255 - lr: 5.0000e-04 Epoch 4/25 473/473 [==============================] - 21s 45ms/step - loss: 1.1108 - accuracy: 0.4621 - val_loss: 1.1026 - val_accuracy: 0.5041 - lr: 5.0000e-04 Epoch 5/25 473/473 [==============================] - 21s 45ms/step - loss: 1.0652 - accuracy: 0.4964 - val_loss: 1.0543 - val_accuracy: 0.5188 - lr: 5.0000e-04 Epoch 6/25 473/473 [==============================] - 21s 45ms/step - loss: 0.9929 - accuracy: 0.5599 - val_loss: 0.9972 - val_accuracy: 0.5821 - lr: 5.0000e-04 Epoch 7/25 473/473 [==============================] - 21s 44ms/step - loss: 0.9165 - accuracy: 0.6112 - val_loss: 1.0254 - val_accuracy: 0.5600 - lr: 5.0000e-04 Epoch 8/25 473/473 [==============================] - 21s 44ms/step - loss: 0.8532 - accuracy: 0.6508 - val_loss: 0.7439 - val_accuracy: 0.6994 - lr: 5.0000e-04 Epoch 9/25 473/473 [==============================] - 21s 44ms/step - loss: 0.8271 - accuracy: 0.6621 - val_loss: 0.7456 - val_accuracy: 0.7036 - lr: 5.0000e-04 Epoch 10/25 473/473 [==============================] - 21s 44ms/step - loss: 0.8005 - accuracy: 0.6705 - val_loss: 0.6997 - val_accuracy: 0.7117 - lr: 5.0000e-04 Epoch 11/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7749 - accuracy: 0.6885 - val_loss: 0.6853 - val_accuracy: 0.7316 - lr: 5.0000e-04 Epoch 12/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7588 - accuracy: 0.6870 - val_loss: 0.6973 - val_accuracy: 0.7151 - lr: 5.0000e-04 Epoch 13/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7347 - accuracy: 0.7069 - val_loss: 0.6655 - val_accuracy: 0.7418 - lr: 5.0000e-04 Epoch 14/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7219 - accuracy: 0.7054 - val_loss: 0.6480 - val_accuracy: 0.7448 - lr: 5.0000e-04 Epoch 15/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7119 - accuracy: 0.7107 - val_loss: 0.6969 - val_accuracy: 0.7267 - lr: 5.0000e-04 Epoch 16/25 473/473 [==============================] - 21s 44ms/step - loss: 0.7008 - accuracy: 0.7190 - val_loss: 0.6514 - val_accuracy: 0.7338 - lr: 5.0000e-04 Epoch 17/25 472/473 [============================>.] - ETA: 0s - loss: 0.6900 - accuracy: 0.7220 Epoch 17: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513. 473/473 [==============================] - 21s 44ms/step - loss: 0.6894 - accuracy: 0.7222 - val_loss: 0.6523 - val_accuracy: 0.7509 - lr: 5.0000e-04 Epoch 18/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6402 - accuracy: 0.7397 - val_loss: 0.5846 - val_accuracy: 0.7709 - lr: 1.0000e-04 Epoch 19/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6118 - accuracy: 0.7498 - val_loss: 0.5854 - val_accuracy: 0.7687 - lr: 1.0000e-04 Epoch 20/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6006 - accuracy: 0.7551 - val_loss: 0.5718 - val_accuracy: 0.7780 - lr: 1.0000e-04 Epoch 21/25 473/473 [==============================] - 21s 44ms/step - loss: 0.6130 - accuracy: 0.7484 - val_loss: 0.5883 - val_accuracy: 0.7675 - lr: 1.0000e-04 Epoch 22/25 473/473 [==============================] - 21s 44ms/step - loss: 0.5937 - accuracy: 0.7576 - val_loss: 0.5962 - val_accuracy: 0.7669 - lr: 1.0000e-04 Epoch 23/25 472/473 [============================>.] - ETA: 0s - loss: 0.5907 - accuracy: 0.7559 Epoch 23: ReduceLROnPlateau reducing learning rate to 2.0000000949949027e-05. 473/473 [==============================] - 21s 44ms/step - loss: 0.5912 - accuracy: 0.7557 - val_loss: 0.5798 - val_accuracy: 0.7673 - lr: 1.0000e-04 Epoch 24/25 473/473 [==============================] - 21s 44ms/step - loss: 0.5820 - accuracy: 0.7642 - val_loss: 0.5684 - val_accuracy: 0.7812 - lr: 2.0000e-05 Epoch 25/25 473/473 [==============================] - 21s 44ms/step - loss: 0.5749 - accuracy: 0.7667 - val_loss: 0.5712 - val_accuracy: 0.7788 - lr: 2.0000e-05
cnn_model_1.save('cnn_model_1bestscores.h5')
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`. saving_api.save_model(
Note: .h5 format is usable with python based backend such as flask however since the web is becoming dominated by javascript frame works such as react for front end client side user interfaces, and node.js backend and serverside functions converting the saved model to json format simplifys the deployment phase.
!tensorflowjs_converter --input_format=keras /content/cnn_model_1bestscores.h5 /content/recallprecisionf1js
2024-02-20 00:53:00.430837: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-02-20 00:53:00.430894: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-02-20 00:53:00.432410: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-02-20 00:53:01.465249: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_1.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 19ms/step - loss: 0.6674 - accuracy: 0.8047 Test Loss: 0.6673559546470642 Test Accuracy: 0.8046875
plot_model_performance_and_confusion_matrix(cnn_model_1, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 0s 15ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.87 0.81 0.84 32 1 0.65 0.88 0.75 32 2 0.81 0.69 0.75 32 3 0.96 0.84 0.90 32 accuracy 0.80 128 macro avg 0.82 0.80 0.81 128 weighted avg 0.82 0.80 0.81 128
from IPython.display import HTML, display
# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_1.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> CNN_MODEL_1 </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 19ms/step - loss: 0.6674 - accuracy: 0.8047 Test Loss: 0.6673559546470642 Test Accuracy: 0.8046875
Test Loss: CNN_MODEL_1
Test Loss: 0.6674
Test Accuracy: 80.4688%
# Fixing the random seed generator to ensure random initilization numbers
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
# Define the CNN model
cnn_model_updated = Sequential([
# First Convolutional Block
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.2), # Slightly reduced dropout
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(128, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Adjusted dropout
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Adjusted dropout
# Fourth Convolutional Block
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Keeping consistent dropout
# Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
GlobalAveragePooling2D(),
Dropout(0.4), # Adjusted dropout
# Simplified Dense Layers
Dense(512, activation='relu'),
Dropout(0.4),
Dense(4, activation='softmax') # Output layer for 4 classes
])
# Model Summary
cnn_model_updated.summary()
cnn_model_updated.compile(optimizer=Adamax(learning_rate=0.001), # Adjusted learning rate
loss='categorical_crossentropy',
metrics=['accuracy'])
# Callbacks
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=1, min_lr=0.0001)
# Use plot_model to create a diagram of the model
plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)
# Display the image in the notebook
from IPython.display import Image
Image(filename='model_architecture.png')
Model: "sequential_3" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_19 (Conv2D) (None, 48, 48, 64) 1792 batch_normalization_10 (Ba (None, 48, 48, 64) 256 tchNormalization) conv2d_20 (Conv2D) (None, 48, 48, 64) 36928 max_pooling2d_7 (MaxPoolin (None, 24, 24, 64) 0 g2D) dropout_12 (Dropout) (None, 24, 24, 64) 0 conv2d_21 (Conv2D) (None, 24, 24, 128) 73856 batch_normalization_11 (Ba (None, 24, 24, 128) 512 tchNormalization) conv2d_22 (Conv2D) (None, 24, 24, 128) 147584 max_pooling2d_8 (MaxPoolin (None, 12, 12, 128) 0 g2D) dropout_13 (Dropout) (None, 12, 12, 128) 0 conv2d_23 (Conv2D) (None, 12, 12, 256) 295168 batch_normalization_12 (Ba (None, 12, 12, 256) 1024 tchNormalization) conv2d_24 (Conv2D) (None, 12, 12, 256) 590080 batch_normalization_13 (Ba (None, 12, 12, 256) 1024 tchNormalization) max_pooling2d_9 (MaxPoolin (None, 6, 6, 256) 0 g2D) dropout_14 (Dropout) (None, 6, 6, 256) 0 conv2d_25 (Conv2D) (None, 6, 6, 512) 1180160 batch_normalization_14 (Ba (None, 6, 6, 512) 2048 tchNormalization) max_pooling2d_10 (MaxPooli (None, 3, 3, 512) 0 ng2D) dropout_15 (Dropout) (None, 3, 3, 512) 0 global_average_pooling2d_2 (None, 512) 0 (GlobalAveragePooling2D) dropout_16 (Dropout) (None, 512) 0 dense_12 (Dense) (None, 512) 262656 dropout_17 (Dropout) (None, 512) 0 dense_13 (Dense) (None, 4) 2052 ================================================================= Total params: 2595140 (9.90 MB) Trainable params: 2592708 (9.89 MB) Non-trainable params: 2432 (9.50 KB) _________________________________________________________________
history = cnn_model_updated.fit(
train_generator,
epochs=50, # Adjust based on early stopping
validation_data=validation_generator,
callbacks=[early_stopping, reduce_lr],
class_weight=class_weights_adjusted
)
Epoch 1/50 473/473 [==============================] - 21s 39ms/step - loss: 1.5330 - accuracy: 0.3007 - val_loss: 1.5124 - val_accuracy: 0.2614 - lr: 0.0010 Epoch 2/50 473/473 [==============================] - 18s 38ms/step - loss: 1.3547 - accuracy: 0.3367 - val_loss: 1.3045 - val_accuracy: 0.4069 - lr: 0.0010 Epoch 3/50 473/473 [==============================] - 18s 37ms/step - loss: 1.2599 - accuracy: 0.3876 - val_loss: 1.2079 - val_accuracy: 0.4392 - lr: 0.0010 Epoch 4/50 473/473 [==============================] - 18s 39ms/step - loss: 1.1803 - accuracy: 0.4438 - val_loss: 1.0475 - val_accuracy: 0.5152 - lr: 0.0010 Epoch 5/50 473/473 [==============================] - 18s 38ms/step - loss: 1.1001 - accuracy: 0.5031 - val_loss: 1.1717 - val_accuracy: 0.5286 - lr: 0.0010 Epoch 6/50 473/473 [==============================] - 18s 38ms/step - loss: 0.9893 - accuracy: 0.5705 - val_loss: 0.8159 - val_accuracy: 0.6614 - lr: 0.0010 Epoch 7/50 473/473 [==============================] - 19s 41ms/step - loss: 0.9038 - accuracy: 0.6192 - val_loss: 0.8654 - val_accuracy: 0.6237 - lr: 0.0010 Epoch 8/50 473/473 [==============================] - 18s 38ms/step - loss: 0.8652 - accuracy: 0.6345 - val_loss: 0.8329 - val_accuracy: 0.6440 - lr: 0.0010 Epoch 9/50 473/473 [==============================] - 18s 38ms/step - loss: 0.8341 - accuracy: 0.6546 - val_loss: 0.6905 - val_accuracy: 0.7279 - lr: 0.0010 Epoch 10/50 473/473 [==============================] - 19s 40ms/step - loss: 0.8032 - accuracy: 0.6685 - val_loss: 0.7138 - val_accuracy: 0.7151 - lr: 0.0010 Epoch 11/50 473/473 [==============================] - 18s 39ms/step - loss: 0.7811 - accuracy: 0.6790 - val_loss: 0.7618 - val_accuracy: 0.6908 - lr: 0.0010 Epoch 12/50 473/473 [==============================] - ETA: 0s - loss: 0.7622 - accuracy: 0.6912 Epoch 12: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 473/473 [==============================] - 18s 39ms/step - loss: 0.7622 - accuracy: 0.6912 - val_loss: 0.8096 - val_accuracy: 0.6610 - lr: 0.0010 Epoch 13/50 473/473 [==============================] - 18s 38ms/step - loss: 0.7197 - accuracy: 0.7091 - val_loss: 0.6167 - val_accuracy: 0.7490 - lr: 2.0000e-04 Epoch 14/50 473/473 [==============================] - 18s 39ms/step - loss: 0.7033 - accuracy: 0.7167 - val_loss: 0.6098 - val_accuracy: 0.7553 - lr: 2.0000e-04 Epoch 15/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6967 - accuracy: 0.7189 - val_loss: 0.6111 - val_accuracy: 0.7494 - lr: 2.0000e-04 Epoch 16/50 473/473 [==============================] - 19s 39ms/step - loss: 0.6935 - accuracy: 0.7200 - val_loss: 0.6152 - val_accuracy: 0.7468 - lr: 2.0000e-04 Epoch 17/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6796 - accuracy: 0.7239 - val_loss: 0.5981 - val_accuracy: 0.7599 - lr: 2.0000e-04 Epoch 18/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6750 - accuracy: 0.7243 - val_loss: 0.5996 - val_accuracy: 0.7577 - lr: 2.0000e-04 Epoch 19/50 473/473 [==============================] - 19s 40ms/step - loss: 0.6636 - accuracy: 0.7290 - val_loss: 0.6017 - val_accuracy: 0.7615 - lr: 2.0000e-04 Epoch 20/50 473/473 [==============================] - ETA: 0s - loss: 0.6692 - accuracy: 0.7312 Epoch 20: ReduceLROnPlateau reducing learning rate to 0.0001. 473/473 [==============================] - 20s 41ms/step - loss: 0.6692 - accuracy: 0.7312 - val_loss: 0.6029 - val_accuracy: 0.7631 - lr: 2.0000e-04 Epoch 21/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6599 - accuracy: 0.7342 - val_loss: 0.5979 - val_accuracy: 0.7593 - lr: 1.0000e-04 Epoch 22/50 473/473 [==============================] - 19s 39ms/step - loss: 0.6470 - accuracy: 0.7419 - val_loss: 0.6020 - val_accuracy: 0.7637 - lr: 1.0000e-04 Epoch 23/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6533 - accuracy: 0.7359 - val_loss: 0.5943 - val_accuracy: 0.7641 - lr: 1.0000e-04 Epoch 24/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6441 - accuracy: 0.7425 - val_loss: 0.5866 - val_accuracy: 0.7681 - lr: 1.0000e-04 Epoch 25/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6351 - accuracy: 0.7438 - val_loss: 0.5952 - val_accuracy: 0.7619 - lr: 1.0000e-04 Epoch 26/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6455 - accuracy: 0.7439 - val_loss: 0.5825 - val_accuracy: 0.7689 - lr: 1.0000e-04 Epoch 27/50 473/473 [==============================] - 19s 40ms/step - loss: 0.6396 - accuracy: 0.7444 - val_loss: 0.5929 - val_accuracy: 0.7609 - lr: 1.0000e-04 Epoch 28/50 473/473 [==============================] - 19s 40ms/step - loss: 0.6401 - accuracy: 0.7401 - val_loss: 0.5803 - val_accuracy: 0.7742 - lr: 1.0000e-04 Epoch 29/50 473/473 [==============================] - 19s 41ms/step - loss: 0.6328 - accuracy: 0.7465 - val_loss: 0.5784 - val_accuracy: 0.7744 - lr: 1.0000e-04 Epoch 30/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6305 - accuracy: 0.7479 - val_loss: 0.5822 - val_accuracy: 0.7715 - lr: 1.0000e-04 Epoch 31/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6291 - accuracy: 0.7468 - val_loss: 0.5808 - val_accuracy: 0.7754 - lr: 1.0000e-04 Epoch 32/50 473/473 [==============================] - 18s 39ms/step - loss: 0.6255 - accuracy: 0.7497 - val_loss: 0.5841 - val_accuracy: 0.7707 - lr: 1.0000e-04 Epoch 33/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6215 - accuracy: 0.7494 - val_loss: 0.5776 - val_accuracy: 0.7677 - lr: 1.0000e-04 Epoch 34/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6125 - accuracy: 0.7530 - val_loss: 0.5772 - val_accuracy: 0.7738 - lr: 1.0000e-04 Epoch 35/50 473/473 [==============================] - 18s 38ms/step - loss: 0.6201 - accuracy: 0.7522 - val_loss: 0.5783 - val_accuracy: 0.7736 - lr: 1.0000e-04 Epoch 36/50 472/473 [============================>.] - ETA: 0s - loss: 0.6184 - accuracy: 0.7519Restoring model weights from the end of the best epoch: 31. 473/473 [==============================] - 18s 38ms/step - loss: 0.6186 - accuracy: 0.7520 - val_loss: 0.5763 - val_accuracy: 0.7705 - lr: 1.0000e-04 Epoch 36: early stopping
plot_model_performance_and_confusion_matrix(cnn_model_updated, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display
# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_updated.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> CNN_MODEL_updated </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 10ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.82 0.84 0.83 32 1 0.72 0.81 0.76 32 2 0.79 0.69 0.73 32 3 0.97 0.94 0.95 32 accuracy 0.82 128 macro avg 0.82 0.82 0.82 128 weighted avg 0.82 0.82 0.82 128
4/4 [==============================] - 0s 11ms/step - loss: 0.5187 - accuracy: 0.8203 Test Loss: 0.5187498927116394 Test Accuracy: 0.8203125
Test Loss: CNN_MODEL_updated
Test Loss: 0.5187
Test Accuracy: 82.0312%
Observations and Insights:__
While building the CNN models, i tried out various optimizers such as SGD, Adamax, AdamW, Adam, After running the model training several times with each and adjusting the learning rate along with the drop out
Model Architecture: The model is a deep convolutional neural network (CNN) designed with multiple convolutional blocks. Each block consists of convolutional layers with increasing filter sizes, BatchNormalization layers, MaxPooling for down-sampling, and Dropout layers to mitigate overfitting. The model concludes with dense layers and a softmax output layer for 4 classes. It uses GlobalAveragePooling2D to reduce the number of parameters and potentially decrease overfitting compared to using a Flatten layer.
Optimization and Callbacks: The model is compiled with the Adamax optimizer, a variant of Adam, with a low learning rate of 0.00018 to facilitate fine-grained updates to the model weights. Ituses callbacks such as EarlyStopping to halt training when validation accuracy stops to improve, ReduceLROnPlateau to decrease the learning rate when a metric has stopped improving, and ModelCheckpoint to save the model at its best performance based on validation accuracy.
Performance Metrics:
Accuracy: The overall accuracy on the validation set is 80%, indicating that the model correctly predicts the class for 80 out of every 100 images it is tested against. Precision, Recall, and F1-Score: The model shows varying levels of precision and recall across different classes: For the 'happy' class, it has a high precision of 0.87 but a slightly lower recall of 0.81, resulting in an F1-score of 0.84. This indicates it is quite good at identifying 'happy' faces but slightly conservative, missing some true 'happy' instances. The 'neutral' class has a precision of 0.65 with a high recall of 0.88, leading to an F1-score of 0.75. The model tends to label too many faces as 'neutral', but when it predicts 'neutral', it is often correct. For 'sad', precision is 0.81 and recall is 0.69, with an F1-score of 0.75. The model is relatively precise in identifying 'sad' faces but misses a significant portion of them. The 'surprise' class shows excellent precision at 0.96 and recall at 0.84, with an F1-score of 0.90. The model is highly effective at identifying 'surprise' faces with minimal false positives. Macro and Weighted Averages: Both macro and weighted averages across metrics hover around 0.82 for precision and 0.81 for the F1-score, suggesting a balanced performance across classes despite the variance in individual class metrics. Analysis: The model demonstrates good general performance with an overall accuracy of 80%. However, there's room for improvement, especially in balancing precision and recall across the different classes. The relatively lower precision for 'neutral' suggests a tendency to over-classify images as 'neutral', whereas the lower recall for 'sad' indicates missed detections of this emotion. The high performance on the 'surprise' class, with both high precision and recall, showcases the model's capability in identifying distinct features effectively.
Recommendations for Improvement:
Data Augmentation: To improve model generalization, especially for underperforming classes. Class Weights: Adjusting class weights during training could help address imbalances in precision and recall. Hyperparameter Tuning: Experimenting with different learning rates, optimizers, and dropout rates could enhance model performance. Model Architecture Adjustments: Tweaking the number of filters, layers, or incorporating additional regularization techniques might yield better results. Overall, cnn_model_2 shows promising results with a solid foundation for further optimization and tuning to enhance its ability to generalize across a range of facial expressions.
Observations and Insights:__ Some breif observaions, the models performed ok, reaching a highpoint of prediction accuracy of about 79 percent with cnn model 2. The precision and recall and f1 scores all are between 78 and 80 which is a good indicator. Still there is plenty of room for improvement and possibly with transferlearing the models will be able to get up to the 90 percent accuracy percent tile range.
In this section, we will create several Transfer Learning architectures. For the pre-trained models, we will select three popular architectures namely, VGG16, ResNet v2, and Efficient Net. The difference between these architectures and the previous architectures is that these will require 3 input channels while the earlier ones worked on 'grayscale' images. Therefore, we need to create new DataLoaders.
In this section, we are creating data loaders that we will use as inputs to our Neural Network. We will have to go with color_mode = 'rgb' as this is the required format for the transfer learning architectures.
# Set the random seed
np.random.seed(42)
train_generator = train_datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
validation_generator = test_val_datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical'
)
test_generator = test_val_datagen.flow_from_directory(
test_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical',
shuffle=False # Keep data in order for evaluation
)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras import Model
vgg = VGG16(include_top = False, weights = 'imagenet', input_shape = (48, 48, 3))
vgg.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5 58889256/58889256 [==============================] - 2s 0us/step Model: "vgg16" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 48, 48, 3)] 0 block1_conv1 (Conv2D) (None, 48, 48, 64) 1792 block1_conv2 (Conv2D) (None, 48, 48, 64) 36928 block1_pool (MaxPooling2D) (None, 24, 24, 64) 0 block2_conv1 (Conv2D) (None, 24, 24, 128) 73856 block2_conv2 (Conv2D) (None, 24, 24, 128) 147584 block2_pool (MaxPooling2D) (None, 12, 12, 128) 0 block3_conv1 (Conv2D) (None, 12, 12, 256) 295168 block3_conv2 (Conv2D) (None, 12, 12, 256) 590080 block3_conv3 (Conv2D) (None, 12, 12, 256) 590080 block3_pool (MaxPooling2D) (None, 6, 6, 256) 0 block4_conv1 (Conv2D) (None, 6, 6, 512) 1180160 block4_conv2 (Conv2D) (None, 6, 6, 512) 2359808 block4_conv3 (Conv2D) (None, 6, 6, 512) 2359808 block4_pool (MaxPooling2D) (None, 3, 3, 512) 0 block5_conv1 (Conv2D) (None, 3, 3, 512) 2359808 block5_conv2 (Conv2D) (None, 3, 3, 512) 2359808 block5_conv3 (Conv2D) (None, 3, 3, 512) 2359808 block5_pool (MaxPooling2D) (None, 1, 1, 512) 0 ================================================================= Total params: 14714688 (56.13 MB) Trainable params: 14714688 (56.13 MB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________
np.random.seed(42)
samples_per_class = {
'surprise': 3173,
'happy': 3976,
'neutral': 3978,
'sad': 3982,
}
# Calculate total samples
total_samples = sum(samples_per_class.values())
# Calculate class weights
class_weights = {class_id: total_samples/(len(samples_per_class)*num_samples)
for class_id, num_samples in enumerate(samples_per_class.values())}
print(class_weights)
{0: 1.1904349196344153, 1: 0.9500125754527163, 2: 0.949534942182001, 3: 0.9485811150175791}
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
# Set the random seed
np.random.seed(42)
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.layers import Dense, Flatten, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import AdamW, SGD, Adamax
# Set the random seed
np.random.seed(42)
#
# Load the VGG16 model
vgg = VGG16(include_top=False, weights='imagenet', input_shape=(48, 48, 3))
# Freeze all layers in the base VGG16 model
for layer in vgg.layers:
layer.trainable = False
# Select the output of the 'block3_pool' layer
x = vgg.get_layer('block3_pool').output
# Continue adding your custom layers from here
x = Flatten()(x) # Flatten the output of 'block3_pool'
x = Dense(512, activation='relu')(x) # Custom dense layer
x = Dropout(0.1)(x) # Dropout layer
x = Dense(1024, activation='relu')(x) # Another dense layer
x = Dropout(0.2)(x) # Dropout layer
x = Dense(32, activation='relu')(x)
# Final layer with softmax for classification
predictions = Dense(4, activation='softmax')(x) # Assuming 4 classes for emotions
# Create the final model using VGG16 input and your custom output
model = Model(inputs=vgg.input, outputs=predictions)
# Compile the model
model.compile(optimizer=Adamax(learning_rate=0.000025), loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()
# Use plot_model to create a diagram of the model
plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)
# Display the image in the notebook
from IPython.display import Image
Image(filename='model_architecture.png')
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5 58889256/58889256 [==============================] - 3s 0us/step Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 48, 48, 3)] 0 block1_conv1 (Conv2D) (None, 48, 48, 64) 1792 block1_conv2 (Conv2D) (None, 48, 48, 64) 36928 block1_pool (MaxPooling2D) (None, 24, 24, 64) 0 block2_conv1 (Conv2D) (None, 24, 24, 128) 73856 block2_conv2 (Conv2D) (None, 24, 24, 128) 147584 block2_pool (MaxPooling2D) (None, 12, 12, 128) 0 block3_conv1 (Conv2D) (None, 12, 12, 256) 295168 block3_conv2 (Conv2D) (None, 12, 12, 256) 590080 block3_conv3 (Conv2D) (None, 12, 12, 256) 590080 block3_pool (MaxPooling2D) (None, 6, 6, 256) 0 flatten_3 (Flatten) (None, 9216) 0 dense_14 (Dense) (None, 512) 4719104 dropout_18 (Dropout) (None, 512) 0 dense_15 (Dense) (None, 1024) 525312 dropout_19 (Dropout) (None, 1024) 0 dense_16 (Dense) (None, 32) 32800 dense_17 (Dense) (None, 4) 132 ================================================================= Total params: 7012836 (26.75 MB) Trainable params: 5277348 (20.13 MB) Non-trainable params: 1735488 (6.62 MB) _________________________________________________________________
from tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)
# Train the model
history = model.fit(
train_generator,
epochs=75,
validation_data=validation_generator,
callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/75 473/473 [==============================] - 10s 17ms/step - loss: 1.8805 - accuracy: 0.2998 - val_loss: 1.3816 - val_accuracy: 0.2620 - lr: 2.5000e-05 Epoch 2/75 473/473 [==============================] - 8s 17ms/step - loss: 1.3644 - accuracy: 0.3341 - val_loss: 1.3035 - val_accuracy: 0.3709 - lr: 2.5000e-05 Epoch 3/75 473/473 [==============================] - 8s 16ms/step - loss: 1.2823 - accuracy: 0.4023 - val_loss: 1.2123 - val_accuracy: 0.4571 - lr: 2.5000e-05 Epoch 4/75 473/473 [==============================] - 8s 16ms/step - loss: 1.2114 - accuracy: 0.4331 - val_loss: 1.1272 - val_accuracy: 0.5120 - lr: 2.5000e-05 Epoch 5/75 473/473 [==============================] - 8s 16ms/step - loss: 1.1606 - accuracy: 0.4671 - val_loss: 1.1128 - val_accuracy: 0.5310 - lr: 2.5000e-05 Epoch 6/75 473/473 [==============================] - 8s 16ms/step - loss: 1.1103 - accuracy: 0.4885 - val_loss: 1.0563 - val_accuracy: 0.5429 - lr: 2.5000e-05 Epoch 7/75 473/473 [==============================] - 7s 16ms/step - loss: 1.0590 - accuracy: 0.5169 - val_loss: 0.9952 - val_accuracy: 0.5626 - lr: 2.5000e-05 Epoch 8/75 473/473 [==============================] - 8s 16ms/step - loss: 1.0040 - accuracy: 0.5544 - val_loss: 0.9309 - val_accuracy: 0.6168 - lr: 2.5000e-05 Epoch 9/75 473/473 [==============================] - 8s 16ms/step - loss: 0.9363 - accuracy: 0.5885 - val_loss: 0.8933 - val_accuracy: 0.6359 - lr: 2.5000e-05 Epoch 10/75 473/473 [==============================] - 8s 16ms/step - loss: 0.8996 - accuracy: 0.6125 - val_loss: 0.8726 - val_accuracy: 0.6418 - lr: 2.5000e-05 Epoch 11/75 473/473 [==============================] - 8s 16ms/step - loss: 0.8622 - accuracy: 0.6273 - val_loss: 0.8536 - val_accuracy: 0.6546 - lr: 2.5000e-05 Epoch 12/75 473/473 [==============================] - 8s 16ms/step - loss: 0.8274 - accuracy: 0.6489 - val_loss: 0.8336 - val_accuracy: 0.6612 - lr: 2.5000e-05 Epoch 13/75 473/473 [==============================] - 8s 17ms/step - loss: 0.8026 - accuracy: 0.6589 - val_loss: 0.8221 - val_accuracy: 0.6620 - lr: 2.5000e-05 Epoch 14/75 473/473 [==============================] - 8s 16ms/step - loss: 0.7751 - accuracy: 0.6736 - val_loss: 0.8141 - val_accuracy: 0.6775 - lr: 2.5000e-05 Epoch 15/75 473/473 [==============================] - 7s 16ms/step - loss: 0.7490 - accuracy: 0.6816 - val_loss: 0.8283 - val_accuracy: 0.6739 - lr: 2.5000e-05 Epoch 16/75 473/473 [==============================] - 8s 16ms/step - loss: 0.7240 - accuracy: 0.6998 - val_loss: 0.7986 - val_accuracy: 0.6801 - lr: 2.5000e-05 Epoch 17/75 473/473 [==============================] - 8s 16ms/step - loss: 0.6963 - accuracy: 0.7090 - val_loss: 0.7856 - val_accuracy: 0.6856 - lr: 2.5000e-05 Epoch 18/75 473/473 [==============================] - 7s 16ms/step - loss: 0.6791 - accuracy: 0.7133 - val_loss: 0.7866 - val_accuracy: 0.6845 - lr: 2.5000e-05 Epoch 19/75 473/473 [==============================] - 8s 17ms/step - loss: 0.6501 - accuracy: 0.7321 - val_loss: 0.7696 - val_accuracy: 0.6922 - lr: 2.5000e-05 Epoch 20/75 473/473 [==============================] - 7s 16ms/step - loss: 0.6265 - accuracy: 0.7390 - val_loss: 0.8019 - val_accuracy: 0.6894 - lr: 2.5000e-05 Epoch 21/75 473/473 [==============================] - 8s 16ms/step - loss: 0.6100 - accuracy: 0.7475 - val_loss: 0.7730 - val_accuracy: 0.6940 - lr: 2.5000e-05 Epoch 22/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5893 - accuracy: 0.7551 - val_loss: 0.7784 - val_accuracy: 0.6920 - lr: 2.5000e-05 Epoch 23/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5662 - accuracy: 0.7686 - val_loss: 0.7709 - val_accuracy: 0.6994 - lr: 2.5000e-05 Epoch 24/75 473/473 [==============================] - 7s 16ms/step - loss: 0.5509 - accuracy: 0.7749 - val_loss: 0.7765 - val_accuracy: 0.6946 - lr: 2.5000e-05 Epoch 25/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5189 - accuracy: 0.7917 - val_loss: 0.7767 - val_accuracy: 0.6966 - lr: 5.0000e-06 Epoch 26/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5130 - accuracy: 0.7920 - val_loss: 0.7657 - val_accuracy: 0.7024 - lr: 5.0000e-06 Epoch 27/75 473/473 [==============================] - 8s 16ms/step - loss: 0.5047 - accuracy: 0.7953 - val_loss: 0.7609 - val_accuracy: 0.7018 - lr: 5.0000e-06 Epoch 28/75 473/473 [==============================] - 8s 16ms/step - loss: 0.4956 - accuracy: 0.8018 - val_loss: 0.7666 - val_accuracy: 0.7036 - lr: 5.0000e-06 Epoch 29/75 473/473 [==============================] - 7s 16ms/step - loss: 0.4924 - accuracy: 0.7997 - val_loss: 0.7728 - val_accuracy: 0.6998 - lr: 5.0000e-06 Epoch 30/75 473/473 [==============================] - 7s 16ms/step - loss: 0.4924 - accuracy: 0.8024 - val_loss: 0.7701 - val_accuracy: 0.7028 - lr: 5.0000e-06 Epoch 31/75 473/473 [==============================] - 8s 16ms/step - loss: 0.4835 - accuracy: 0.8064 - val_loss: 0.7681 - val_accuracy: 0.7018 - lr: 5.0000e-06 Epoch 32/75 473/473 [==============================] - 7s 16ms/step - loss: 0.4756 - accuracy: 0.8108 - val_loss: 0.7696 - val_accuracy: 0.7012 - lr: 5.0000e-06 Epoch 33/75 473/473 [==============================] - 8s 16ms/step - loss: 0.4751 - accuracy: 0.8079 - val_loss: 0.7710 - val_accuracy: 0.7026 - lr: 1.0000e-06 Epoch 34/75 473/473 [==============================] - 7s 15ms/step - loss: 0.4773 - accuracy: 0.8082 - val_loss: 0.7680 - val_accuracy: 0.7032 - lr: 1.0000e-06
from sklearn.metrics import confusion_matrix, classification_report
cm = confusion_matrix(test_generator.classes, np.argmax(model.predict(test_generator), axis=1))
# Confusion matrix
print("Confusion Matrix:")
print(cm)
# Classification report
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('VGG_Model_1 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('VGG_Model_1 Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for VGG_MODEL_1")
plt.show()
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
predicted_labels = np.argmax(model.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 0s 11ms/step Confusion Matrix: [[25 6 0 1] [ 0 28 3 1] [ 3 8 18 3] [ 0 2 2 28]]
4/4 [==============================] - 0s 13ms/step dict_keys(['happy', 'neutral', 'sad', 'surprise']) precision recall f1-score support 0 0.89 0.78 0.83 32 1 0.64 0.88 0.74 32 2 0.78 0.56 0.65 32 3 0.85 0.88 0.86 32 accuracy 0.77 128 macro avg 0.79 0.77 0.77 128 weighted avg 0.79 0.77 0.77 128
The customized VGG16 model, adapted for emotion classification into four categories (happy, neutral, sad, surprise), demonstrates an overall accuracy of 77%. The model shows varying precision and recall across classes, with notable strengths in identifying 'surprise' emotions accurately. Despite its solid foundation, there's room for improvement, especially in balancing performance across all classes. Recommendations include data augmentation, fine-tuning of VGG16 layers, optimization of hyperparameters, and adjustment of class weights to enhance generalization and performance uniformly across categories.
Think About It:
**Observations and Insights:__It is seeming to prove to be difficult to get the model performance above 75-80 percent accuracy. Additionally, The Category of neutral is often what gets predicted when the actual class is that of 'sad', The largest confusion point seems to be a sad face being miss classified for 'neutral'
# Set the random Seed
np.random.seed(42)
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Flatten, Dropout
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.resnet50 import preprocess_input, ResNet50
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
import tensorflow as tf
import tensorflow.keras.applications as ap
from tensorflow.keras import Model
Resnet = ap.ResNet101(include_top = False, weights = "imagenet", input_shape=(48, 48, 3))
Resnet.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet101_weights_tf_dim_ordering_tf_kernels_notop.h5 171446536/171446536 [==============================] - 10s 0us/step Model: "resnet101" __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_2 (InputLayer) [(None, 48, 48, 3)] 0 [] conv1_pad (ZeroPadding2D) (None, 54, 54, 3) 0 ['input_2[0][0]'] conv1_conv (Conv2D) (None, 24, 24, 64) 9472 ['conv1_pad[0][0]'] conv1_bn (BatchNormalizati (None, 24, 24, 64) 256 ['conv1_conv[0][0]'] on) conv1_relu (Activation) (None, 24, 24, 64) 0 ['conv1_bn[0][0]'] pool1_pad (ZeroPadding2D) (None, 26, 26, 64) 0 ['conv1_relu[0][0]'] pool1_pool (MaxPooling2D) (None, 12, 12, 64) 0 ['pool1_pad[0][0]'] conv2_block1_1_conv (Conv2 (None, 12, 12, 64) 4160 ['pool1_pool[0][0]'] D) conv2_block1_1_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block1_1_conv[0][0]'] rmalization) conv2_block1_1_relu (Activ (None, 12, 12, 64) 0 ['conv2_block1_1_bn[0][0]'] ation) conv2_block1_2_conv (Conv2 (None, 12, 12, 64) 36928 ['conv2_block1_1_relu[0][0]'] D) conv2_block1_2_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block1_2_conv[0][0]'] rmalization) conv2_block1_2_relu (Activ (None, 12, 12, 64) 0 ['conv2_block1_2_bn[0][0]'] ation) conv2_block1_0_conv (Conv2 (None, 12, 12, 256) 16640 ['pool1_pool[0][0]'] D) conv2_block1_3_conv (Conv2 (None, 12, 12, 256) 16640 ['conv2_block1_2_relu[0][0]'] D) conv2_block1_0_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block1_0_conv[0][0]'] rmalization) conv2_block1_3_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block1_3_conv[0][0]'] rmalization) conv2_block1_add (Add) (None, 12, 12, 256) 0 ['conv2_block1_0_bn[0][0]', 'conv2_block1_3_bn[0][0]'] conv2_block1_out (Activati (None, 12, 12, 256) 0 ['conv2_block1_add[0][0]'] on) conv2_block2_1_conv (Conv2 (None, 12, 12, 64) 16448 ['conv2_block1_out[0][0]'] D) conv2_block2_1_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block2_1_conv[0][0]'] rmalization) conv2_block2_1_relu (Activ (None, 12, 12, 64) 0 ['conv2_block2_1_bn[0][0]'] ation) conv2_block2_2_conv (Conv2 (None, 12, 12, 64) 36928 ['conv2_block2_1_relu[0][0]'] D) conv2_block2_2_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block2_2_conv[0][0]'] rmalization) conv2_block2_2_relu (Activ (None, 12, 12, 64) 0 ['conv2_block2_2_bn[0][0]'] ation) conv2_block2_3_conv (Conv2 (None, 12, 12, 256) 16640 ['conv2_block2_2_relu[0][0]'] D) conv2_block2_3_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block2_3_conv[0][0]'] rmalization) conv2_block2_add (Add) (None, 12, 12, 256) 0 ['conv2_block1_out[0][0]', 'conv2_block2_3_bn[0][0]'] conv2_block2_out (Activati (None, 12, 12, 256) 0 ['conv2_block2_add[0][0]'] on) conv2_block3_1_conv (Conv2 (None, 12, 12, 64) 16448 ['conv2_block2_out[0][0]'] D) conv2_block3_1_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block3_1_conv[0][0]'] rmalization) conv2_block3_1_relu (Activ (None, 12, 12, 64) 0 ['conv2_block3_1_bn[0][0]'] ation) conv2_block3_2_conv (Conv2 (None, 12, 12, 64) 36928 ['conv2_block3_1_relu[0][0]'] D) conv2_block3_2_bn (BatchNo (None, 12, 12, 64) 256 ['conv2_block3_2_conv[0][0]'] rmalization) conv2_block3_2_relu (Activ (None, 12, 12, 64) 0 ['conv2_block3_2_bn[0][0]'] ation) conv2_block3_3_conv (Conv2 (None, 12, 12, 256) 16640 ['conv2_block3_2_relu[0][0]'] D) conv2_block3_3_bn (BatchNo (None, 12, 12, 256) 1024 ['conv2_block3_3_conv[0][0]'] rmalization) conv2_block3_add (Add) (None, 12, 12, 256) 0 ['conv2_block2_out[0][0]', 'conv2_block3_3_bn[0][0]'] conv2_block3_out (Activati (None, 12, 12, 256) 0 ['conv2_block3_add[0][0]'] on) conv3_block1_1_conv (Conv2 (None, 6, 6, 128) 32896 ['conv2_block3_out[0][0]'] D) conv3_block1_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block1_1_conv[0][0]'] rmalization) conv3_block1_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block1_1_bn[0][0]'] ation) conv3_block1_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block1_1_relu[0][0]'] D) conv3_block1_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block1_2_conv[0][0]'] rmalization) conv3_block1_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block1_2_bn[0][0]'] ation) conv3_block1_0_conv (Conv2 (None, 6, 6, 512) 131584 ['conv2_block3_out[0][0]'] D) conv3_block1_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block1_2_relu[0][0]'] D) conv3_block1_0_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block1_0_conv[0][0]'] rmalization) conv3_block1_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block1_3_conv[0][0]'] rmalization) conv3_block1_add (Add) (None, 6, 6, 512) 0 ['conv3_block1_0_bn[0][0]', 'conv3_block1_3_bn[0][0]'] conv3_block1_out (Activati (None, 6, 6, 512) 0 ['conv3_block1_add[0][0]'] on) conv3_block2_1_conv (Conv2 (None, 6, 6, 128) 65664 ['conv3_block1_out[0][0]'] D) conv3_block2_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block2_1_conv[0][0]'] rmalization) conv3_block2_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block2_1_bn[0][0]'] ation) conv3_block2_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block2_1_relu[0][0]'] D) conv3_block2_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block2_2_conv[0][0]'] rmalization) conv3_block2_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block2_2_bn[0][0]'] ation) conv3_block2_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block2_2_relu[0][0]'] D) conv3_block2_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block2_3_conv[0][0]'] rmalization) conv3_block2_add (Add) (None, 6, 6, 512) 0 ['conv3_block1_out[0][0]', 'conv3_block2_3_bn[0][0]'] conv3_block2_out (Activati (None, 6, 6, 512) 0 ['conv3_block2_add[0][0]'] on) conv3_block3_1_conv (Conv2 (None, 6, 6, 128) 65664 ['conv3_block2_out[0][0]'] D) conv3_block3_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block3_1_conv[0][0]'] rmalization) conv3_block3_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block3_1_bn[0][0]'] ation) conv3_block3_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block3_1_relu[0][0]'] D) conv3_block3_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block3_2_conv[0][0]'] rmalization) conv3_block3_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block3_2_bn[0][0]'] ation) conv3_block3_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block3_2_relu[0][0]'] D) conv3_block3_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block3_3_conv[0][0]'] rmalization) conv3_block3_add (Add) (None, 6, 6, 512) 0 ['conv3_block2_out[0][0]', 'conv3_block3_3_bn[0][0]'] conv3_block3_out (Activati (None, 6, 6, 512) 0 ['conv3_block3_add[0][0]'] on) conv3_block4_1_conv (Conv2 (None, 6, 6, 128) 65664 ['conv3_block3_out[0][0]'] D) conv3_block4_1_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block4_1_conv[0][0]'] rmalization) conv3_block4_1_relu (Activ (None, 6, 6, 128) 0 ['conv3_block4_1_bn[0][0]'] ation) conv3_block4_2_conv (Conv2 (None, 6, 6, 128) 147584 ['conv3_block4_1_relu[0][0]'] D) conv3_block4_2_bn (BatchNo (None, 6, 6, 128) 512 ['conv3_block4_2_conv[0][0]'] rmalization) conv3_block4_2_relu (Activ (None, 6, 6, 128) 0 ['conv3_block4_2_bn[0][0]'] ation) conv3_block4_3_conv (Conv2 (None, 6, 6, 512) 66048 ['conv3_block4_2_relu[0][0]'] D) conv3_block4_3_bn (BatchNo (None, 6, 6, 512) 2048 ['conv3_block4_3_conv[0][0]'] rmalization) conv3_block4_add (Add) (None, 6, 6, 512) 0 ['conv3_block3_out[0][0]', 'conv3_block4_3_bn[0][0]'] conv3_block4_out (Activati (None, 6, 6, 512) 0 ['conv3_block4_add[0][0]'] on) conv4_block1_1_conv (Conv2 (None, 3, 3, 256) 131328 ['conv3_block4_out[0][0]'] D) conv4_block1_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block1_1_conv[0][0]'] rmalization) conv4_block1_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block1_1_bn[0][0]'] ation) conv4_block1_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block1_1_relu[0][0]'] D) conv4_block1_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block1_2_conv[0][0]'] rmalization) conv4_block1_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block1_2_bn[0][0]'] ation) conv4_block1_0_conv (Conv2 (None, 3, 3, 1024) 525312 ['conv3_block4_out[0][0]'] D) conv4_block1_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block1_2_relu[0][0]'] D) conv4_block1_0_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block1_0_conv[0][0]'] rmalization) conv4_block1_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block1_3_conv[0][0]'] rmalization) conv4_block1_add (Add) (None, 3, 3, 1024) 0 ['conv4_block1_0_bn[0][0]', 'conv4_block1_3_bn[0][0]'] conv4_block1_out (Activati (None, 3, 3, 1024) 0 ['conv4_block1_add[0][0]'] on) conv4_block2_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block1_out[0][0]'] D) conv4_block2_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block2_1_conv[0][0]'] rmalization) conv4_block2_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block2_1_bn[0][0]'] ation) conv4_block2_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block2_1_relu[0][0]'] D) conv4_block2_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block2_2_conv[0][0]'] rmalization) conv4_block2_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block2_2_bn[0][0]'] ation) conv4_block2_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block2_2_relu[0][0]'] D) conv4_block2_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block2_3_conv[0][0]'] rmalization) conv4_block2_add (Add) (None, 3, 3, 1024) 0 ['conv4_block1_out[0][0]', 'conv4_block2_3_bn[0][0]'] conv4_block2_out (Activati (None, 3, 3, 1024) 0 ['conv4_block2_add[0][0]'] on) conv4_block3_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block2_out[0][0]'] D) conv4_block3_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block3_1_conv[0][0]'] rmalization) conv4_block3_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block3_1_bn[0][0]'] ation) conv4_block3_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block3_1_relu[0][0]'] D) conv4_block3_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block3_2_conv[0][0]'] rmalization) conv4_block3_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block3_2_bn[0][0]'] ation) conv4_block3_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block3_2_relu[0][0]'] D) conv4_block3_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block3_3_conv[0][0]'] rmalization) conv4_block3_add (Add) (None, 3, 3, 1024) 0 ['conv4_block2_out[0][0]', 'conv4_block3_3_bn[0][0]'] conv4_block3_out (Activati (None, 3, 3, 1024) 0 ['conv4_block3_add[0][0]'] on) conv4_block4_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block3_out[0][0]'] D) conv4_block4_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block4_1_conv[0][0]'] rmalization) conv4_block4_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block4_1_bn[0][0]'] ation) conv4_block4_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block4_1_relu[0][0]'] D) conv4_block4_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block4_2_conv[0][0]'] rmalization) conv4_block4_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block4_2_bn[0][0]'] ation) conv4_block4_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block4_2_relu[0][0]'] D) conv4_block4_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block4_3_conv[0][0]'] rmalization) conv4_block4_add (Add) (None, 3, 3, 1024) 0 ['conv4_block3_out[0][0]', 'conv4_block4_3_bn[0][0]'] conv4_block4_out (Activati (None, 3, 3, 1024) 0 ['conv4_block4_add[0][0]'] on) conv4_block5_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block4_out[0][0]'] D) conv4_block5_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block5_1_conv[0][0]'] rmalization) conv4_block5_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block5_1_bn[0][0]'] ation) conv4_block5_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block5_1_relu[0][0]'] D) conv4_block5_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block5_2_conv[0][0]'] rmalization) conv4_block5_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block5_2_bn[0][0]'] ation) conv4_block5_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block5_2_relu[0][0]'] D) conv4_block5_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block5_3_conv[0][0]'] rmalization) conv4_block5_add (Add) (None, 3, 3, 1024) 0 ['conv4_block4_out[0][0]', 'conv4_block5_3_bn[0][0]'] conv4_block5_out (Activati (None, 3, 3, 1024) 0 ['conv4_block5_add[0][0]'] on) conv4_block6_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block5_out[0][0]'] D) conv4_block6_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block6_1_conv[0][0]'] rmalization) conv4_block6_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block6_1_bn[0][0]'] ation) conv4_block6_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block6_1_relu[0][0]'] D) conv4_block6_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block6_2_conv[0][0]'] rmalization) conv4_block6_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block6_2_bn[0][0]'] ation) conv4_block6_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block6_2_relu[0][0]'] D) conv4_block6_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block6_3_conv[0][0]'] rmalization) conv4_block6_add (Add) (None, 3, 3, 1024) 0 ['conv4_block5_out[0][0]', 'conv4_block6_3_bn[0][0]'] conv4_block6_out (Activati (None, 3, 3, 1024) 0 ['conv4_block6_add[0][0]'] on) conv4_block7_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block6_out[0][0]'] D) conv4_block7_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block7_1_conv[0][0]'] rmalization) conv4_block7_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block7_1_bn[0][0]'] ation) conv4_block7_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block7_1_relu[0][0]'] D) conv4_block7_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block7_2_conv[0][0]'] rmalization) conv4_block7_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block7_2_bn[0][0]'] ation) conv4_block7_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block7_2_relu[0][0]'] D) conv4_block7_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block7_3_conv[0][0]'] rmalization) conv4_block7_add (Add) (None, 3, 3, 1024) 0 ['conv4_block6_out[0][0]', 'conv4_block7_3_bn[0][0]'] conv4_block7_out (Activati (None, 3, 3, 1024) 0 ['conv4_block7_add[0][0]'] on) conv4_block8_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block7_out[0][0]'] D) conv4_block8_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block8_1_conv[0][0]'] rmalization) conv4_block8_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block8_1_bn[0][0]'] ation) conv4_block8_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block8_1_relu[0][0]'] D) conv4_block8_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block8_2_conv[0][0]'] rmalization) conv4_block8_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block8_2_bn[0][0]'] ation) conv4_block8_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block8_2_relu[0][0]'] D) conv4_block8_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block8_3_conv[0][0]'] rmalization) conv4_block8_add (Add) (None, 3, 3, 1024) 0 ['conv4_block7_out[0][0]', 'conv4_block8_3_bn[0][0]'] conv4_block8_out (Activati (None, 3, 3, 1024) 0 ['conv4_block8_add[0][0]'] on) conv4_block9_1_conv (Conv2 (None, 3, 3, 256) 262400 ['conv4_block8_out[0][0]'] D) conv4_block9_1_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block9_1_conv[0][0]'] rmalization) conv4_block9_1_relu (Activ (None, 3, 3, 256) 0 ['conv4_block9_1_bn[0][0]'] ation) conv4_block9_2_conv (Conv2 (None, 3, 3, 256) 590080 ['conv4_block9_1_relu[0][0]'] D) conv4_block9_2_bn (BatchNo (None, 3, 3, 256) 1024 ['conv4_block9_2_conv[0][0]'] rmalization) conv4_block9_2_relu (Activ (None, 3, 3, 256) 0 ['conv4_block9_2_bn[0][0]'] ation) conv4_block9_3_conv (Conv2 (None, 3, 3, 1024) 263168 ['conv4_block9_2_relu[0][0]'] D) conv4_block9_3_bn (BatchNo (None, 3, 3, 1024) 4096 ['conv4_block9_3_conv[0][0]'] rmalization) conv4_block9_add (Add) (None, 3, 3, 1024) 0 ['conv4_block8_out[0][0]', 'conv4_block9_3_bn[0][0]'] conv4_block9_out (Activati (None, 3, 3, 1024) 0 ['conv4_block9_add[0][0]'] on) conv4_block10_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block9_out[0][0]'] 2D) conv4_block10_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block10_1_conv[0][0]'] ormalization) conv4_block10_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block10_1_bn[0][0]'] vation) conv4_block10_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block10_1_relu[0][0]'] 2D) conv4_block10_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block10_2_conv[0][0]'] ormalization) conv4_block10_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block10_2_bn[0][0]'] vation) conv4_block10_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block10_2_relu[0][0]'] 2D) conv4_block10_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block10_3_conv[0][0]'] ormalization) conv4_block10_add (Add) (None, 3, 3, 1024) 0 ['conv4_block9_out[0][0]', 'conv4_block10_3_bn[0][0]'] conv4_block10_out (Activat (None, 3, 3, 1024) 0 ['conv4_block10_add[0][0]'] ion) conv4_block11_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block10_out[0][0]'] 2D) conv4_block11_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block11_1_conv[0][0]'] ormalization) conv4_block11_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block11_1_bn[0][0]'] vation) conv4_block11_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block11_1_relu[0][0]'] 2D) conv4_block11_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block11_2_conv[0][0]'] ormalization) conv4_block11_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block11_2_bn[0][0]'] vation) conv4_block11_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block11_2_relu[0][0]'] 2D) conv4_block11_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block11_3_conv[0][0]'] ormalization) conv4_block11_add (Add) (None, 3, 3, 1024) 0 ['conv4_block10_out[0][0]', 'conv4_block11_3_bn[0][0]'] conv4_block11_out (Activat (None, 3, 3, 1024) 0 ['conv4_block11_add[0][0]'] ion) conv4_block12_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block11_out[0][0]'] 2D) conv4_block12_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block12_1_conv[0][0]'] ormalization) conv4_block12_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block12_1_bn[0][0]'] vation) conv4_block12_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block12_1_relu[0][0]'] 2D) conv4_block12_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block12_2_conv[0][0]'] ormalization) conv4_block12_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block12_2_bn[0][0]'] vation) conv4_block12_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block12_2_relu[0][0]'] 2D) conv4_block12_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block12_3_conv[0][0]'] ormalization) conv4_block12_add (Add) (None, 3, 3, 1024) 0 ['conv4_block11_out[0][0]', 'conv4_block12_3_bn[0][0]'] conv4_block12_out (Activat (None, 3, 3, 1024) 0 ['conv4_block12_add[0][0]'] ion) conv4_block13_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block12_out[0][0]'] 2D) conv4_block13_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block13_1_conv[0][0]'] ormalization) conv4_block13_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block13_1_bn[0][0]'] vation) conv4_block13_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block13_1_relu[0][0]'] 2D) conv4_block13_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block13_2_conv[0][0]'] ormalization) conv4_block13_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block13_2_bn[0][0]'] vation) conv4_block13_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block13_2_relu[0][0]'] 2D) conv4_block13_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block13_3_conv[0][0]'] ormalization) conv4_block13_add (Add) (None, 3, 3, 1024) 0 ['conv4_block12_out[0][0]', 'conv4_block13_3_bn[0][0]'] conv4_block13_out (Activat (None, 3, 3, 1024) 0 ['conv4_block13_add[0][0]'] ion) conv4_block14_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block13_out[0][0]'] 2D) conv4_block14_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block14_1_conv[0][0]'] ormalization) conv4_block14_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block14_1_bn[0][0]'] vation) conv4_block14_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block14_1_relu[0][0]'] 2D) conv4_block14_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block14_2_conv[0][0]'] ormalization) conv4_block14_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block14_2_bn[0][0]'] vation) conv4_block14_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block14_2_relu[0][0]'] 2D) conv4_block14_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block14_3_conv[0][0]'] ormalization) conv4_block14_add (Add) (None, 3, 3, 1024) 0 ['conv4_block13_out[0][0]', 'conv4_block14_3_bn[0][0]'] conv4_block14_out (Activat (None, 3, 3, 1024) 0 ['conv4_block14_add[0][0]'] ion) conv4_block15_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block14_out[0][0]'] 2D) conv4_block15_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block15_1_conv[0][0]'] ormalization) conv4_block15_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block15_1_bn[0][0]'] vation) conv4_block15_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block15_1_relu[0][0]'] 2D) conv4_block15_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block15_2_conv[0][0]'] ormalization) conv4_block15_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block15_2_bn[0][0]'] vation) conv4_block15_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block15_2_relu[0][0]'] 2D) conv4_block15_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block15_3_conv[0][0]'] ormalization) conv4_block15_add (Add) (None, 3, 3, 1024) 0 ['conv4_block14_out[0][0]', 'conv4_block15_3_bn[0][0]'] conv4_block15_out (Activat (None, 3, 3, 1024) 0 ['conv4_block15_add[0][0]'] ion) conv4_block16_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block15_out[0][0]'] 2D) conv4_block16_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block16_1_conv[0][0]'] ormalization) conv4_block16_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block16_1_bn[0][0]'] vation) conv4_block16_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block16_1_relu[0][0]'] 2D) conv4_block16_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block16_2_conv[0][0]'] ormalization) conv4_block16_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block16_2_bn[0][0]'] vation) conv4_block16_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block16_2_relu[0][0]'] 2D) conv4_block16_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block16_3_conv[0][0]'] ormalization) conv4_block16_add (Add) (None, 3, 3, 1024) 0 ['conv4_block15_out[0][0]', 'conv4_block16_3_bn[0][0]'] conv4_block16_out (Activat (None, 3, 3, 1024) 0 ['conv4_block16_add[0][0]'] ion) conv4_block17_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block16_out[0][0]'] 2D) conv4_block17_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block17_1_conv[0][0]'] ormalization) conv4_block17_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block17_1_bn[0][0]'] vation) conv4_block17_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block17_1_relu[0][0]'] 2D) conv4_block17_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block17_2_conv[0][0]'] ormalization) conv4_block17_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block17_2_bn[0][0]'] vation) conv4_block17_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block17_2_relu[0][0]'] 2D) conv4_block17_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block17_3_conv[0][0]'] ormalization) conv4_block17_add (Add) (None, 3, 3, 1024) 0 ['conv4_block16_out[0][0]', 'conv4_block17_3_bn[0][0]'] conv4_block17_out (Activat (None, 3, 3, 1024) 0 ['conv4_block17_add[0][0]'] ion) conv4_block18_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block17_out[0][0]'] 2D) conv4_block18_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block18_1_conv[0][0]'] ormalization) conv4_block18_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block18_1_bn[0][0]'] vation) conv4_block18_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block18_1_relu[0][0]'] 2D) conv4_block18_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block18_2_conv[0][0]'] ormalization) conv4_block18_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block18_2_bn[0][0]'] vation) conv4_block18_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block18_2_relu[0][0]'] 2D) conv4_block18_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block18_3_conv[0][0]'] ormalization) conv4_block18_add (Add) (None, 3, 3, 1024) 0 ['conv4_block17_out[0][0]', 'conv4_block18_3_bn[0][0]'] conv4_block18_out (Activat (None, 3, 3, 1024) 0 ['conv4_block18_add[0][0]'] ion) conv4_block19_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block18_out[0][0]'] 2D) conv4_block19_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block19_1_conv[0][0]'] ormalization) conv4_block19_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block19_1_bn[0][0]'] vation) conv4_block19_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block19_1_relu[0][0]'] 2D) conv4_block19_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block19_2_conv[0][0]'] ormalization) conv4_block19_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block19_2_bn[0][0]'] vation) conv4_block19_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block19_2_relu[0][0]'] 2D) conv4_block19_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block19_3_conv[0][0]'] ormalization) conv4_block19_add (Add) (None, 3, 3, 1024) 0 ['conv4_block18_out[0][0]', 'conv4_block19_3_bn[0][0]'] conv4_block19_out (Activat (None, 3, 3, 1024) 0 ['conv4_block19_add[0][0]'] ion) conv4_block20_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block19_out[0][0]'] 2D) conv4_block20_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block20_1_conv[0][0]'] ormalization) conv4_block20_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block20_1_bn[0][0]'] vation) conv4_block20_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block20_1_relu[0][0]'] 2D) conv4_block20_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block20_2_conv[0][0]'] ormalization) conv4_block20_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block20_2_bn[0][0]'] vation) conv4_block20_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block20_2_relu[0][0]'] 2D) conv4_block20_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block20_3_conv[0][0]'] ormalization) conv4_block20_add (Add) (None, 3, 3, 1024) 0 ['conv4_block19_out[0][0]', 'conv4_block20_3_bn[0][0]'] conv4_block20_out (Activat (None, 3, 3, 1024) 0 ['conv4_block20_add[0][0]'] ion) conv4_block21_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block20_out[0][0]'] 2D) conv4_block21_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block21_1_conv[0][0]'] ormalization) conv4_block21_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block21_1_bn[0][0]'] vation) conv4_block21_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block21_1_relu[0][0]'] 2D) conv4_block21_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block21_2_conv[0][0]'] ormalization) conv4_block21_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block21_2_bn[0][0]'] vation) conv4_block21_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block21_2_relu[0][0]'] 2D) conv4_block21_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block21_3_conv[0][0]'] ormalization) conv4_block21_add (Add) (None, 3, 3, 1024) 0 ['conv4_block20_out[0][0]', 'conv4_block21_3_bn[0][0]'] conv4_block21_out (Activat (None, 3, 3, 1024) 0 ['conv4_block21_add[0][0]'] ion) conv4_block22_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block21_out[0][0]'] 2D) conv4_block22_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block22_1_conv[0][0]'] ormalization) conv4_block22_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block22_1_bn[0][0]'] vation) conv4_block22_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block22_1_relu[0][0]'] 2D) conv4_block22_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block22_2_conv[0][0]'] ormalization) conv4_block22_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block22_2_bn[0][0]'] vation) conv4_block22_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block22_2_relu[0][0]'] 2D) conv4_block22_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block22_3_conv[0][0]'] ormalization) conv4_block22_add (Add) (None, 3, 3, 1024) 0 ['conv4_block21_out[0][0]', 'conv4_block22_3_bn[0][0]'] conv4_block22_out (Activat (None, 3, 3, 1024) 0 ['conv4_block22_add[0][0]'] ion) conv4_block23_1_conv (Conv (None, 3, 3, 256) 262400 ['conv4_block22_out[0][0]'] 2D) conv4_block23_1_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block23_1_conv[0][0]'] ormalization) conv4_block23_1_relu (Acti (None, 3, 3, 256) 0 ['conv4_block23_1_bn[0][0]'] vation) conv4_block23_2_conv (Conv (None, 3, 3, 256) 590080 ['conv4_block23_1_relu[0][0]'] 2D) conv4_block23_2_bn (BatchN (None, 3, 3, 256) 1024 ['conv4_block23_2_conv[0][0]'] ormalization) conv4_block23_2_relu (Acti (None, 3, 3, 256) 0 ['conv4_block23_2_bn[0][0]'] vation) conv4_block23_3_conv (Conv (None, 3, 3, 1024) 263168 ['conv4_block23_2_relu[0][0]'] 2D) conv4_block23_3_bn (BatchN (None, 3, 3, 1024) 4096 ['conv4_block23_3_conv[0][0]'] ormalization) conv4_block23_add (Add) (None, 3, 3, 1024) 0 ['conv4_block22_out[0][0]', 'conv4_block23_3_bn[0][0]'] conv4_block23_out (Activat (None, 3, 3, 1024) 0 ['conv4_block23_add[0][0]'] ion) conv5_block1_1_conv (Conv2 (None, 2, 2, 512) 524800 ['conv4_block23_out[0][0]'] D) conv5_block1_1_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block1_1_conv[0][0]'] rmalization) conv5_block1_1_relu (Activ (None, 2, 2, 512) 0 ['conv5_block1_1_bn[0][0]'] ation) conv5_block1_2_conv (Conv2 (None, 2, 2, 512) 2359808 ['conv5_block1_1_relu[0][0]'] D) conv5_block1_2_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block1_2_conv[0][0]'] rmalization) conv5_block1_2_relu (Activ (None, 2, 2, 512) 0 ['conv5_block1_2_bn[0][0]'] ation) conv5_block1_0_conv (Conv2 (None, 2, 2, 2048) 2099200 ['conv4_block23_out[0][0]'] D) conv5_block1_3_conv (Conv2 (None, 2, 2, 2048) 1050624 ['conv5_block1_2_relu[0][0]'] D) conv5_block1_0_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block1_0_conv[0][0]'] rmalization) conv5_block1_3_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block1_3_conv[0][0]'] rmalization) conv5_block1_add (Add) (None, 2, 2, 2048) 0 ['conv5_block1_0_bn[0][0]', 'conv5_block1_3_bn[0][0]'] conv5_block1_out (Activati (None, 2, 2, 2048) 0 ['conv5_block1_add[0][0]'] on) conv5_block2_1_conv (Conv2 (None, 2, 2, 512) 1049088 ['conv5_block1_out[0][0]'] D) conv5_block2_1_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block2_1_conv[0][0]'] rmalization) conv5_block2_1_relu (Activ (None, 2, 2, 512) 0 ['conv5_block2_1_bn[0][0]'] ation) conv5_block2_2_conv (Conv2 (None, 2, 2, 512) 2359808 ['conv5_block2_1_relu[0][0]'] D) conv5_block2_2_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block2_2_conv[0][0]'] rmalization) conv5_block2_2_relu (Activ (None, 2, 2, 512) 0 ['conv5_block2_2_bn[0][0]'] ation) conv5_block2_3_conv (Conv2 (None, 2, 2, 2048) 1050624 ['conv5_block2_2_relu[0][0]'] D) conv5_block2_3_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block2_3_conv[0][0]'] rmalization) conv5_block2_add (Add) (None, 2, 2, 2048) 0 ['conv5_block1_out[0][0]', 'conv5_block2_3_bn[0][0]'] conv5_block2_out (Activati (None, 2, 2, 2048) 0 ['conv5_block2_add[0][0]'] on) conv5_block3_1_conv (Conv2 (None, 2, 2, 512) 1049088 ['conv5_block2_out[0][0]'] D) conv5_block3_1_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block3_1_conv[0][0]'] rmalization) conv5_block3_1_relu (Activ (None, 2, 2, 512) 0 ['conv5_block3_1_bn[0][0]'] ation) conv5_block3_2_conv (Conv2 (None, 2, 2, 512) 2359808 ['conv5_block3_1_relu[0][0]'] D) conv5_block3_2_bn (BatchNo (None, 2, 2, 512) 2048 ['conv5_block3_2_conv[0][0]'] rmalization) conv5_block3_2_relu (Activ (None, 2, 2, 512) 0 ['conv5_block3_2_bn[0][0]'] ation) conv5_block3_3_conv (Conv2 (None, 2, 2, 2048) 1050624 ['conv5_block3_2_relu[0][0]'] D) conv5_block3_3_bn (BatchNo (None, 2, 2, 2048) 8192 ['conv5_block3_3_conv[0][0]'] rmalization) conv5_block3_add (Add) (None, 2, 2, 2048) 0 ['conv5_block2_out[0][0]', 'conv5_block3_3_bn[0][0]'] conv5_block3_out (Activati (None, 2, 2, 2048) 0 ['conv5_block3_add[0][0]'] on) ================================================================================================== Total params: 42658176 (162.73 MB) Trainable params: 42552832 (162.33 MB) Non-trainable params: 105344 (411.50 KB) __________________________________________________________________________________________________
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
np.random.seed(42)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import AdamW, SGD, Adamax
# Load ResNet50 model pre-trained on ImageNet without the top layer
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(48, 48, 3))
# Add custom layers on top of the base model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.4)(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.4)(x)
predictions = Dense(4, activation='softmax')(x)
resnet_model2 = Model(inputs=base_model.input, outputs=predictions)
# Compile the model
resnet_model2.compile(optimizer=Adamax(learning_rate=0.00025), loss='categorical_crossentropy', metrics=['accuracy'])
# Use plot_model to create a diagram of the model
plot_model(model, to_file='model_architecture.png', show_shapes=True, show_layer_names=True)
# Display the image in the notebook
from IPython.display import Image
Image(filename='model_architecture.png')
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5 94765736/94765736 [==============================] - 4s 0us/step
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)
history = resnet_model2.fit(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=20,
validation_data=validation_generator,
validation_steps=validation_generator.samples // validation_generator.batch_size,
class_weight=class_weights_adjusted,
callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/20 472/472 [==============================] - ETA: 0s - loss: 1.2432 - accuracy: 0.4541
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`. saving_api.save_model(
472/472 [==============================] - 53s 62ms/step - loss: 1.2432 - accuracy: 0.4541 - val_loss: 148.1311 - val_accuracy: 0.2230 - lr: 2.5000e-04 Epoch 2/20 472/472 [==============================] - 27s 57ms/step - loss: 0.8767 - accuracy: 0.6330 - val_loss: 1.2080 - val_accuracy: 0.4980 - lr: 2.5000e-04 Epoch 3/20 472/472 [==============================] - 27s 57ms/step - loss: 0.6773 - accuracy: 0.7215 - val_loss: 0.8134 - val_accuracy: 0.6853 - lr: 2.5000e-04 Epoch 4/20 472/472 [==============================] - 26s 55ms/step - loss: 0.5270 - accuracy: 0.7853 - val_loss: 0.8374 - val_accuracy: 0.6808 - lr: 2.5000e-04 Epoch 5/20 472/472 [==============================] - 27s 58ms/step - loss: 0.3790 - accuracy: 0.8498 - val_loss: 0.8913 - val_accuracy: 0.7121 - lr: 2.5000e-04 Epoch 6/20 472/472 [==============================] - 26s 55ms/step - loss: 0.2578 - accuracy: 0.9021 - val_loss: 1.2126 - val_accuracy: 0.6875 - lr: 2.5000e-04 Epoch 7/20 472/472 [==============================] - 27s 57ms/step - loss: 0.1923 - accuracy: 0.9292 - val_loss: 0.9888 - val_accuracy: 0.7210 - lr: 2.5000e-04 Epoch 8/20 472/472 [==============================] - 26s 54ms/step - loss: 0.1055 - accuracy: 0.9625 - val_loss: 1.2811 - val_accuracy: 0.7185 - lr: 2.5000e-04 Epoch 9/20 472/472 [==============================] - 27s 57ms/step - loss: 0.0546 - accuracy: 0.9821 - val_loss: 1.2760 - val_accuracy: 0.7306 - lr: 5.0000e-05 Epoch 10/20 472/472 [==============================] - 27s 57ms/step - loss: 0.0351 - accuracy: 0.9884 - val_loss: 1.3550 - val_accuracy: 0.7310 - lr: 5.0000e-05
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(test_generator.classes, np.argmax(resnet_model2.predict(test_generator), axis=1))
# Confusion matrix
print("Confusion Matrix:")
print(cm)
# Classification report
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model_resnet Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model_resnet Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for Resnet Model")
plt.show()
predicted_labels = np.argmax(resnet_model2.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 1s 20ms/step Confusion Matrix: [[26 2 2 2] [ 2 25 5 0] [ 1 12 17 2] [ 0 3 1 28]]
4/4 [==============================] - 0s 22ms/step dict_keys(['happy', 'neutral', 'sad', 'surprise']) precision recall f1-score support 0 0.90 0.81 0.85 32 1 0.60 0.78 0.68 32 2 0.68 0.53 0.60 32 3 0.88 0.88 0.88 32 accuracy 0.75 128 macro avg 0.76 0.75 0.75 128 weighted avg 0.76 0.75 0.75 128
# Evaluate the model on the test set
val_loss, val_accuracy = resnet_model2.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
The ResNet50-based custom model, adapted for classifying four emotional states, achieved an overall accuracy of 75%. The model's precision and recall suggest it performs best in recognizing 'happy' and 'surprise' emotions, with scores indicating high reliability in these predictions. However, it shows a need for improvement in accurately classifying 'neutral' and 'sad' states, as evidenced by lower precision for 'neutral' and lower recall for 'sad'. The confusion matrix details the model's performance across all classes, highlighting specific areas where misclassifications occur, particularly in distinguishing 'neutral' from 'sad'. Optimizations could focus on enhancing feature extraction and addressing class imbalance to improve accuracy and balance across all emotion categories.
np.random.seed(42)
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.efficientnet import EfficientNetB0, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras import regularizers
import tensorflow as tf
import tensorflow.keras.applications as ap
EfficientNet = ap.EfficientNetV2B2(include_top=False,weights="imagenet", input_shape= (48, 48, 3))
EfficientNet.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/efficientnet_v2/efficientnetv2-b2_notop.h5 35839040/35839040 [==============================] - 3s 0us/step Model: "efficientnetv2-b2" __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) [(None, 48, 48, 3)] 0 [] rescaling (Rescaling) (None, 48, 48, 3) 0 ['input_1[0][0]'] normalization (Normalizati (None, 48, 48, 3) 0 ['rescaling[0][0]'] on) stem_conv (Conv2D) (None, 24, 24, 32) 864 ['normalization[0][0]'] stem_bn (BatchNormalizatio (None, 24, 24, 32) 128 ['stem_conv[0][0]'] n) stem_activation (Activatio (None, 24, 24, 32) 0 ['stem_bn[0][0]'] n) block1a_project_conv (Conv (None, 24, 24, 16) 4608 ['stem_activation[0][0]'] 2D) block1a_project_bn (BatchN (None, 24, 24, 16) 64 ['block1a_project_conv[0][0]'] ormalization) block1a_project_activation (None, 24, 24, 16) 0 ['block1a_project_bn[0][0]'] (Activation) block1b_project_conv (Conv (None, 24, 24, 16) 2304 ['block1a_project_activation[0 2D) ][0]'] block1b_project_bn (BatchN (None, 24, 24, 16) 64 ['block1b_project_conv[0][0]'] ormalization) block1b_project_activation (None, 24, 24, 16) 0 ['block1b_project_bn[0][0]'] (Activation) block1b_drop (Dropout) (None, 24, 24, 16) 0 ['block1b_project_activation[0 ][0]'] block1b_add (Add) (None, 24, 24, 16) 0 ['block1b_drop[0][0]', 'block1a_project_activation[0 ][0]'] block2a_expand_conv (Conv2 (None, 12, 12, 64) 9216 ['block1b_add[0][0]'] D) block2a_expand_bn (BatchNo (None, 12, 12, 64) 256 ['block2a_expand_conv[0][0]'] rmalization) block2a_expand_activation (None, 12, 12, 64) 0 ['block2a_expand_bn[0][0]'] (Activation) block2a_project_conv (Conv (None, 12, 12, 32) 2048 ['block2a_expand_activation[0] 2D) [0]'] block2a_project_bn (BatchN (None, 12, 12, 32) 128 ['block2a_project_conv[0][0]'] ormalization) block2b_expand_conv (Conv2 (None, 12, 12, 128) 36864 ['block2a_project_bn[0][0]'] D) block2b_expand_bn (BatchNo (None, 12, 12, 128) 512 ['block2b_expand_conv[0][0]'] rmalization) block2b_expand_activation (None, 12, 12, 128) 0 ['block2b_expand_bn[0][0]'] (Activation) block2b_project_conv (Conv (None, 12, 12, 32) 4096 ['block2b_expand_activation[0] 2D) [0]'] block2b_project_bn (BatchN (None, 12, 12, 32) 128 ['block2b_project_conv[0][0]'] ormalization) block2b_drop (Dropout) (None, 12, 12, 32) 0 ['block2b_project_bn[0][0]'] block2b_add (Add) (None, 12, 12, 32) 0 ['block2b_drop[0][0]', 'block2a_project_bn[0][0]'] block2c_expand_conv (Conv2 (None, 12, 12, 128) 36864 ['block2b_add[0][0]'] D) block2c_expand_bn (BatchNo (None, 12, 12, 128) 512 ['block2c_expand_conv[0][0]'] rmalization) block2c_expand_activation (None, 12, 12, 128) 0 ['block2c_expand_bn[0][0]'] (Activation) block2c_project_conv (Conv (None, 12, 12, 32) 4096 ['block2c_expand_activation[0] 2D) [0]'] block2c_project_bn (BatchN (None, 12, 12, 32) 128 ['block2c_project_conv[0][0]'] ormalization) block2c_drop (Dropout) (None, 12, 12, 32) 0 ['block2c_project_bn[0][0]'] block2c_add (Add) (None, 12, 12, 32) 0 ['block2c_drop[0][0]', 'block2b_add[0][0]'] block3a_expand_conv (Conv2 (None, 6, 6, 128) 36864 ['block2c_add[0][0]'] D) block3a_expand_bn (BatchNo (None, 6, 6, 128) 512 ['block3a_expand_conv[0][0]'] rmalization) block3a_expand_activation (None, 6, 6, 128) 0 ['block3a_expand_bn[0][0]'] (Activation) block3a_project_conv (Conv (None, 6, 6, 56) 7168 ['block3a_expand_activation[0] 2D) [0]'] block3a_project_bn (BatchN (None, 6, 6, 56) 224 ['block3a_project_conv[0][0]'] ormalization) block3b_expand_conv (Conv2 (None, 6, 6, 224) 112896 ['block3a_project_bn[0][0]'] D) block3b_expand_bn (BatchNo (None, 6, 6, 224) 896 ['block3b_expand_conv[0][0]'] rmalization) block3b_expand_activation (None, 6, 6, 224) 0 ['block3b_expand_bn[0][0]'] (Activation) block3b_project_conv (Conv (None, 6, 6, 56) 12544 ['block3b_expand_activation[0] 2D) [0]'] block3b_project_bn (BatchN (None, 6, 6, 56) 224 ['block3b_project_conv[0][0]'] ormalization) block3b_drop (Dropout) (None, 6, 6, 56) 0 ['block3b_project_bn[0][0]'] block3b_add (Add) (None, 6, 6, 56) 0 ['block3b_drop[0][0]', 'block3a_project_bn[0][0]'] block3c_expand_conv (Conv2 (None, 6, 6, 224) 112896 ['block3b_add[0][0]'] D) block3c_expand_bn (BatchNo (None, 6, 6, 224) 896 ['block3c_expand_conv[0][0]'] rmalization) block3c_expand_activation (None, 6, 6, 224) 0 ['block3c_expand_bn[0][0]'] (Activation) block3c_project_conv (Conv (None, 6, 6, 56) 12544 ['block3c_expand_activation[0] 2D) [0]'] block3c_project_bn (BatchN (None, 6, 6, 56) 224 ['block3c_project_conv[0][0]'] ormalization) block3c_drop (Dropout) (None, 6, 6, 56) 0 ['block3c_project_bn[0][0]'] block3c_add (Add) (None, 6, 6, 56) 0 ['block3c_drop[0][0]', 'block3b_add[0][0]'] block4a_expand_conv (Conv2 (None, 6, 6, 224) 12544 ['block3c_add[0][0]'] D) block4a_expand_bn (BatchNo (None, 6, 6, 224) 896 ['block4a_expand_conv[0][0]'] rmalization) block4a_expand_activation (None, 6, 6, 224) 0 ['block4a_expand_bn[0][0]'] (Activation) block4a_dwconv2 (Depthwise (None, 3, 3, 224) 2016 ['block4a_expand_activation[0] Conv2D) [0]'] block4a_bn (BatchNormaliza (None, 3, 3, 224) 896 ['block4a_dwconv2[0][0]'] tion) block4a_activation (Activa (None, 3, 3, 224) 0 ['block4a_bn[0][0]'] tion) block4a_se_squeeze (Global (None, 224) 0 ['block4a_activation[0][0]'] AveragePooling2D) block4a_se_reshape (Reshap (None, 1, 1, 224) 0 ['block4a_se_squeeze[0][0]'] e) block4a_se_reduce (Conv2D) (None, 1, 1, 14) 3150 ['block4a_se_reshape[0][0]'] block4a_se_expand (Conv2D) (None, 1, 1, 224) 3360 ['block4a_se_reduce[0][0]'] block4a_se_excite (Multipl (None, 3, 3, 224) 0 ['block4a_activation[0][0]', y) 'block4a_se_expand[0][0]'] block4a_project_conv (Conv (None, 3, 3, 104) 23296 ['block4a_se_excite[0][0]'] 2D) block4a_project_bn (BatchN (None, 3, 3, 104) 416 ['block4a_project_conv[0][0]'] ormalization) block4b_expand_conv (Conv2 (None, 3, 3, 416) 43264 ['block4a_project_bn[0][0]'] D) block4b_expand_bn (BatchNo (None, 3, 3, 416) 1664 ['block4b_expand_conv[0][0]'] rmalization) block4b_expand_activation (None, 3, 3, 416) 0 ['block4b_expand_bn[0][0]'] (Activation) block4b_dwconv2 (Depthwise (None, 3, 3, 416) 3744 ['block4b_expand_activation[0] Conv2D) [0]'] block4b_bn (BatchNormaliza (None, 3, 3, 416) 1664 ['block4b_dwconv2[0][0]'] tion) block4b_activation (Activa (None, 3, 3, 416) 0 ['block4b_bn[0][0]'] tion) block4b_se_squeeze (Global (None, 416) 0 ['block4b_activation[0][0]'] AveragePooling2D) block4b_se_reshape (Reshap (None, 1, 1, 416) 0 ['block4b_se_squeeze[0][0]'] e) block4b_se_reduce (Conv2D) (None, 1, 1, 26) 10842 ['block4b_se_reshape[0][0]'] block4b_se_expand (Conv2D) (None, 1, 1, 416) 11232 ['block4b_se_reduce[0][0]'] block4b_se_excite (Multipl (None, 3, 3, 416) 0 ['block4b_activation[0][0]', y) 'block4b_se_expand[0][0]'] block4b_project_conv (Conv (None, 3, 3, 104) 43264 ['block4b_se_excite[0][0]'] 2D) block4b_project_bn (BatchN (None, 3, 3, 104) 416 ['block4b_project_conv[0][0]'] ormalization) block4b_drop (Dropout) (None, 3, 3, 104) 0 ['block4b_project_bn[0][0]'] block4b_add (Add) (None, 3, 3, 104) 0 ['block4b_drop[0][0]', 'block4a_project_bn[0][0]'] block4c_expand_conv (Conv2 (None, 3, 3, 416) 43264 ['block4b_add[0][0]'] D) block4c_expand_bn (BatchNo (None, 3, 3, 416) 1664 ['block4c_expand_conv[0][0]'] rmalization) block4c_expand_activation (None, 3, 3, 416) 0 ['block4c_expand_bn[0][0]'] (Activation) block4c_dwconv2 (Depthwise (None, 3, 3, 416) 3744 ['block4c_expand_activation[0] Conv2D) [0]'] block4c_bn (BatchNormaliza (None, 3, 3, 416) 1664 ['block4c_dwconv2[0][0]'] tion) block4c_activation (Activa (None, 3, 3, 416) 0 ['block4c_bn[0][0]'] tion) block4c_se_squeeze (Global (None, 416) 0 ['block4c_activation[0][0]'] AveragePooling2D) block4c_se_reshape (Reshap (None, 1, 1, 416) 0 ['block4c_se_squeeze[0][0]'] e) block4c_se_reduce (Conv2D) (None, 1, 1, 26) 10842 ['block4c_se_reshape[0][0]'] block4c_se_expand (Conv2D) (None, 1, 1, 416) 11232 ['block4c_se_reduce[0][0]'] block4c_se_excite (Multipl (None, 3, 3, 416) 0 ['block4c_activation[0][0]', y) 'block4c_se_expand[0][0]'] block4c_project_conv (Conv (None, 3, 3, 104) 43264 ['block4c_se_excite[0][0]'] 2D) block4c_project_bn (BatchN (None, 3, 3, 104) 416 ['block4c_project_conv[0][0]'] ormalization) block4c_drop (Dropout) (None, 3, 3, 104) 0 ['block4c_project_bn[0][0]'] block4c_add (Add) (None, 3, 3, 104) 0 ['block4c_drop[0][0]', 'block4b_add[0][0]'] block4d_expand_conv (Conv2 (None, 3, 3, 416) 43264 ['block4c_add[0][0]'] D) block4d_expand_bn (BatchNo (None, 3, 3, 416) 1664 ['block4d_expand_conv[0][0]'] rmalization) block4d_expand_activation (None, 3, 3, 416) 0 ['block4d_expand_bn[0][0]'] (Activation) block4d_dwconv2 (Depthwise (None, 3, 3, 416) 3744 ['block4d_expand_activation[0] Conv2D) [0]'] block4d_bn (BatchNormaliza (None, 3, 3, 416) 1664 ['block4d_dwconv2[0][0]'] tion) block4d_activation (Activa (None, 3, 3, 416) 0 ['block4d_bn[0][0]'] tion) block4d_se_squeeze (Global (None, 416) 0 ['block4d_activation[0][0]'] AveragePooling2D) block4d_se_reshape (Reshap (None, 1, 1, 416) 0 ['block4d_se_squeeze[0][0]'] e) block4d_se_reduce (Conv2D) (None, 1, 1, 26) 10842 ['block4d_se_reshape[0][0]'] block4d_se_expand (Conv2D) (None, 1, 1, 416) 11232 ['block4d_se_reduce[0][0]'] block4d_se_excite (Multipl (None, 3, 3, 416) 0 ['block4d_activation[0][0]', y) 'block4d_se_expand[0][0]'] block4d_project_conv (Conv (None, 3, 3, 104) 43264 ['block4d_se_excite[0][0]'] 2D) block4d_project_bn (BatchN (None, 3, 3, 104) 416 ['block4d_project_conv[0][0]'] ormalization) block4d_drop (Dropout) (None, 3, 3, 104) 0 ['block4d_project_bn[0][0]'] block4d_add (Add) (None, 3, 3, 104) 0 ['block4d_drop[0][0]', 'block4c_add[0][0]'] block5a_expand_conv (Conv2 (None, 3, 3, 624) 64896 ['block4d_add[0][0]'] D) block5a_expand_bn (BatchNo (None, 3, 3, 624) 2496 ['block5a_expand_conv[0][0]'] rmalization) block5a_expand_activation (None, 3, 3, 624) 0 ['block5a_expand_bn[0][0]'] (Activation) block5a_dwconv2 (Depthwise (None, 3, 3, 624) 5616 ['block5a_expand_activation[0] Conv2D) [0]'] block5a_bn (BatchNormaliza (None, 3, 3, 624) 2496 ['block5a_dwconv2[0][0]'] tion) block5a_activation (Activa (None, 3, 3, 624) 0 ['block5a_bn[0][0]'] tion) block5a_se_squeeze (Global (None, 624) 0 ['block5a_activation[0][0]'] AveragePooling2D) block5a_se_reshape (Reshap (None, 1, 1, 624) 0 ['block5a_se_squeeze[0][0]'] e) block5a_se_reduce (Conv2D) (None, 1, 1, 26) 16250 ['block5a_se_reshape[0][0]'] block5a_se_expand (Conv2D) (None, 1, 1, 624) 16848 ['block5a_se_reduce[0][0]'] block5a_se_excite (Multipl (None, 3, 3, 624) 0 ['block5a_activation[0][0]', y) 'block5a_se_expand[0][0]'] block5a_project_conv (Conv (None, 3, 3, 120) 74880 ['block5a_se_excite[0][0]'] 2D) block5a_project_bn (BatchN (None, 3, 3, 120) 480 ['block5a_project_conv[0][0]'] ormalization) block5b_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5a_project_bn[0][0]'] D) block5b_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5b_expand_conv[0][0]'] rmalization) block5b_expand_activation (None, 3, 3, 720) 0 ['block5b_expand_bn[0][0]'] (Activation) block5b_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5b_expand_activation[0] Conv2D) [0]'] block5b_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5b_dwconv2[0][0]'] tion) block5b_activation (Activa (None, 3, 3, 720) 0 ['block5b_bn[0][0]'] tion) block5b_se_squeeze (Global (None, 720) 0 ['block5b_activation[0][0]'] AveragePooling2D) block5b_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5b_se_squeeze[0][0]'] e) block5b_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5b_se_reshape[0][0]'] block5b_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5b_se_reduce[0][0]'] block5b_se_excite (Multipl (None, 3, 3, 720) 0 ['block5b_activation[0][0]', y) 'block5b_se_expand[0][0]'] block5b_project_conv (Conv (None, 3, 3, 120) 86400 ['block5b_se_excite[0][0]'] 2D) block5b_project_bn (BatchN (None, 3, 3, 120) 480 ['block5b_project_conv[0][0]'] ormalization) block5b_drop (Dropout) (None, 3, 3, 120) 0 ['block5b_project_bn[0][0]'] block5b_add (Add) (None, 3, 3, 120) 0 ['block5b_drop[0][0]', 'block5a_project_bn[0][0]'] block5c_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5b_add[0][0]'] D) block5c_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5c_expand_conv[0][0]'] rmalization) block5c_expand_activation (None, 3, 3, 720) 0 ['block5c_expand_bn[0][0]'] (Activation) block5c_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5c_expand_activation[0] Conv2D) [0]'] block5c_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5c_dwconv2[0][0]'] tion) block5c_activation (Activa (None, 3, 3, 720) 0 ['block5c_bn[0][0]'] tion) block5c_se_squeeze (Global (None, 720) 0 ['block5c_activation[0][0]'] AveragePooling2D) block5c_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5c_se_squeeze[0][0]'] e) block5c_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5c_se_reshape[0][0]'] block5c_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5c_se_reduce[0][0]'] block5c_se_excite (Multipl (None, 3, 3, 720) 0 ['block5c_activation[0][0]', y) 'block5c_se_expand[0][0]'] block5c_project_conv (Conv (None, 3, 3, 120) 86400 ['block5c_se_excite[0][0]'] 2D) block5c_project_bn (BatchN (None, 3, 3, 120) 480 ['block5c_project_conv[0][0]'] ormalization) block5c_drop (Dropout) (None, 3, 3, 120) 0 ['block5c_project_bn[0][0]'] block5c_add (Add) (None, 3, 3, 120) 0 ['block5c_drop[0][0]', 'block5b_add[0][0]'] block5d_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5c_add[0][0]'] D) block5d_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5d_expand_conv[0][0]'] rmalization) block5d_expand_activation (None, 3, 3, 720) 0 ['block5d_expand_bn[0][0]'] (Activation) block5d_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5d_expand_activation[0] Conv2D) [0]'] block5d_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5d_dwconv2[0][0]'] tion) block5d_activation (Activa (None, 3, 3, 720) 0 ['block5d_bn[0][0]'] tion) block5d_se_squeeze (Global (None, 720) 0 ['block5d_activation[0][0]'] AveragePooling2D) block5d_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5d_se_squeeze[0][0]'] e) block5d_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5d_se_reshape[0][0]'] block5d_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5d_se_reduce[0][0]'] block5d_se_excite (Multipl (None, 3, 3, 720) 0 ['block5d_activation[0][0]', y) 'block5d_se_expand[0][0]'] block5d_project_conv (Conv (None, 3, 3, 120) 86400 ['block5d_se_excite[0][0]'] 2D) block5d_project_bn (BatchN (None, 3, 3, 120) 480 ['block5d_project_conv[0][0]'] ormalization) block5d_drop (Dropout) (None, 3, 3, 120) 0 ['block5d_project_bn[0][0]'] block5d_add (Add) (None, 3, 3, 120) 0 ['block5d_drop[0][0]', 'block5c_add[0][0]'] block5e_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5d_add[0][0]'] D) block5e_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5e_expand_conv[0][0]'] rmalization) block5e_expand_activation (None, 3, 3, 720) 0 ['block5e_expand_bn[0][0]'] (Activation) block5e_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5e_expand_activation[0] Conv2D) [0]'] block5e_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5e_dwconv2[0][0]'] tion) block5e_activation (Activa (None, 3, 3, 720) 0 ['block5e_bn[0][0]'] tion) block5e_se_squeeze (Global (None, 720) 0 ['block5e_activation[0][0]'] AveragePooling2D) block5e_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5e_se_squeeze[0][0]'] e) block5e_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5e_se_reshape[0][0]'] block5e_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5e_se_reduce[0][0]'] block5e_se_excite (Multipl (None, 3, 3, 720) 0 ['block5e_activation[0][0]', y) 'block5e_se_expand[0][0]'] block5e_project_conv (Conv (None, 3, 3, 120) 86400 ['block5e_se_excite[0][0]'] 2D) block5e_project_bn (BatchN (None, 3, 3, 120) 480 ['block5e_project_conv[0][0]'] ormalization) block5e_drop (Dropout) (None, 3, 3, 120) 0 ['block5e_project_bn[0][0]'] block5e_add (Add) (None, 3, 3, 120) 0 ['block5e_drop[0][0]', 'block5d_add[0][0]'] block5f_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5e_add[0][0]'] D) block5f_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block5f_expand_conv[0][0]'] rmalization) block5f_expand_activation (None, 3, 3, 720) 0 ['block5f_expand_bn[0][0]'] (Activation) block5f_dwconv2 (Depthwise (None, 3, 3, 720) 6480 ['block5f_expand_activation[0] Conv2D) [0]'] block5f_bn (BatchNormaliza (None, 3, 3, 720) 2880 ['block5f_dwconv2[0][0]'] tion) block5f_activation (Activa (None, 3, 3, 720) 0 ['block5f_bn[0][0]'] tion) block5f_se_squeeze (Global (None, 720) 0 ['block5f_activation[0][0]'] AveragePooling2D) block5f_se_reshape (Reshap (None, 1, 1, 720) 0 ['block5f_se_squeeze[0][0]'] e) block5f_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block5f_se_reshape[0][0]'] block5f_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block5f_se_reduce[0][0]'] block5f_se_excite (Multipl (None, 3, 3, 720) 0 ['block5f_activation[0][0]', y) 'block5f_se_expand[0][0]'] block5f_project_conv (Conv (None, 3, 3, 120) 86400 ['block5f_se_excite[0][0]'] 2D) block5f_project_bn (BatchN (None, 3, 3, 120) 480 ['block5f_project_conv[0][0]'] ormalization) block5f_drop (Dropout) (None, 3, 3, 120) 0 ['block5f_project_bn[0][0]'] block5f_add (Add) (None, 3, 3, 120) 0 ['block5f_drop[0][0]', 'block5e_add[0][0]'] block6a_expand_conv (Conv2 (None, 3, 3, 720) 86400 ['block5f_add[0][0]'] D) block6a_expand_bn (BatchNo (None, 3, 3, 720) 2880 ['block6a_expand_conv[0][0]'] rmalization) block6a_expand_activation (None, 3, 3, 720) 0 ['block6a_expand_bn[0][0]'] (Activation) block6a_dwconv2 (Depthwise (None, 2, 2, 720) 6480 ['block6a_expand_activation[0] Conv2D) [0]'] block6a_bn (BatchNormaliza (None, 2, 2, 720) 2880 ['block6a_dwconv2[0][0]'] tion) block6a_activation (Activa (None, 2, 2, 720) 0 ['block6a_bn[0][0]'] tion) block6a_se_squeeze (Global (None, 720) 0 ['block6a_activation[0][0]'] AveragePooling2D) block6a_se_reshape (Reshap (None, 1, 1, 720) 0 ['block6a_se_squeeze[0][0]'] e) block6a_se_reduce (Conv2D) (None, 1, 1, 30) 21630 ['block6a_se_reshape[0][0]'] block6a_se_expand (Conv2D) (None, 1, 1, 720) 22320 ['block6a_se_reduce[0][0]'] block6a_se_excite (Multipl (None, 2, 2, 720) 0 ['block6a_activation[0][0]', y) 'block6a_se_expand[0][0]'] block6a_project_conv (Conv (None, 2, 2, 208) 149760 ['block6a_se_excite[0][0]'] 2D) block6a_project_bn (BatchN (None, 2, 2, 208) 832 ['block6a_project_conv[0][0]'] ormalization) block6b_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6a_project_bn[0][0]'] D) block6b_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6b_expand_conv[0][0]'] rmalization) block6b_expand_activation (None, 2, 2, 1248) 0 ['block6b_expand_bn[0][0]'] (Activation) block6b_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6b_expand_activation[0] Conv2D) [0]'] block6b_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6b_dwconv2[0][0]'] tion) block6b_activation (Activa (None, 2, 2, 1248) 0 ['block6b_bn[0][0]'] tion) block6b_se_squeeze (Global (None, 1248) 0 ['block6b_activation[0][0]'] AveragePooling2D) block6b_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6b_se_squeeze[0][0]'] e) block6b_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6b_se_reshape[0][0]'] block6b_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6b_se_reduce[0][0]'] block6b_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6b_activation[0][0]', y) 'block6b_se_expand[0][0]'] block6b_project_conv (Conv (None, 2, 2, 208) 259584 ['block6b_se_excite[0][0]'] 2D) block6b_project_bn (BatchN (None, 2, 2, 208) 832 ['block6b_project_conv[0][0]'] ormalization) block6b_drop (Dropout) (None, 2, 2, 208) 0 ['block6b_project_bn[0][0]'] block6b_add (Add) (None, 2, 2, 208) 0 ['block6b_drop[0][0]', 'block6a_project_bn[0][0]'] block6c_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6b_add[0][0]'] D) block6c_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6c_expand_conv[0][0]'] rmalization) block6c_expand_activation (None, 2, 2, 1248) 0 ['block6c_expand_bn[0][0]'] (Activation) block6c_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6c_expand_activation[0] Conv2D) [0]'] block6c_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6c_dwconv2[0][0]'] tion) block6c_activation (Activa (None, 2, 2, 1248) 0 ['block6c_bn[0][0]'] tion) block6c_se_squeeze (Global (None, 1248) 0 ['block6c_activation[0][0]'] AveragePooling2D) block6c_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6c_se_squeeze[0][0]'] e) block6c_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6c_se_reshape[0][0]'] block6c_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6c_se_reduce[0][0]'] block6c_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6c_activation[0][0]', y) 'block6c_se_expand[0][0]'] block6c_project_conv (Conv (None, 2, 2, 208) 259584 ['block6c_se_excite[0][0]'] 2D) block6c_project_bn (BatchN (None, 2, 2, 208) 832 ['block6c_project_conv[0][0]'] ormalization) block6c_drop (Dropout) (None, 2, 2, 208) 0 ['block6c_project_bn[0][0]'] block6c_add (Add) (None, 2, 2, 208) 0 ['block6c_drop[0][0]', 'block6b_add[0][0]'] block6d_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6c_add[0][0]'] D) block6d_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6d_expand_conv[0][0]'] rmalization) block6d_expand_activation (None, 2, 2, 1248) 0 ['block6d_expand_bn[0][0]'] (Activation) block6d_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6d_expand_activation[0] Conv2D) [0]'] block6d_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6d_dwconv2[0][0]'] tion) block6d_activation (Activa (None, 2, 2, 1248) 0 ['block6d_bn[0][0]'] tion) block6d_se_squeeze (Global (None, 1248) 0 ['block6d_activation[0][0]'] AveragePooling2D) block6d_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6d_se_squeeze[0][0]'] e) block6d_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6d_se_reshape[0][0]'] block6d_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6d_se_reduce[0][0]'] block6d_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6d_activation[0][0]', y) 'block6d_se_expand[0][0]'] block6d_project_conv (Conv (None, 2, 2, 208) 259584 ['block6d_se_excite[0][0]'] 2D) block6d_project_bn (BatchN (None, 2, 2, 208) 832 ['block6d_project_conv[0][0]'] ormalization) block6d_drop (Dropout) (None, 2, 2, 208) 0 ['block6d_project_bn[0][0]'] block6d_add (Add) (None, 2, 2, 208) 0 ['block6d_drop[0][0]', 'block6c_add[0][0]'] block6e_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6d_add[0][0]'] D) block6e_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6e_expand_conv[0][0]'] rmalization) block6e_expand_activation (None, 2, 2, 1248) 0 ['block6e_expand_bn[0][0]'] (Activation) block6e_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6e_expand_activation[0] Conv2D) [0]'] block6e_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6e_dwconv2[0][0]'] tion) block6e_activation (Activa (None, 2, 2, 1248) 0 ['block6e_bn[0][0]'] tion) block6e_se_squeeze (Global (None, 1248) 0 ['block6e_activation[0][0]'] AveragePooling2D) block6e_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6e_se_squeeze[0][0]'] e) block6e_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6e_se_reshape[0][0]'] block6e_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6e_se_reduce[0][0]'] block6e_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6e_activation[0][0]', y) 'block6e_se_expand[0][0]'] block6e_project_conv (Conv (None, 2, 2, 208) 259584 ['block6e_se_excite[0][0]'] 2D) block6e_project_bn (BatchN (None, 2, 2, 208) 832 ['block6e_project_conv[0][0]'] ormalization) block6e_drop (Dropout) (None, 2, 2, 208) 0 ['block6e_project_bn[0][0]'] block6e_add (Add) (None, 2, 2, 208) 0 ['block6e_drop[0][0]', 'block6d_add[0][0]'] block6f_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6e_add[0][0]'] D) block6f_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6f_expand_conv[0][0]'] rmalization) block6f_expand_activation (None, 2, 2, 1248) 0 ['block6f_expand_bn[0][0]'] (Activation) block6f_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6f_expand_activation[0] Conv2D) [0]'] block6f_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6f_dwconv2[0][0]'] tion) block6f_activation (Activa (None, 2, 2, 1248) 0 ['block6f_bn[0][0]'] tion) block6f_se_squeeze (Global (None, 1248) 0 ['block6f_activation[0][0]'] AveragePooling2D) block6f_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6f_se_squeeze[0][0]'] e) block6f_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6f_se_reshape[0][0]'] block6f_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6f_se_reduce[0][0]'] block6f_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6f_activation[0][0]', y) 'block6f_se_expand[0][0]'] block6f_project_conv (Conv (None, 2, 2, 208) 259584 ['block6f_se_excite[0][0]'] 2D) block6f_project_bn (BatchN (None, 2, 2, 208) 832 ['block6f_project_conv[0][0]'] ormalization) block6f_drop (Dropout) (None, 2, 2, 208) 0 ['block6f_project_bn[0][0]'] block6f_add (Add) (None, 2, 2, 208) 0 ['block6f_drop[0][0]', 'block6e_add[0][0]'] block6g_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6f_add[0][0]'] D) block6g_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6g_expand_conv[0][0]'] rmalization) block6g_expand_activation (None, 2, 2, 1248) 0 ['block6g_expand_bn[0][0]'] (Activation) block6g_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6g_expand_activation[0] Conv2D) [0]'] block6g_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6g_dwconv2[0][0]'] tion) block6g_activation (Activa (None, 2, 2, 1248) 0 ['block6g_bn[0][0]'] tion) block6g_se_squeeze (Global (None, 1248) 0 ['block6g_activation[0][0]'] AveragePooling2D) block6g_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6g_se_squeeze[0][0]'] e) block6g_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6g_se_reshape[0][0]'] block6g_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6g_se_reduce[0][0]'] block6g_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6g_activation[0][0]', y) 'block6g_se_expand[0][0]'] block6g_project_conv (Conv (None, 2, 2, 208) 259584 ['block6g_se_excite[0][0]'] 2D) block6g_project_bn (BatchN (None, 2, 2, 208) 832 ['block6g_project_conv[0][0]'] ormalization) block6g_drop (Dropout) (None, 2, 2, 208) 0 ['block6g_project_bn[0][0]'] block6g_add (Add) (None, 2, 2, 208) 0 ['block6g_drop[0][0]', 'block6f_add[0][0]'] block6h_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6g_add[0][0]'] D) block6h_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6h_expand_conv[0][0]'] rmalization) block6h_expand_activation (None, 2, 2, 1248) 0 ['block6h_expand_bn[0][0]'] (Activation) block6h_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6h_expand_activation[0] Conv2D) [0]'] block6h_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6h_dwconv2[0][0]'] tion) block6h_activation (Activa (None, 2, 2, 1248) 0 ['block6h_bn[0][0]'] tion) block6h_se_squeeze (Global (None, 1248) 0 ['block6h_activation[0][0]'] AveragePooling2D) block6h_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6h_se_squeeze[0][0]'] e) block6h_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6h_se_reshape[0][0]'] block6h_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6h_se_reduce[0][0]'] block6h_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6h_activation[0][0]', y) 'block6h_se_expand[0][0]'] block6h_project_conv (Conv (None, 2, 2, 208) 259584 ['block6h_se_excite[0][0]'] 2D) block6h_project_bn (BatchN (None, 2, 2, 208) 832 ['block6h_project_conv[0][0]'] ormalization) block6h_drop (Dropout) (None, 2, 2, 208) 0 ['block6h_project_bn[0][0]'] block6h_add (Add) (None, 2, 2, 208) 0 ['block6h_drop[0][0]', 'block6g_add[0][0]'] block6i_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6h_add[0][0]'] D) block6i_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6i_expand_conv[0][0]'] rmalization) block6i_expand_activation (None, 2, 2, 1248) 0 ['block6i_expand_bn[0][0]'] (Activation) block6i_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6i_expand_activation[0] Conv2D) [0]'] block6i_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6i_dwconv2[0][0]'] tion) block6i_activation (Activa (None, 2, 2, 1248) 0 ['block6i_bn[0][0]'] tion) block6i_se_squeeze (Global (None, 1248) 0 ['block6i_activation[0][0]'] AveragePooling2D) block6i_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6i_se_squeeze[0][0]'] e) block6i_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6i_se_reshape[0][0]'] block6i_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6i_se_reduce[0][0]'] block6i_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6i_activation[0][0]', y) 'block6i_se_expand[0][0]'] block6i_project_conv (Conv (None, 2, 2, 208) 259584 ['block6i_se_excite[0][0]'] 2D) block6i_project_bn (BatchN (None, 2, 2, 208) 832 ['block6i_project_conv[0][0]'] ormalization) block6i_drop (Dropout) (None, 2, 2, 208) 0 ['block6i_project_bn[0][0]'] block6i_add (Add) (None, 2, 2, 208) 0 ['block6i_drop[0][0]', 'block6h_add[0][0]'] block6j_expand_conv (Conv2 (None, 2, 2, 1248) 259584 ['block6i_add[0][0]'] D) block6j_expand_bn (BatchNo (None, 2, 2, 1248) 4992 ['block6j_expand_conv[0][0]'] rmalization) block6j_expand_activation (None, 2, 2, 1248) 0 ['block6j_expand_bn[0][0]'] (Activation) block6j_dwconv2 (Depthwise (None, 2, 2, 1248) 11232 ['block6j_expand_activation[0] Conv2D) [0]'] block6j_bn (BatchNormaliza (None, 2, 2, 1248) 4992 ['block6j_dwconv2[0][0]'] tion) block6j_activation (Activa (None, 2, 2, 1248) 0 ['block6j_bn[0][0]'] tion) block6j_se_squeeze (Global (None, 1248) 0 ['block6j_activation[0][0]'] AveragePooling2D) block6j_se_reshape (Reshap (None, 1, 1, 1248) 0 ['block6j_se_squeeze[0][0]'] e) block6j_se_reduce (Conv2D) (None, 1, 1, 52) 64948 ['block6j_se_reshape[0][0]'] block6j_se_expand (Conv2D) (None, 1, 1, 1248) 66144 ['block6j_se_reduce[0][0]'] block6j_se_excite (Multipl (None, 2, 2, 1248) 0 ['block6j_activation[0][0]', y) 'block6j_se_expand[0][0]'] block6j_project_conv (Conv (None, 2, 2, 208) 259584 ['block6j_se_excite[0][0]'] 2D) block6j_project_bn (BatchN (None, 2, 2, 208) 832 ['block6j_project_conv[0][0]'] ormalization) block6j_drop (Dropout) (None, 2, 2, 208) 0 ['block6j_project_bn[0][0]'] block6j_add (Add) (None, 2, 2, 208) 0 ['block6j_drop[0][0]', 'block6i_add[0][0]'] top_conv (Conv2D) (None, 2, 2, 1408) 292864 ['block6j_add[0][0]'] top_bn (BatchNormalization (None, 2, 2, 1408) 5632 ['top_conv[0][0]'] ) top_activation (Activation (None, 2, 2, 1408) 0 ['top_bn[0][0]'] ) ================================================================================================== Total params: 8769374 (33.45 MB) Trainable params: 8687086 (33.14 MB) Non-trainable params: 82288 (321.44 KB) __________________________________________________________________________________________________
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
# Load EfficientNetB0 model pre-trained on ImageNet without the top layer
base_model = EfficientNetB0(weights='imagenet', include_top=False, input_shape=(48, 48, 3))
# Add custom layers on top of the base model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.2)(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.2)(x) # Adjusted intermediate layer
predictions = Dense(4, activation='softmax')(x) # Adjust the number of units to match the number of classes
# Define the new model
efficient_model2 = Model(inputs=base_model.input, outputs=predictions)
efficient_model2.compile(optimizer=AdamW(learning_rate=0.000025), loss='categorical_crossentropy', metrics=['accuracy'])
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)
history = efficient_model2.fit(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=50,
validation_data=validation_generator,
validation_steps=validation_generator.samples // validation_generator.batch_size,
class_weight=class_weights_adjusted,
callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/50 471/472 [============================>.] - ETA: 0s - loss: 1.3364 - accuracy: 0.3444
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`. saving_api.save_model(
472/472 [==============================] - 61s 56ms/step - loss: 1.3362 - accuracy: 0.3445 - val_loss: 1.4205 - val_accuracy: 0.2494 - lr: 2.5000e-05 Epoch 2/50 472/472 [==============================] - 24s 51ms/step - loss: 1.1789 - accuracy: 0.4744 - val_loss: 1.5377 - val_accuracy: 0.2607 - lr: 2.5000e-05 Epoch 3/50 472/472 [==============================] - 24s 50ms/step - loss: 1.0906 - accuracy: 0.5233 - val_loss: 1.3381 - val_accuracy: 0.3417 - lr: 2.5000e-05 Epoch 4/50 472/472 [==============================] - 24s 51ms/step - loss: 1.0268 - accuracy: 0.5559 - val_loss: 1.3126 - val_accuracy: 0.3831 - lr: 2.5000e-05 Epoch 5/50 472/472 [==============================] - 23s 50ms/step - loss: 0.9674 - accuracy: 0.5873 - val_loss: 1.4013 - val_accuracy: 0.3456 - lr: 2.5000e-05 Epoch 6/50 472/472 [==============================] - 24s 52ms/step - loss: 0.9249 - accuracy: 0.6089 - val_loss: 1.3047 - val_accuracy: 0.4036 - lr: 2.5000e-05 Epoch 7/50 472/472 [==============================] - 24s 51ms/step - loss: 0.8887 - accuracy: 0.6251 - val_loss: 1.2046 - val_accuracy: 0.4643 - lr: 2.5000e-05 Epoch 8/50 472/472 [==============================] - 23s 49ms/step - loss: 0.8461 - accuracy: 0.6501 - val_loss: 1.2797 - val_accuracy: 0.4194 - lr: 2.5000e-05 Epoch 9/50 472/472 [==============================] - 23s 50ms/step - loss: 0.8105 - accuracy: 0.6625 - val_loss: 1.6911 - val_accuracy: 0.2988 - lr: 2.5000e-05 Epoch 10/50 472/472 [==============================] - 23s 49ms/step - loss: 0.7762 - accuracy: 0.6786 - val_loss: 1.5159 - val_accuracy: 0.3510 - lr: 2.5000e-05 Epoch 11/50 472/472 [==============================] - 23s 49ms/step - loss: 0.7396 - accuracy: 0.6985 - val_loss: 1.3115 - val_accuracy: 0.4635 - lr: 2.5000e-05 Epoch 12/50 472/472 [==============================] - 23s 49ms/step - loss: 0.7042 - accuracy: 0.7141 - val_loss: 1.4750 - val_accuracy: 0.4284 - lr: 2.5000e-05 Epoch 13/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6657 - accuracy: 0.7307 - val_loss: 1.2258 - val_accuracy: 0.4984 - lr: 5.0000e-06 Epoch 14/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6549 - accuracy: 0.7325 - val_loss: 1.1271 - val_accuracy: 0.5583 - lr: 5.0000e-06 Epoch 15/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6461 - accuracy: 0.7392 - val_loss: 1.0258 - val_accuracy: 0.5927 - lr: 5.0000e-06 Epoch 16/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6434 - accuracy: 0.7391 - val_loss: 0.9484 - val_accuracy: 0.6177 - lr: 5.0000e-06 Epoch 17/50 472/472 [==============================] - 23s 50ms/step - loss: 0.6222 - accuracy: 0.7500 - val_loss: 1.1752 - val_accuracy: 0.5224 - lr: 5.0000e-06 Epoch 18/50 472/472 [==============================] - 23s 49ms/step - loss: 0.6238 - accuracy: 0.7521 - val_loss: 1.0146 - val_accuracy: 0.6067 - lr: 5.0000e-06 Epoch 19/50 472/472 [==============================] - 24s 50ms/step - loss: 0.6201 - accuracy: 0.7529 - val_loss: 1.0254 - val_accuracy: 0.6173 - lr: 5.0000e-06 Epoch 20/50 472/472 [==============================] - 24s 51ms/step - loss: 0.6098 - accuracy: 0.7553 - val_loss: 0.9382 - val_accuracy: 0.6367 - lr: 5.0000e-06 Epoch 21/50 472/472 [==============================] - 23s 49ms/step - loss: 0.6099 - accuracy: 0.7577 - val_loss: 1.0727 - val_accuracy: 0.5831 - lr: 5.0000e-06 Epoch 22/50 472/472 [==============================] - 24s 50ms/step - loss: 0.5948 - accuracy: 0.7632 - val_loss: 0.9776 - val_accuracy: 0.6135 - lr: 5.0000e-06 Epoch 23/50 472/472 [==============================] - 23s 49ms/step - loss: 0.5924 - accuracy: 0.7661 - val_loss: 1.0028 - val_accuracy: 0.6190 - lr: 5.0000e-06 Epoch 24/50 472/472 [==============================] - 23s 49ms/step - loss: 0.5832 - accuracy: 0.7723 - val_loss: 1.0490 - val_accuracy: 0.6113 - lr: 5.0000e-06 Epoch 25/50 472/472 [==============================] - 24s 51ms/step - loss: 0.5757 - accuracy: 0.7692 - val_loss: 0.9597 - val_accuracy: 0.6375 - lr: 5.0000e-06 Epoch 26/50 472/472 [==============================] - 24s 50ms/step - loss: 0.5759 - accuracy: 0.7736 - val_loss: 0.9805 - val_accuracy: 0.6216 - lr: 1.0000e-06 Epoch 27/50 472/472 [==============================] - 23s 50ms/step - loss: 0.5630 - accuracy: 0.7769 - val_loss: 0.9584 - val_accuracy: 0.6250 - lr: 1.0000e-06
# Evaluate the model on the test set
val_loss, val_accuracy = efficient_model2.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 17ms/step - loss: 0.8464 - accuracy: 0.6484 Test Loss: 0.84635990858078 Test Accuracy: 0.6484375
plot_model_performance_and_confusion_matrix(efficient_model2, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display
# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = efficient_model2.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> efficient_model2 </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.2f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.2%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 15ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.67 0.81 0.73 32 1 0.57 0.72 0.64 32 2 0.54 0.44 0.48 32 3 0.87 0.62 0.73 32 accuracy 0.65 128 macro avg 0.66 0.65 0.65 128 weighted avg 0.66 0.65 0.65 128
4/4 [==============================] - 0s 21ms/step - loss: 0.8464 - accuracy: 0.6484 Test Loss: 0.84635990858078 Test Accuracy: 0.6484375
Test Loss: efficient_model2
Test Loss: 0.85
Test Accuracy: 64.84%
Observations and Insights:__
Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance.
Think About It:
The EfficientNetB0-based model tailored for a 4-class emotion classification task exhibits an overall accuracy of 62%. Performance metrics indicate varied effectiveness across classes:
Happy class shows moderate precision and slightly higher recall, achieving a balance reflected in an F1-score of 0.70. Neutral and Sad classes have lower precision and recall, resulting in F1-scores of 0.56 and 0.56, respectively. This suggests challenges in distinguishing these emotions accurately. Surprise class demonstrates higher precision but reduced recall, with an F1-score of 0.67, indicating better identification but room for improved coverage. These results suggest that while the model can identify certain emotions with reasonable accuracy, its performance is inconsistent across all classes. The lower overall accuracy and mixed precision and recall scores highlight areas for improvement, particularly in enhancing the model's ability to recognize 'neutral' and 'sad' emotions more accurately. Optimization strategies might include adjusting the model's architecture, fine-tuning the learning rate, and employing targeted data augmentation to address these disparities.
:Observations and Insights:__
To evaluate the overall performance of the transfer learning architectures mentioned (VGG16, ResNet50, and EfficientNetB0) for the emotion classification task, let's compare their reported metrics:
VGG16-based Model:
Accuracy: 77% Best performing class: 'Surprise' with high precision and recall. Challenges: Lower precision in 'neutral' and lower recall in 'sad'. ResNet50-based Model:
Accuracy: 75% Best performing class: 'Happy' and 'Surprise' with high precision and recall. Challenges: Lower precision for 'neutral' and 'sad', indicating difficulty in distinguishing these emotions accurately. EfficientNetB0-based Model:
Accuracy: 62% Shows moderate to good performance in recognizing 'happy' and 'surprise' but struggles with 'neutral' and 'sad'. The lower overall accuracy compared to VGG16 and ResNet50.
The VGG16 and ResNet50 models have shown similar performance levels, with both achieving mid to high 70s in accuracy. They are relatively close in their ability to classify emotions, with strengths in identifying 'happy' and 'surprise' emotions accurately. The EfficientNetB0 model lagged behind in performance, with an accuracy of 62%, indicating that it might not have been as effective for this particular task or dataset. While accuracies are decent for VGG16 and ResNet50,there is still alot of room for improvement.
The task of computer vision is i
CONVOLUTIONAL NETWORK #2
plot_model_performance_and_confusion_matrix(cnn_model_updated, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 1s 11ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.82 0.84 0.83 32 1 0.72 0.81 0.76 32 2 0.79 0.69 0.73 32 3 0.97 0.94 0.95 32 accuracy 0.82 128 macro avg 0.82 0.82 0.82 128 weighted avg 0.82 0.82 0.82 128
Now that we have tried multiple pre-trained models, let's build a complex CNN architecture and see if we can get better performance.
In this section, we will build a more complex Convolutional Neural Network Model that has close to as many parameters as we had in our Transfer Learning Models. However, we will have only 1 input channel for our input images.
In this section, we are creating data loaders which we will use as inputs to the more Complicated Convolutional Neural Network. We will go ahead with color_mode = 'grayscale'.
# Set the random Seed
np.random.seed(42)
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
from tensorflow.keras.preprocessing.image import ImageDataGenerator
# Create more augmented images for 'sad' and 'neutral' classes
augmentation_for_sad_neutral = ImageDataGenerator(
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.15,
zoom_range=0.15,
horizontal_flip=True,
fill_mode='nearest'
)
cnn_model_4 = Sequential([
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Slightly reduced dropout
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Adjusted dropout
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Adjusted dropout
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Convolutional Block 5
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
GlobalAveragePooling2D(),
Flatten(),
# Dense Layers
Dense(128, activation='relu'),
Dense(512, activation='relu'),
Dropout(0.2),
Dense(1024, activation='relu'),
Dense(2048, activation='relu'),
Dropout(0.2),
Dense(128, activation='relu'),
Dropout(0.2),
Dense(4, activation='softmax')
])
cnn_model_4.summary() # Print the model summary
Model: "sequential_47" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_218 (Conv2D) (None, 48, 48, 64) 1792 batch_normalization_154 (B (None, 48, 48, 64) 256 atchNormalization) conv2d_219 (Conv2D) (None, 48, 48, 64) 36928 max_pooling2d_128 (MaxPool (None, 24, 24, 64) 0 ing2D) dropout_186 (Dropout) (None, 24, 24, 64) 0 conv2d_220 (Conv2D) (None, 24, 24, 128) 73856 batch_normalization_155 (B (None, 24, 24, 128) 512 atchNormalization) conv2d_221 (Conv2D) (None, 24, 24, 256) 295168 max_pooling2d_129 (MaxPool (None, 12, 12, 256) 0 ing2D) dropout_187 (Dropout) (None, 12, 12, 256) 0 conv2d_222 (Conv2D) (None, 12, 12, 256) 590080 batch_normalization_156 (B (None, 12, 12, 256) 1024 atchNormalization) conv2d_223 (Conv2D) (None, 12, 12, 512) 1180160 batch_normalization_157 (B (None, 12, 12, 512) 2048 atchNormalization) max_pooling2d_130 (MaxPool (None, 6, 6, 512) 0 ing2D) dropout_188 (Dropout) (None, 6, 6, 512) 0 conv2d_224 (Conv2D) (None, 6, 6, 1024) 4719616 batch_normalization_158 (B (None, 6, 6, 1024) 4096 atchNormalization) max_pooling2d_131 (MaxPool (None, 3, 3, 1024) 0 ing2D) dropout_189 (Dropout) (None, 3, 3, 1024) 0 conv2d_225 (Conv2D) (None, 3, 3, 1024) 9438208 batch_normalization_159 (B (None, 3, 3, 1024) 4096 atchNormalization) max_pooling2d_132 (MaxPool (None, 1, 1, 1024) 0 ing2D) dropout_190 (Dropout) (None, 1, 1, 1024) 0 global_average_pooling2d_3 (None, 1024) 0 1 (GlobalAveragePooling2D) flatten_34 (Flatten) (None, 1024) 0 dense_158 (Dense) (None, 128) 131200 dense_159 (Dense) (None, 512) 66048 dropout_191 (Dropout) (None, 512) 0 dense_160 (Dense) (None, 1024) 525312 dense_161 (Dense) (None, 2048) 2099200 dropout_192 (Dropout) (None, 2048) 0 dense_162 (Dense) (None, 128) 262272 dropout_193 (Dropout) (None, 128) 0 dense_163 (Dense) (None, 4) 516 ================================================================= Total params: 19432388 (74.13 MB) Trainable params: 19426372 (74.11 MB) Non-trainable params: 6016 (23.50 KB) _________________________________________________________________
cnn_model_4.compile(optimizer=AdamW(learning_rate=0.00003),
loss='categorical_crossentropy',
metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)
history = cnn_model_4.fit(
train_generator,
epochs=30,
verbose=1,
callbacks=[early_stopping, reduce_lr],
validation_data=validation_generator,
class_weight=class_weights_adjusted
)
Epoch 1/30 473/473 [==============================] - 33s 57ms/step - loss: 1.3675 - accuracy: 0.2974 - val_loss: 1.4441 - val_accuracy: 0.2465 - lr: 3.0000e-05 Epoch 2/30 473/473 [==============================] - 27s 57ms/step - loss: 1.3172 - accuracy: 0.3478 - val_loss: 1.4579 - val_accuracy: 0.2769 - lr: 3.0000e-05 Epoch 3/30 473/473 [==============================] - 28s 58ms/step - loss: 1.2367 - accuracy: 0.4090 - val_loss: 1.1947 - val_accuracy: 0.4621 - lr: 3.0000e-05 Epoch 4/30 473/473 [==============================] - 28s 59ms/step - loss: 1.1550 - accuracy: 0.4691 - val_loss: 1.0658 - val_accuracy: 0.5182 - lr: 3.0000e-05 Epoch 5/30 473/473 [==============================] - 28s 59ms/step - loss: 1.0598 - accuracy: 0.5358 - val_loss: 1.0690 - val_accuracy: 0.5339 - lr: 3.0000e-05 Epoch 6/30 473/473 [==============================] - 28s 59ms/step - loss: 0.9991 - accuracy: 0.5732 - val_loss: 0.9468 - val_accuracy: 0.6020 - lr: 3.0000e-05 Epoch 7/30 473/473 [==============================] - 28s 59ms/step - loss: 0.9407 - accuracy: 0.5997 - val_loss: 0.9827 - val_accuracy: 0.5990 - lr: 3.0000e-05 Epoch 8/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8964 - accuracy: 0.6178 - val_loss: 0.9243 - val_accuracy: 0.6401 - lr: 3.0000e-05 Epoch 9/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8639 - accuracy: 0.6409 - val_loss: 0.8495 - val_accuracy: 0.6703 - lr: 3.0000e-05 Epoch 10/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8245 - accuracy: 0.6535 - val_loss: 0.8480 - val_accuracy: 0.6773 - lr: 3.0000e-05 Epoch 11/30 473/473 [==============================] - 28s 59ms/step - loss: 0.8008 - accuracy: 0.6653 - val_loss: 0.8084 - val_accuracy: 0.6900 - lr: 3.0000e-05 Epoch 12/30 473/473 [==============================] - 28s 59ms/step - loss: 0.7659 - accuracy: 0.6814 - val_loss: 0.7696 - val_accuracy: 0.7085 - lr: 3.0000e-05 Epoch 13/30 473/473 [==============================] - 28s 59ms/step - loss: 0.7424 - accuracy: 0.6950 - val_loss: 0.7819 - val_accuracy: 0.7012 - lr: 3.0000e-05 Epoch 14/30 473/473 [==============================] - 28s 59ms/step - loss: 0.7101 - accuracy: 0.7108 - val_loss: 0.7668 - val_accuracy: 0.6992 - lr: 3.0000e-05 Epoch 15/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6990 - accuracy: 0.7163 - val_loss: 0.7166 - val_accuracy: 0.7219 - lr: 3.0000e-05 Epoch 16/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6658 - accuracy: 0.7249 - val_loss: 0.6994 - val_accuracy: 0.7267 - lr: 3.0000e-05 Epoch 17/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6449 - accuracy: 0.7385 - val_loss: 0.7149 - val_accuracy: 0.7312 - lr: 3.0000e-05 Epoch 18/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6245 - accuracy: 0.7470 - val_loss: 0.7161 - val_accuracy: 0.7346 - lr: 3.0000e-05 Epoch 19/30 473/473 [==============================] - 28s 59ms/step - loss: 0.6038 - accuracy: 0.7531 - val_loss: 0.6973 - val_accuracy: 0.7378 - lr: 3.0000e-05 Epoch 20/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5787 - accuracy: 0.7652 - val_loss: 0.7712 - val_accuracy: 0.7233 - lr: 3.0000e-05 Epoch 21/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5627 - accuracy: 0.7734 - val_loss: 0.6959 - val_accuracy: 0.7380 - lr: 3.0000e-05 Epoch 22/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5365 - accuracy: 0.7804 - val_loss: 0.7029 - val_accuracy: 0.7408 - lr: 3.0000e-05 Epoch 23/30 473/473 [==============================] - 28s 59ms/step - loss: 0.5206 - accuracy: 0.7927 - val_loss: 0.7007 - val_accuracy: 0.7454 - lr: 3.0000e-05 Epoch 24/30 473/473 [==============================] - 28s 59ms/step - loss: 0.4909 - accuracy: 0.8044 - val_loss: 0.7170 - val_accuracy: 0.7478 - lr: 3.0000e-05 Epoch 25/30 473/473 [==============================] - 28s 59ms/step - loss: 0.4651 - accuracy: 0.8154 - val_loss: 0.7634 - val_accuracy: 0.7370 - lr: 3.0000e-05 Epoch 26/30 473/473 [==============================] - ETA: 0s - loss: 0.4524 - accuracy: 0.8224 Epoch 26: ReduceLROnPlateau reducing learning rate to 1e-05. 473/473 [==============================] - 28s 59ms/step - loss: 0.4524 - accuracy: 0.8224 - val_loss: 0.7263 - val_accuracy: 0.7458 - lr: 3.0000e-05 Epoch 27/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3850 - accuracy: 0.8508 - val_loss: 0.7633 - val_accuracy: 0.7456 - lr: 1.0000e-05 Epoch 28/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3626 - accuracy: 0.8564 - val_loss: 0.7651 - val_accuracy: 0.7468 - lr: 1.0000e-05 Epoch 29/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3502 - accuracy: 0.8602 - val_loss: 0.7947 - val_accuracy: 0.7454 - lr: 1.0000e-05 Epoch 30/30 473/473 [==============================] - 28s 59ms/step - loss: 0.3374 - accuracy: 0.8695 - val_loss: 0.8206 - val_accuracy: 0.7430 - lr: 1.0000e-05
history = cnn_model_4.fit(
train_generator,
epochs=75,
verbose=1,
callbacks=[reduce_lr],
validation_data=validation_generator,
class_weight=class_weights_adjusted
)
Epoch 1/75 473/473 [==============================] - 32s 57ms/step - loss: 1.3529 - accuracy: 0.3203 - val_loss: 1.3962 - val_accuracy: 0.3649 - lr: 3.0000e-05 Epoch 2/75 473/473 [==============================] - 27s 57ms/step - loss: 1.3064 - accuracy: 0.3630 - val_loss: 1.3467 - val_accuracy: 0.3757 - lr: 3.0000e-05 Epoch 3/75 473/473 [==============================] - 27s 58ms/step - loss: 1.2189 - accuracy: 0.4201 - val_loss: 1.1732 - val_accuracy: 0.4531 - lr: 3.0000e-05 Epoch 4/75 473/473 [==============================] - 28s 58ms/step - loss: 1.1095 - accuracy: 0.5029 - val_loss: 1.1512 - val_accuracy: 0.5142 - lr: 3.0000e-05 Epoch 5/75 473/473 [==============================] - 28s 59ms/step - loss: 1.0259 - accuracy: 0.5528 - val_loss: 1.0352 - val_accuracy: 0.5738 - lr: 3.0000e-05 Epoch 6/75 473/473 [==============================] - 28s 58ms/step - loss: 0.9608 - accuracy: 0.5862 - val_loss: 1.0371 - val_accuracy: 0.5799 - lr: 3.0000e-05 Epoch 7/75 473/473 [==============================] - 28s 58ms/step - loss: 0.9163 - accuracy: 0.6108 - val_loss: 0.9709 - val_accuracy: 0.6180 - lr: 3.0000e-05 Epoch 8/75 473/473 [==============================] - 28s 58ms/step - loss: 0.8684 - accuracy: 0.6329 - val_loss: 0.9718 - val_accuracy: 0.6339 - lr: 3.0000e-05 Epoch 9/75 473/473 [==============================] - 28s 58ms/step - loss: 0.8387 - accuracy: 0.6496 - val_loss: 0.9744 - val_accuracy: 0.6349 - lr: 3.0000e-05 Epoch 10/75 473/473 [==============================] - 28s 58ms/step - loss: 0.8003 - accuracy: 0.6652 - val_loss: 0.8822 - val_accuracy: 0.6787 - lr: 3.0000e-05 Epoch 11/75 473/473 [==============================] - 28s 59ms/step - loss: 0.7792 - accuracy: 0.6750 - val_loss: 0.8308 - val_accuracy: 0.6751 - lr: 3.0000e-05 Epoch 12/75 473/473 [==============================] - 28s 58ms/step - loss: 0.7439 - accuracy: 0.6922 - val_loss: 0.7526 - val_accuracy: 0.7028 - lr: 3.0000e-05 Epoch 13/75 473/473 [==============================] - 28s 58ms/step - loss: 0.7282 - accuracy: 0.6992 - val_loss: 0.7748 - val_accuracy: 0.6801 - lr: 3.0000e-05 Epoch 14/75 473/473 [==============================] - 28s 59ms/step - loss: 0.7001 - accuracy: 0.7132 - val_loss: 0.7531 - val_accuracy: 0.7030 - lr: 3.0000e-05 Epoch 15/75 473/473 [==============================] - 28s 58ms/step - loss: 0.6800 - accuracy: 0.7193 - val_loss: 0.7626 - val_accuracy: 0.7213 - lr: 3.0000e-05 Epoch 16/75 473/473 [==============================] - 28s 59ms/step - loss: 0.6534 - accuracy: 0.7336 - val_loss: 0.7654 - val_accuracy: 0.7189 - lr: 3.0000e-05 Epoch 17/75 473/473 [==============================] - 28s 58ms/step - loss: 0.6306 - accuracy: 0.7451 - val_loss: 0.7242 - val_accuracy: 0.7286 - lr: 3.0000e-05 Epoch 18/75 473/473 [==============================] - 28s 58ms/step - loss: 0.6145 - accuracy: 0.7500 - val_loss: 0.7546 - val_accuracy: 0.7091 - lr: 3.0000e-05 Epoch 19/75 473/473 [==============================] - 28s 58ms/step - loss: 0.5924 - accuracy: 0.7586 - val_loss: 0.6986 - val_accuracy: 0.7360 - lr: 3.0000e-05 Epoch 20/75 473/473 [==============================] - 28s 58ms/step - loss: 0.5708 - accuracy: 0.7662 - val_loss: 0.7054 - val_accuracy: 0.7362 - lr: 3.0000e-05 Epoch 21/75 473/473 [==============================] - 28s 58ms/step - loss: 0.5444 - accuracy: 0.7811 - val_loss: 0.7600 - val_accuracy: 0.7330 - lr: 3.0000e-05 Epoch 22/75 473/473 [==============================] - 28s 59ms/step - loss: 0.5204 - accuracy: 0.7886 - val_loss: 0.7341 - val_accuracy: 0.7334 - lr: 3.0000e-05 Epoch 23/75 473/473 [==============================] - 28s 58ms/step - loss: 0.4989 - accuracy: 0.7985 - val_loss: 0.7955 - val_accuracy: 0.7273 - lr: 3.0000e-05 Epoch 24/75 473/473 [==============================] - ETA: 0s - loss: 0.4749 - accuracy: 0.8068 Epoch 24: ReduceLROnPlateau reducing learning rate to 1e-05. 473/473 [==============================] - 28s 58ms/step - loss: 0.4749 - accuracy: 0.8068 - val_loss: 0.7177 - val_accuracy: 0.7454 - lr: 3.0000e-05 Epoch 25/75 473/473 [==============================] - 28s 58ms/step - loss: 0.4228 - accuracy: 0.8315 - val_loss: 0.7474 - val_accuracy: 0.7470 - lr: 1.0000e-05 Epoch 26/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3946 - accuracy: 0.8453 - val_loss: 0.7789 - val_accuracy: 0.7432 - lr: 1.0000e-05 Epoch 27/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3869 - accuracy: 0.8459 - val_loss: 0.7746 - val_accuracy: 0.7444 - lr: 1.0000e-05 Epoch 28/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3722 - accuracy: 0.8547 - val_loss: 0.7694 - val_accuracy: 0.7559 - lr: 1.0000e-05 Epoch 29/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3528 - accuracy: 0.8617 - val_loss: 0.8082 - val_accuracy: 0.7454 - lr: 1.0000e-05 Epoch 30/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3439 - accuracy: 0.8594 - val_loss: 0.7955 - val_accuracy: 0.7480 - lr: 1.0000e-05 Epoch 31/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3311 - accuracy: 0.8717 - val_loss: 0.8314 - val_accuracy: 0.7464 - lr: 1.0000e-05 Epoch 32/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3227 - accuracy: 0.8716 - val_loss: 0.8423 - val_accuracy: 0.7406 - lr: 1.0000e-05 Epoch 33/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3145 - accuracy: 0.8768 - val_loss: 0.8531 - val_accuracy: 0.7446 - lr: 1.0000e-05 Epoch 34/75 473/473 [==============================] - 28s 58ms/step - loss: 0.3001 - accuracy: 0.8834 - val_loss: 0.8843 - val_accuracy: 0.7482 - lr: 1.0000e-05 Epoch 35/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2829 - accuracy: 0.8889 - val_loss: 0.9057 - val_accuracy: 0.7402 - lr: 1.0000e-05 Epoch 36/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2840 - accuracy: 0.8887 - val_loss: 0.8644 - val_accuracy: 0.7440 - lr: 1.0000e-05 Epoch 37/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2776 - accuracy: 0.8927 - val_loss: 0.8776 - val_accuracy: 0.7388 - lr: 1.0000e-05 Epoch 38/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2650 - accuracy: 0.8952 - val_loss: 0.9019 - val_accuracy: 0.7422 - lr: 1.0000e-05 Epoch 39/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2612 - accuracy: 0.8976 - val_loss: 0.9259 - val_accuracy: 0.7448 - lr: 1.0000e-05 Epoch 40/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2482 - accuracy: 0.9060 - val_loss: 0.9300 - val_accuracy: 0.7476 - lr: 1.0000e-05 Epoch 41/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2518 - accuracy: 0.9027 - val_loss: 0.8982 - val_accuracy: 0.7476 - lr: 1.0000e-05 Epoch 42/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2349 - accuracy: 0.9086 - val_loss: 0.9488 - val_accuracy: 0.7470 - lr: 1.0000e-05 Epoch 43/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2288 - accuracy: 0.9130 - val_loss: 0.9425 - val_accuracy: 0.7519 - lr: 1.0000e-05 Epoch 44/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2199 - accuracy: 0.9153 - val_loss: 0.9910 - val_accuracy: 0.7416 - lr: 1.0000e-05 Epoch 45/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2143 - accuracy: 0.9180 - val_loss: 0.9927 - val_accuracy: 0.7480 - lr: 1.0000e-05 Epoch 46/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2127 - accuracy: 0.9192 - val_loss: 0.9895 - val_accuracy: 0.7498 - lr: 1.0000e-05 Epoch 47/75 473/473 [==============================] - 28s 58ms/step - loss: 0.2106 - accuracy: 0.9196 - val_loss: 1.0095 - val_accuracy: 0.7438 - lr: 1.0000e-05 Epoch 48/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1981 - accuracy: 0.9259 - val_loss: 1.0120 - val_accuracy: 0.7505 - lr: 1.0000e-05 Epoch 49/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1980 - accuracy: 0.9231 - val_loss: 0.9901 - val_accuracy: 0.7490 - lr: 1.0000e-05 Epoch 50/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1898 - accuracy: 0.9265 - val_loss: 1.0584 - val_accuracy: 0.7418 - lr: 1.0000e-05 Epoch 51/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1845 - accuracy: 0.9279 - val_loss: 1.0573 - val_accuracy: 0.7533 - lr: 1.0000e-05 Epoch 52/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1808 - accuracy: 0.9321 - val_loss: 1.0527 - val_accuracy: 0.7472 - lr: 1.0000e-05 Epoch 53/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1741 - accuracy: 0.9338 - val_loss: 1.0805 - val_accuracy: 0.7430 - lr: 1.0000e-05 Epoch 54/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1675 - accuracy: 0.9385 - val_loss: 1.0889 - val_accuracy: 0.7458 - lr: 1.0000e-05 Epoch 55/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1632 - accuracy: 0.9375 - val_loss: 1.0601 - val_accuracy: 0.7454 - lr: 1.0000e-05 Epoch 56/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1649 - accuracy: 0.9381 - val_loss: 1.0953 - val_accuracy: 0.7438 - lr: 1.0000e-05 Epoch 57/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1606 - accuracy: 0.9389 - val_loss: 1.0946 - val_accuracy: 0.7482 - lr: 1.0000e-05 Epoch 58/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1558 - accuracy: 0.9386 - val_loss: 1.1140 - val_accuracy: 0.7452 - lr: 1.0000e-05 Epoch 59/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1515 - accuracy: 0.9437 - val_loss: 1.1096 - val_accuracy: 0.7472 - lr: 1.0000e-05 Epoch 60/75 473/473 [==============================] - 28s 59ms/step - loss: 0.1411 - accuracy: 0.9466 - val_loss: 1.1481 - val_accuracy: 0.7390 - lr: 1.0000e-05 Epoch 61/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1419 - accuracy: 0.9473 - val_loss: 1.1623 - val_accuracy: 0.7410 - lr: 1.0000e-05 Epoch 62/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1389 - accuracy: 0.9478 - val_loss: 1.1749 - val_accuracy: 0.7426 - lr: 1.0000e-05 Epoch 63/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1333 - accuracy: 0.9508 - val_loss: 1.1665 - val_accuracy: 0.7472 - lr: 1.0000e-05 Epoch 64/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1287 - accuracy: 0.9514 - val_loss: 1.1912 - val_accuracy: 0.7503 - lr: 1.0000e-05 Epoch 65/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1321 - accuracy: 0.9507 - val_loss: 1.1520 - val_accuracy: 0.7386 - lr: 1.0000e-05 Epoch 66/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1255 - accuracy: 0.9532 - val_loss: 1.1690 - val_accuracy: 0.7535 - lr: 1.0000e-05 Epoch 67/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1196 - accuracy: 0.9538 - val_loss: 1.1468 - val_accuracy: 0.7527 - lr: 1.0000e-05 Epoch 68/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1157 - accuracy: 0.9570 - val_loss: 1.2080 - val_accuracy: 0.7444 - lr: 1.0000e-05 Epoch 69/75 473/473 [==============================] - 28s 59ms/step - loss: 0.1148 - accuracy: 0.9577 - val_loss: 1.2205 - val_accuracy: 0.7420 - lr: 1.0000e-05 Epoch 70/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1019 - accuracy: 0.9632 - val_loss: 1.2842 - val_accuracy: 0.7430 - lr: 1.0000e-05 Epoch 71/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1089 - accuracy: 0.9602 - val_loss: 1.2643 - val_accuracy: 0.7450 - lr: 1.0000e-05 Epoch 72/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1109 - accuracy: 0.9591 - val_loss: 1.2686 - val_accuracy: 0.7517 - lr: 1.0000e-05 Epoch 73/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1053 - accuracy: 0.9628 - val_loss: 1.2550 - val_accuracy: 0.7428 - lr: 1.0000e-05 Epoch 74/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1052 - accuracy: 0.9600 - val_loss: 1.2497 - val_accuracy: 0.7456 - lr: 1.0000e-05 Epoch 75/75 473/473 [==============================] - 28s 58ms/step - loss: 0.1007 - accuracy: 0.9621 - val_loss: 1.2674 - val_accuracy: 0.7392 - lr: 1.0000e-05
plot_model_performance_and_confusion_matrix(cnn_model_4, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display
val_loss, val_accuracy = cnn_model_4.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
<p style='color: red;'><b>Test Loss:</b> CNN_MODEL_4 </p>
<p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
<p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""
# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 17ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.86 0.78 0.82 32 1 0.65 0.75 0.70 32 2 0.70 0.66 0.68 32 3 0.84 0.84 0.84 32 accuracy 0.76 128 macro avg 0.76 0.76 0.76 128 weighted avg 0.76 0.76 0.76 128
4/4 [==============================] - 0s 20ms/step - loss: 0.6133 - accuracy: 0.7578 Test Loss: 0.6132991909980774 Test Accuracy: 0.7578125
Test Loss: CNN_MODEL_4
Test Loss: 0.6133
Test Accuracy: 75.7812%
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
predicted_labels = np.argmax(cnn_model_4.predict(test_generator), axis=1)
# correcting mislabeling
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 0s 17ms/step dict_keys(['happy', 'neutral', 'sad', 'surprise']) precision recall f1-score support 0 0.96 0.81 0.88 32 1 0.60 0.81 0.69 32 2 0.71 0.69 0.70 32 3 1.00 0.84 0.92 32 accuracy 0.79 128 macro avg 0.82 0.79 0.80 128 weighted avg 0.82 0.79 0.80 128
It is then used to perform feature extraction on the images in the pipeline
the Siamese network outputs the features from the last layer before the final similarity score calculation.
Then those output feature vectors are used as the inputs to the CNN_MODEL_5
Start with defining the input shape for our images, making sure it matches the resolution we're working with, like (48, 48, 3) for 48x48 RGB images.
Create the base CNN model that learns feature representations from the images. This model is crucial as it captures the essence of the images.
Define two input tensors for the Siamese network, representing the pair of images to compare.
Reuse the base CNN model for both inputs to ensure the feature extraction process is identical for each image in the pair.
Process both inputs through the base CNN independently, obtaining their feature representations.
Calculate the distance between the two sets of features to understand how similar or dissimilar the images are. The absolute difference works well for this.
Feed the distance through a dense layer with a sigmoid activation to get a similarity score, indicating the likelihood of the images being the same class. Compile the Siamese network, choosing an optimizer and a loss function tailored for comparison tasks, like contrastive loss.
Prepare the image pairs for training, ensuring we have both positive pairs (same class) and negative pairs (different classes).
Augment the dataset if needed, especially to address class imbalance or to increase the diversity of training examples.
Train the Siamese network on these pairs, focusing on learning the subtle differences and similarities between classes.
Evaluate the model on a separate validation set to ensure it generalizes well to new, unseen image pairs.
Siamese for Feature Vectors: I use the trained Siamese model, or specifically its base CNN part, to extract feature vectors from images. This involves modifying the Siamese network to output features from the layer just before the final similarity score calculation.
CNN for Classification: The extracted feature vectors serve as inputs to cnn_model_4 or a similar classification model. This requires me to adjust cnn_model_4 to accept the feature vector size as its input shape, possibly changing the first layer to a Dense layer from a Conv2D, or using the features directly in a custom training loop to influence the classification decision. Using Siamese Model for Data Augmentation:
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False,
fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"
# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)
# Prepare the data generators
train_generator = datagen.flow_from_directory(
train_path,
target_size=(48, 48),
batch_size=32,
class_mode='categorical')
validation_generator = datagen.flow_from_directory(
validation_path,
target_size=(48, 48),
batch_size=32, # Keep the same batch size for simplicity
class_mode='categorical')
test_generator = datagen.flow_from_directory(
test_path,
target_size=(48, 48),
shuffle=False)
Found 15109 images belonging to 4 classes. Found 4977 images belonging to 4 classes. Found 128 images belonging to 4 classes.
cnn_model_4 = Sequential([
Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Slightly reduced dropout
# Second Convolutional Block
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3), # Adjusted dropout
# Third Convolutional Block
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4), # Adjusted dropout
# Fourth Convolutional Block
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Convolutional Block 5
Conv2D(1024, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.2), # Keeping consistent dropout
# Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
GlobalAveragePooling2D(),
Flatten(),
# Dense Layers
Dense(128, activation='relu'),
Dense(512, activation='relu'),
Dropout(0.2),
Dense(1024, activation='relu'),
Dense(2048, activation='relu'),
Dropout(0.4),
Dense(128, activation='relu'),
Dropout(0.2),
Dense(4, activation='softmax')
])
cnn_model_4.compile(optimizer=AdamW(learning_rate=0.00003),
loss='categorical_crossentropy',
metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True) # Increased patience
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001) # Adjusted patience
cnn_model_4.summary()
Model: "sequential_43" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_210 (Conv2D) (None, 48, 48, 64) 1792 batch_normalization_148 (B (None, 48, 48, 64) 256 atchNormalization) conv2d_211 (Conv2D) (None, 48, 48, 64) 36928 max_pooling2d_123 (MaxPool (None, 24, 24, 64) 0 ing2D) dropout_178 (Dropout) (None, 24, 24, 64) 0 conv2d_212 (Conv2D) (None, 24, 24, 128) 73856 batch_normalization_149 (B (None, 24, 24, 128) 512 atchNormalization) conv2d_213 (Conv2D) (None, 24, 24, 256) 295168 max_pooling2d_124 (MaxPool (None, 12, 12, 256) 0 ing2D) dropout_179 (Dropout) (None, 12, 12, 256) 0 conv2d_214 (Conv2D) (None, 12, 12, 256) 590080 batch_normalization_150 (B (None, 12, 12, 256) 1024 atchNormalization) conv2d_215 (Conv2D) (None, 12, 12, 512) 1180160 batch_normalization_151 (B (None, 12, 12, 512) 2048 atchNormalization) max_pooling2d_125 (MaxPool (None, 6, 6, 512) 0 ing2D) dropout_180 (Dropout) (None, 6, 6, 512) 0 conv2d_216 (Conv2D) (None, 6, 6, 1024) 4719616 batch_normalization_152 (B (None, 6, 6, 1024) 4096 atchNormalization) max_pooling2d_126 (MaxPool (None, 3, 3, 1024) 0 ing2D) dropout_181 (Dropout) (None, 3, 3, 1024) 0 conv2d_217 (Conv2D) (None, 3, 3, 1024) 9438208 batch_normalization_153 (B (None, 3, 3, 1024) 4096 atchNormalization) max_pooling2d_127 (MaxPool (None, 1, 1, 1024) 0 ing2D) dropout_182 (Dropout) (None, 1, 1, 1024) 0 global_average_pooling2d_3 (None, 1024) 0 0 (GlobalAveragePooling2D) flatten_30 (Flatten) (None, 1024) 0 dense_146 (Dense) (None, 128) 131200 dense_147 (Dense) (None, 512) 66048 dropout_183 (Dropout) (None, 512) 0 dense_148 (Dense) (None, 1024) 525312 dense_149 (Dense) (None, 2048) 2099200 dropout_184 (Dropout) (None, 2048) 0 dense_150 (Dense) (None, 128) 262272 dropout_185 (Dropout) (None, 128) 0 dense_151 (Dense) (None, 4) 516 ================================================================= Total params: 19432388 (74.13 MB) Trainable params: 19426372 (74.11 MB) Non-trainable params: 6016 (23.50 KB) _________________________________________________________________
input_shape = (48, 48, 3)
base_cnn = create_base_cnn(input_shape)
feature_model = Model(inputs=base_cnn.input, outputs=base_cnn.layers[-5].output)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, GlobalAveragePooling2D, Dense, BatchNormalization, Flatten
from tensorflow.keras.regularizers import l2
def create_base_cnn(input_shape):
model = Sequential([
Conv2D(64, (3, 3), activation='relu', input_shape=input_shape, padding='same'),
BatchNormalization(),
Conv2D(64, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Conv2D(128, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(256, (3, 3), activation='relu', padding='same'),
MaxPooling2D((2, 2)),
Dropout(0.3),
Conv2D(256, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
Conv2D(512, (3, 3), activation='relu', padding='same'),
BatchNormalization(),
MaxPooling2D((2, 2)),
Dropout(0.4),
GlobalAveragePooling2D(),
Dense(512, activation='relu'),
Dropout(0.4),
Dense(512, activation='relu', kernel_regularizer=l2(0.001)),
])
return model
from tensorflow.keras.layers import Input, Lambda
from tensorflow.keras.models import Model
import tensorflow.keras.backend as K
input_shape = (48, 48, 3)
def create_siamese_network(input_shape):
input_a = Input(shape=input_shape)
input_b = Input(shape=input_shape)
base_cnn = create_base_cnn(input_shape)
processed_a = base_cnn(input_a)
processed_b = base_cnn(input_b)
distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]))([processed_a, processed_b])
outputs = Dense(1, activation='sigmoid')(distance)
model = Model(inputs=[input_a, input_b], outputs=outputs)
return model
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Lambda
from tensorflow.keras.optimizers import Adam
import tensorflow.keras.backend as K
from sklearn.model_selection import train_test_split
def create_base_cnn(input_shape):
input = Input(shape=input_shape)
x = Conv2D(64, (3, 3), activation='relu')(input)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Conv2D(128, (3, 3), activation='relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Flatten()(x)
x = Dense(128, activation='relu')(x)
return Model(inputs=input, outputs=x)
def create_siamese_network(input_shape):
base_network = create_base_cnn(input_shape)
input_a = Input(shape=input_shape)
input_b = Input(shape=input_shape)
processed_a = base_network(input_a)
processed_b = base_network(input_b)
distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]))([processed_a, processed_b])
outputs = Dense(1, activation='sigmoid')(distance)
model = Model(inputs=[input_a, input_b], outputs=outputs)
return model
input_shape = (48, 48, 3) # Adjust based on your dataset
siamese_network = create_siamese_network(input_shape)
siamese_network.compile(optimizer=Adam(0.0001), loss='binary_crossentropy', metrics=['accuracy'])
# Define callbacks
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)
# Fit the model
history = cnn_model_4.fit(
train_generator,
steps_per_epoch=train_generator.samples // train_generator.batch_size,
epochs=100,
validation_data=validation_generator,# Adjusted data loading
validation_steps=validation_generator.samples // validation_generator.batch_size,# adjusted data loading
callbacks=[early_stopping, reduce_lr],
verbose=1
)
Epoch 1/100 472/472 [==============================] - 25s 43ms/step - loss: 1.3865 - accuracy: 0.3460 - val_loss: 1.6837 - val_accuracy: 0.3663 - lr: 3.0000e-05 Epoch 2/100 472/472 [==============================] - 20s 43ms/step - loss: 1.2545 - accuracy: 0.4210 - val_loss: 1.5015 - val_accuracy: 0.3835 - lr: 3.0000e-05 Epoch 3/100 472/472 [==============================] - 20s 43ms/step - loss: 1.1560 - accuracy: 0.4819 - val_loss: 1.3127 - val_accuracy: 0.4194 - lr: 3.0000e-05 Epoch 4/100 472/472 [==============================] - 20s 43ms/step - loss: 1.0680 - accuracy: 0.5405 - val_loss: 1.1432 - val_accuracy: 0.5143 - lr: 3.0000e-05 Epoch 5/100 472/472 [==============================] - 20s 42ms/step - loss: 1.0107 - accuracy: 0.5706 - val_loss: 0.9674 - val_accuracy: 0.5817 - lr: 3.0000e-05 Epoch 6/100 472/472 [==============================] - 20s 42ms/step - loss: 0.9745 - accuracy: 0.5841 - val_loss: 1.1597 - val_accuracy: 0.5379 - lr: 3.0000e-05 Epoch 7/100 472/472 [==============================] - 20s 43ms/step - loss: 0.9306 - accuracy: 0.6095 - val_loss: 0.9213 - val_accuracy: 0.6222 - lr: 3.0000e-05 Epoch 8/100 472/472 [==============================] - 20s 43ms/step - loss: 0.8935 - accuracy: 0.6266 - val_loss: 0.8629 - val_accuracy: 0.6468 - lr: 3.0000e-05 Epoch 9/100 472/472 [==============================] - 20s 43ms/step - loss: 0.8591 - accuracy: 0.6397 - val_loss: 0.9322 - val_accuracy: 0.6288 - lr: 3.0000e-05 Epoch 10/100 472/472 [==============================] - 20s 42ms/step - loss: 0.8364 - accuracy: 0.6540 - val_loss: 0.8387 - val_accuracy: 0.6633 - lr: 3.0000e-05 Epoch 11/100 472/472 [==============================] - 20s 43ms/step - loss: 0.8130 - accuracy: 0.6615 - val_loss: 0.8049 - val_accuracy: 0.6722 - lr: 3.0000e-05 Epoch 12/100 472/472 [==============================] - 20s 42ms/step - loss: 0.7941 - accuracy: 0.6763 - val_loss: 0.7672 - val_accuracy: 0.7006 - lr: 3.0000e-05 Epoch 13/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7713 - accuracy: 0.6821 - val_loss: 0.7706 - val_accuracy: 0.6976 - lr: 3.0000e-05 Epoch 14/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7454 - accuracy: 0.6914 - val_loss: 0.7470 - val_accuracy: 0.7002 - lr: 3.0000e-05 Epoch 15/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7268 - accuracy: 0.7003 - val_loss: 0.7572 - val_accuracy: 0.6978 - lr: 3.0000e-05 Epoch 16/100 472/472 [==============================] - 20s 43ms/step - loss: 0.7109 - accuracy: 0.7088 - val_loss: 0.7452 - val_accuracy: 0.7026 - lr: 3.0000e-05 Epoch 17/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6928 - accuracy: 0.7184 - val_loss: 0.7401 - val_accuracy: 0.7079 - lr: 3.0000e-05 Epoch 18/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6757 - accuracy: 0.7249 - val_loss: 0.7120 - val_accuracy: 0.7216 - lr: 3.0000e-05 Epoch 19/100 472/472 [==============================] - 21s 44ms/step - loss: 0.6559 - accuracy: 0.7340 - val_loss: 0.6963 - val_accuracy: 0.7278 - lr: 3.0000e-05 Epoch 20/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6376 - accuracy: 0.7405 - val_loss: 0.6986 - val_accuracy: 0.7262 - lr: 3.0000e-05 Epoch 21/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6212 - accuracy: 0.7464 - val_loss: 0.7125 - val_accuracy: 0.7254 - lr: 3.0000e-05 Epoch 22/100 472/472 [==============================] - 20s 43ms/step - loss: 0.6017 - accuracy: 0.7613 - val_loss: 0.6822 - val_accuracy: 0.7323 - lr: 3.0000e-05 Epoch 23/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5885 - accuracy: 0.7675 - val_loss: 0.6721 - val_accuracy: 0.7397 - lr: 3.0000e-05 Epoch 24/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5641 - accuracy: 0.7730 - val_loss: 0.6693 - val_accuracy: 0.7454 - lr: 3.0000e-05 Epoch 25/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5524 - accuracy: 0.7795 - val_loss: 0.6849 - val_accuracy: 0.7361 - lr: 3.0000e-05 Epoch 26/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5410 - accuracy: 0.7822 - val_loss: 0.6808 - val_accuracy: 0.7345 - lr: 3.0000e-05 Epoch 27/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5218 - accuracy: 0.7951 - val_loss: 0.6800 - val_accuracy: 0.7399 - lr: 3.0000e-05 Epoch 28/100 472/472 [==============================] - 20s 43ms/step - loss: 0.5034 - accuracy: 0.8005 - val_loss: 0.6799 - val_accuracy: 0.7407 - lr: 3.0000e-05 Epoch 29/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4944 - accuracy: 0.8042 - val_loss: 0.6570 - val_accuracy: 0.7532 - lr: 3.0000e-05 Epoch 30/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4700 - accuracy: 0.8151 - val_loss: 0.6590 - val_accuracy: 0.7516 - lr: 3.0000e-05 Epoch 31/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4588 - accuracy: 0.8213 - val_loss: 0.6479 - val_accuracy: 0.7506 - lr: 3.0000e-05 Epoch 32/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4400 - accuracy: 0.8301 - val_loss: 0.6796 - val_accuracy: 0.7468 - lr: 3.0000e-05 Epoch 33/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4303 - accuracy: 0.8328 - val_loss: 0.6759 - val_accuracy: 0.7433 - lr: 3.0000e-05 Epoch 34/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4136 - accuracy: 0.8395 - val_loss: 0.6724 - val_accuracy: 0.7496 - lr: 3.0000e-05 Epoch 35/100 472/472 [==============================] - 20s 43ms/step - loss: 0.4050 - accuracy: 0.8461 - val_loss: 0.7359 - val_accuracy: 0.7252 - lr: 3.0000e-05 Epoch 36/100 471/472 [============================>.] - ETA: 0s - loss: 0.3864 - accuracy: 0.8512 Epoch 36: ReduceLROnPlateau reducing learning rate to 1e-05. 472/472 [==============================] - 20s 43ms/step - loss: 0.3865 - accuracy: 0.8512 - val_loss: 0.6757 - val_accuracy: 0.7506 - lr: 3.0000e-05 Epoch 37/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3486 - accuracy: 0.8706 - val_loss: 0.6466 - val_accuracy: 0.7641 - lr: 1.0000e-05 Epoch 38/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3408 - accuracy: 0.8740 - val_loss: 0.6653 - val_accuracy: 0.7571 - lr: 1.0000e-05 Epoch 39/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3252 - accuracy: 0.8841 - val_loss: 0.6793 - val_accuracy: 0.7526 - lr: 1.0000e-05 Epoch 40/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3270 - accuracy: 0.8804 - val_loss: 0.6592 - val_accuracy: 0.7647 - lr: 1.0000e-05 Epoch 41/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3145 - accuracy: 0.8842 - val_loss: 0.6528 - val_accuracy: 0.7663 - lr: 1.0000e-05 Epoch 42/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3107 - accuracy: 0.8898 - val_loss: 0.6633 - val_accuracy: 0.7649 - lr: 1.0000e-05 Epoch 43/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3040 - accuracy: 0.8900 - val_loss: 0.6761 - val_accuracy: 0.7603 - lr: 1.0000e-05 Epoch 44/100 472/472 [==============================] - 20s 43ms/step - loss: 0.3054 - accuracy: 0.8869 - val_loss: 0.6615 - val_accuracy: 0.7637 - lr: 1.0000e-05 Epoch 45/100 472/472 [==============================] - 20s 43ms/step - loss: 0.2913 - accuracy: 0.8928 - val_loss: 0.6561 - val_accuracy: 0.7681 - lr: 1.0000e-05 Epoch 46/100 472/472 [==============================] - 20s 43ms/step - loss: 0.2858 - accuracy: 0.8963 - val_loss: 0.6878 - val_accuracy: 0.7579 - lr: 1.0000e-05 Epoch 47/100 471/472 [============================>.] - ETA: 0s - loss: 0.2852 - accuracy: 0.8956Restoring model weights from the end of the best epoch: 37. 472/472 [==============================] - 20s 43ms/step - loss: 0.2857 - accuracy: 0.8954 - val_loss: 0.6985 - val_accuracy: 0.7550 - lr: 1.0000e-05 Epoch 47: early stopping
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_4.evaluate(
test_generator,
steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 16ms/step - loss: 0.5081 - accuracy: 0.7969 Test Loss: 0.5081095695495605 Test Accuracy: 0.796875
from sklearn.metrics import confusion_matrix, classification_report
# Plot the training and validation accuracy values
plt.figure(figsize=(35, 6))
plt.subplot(1, 4, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('cnn_model_4 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 4, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('cnn_model_4 Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
# Plot the training and validation loss values
plt.subplot(1, 4, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for cnn_model_4")
predicted_labels = np.argmax(cnn_model_4.predict(test_generator), axis=1)
# Get the true labels for the test set
true_labels = test_generator.classes
# Get the class names
class_names_list = test_generator.class_indices.keys()
# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
plt.show()
4/4 [==============================] - 0s 16ms/step precision recall f1-score support 0 0.83 0.91 0.87 32 1 0.69 0.75 0.72 32 2 0.72 0.66 0.69 32 3 0.97 0.88 0.92 32 accuracy 0.80 128 macro avg 0.80 0.80 0.80 128 weighted avg 0.80 0.80 0.80 128
!tensorflowjs_converter --input_format=keras /content/cnn_model_4.h5 /content/siamese_model_1
2024-02-22 05:12:35.043733: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-02-22 05:12:35.043782: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-02-22 05:12:35.044911: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-02-22 05:12:36.086643: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
However, the methods used show that substantial gains can be made:
Data augmentation for the training data helps to make the smaller dataset more robust. Weighting the classes helped with making the class distribution skew less detrimental to the overall performance. Transfer learning on the dataset proved to be difficult; however, after a good amount of hyperparameter adjusting, two of the three transfer learning models started to show promising results. The second convolutional neural network constructed is the overall best performer among the models built here.
The final model tries to extract useful feature vector data and then fuse its learned properties with the training process of a standard CNN. This is an attempt to take a more specialized method to really learn and embed the deep underlying properties that differentiate the classes of neutral and sad. Those two classes gave all the models the most difficulty during testing, as seen in the included confusion matrices for each model's testing results.
This task is important. Being able to detect basic emotions rests upon the base technologies that will enable more impactful use cases, such as driver monitoring in automobiles, to check if the driver is paying attention. This is especially important with level 2 and 3 semi-autonomous vehicles where there is a notable vigilance decrement by drivers. Drivers can get the sense that the car is fully in control and will start not only to not pay attention but, in some extreme cases, even fall asleep. The methods to be able to detect a human face and determine the emotion of an individual can easily be extended and used to track the driver's attention span and ensure that they are ready to take back control of the vehicle.
In this project, we developed a facial recognition and emotion classification model using a convolutional neural network (CNN). The model, a 5-layer CNN with deep dense layers and optimized convolution setup, achieved 82% accuracy.
The main challenge was accurately classifying facial emotions due to the subtle differences between expressions. This is critical for sectors like security and customer service, where understanding human emotions is key.
The chosen CNN architecture relys on deep learning's pattern recognition capabilities, especially for image data, enhancing the model's ability to detect subtle emotional features. Implementing this model could significantly improve human-computer interactions, making them more responsive and empathetic. For customer service, it could lead to tailored communications and solutions, boosting satisfaction and loyalty. In security, it could help identify distress or suspicious behaviors more effectively. Recommendations for Implementation Integration with Interactive Systems: Deploy the model in platforms for enhanced user interaction, such as customer service kiosks or online help desks. Enhancement of Security Systems: Use the model for improved surveillance and monitoring, flagging unusual or suspicious behavior. Actionables for Stakeholders:
Deployment and Training: Implement the model on targeted platforms and train staff to use it effectively. Continuous Model Evaluation and Improvement: Regularly review and update the model with new data to ensure its accuracy and relevance. Privacy and Ethical Considerations: Set guidelines for ethical use, respecting privacy and data protection laws.
Costs: The training of these models required backend gpu services provided through google. For a larger scale version the costs would go up accordingly. This project alone used 200 compute units over the period of a week training and re-training models. For a more complex model that would be required to scale up the services explored here cost analysis should be taken into consideration depending on the rate of scaling.
Available: 16.86 compute units Usage rate: approximately 2.05 per hour
To conclude, Computer Vision, is where all the code and software actually make contact with our world. Unlike robotics, for computer vision all that is required is a simple camera, giving the technology the ability to reach and touch most of the population on the planet depening on the use case.
Thank you -- Robert A, Sloan
plot_model_performance_and_confusion_matrix(cnn_model_1, history, test_generator,
metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 0s 15ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.87 0.81 0.84 32 1 0.65 0.88 0.75 32 2 0.81 0.69 0.75 32 3 0.96 0.84 0.90 32 accuracy 0.80 128 macro avg 0.82 0.80 0.81 128 weighted avg 0.82 0.80 0.81 128
4/4 [==============================] - 0s 10ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.82 0.84 0.83 32 1 0.72 0.81 0.76 32 2 0.79 0.69 0.73 32 3 0.97 0.94 0.95 32 accuracy 0.82 128 macro avg 0.82 0.82 0.82 128 weighted avg 0.82 0.82 0.82 128
4/4 [==============================] - 0s 11ms/step - loss: 0.5187 - accuracy: 0.8203 Test Loss: 0.5187498927116394 Test Accuracy: 0.8203125
Test Loss: CNN_MODEL_updated
Test Loss: 0.5187
Test Accuracy: 82.0312%
4/4 [==============================] - 0s 11ms/step Confusion Matrix: [[25 6 0 1] [ 0 28 3 1] [ 3 8 18 3] [ 0 2 2 28]]
4/4 [==============================] - 0s 13ms/step dict_keys(['happy', 'neutral', 'sad', 'surprise']) precision recall f1-score support 0 0.89 0.78 0.83 32 1 0.64 0.88 0.74 32 2 0.78 0.56 0.65 32 3 0.85 0.88 0.86 32 accuracy 0.77 128 macro avg 0.79 0.77 0.77 128 weighted avg 0.79 0.77 0.77 128
4/4 [==============================] - 1s 20ms/step Confusion Matrix: [[26 2 2 2] [ 2 25 5 0] [ 1 12 17 2] [ 0 3 1 28]]
4/4 [==============================] - 0s 22ms/step dict_keys(['happy', 'neutral', 'sad', 'surprise']) precision recall f1-score support 0 0.90 0.81 0.85 32 1 0.60 0.78 0.68 32 2 0.68 0.53 0.60 32 3 0.88 0.88 0.88 32 accuracy 0.75 128 macro avg 0.76 0.75 0.75 128 weighted avg 0.76 0.75 0.75 128
4/4 [==============================] - 0s 12ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.67 0.81 0.73 32 1 0.57 0.72 0.64 32 2 0.54 0.44 0.48 32 3 0.87 0.62 0.73 32 accuracy 0.65 128 macro avg 0.66 0.65 0.65 128 weighted avg 0.66 0.65 0.65 128
4/4 [==============================] - 0s 18ms/step - loss: 0.8464 - accuracy: 0.6484 Test Loss: 0.84635990858078 Test Accuracy: 0.6484375
Test Loss: efficient_model2
Test Loss: 0.85
Test Accuracy: 64.84%
4/4 [==============================] - 0s 17ms/step Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise'] precision recall f1-score support 0 0.86 0.78 0.82 32 1 0.65 0.75 0.70 32 2 0.70 0.66 0.68 32 3 0.84 0.84 0.84 32 accuracy 0.76 128 macro avg 0.76 0.76 0.76 128 weighted avg 0.76 0.76 0.76 128
4/4 [==============================] - 0s 20ms/step - loss: 0.6133 - accuracy: 0.7578 Test Loss: 0.6132991909980774 Test Accuracy: 0.7578125
Test Loss: CNN_MODEL_4
Test Loss: 0.6133
Test Accuracy: 75.7812%
4/4 [==============================] - 0s 16ms/step precision recall f1-score support 0 0.83 0.91 0.87 32 1 0.69 0.75 0.72 32 2 0.72 0.66 0.69 32 3 0.97 0.88 0.92 32 accuracy 0.80 128 macro avg 0.80 0.80 0.80 128 weighted avg 0.80 0.80 0.80 128
!tensorflowjs_converter --input_format=keras /content/cnn_model_4.h5 /content/siamese_model_1
2024-02-22 05:12:35.043733: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-02-22 05:12:35.043782: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-02-22 05:12:35.044911: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-02-22 05:12:36.086643: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
# prompt: using the nbconvert tool convert the notebook to html
!jupyter nbconvert --to html 'cnn_model_4.ipynb'
[NbConvertApp] Converting notebook /content/Capstone_Facial_emotion_detection.ipynb to html [NbConvertApp] Writing 4837557 bytes to /content/Capstone_Facial_emotion_detection.html