Training Neural Networks Part 1

In [1]:
%pylab inline
from ipypublish import nb_setup

import keras
keras.__version__
from keras import models
from keras import layers
Populating the interactive namespace from numpy and matplotlib

Backprop recap

The Backprop algorithm was known by the mid-1980s, but it toook almost two more decades before the field of Deep Learning entered the mainstream. There were several reasons for this delay, including the fact that the processing power was not yet there, but the main reason was that Backprop simply did not work for large models that arise in practical problems. In these cases it was observed that the gradients died away before the training was complete, thus limiting the accuracy of the model. In this chapter and in the next, our objective is go through individual elements of the Gradient Descent algorithm and make improvements so that it is able to work for large models. We start by examining the parameter update equation and present several modifications that improve upon it in Sections LearningRateSelection and ParameterUpdate . We then investigate the role of the Activation Function in Section ActivationLossFunctions, and also come up with a number of alternative functions that improve performance. The correct initialization of weights at the start of the algorithm is also a huge issue, and is discussed in Section InitializingWeights. In Section DataPreprocessing we show that if the training data is pre-conditioned before being fed into the model, then it leads to several benefits in the training process. We end the chapter with a discussion of Batch Normalization which is a recently discovered technique to improve the training process, but which has already had a big impact on the field.

In [2]:
#TNN1
nb_setup.images_hconcat(["DL_images/TNN1.png"], width=600)
Out[2]:

Issues with Gradient Descent

Recall that the Gradient Descent based parameter update equation in one dimension is given by:

$$ w\leftarrow w - \eta\frac{\partial {\mathcal L}}{\partial w} $$

As shown in Figure TNN1, $\frac{\partial {\mathcal L}}{\partial w} > 0$ for points on the curve to the right of the minimum. This causes the value of $w$ to decrease, until it converges to the minimum point at which the gradient is zero. Conversely $\frac{\partial {\mathcal L}}{\partial w} < 0$ for points to the left of the minimum, which causes $w$ to increase with each iteration. There are a number of cases in which this simple iteration does not work very well, and we will describe these next:

In [3]:
#TNN2
nb_setup.images_hconcat(["DL_images/TNN2.png"], width=600)
Out[3]:
  1. Even in the simple one dimensional case, it is easy to see that the learning rate parameter $\eta$ exerts a powerful infuence on the convergence process (see Figure TNN2). If $\eta$ is too small, then the convergence happens very slowly as shown in the left hand side of the figure. On the other hand if $\eta$ is too large, then the algorithm starts to oscillate and may even diverge.
In [4]:
#TNN4
nb_setup.images_hconcat(["DL_images/TNN4.png"], width=600)
Out[4]:
  1. If Gradient Descent is run in multiple dimensions, then other problems can arise. One such problem is illustrated in Figure TNN4. The figure illustrates a two dimensional scenario in which te Loss Function $\mathcal L$ has a very steep slope along one dimension and a shallow slope along the other: i.e., it has the shape of a Narrow Steep Valley. If we run Gradient Descent in this system, we get the behavior shown in the diagram on the LHS (this is a 2-D analog of the behavior shown in the RHS of Figure TNN2). The parameter that lies along the steep part of the objective function oscillates back and forth between the valley slopes, while the parameter that lies along the shallow part of the Loss Function moves slowly down the valley. The net effect of this is that convergence happens very slowly. The right hand side of the figure shows a more ideal convergence behavior, which we will show how to achieve in Section ParameterUpdate.
In [5]:
#TNN3
nb_setup.images_hconcat(["DL_images/TNN3.png"], width=600)
Out[5]:
In [6]:
#TNN8
nb_setup.images_hconcat(["DL_images/TNN8.png"], width=600)
Out[6]:
  1. Another issue that arises when Gradient Descent is run in multiple dimensions, is that of Saddle Points. These are defined as areas on the surface of the Loss Function, which are a minima when observed along one of the dimensions, and simultaneously serve as a maxima when observes along another dimension. This is illustrated for the the two dimensional case in Figure TNN3. When the iteration approaches a saddle point from above, then even though it is not at a minimum, it slope hits zero. As a result the ieration comes to halt and the algorithm gets stuck a non-optimal point. This bevavior is further illustrated in Figure TNN8.

All the issues with Gradient Descent that were raised in this section are addressed by techniques described in the next two sections.

Learning Rate Annealing

In [7]:
#TNN6
nb_setup.images_hconcat(["DL_images/TNN6.png"], width=600)
Out[7]:

As we mentioned in the previous section, the Learning Rate parameter has a big influence on the effectiveness of the Gradient Descent algorithm. If it is set to a large value then the algorithm moves quickly at the start of the iteration, but the large step size can cause a parameter overshoot as the system approaches minimum which can lead to oscillations. If set too small then the algorithm converges with high likelihood, however it can take a very long time to do so (see Fig. TNN2). Hence ideally $\eta$ should be set adaptively such it is large in the initial stages of the optimization and becomes smaller as it gets closer to the minimum.

Figure TNN6 illustrates the effect of the Learning Rate on the Loss Function during training and can be used to do a quick check on the suitability of the rate being used. A very high Learning Rate can cause the Loss Function to start to increase after a few iterations, while a moderately high rate causes the Loss to plateau at a high value after an initial rapid decrease. A very low Learning Rate on the other hand can be identified by a slow decrease in the Loss Function over training epochs. A good Learning Rate on the other hand combines a quick decrease during the initial epochs with a lower steady state value.

In [8]:
#TNN7
nb_setup.images_hconcat(["DL_images/TNN7.png"], width=400)
Out[8]:

A well known technique for achieving the best Learning Rate behavior is called Learning Rate Annealing. This is the strategy of reducing the Learning Rate as the system approaches the minimum (see Figure TNN7), such that rate is high at the start of the training and gradually falls as the training progresses. This reduction can be done in several ways, popular approaches are:

  • Track the validation accuracy and decrease the Learning Rate when it appears to plateau.

  • Automatically anneal the Learning Rate based on the number of epochs that the Gradient Descent algorithm has been through.

Instead of using the same Learning Rate for every parameter, in the next Section we will learn about techniques that tailor the rate to the parameter. Thus parameters possessing a steep gradient get lower rates compared to parameters with a smaller gradient.

Keras provides a feature called callbacks which can be used to implement Learning Rate Annealing. Callbacks is an object that is passed to the fit routine, and then gets called by the model while the training is still going on. Some of the uses for this feature incluse:

  • Interrupt training
  • Save the model parameters
  • Load a different set of parameters
  • etc..

Callbacks can be used to do Learning Rate Annealing, as illustrated in the example below. We use one of the built-in callbacks called ReduceLROnPlateau, which has three parameters: (1) The performance measure to be monitored, (2) The factor by which the Learning Rate is reduced everytime the callback is triggered, and (3) The number of epochs for which the performance measure is seen to be stationary before the callback is triggered.

In [5]:
import keras
keras.__version__
from keras import models
from keras import layers
from keras import callbacks
from tensorflow.keras import optimizers

from keras.datasets import cifar10

(train_images, train_labels), (test_images, test_labels) = cifar10.load_data()

train_images = train_images.reshape((50000, 32 * 32 * 3))
train_images = train_images.astype('float32') / 255

test_images = test_images.reshape((10000, 32 * 32 * 3))
test_images = test_images.astype('float32') / 255

from tensorflow.keras.utils import to_categorical

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

network = models.Sequential()
network.add(layers.Dense(20, activation='relu', input_shape=(32 * 32 * 3,)))
network.add(layers.Dense(15, activation='relu'))
network.add(layers.Dense(10, activation='softmax'))

network.compile(optimizer='sgd',
                loss='categorical_crossentropy',
                metrics=['accuracy'])

callbacks_list = [
     keras.callbacks.ReduceLROnPlateau(
          monitor = 'val_loss',
          factor = 0.1,
          patience = 5,
      )
]

sgd = optimizers.SGD(learning_rate=0.01, decay=1e-6, momentum=0.9, nesterov=True)

network.compile(optimizer = sgd,
                loss='categorical_crossentropy',
                metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=500, batch_size=128, 
                      callbacks = callbacks_list, validation_split=0.2)

history_dict = history.history
history_dict.keys()
//anaconda/envs/miniconda3/lib/python3.6/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  "The `lr` argument is deprecated, use `learning_rate` instead.")
Epoch 1/500
313/313 [==============================] - 2s 4ms/step - loss: 1.9762 - accuracy: 0.2794 - val_loss: 1.8635 - val_accuracy: 0.3287
Epoch 2/500
313/313 [==============================] - 1s 3ms/step - loss: 1.8028 - accuracy: 0.3539 - val_loss: 1.8617 - val_accuracy: 0.3289
Epoch 3/500
313/313 [==============================] - 1s 3ms/step - loss: 1.7598 - accuracy: 0.3658 - val_loss: 1.7539 - val_accuracy: 0.3736
Epoch 4/500
313/313 [==============================] - 1s 3ms/step - loss: 1.7268 - accuracy: 0.3805 - val_loss: 1.7565 - val_accuracy: 0.3702
Epoch 5/500
313/313 [==============================] - 1s 3ms/step - loss: 1.7037 - accuracy: 0.3916 - val_loss: 1.7178 - val_accuracy: 0.3897
Epoch 6/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6834 - accuracy: 0.3975 - val_loss: 1.7238 - val_accuracy: 0.3849
Epoch 7/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6762 - accuracy: 0.4012 - val_loss: 1.7840 - val_accuracy: 0.3620
Epoch 8/500
313/313 [==============================] - 1s 4ms/step - loss: 1.6614 - accuracy: 0.4055 - val_loss: 1.7091 - val_accuracy: 0.3938
Epoch 9/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6542 - accuracy: 0.4069 - val_loss: 1.8199 - val_accuracy: 0.3479
Epoch 10/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6475 - accuracy: 0.4086 - val_loss: 1.7347 - val_accuracy: 0.3772
Epoch 11/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6416 - accuracy: 0.4111 - val_loss: 1.7961 - val_accuracy: 0.3486
Epoch 12/500
313/313 [==============================] - 1s 4ms/step - loss: 1.6337 - accuracy: 0.4142 - val_loss: 1.6660 - val_accuracy: 0.4056
Epoch 13/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6304 - accuracy: 0.4132 - val_loss: 1.7423 - val_accuracy: 0.3767
Epoch 14/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6241 - accuracy: 0.4151 - val_loss: 1.7761 - val_accuracy: 0.3568
Epoch 15/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6191 - accuracy: 0.4180 - val_loss: 1.7345 - val_accuracy: 0.3729
Epoch 16/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6162 - accuracy: 0.4194 - val_loss: 1.6833 - val_accuracy: 0.3978
Epoch 17/500
313/313 [==============================] - 1s 3ms/step - loss: 1.6105 - accuracy: 0.4230 - val_loss: 1.7222 - val_accuracy: 0.3859
Epoch 18/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5772 - accuracy: 0.4338 - val_loss: 1.6482 - val_accuracy: 0.4117
Epoch 19/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5680 - accuracy: 0.4391 - val_loss: 1.6503 - val_accuracy: 0.4117
Epoch 20/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5677 - accuracy: 0.4361 - val_loss: 1.6445 - val_accuracy: 0.4115
Epoch 21/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5665 - accuracy: 0.4372 - val_loss: 1.6397 - val_accuracy: 0.4163
Epoch 22/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5653 - accuracy: 0.4381 - val_loss: 1.6427 - val_accuracy: 0.4149
Epoch 23/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5646 - accuracy: 0.4389 - val_loss: 1.6495 - val_accuracy: 0.4108
Epoch 24/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5644 - accuracy: 0.4384 - val_loss: 1.6430 - val_accuracy: 0.4153
Epoch 25/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5631 - accuracy: 0.4413 - val_loss: 1.6417 - val_accuracy: 0.4143
Epoch 26/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5628 - accuracy: 0.4396 - val_loss: 1.6430 - val_accuracy: 0.4132
Epoch 27/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5575 - accuracy: 0.4413 - val_loss: 1.6390 - val_accuracy: 0.4154
Epoch 28/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5565 - accuracy: 0.4419 - val_loss: 1.6391 - val_accuracy: 0.4154
Epoch 29/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5561 - accuracy: 0.4413 - val_loss: 1.6389 - val_accuracy: 0.4161
Epoch 30/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5561 - accuracy: 0.4414 - val_loss: 1.6381 - val_accuracy: 0.4157
Epoch 31/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5560 - accuracy: 0.4424 - val_loss: 1.6387 - val_accuracy: 0.4164
Epoch 32/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5557 - accuracy: 0.4426 - val_loss: 1.6384 - val_accuracy: 0.4175
Epoch 33/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5559 - accuracy: 0.4415 - val_loss: 1.6384 - val_accuracy: 0.4150
Epoch 34/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5556 - accuracy: 0.4420 - val_loss: 1.6388 - val_accuracy: 0.4156
Epoch 35/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5555 - accuracy: 0.4418 - val_loss: 1.6393 - val_accuracy: 0.4170
Epoch 36/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5552 - accuracy: 0.4421 - val_loss: 1.6386 - val_accuracy: 0.4163
Epoch 37/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5548 - accuracy: 0.4421 - val_loss: 1.6384 - val_accuracy: 0.4166
Epoch 38/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5548 - accuracy: 0.4421 - val_loss: 1.6384 - val_accuracy: 0.4168
Epoch 39/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5547 - accuracy: 0.4422 - val_loss: 1.6383 - val_accuracy: 0.4162
Epoch 40/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5547 - accuracy: 0.4425 - val_loss: 1.6384 - val_accuracy: 0.4169
Epoch 41/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6384 - val_accuracy: 0.4163
Epoch 42/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4425 - val_loss: 1.6384 - val_accuracy: 0.4164
Epoch 43/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6384 - val_accuracy: 0.4161
Epoch 44/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4423 - val_loss: 1.6384 - val_accuracy: 0.4158
Epoch 45/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4423 - val_loss: 1.6383 - val_accuracy: 0.4159
Epoch 46/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4423 - val_loss: 1.6383 - val_accuracy: 0.4159
Epoch 47/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4159
Epoch 48/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4159
Epoch 49/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 50/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 51/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 52/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 53/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 54/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 55/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 56/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 57/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 58/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 59/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 60/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 61/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 62/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 63/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 64/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 65/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 66/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 67/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 68/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 69/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 70/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 71/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 72/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 73/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 74/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 75/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 76/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 77/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 78/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 79/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 80/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 81/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 82/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 83/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 84/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 85/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 86/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 87/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 88/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 89/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 90/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 91/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 92/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 93/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 94/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 95/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 96/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 97/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 98/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 99/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 100/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 101/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 102/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 103/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 104/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 105/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 106/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 107/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 108/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 109/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 110/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 111/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 112/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 113/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 114/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 115/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 116/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 117/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 118/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 119/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 120/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 121/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 122/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 123/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 124/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 125/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 126/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 127/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 128/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 129/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 130/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 131/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 132/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 133/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 134/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 135/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 136/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 137/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 138/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 139/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 140/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 141/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 142/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 143/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 144/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 145/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 146/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 147/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 148/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 149/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 150/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 151/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 152/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 153/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 154/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 155/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 156/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 157/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 158/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 159/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 160/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 161/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 162/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 163/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 164/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 165/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 166/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 167/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 168/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 169/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 170/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 171/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 172/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 173/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 174/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 175/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 176/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 177/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 178/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 179/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 180/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 181/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 182/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 183/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 184/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 185/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 186/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 187/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 188/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 189/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 190/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 191/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 192/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 193/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 194/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 195/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 196/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 197/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 198/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 199/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 200/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 201/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 202/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 203/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 204/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 205/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 206/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 207/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 208/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 209/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 210/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 211/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 212/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 213/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 214/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 215/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 216/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 217/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 218/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 219/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 220/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 221/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 222/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 223/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 224/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 225/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 226/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 227/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 228/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 229/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 230/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 231/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 232/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 233/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 234/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 235/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 236/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 237/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 238/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 239/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 240/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 241/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 242/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 243/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 244/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 245/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 246/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 247/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 248/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 249/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 250/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 251/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 252/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 253/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 254/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 255/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 256/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 257/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 258/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 259/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 260/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 261/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 262/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 263/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 264/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 265/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 266/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 267/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 268/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 269/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 270/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 271/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 272/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 273/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 274/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 275/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 276/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 277/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 278/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 279/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 280/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 281/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 282/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 283/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 284/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 285/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 286/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 287/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 288/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 289/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 290/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 291/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 292/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 293/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 294/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 295/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 296/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 297/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 298/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 299/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 300/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 301/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 302/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 303/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 304/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 305/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 306/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 307/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 308/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 309/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 310/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 311/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 312/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 313/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 314/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 315/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 316/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 317/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 318/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 319/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 320/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 321/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 322/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 323/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 324/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 325/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 326/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 327/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 328/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 329/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 330/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 331/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 332/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 333/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 334/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 335/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 336/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 337/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 338/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 339/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 340/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 341/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 342/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 343/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 344/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 345/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 346/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 347/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 348/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 349/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 350/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 351/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 352/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 353/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 354/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 355/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 356/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 357/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 358/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 359/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 360/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 361/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 362/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 363/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 364/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 365/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 366/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 367/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 368/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 369/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 370/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 371/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 372/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 373/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 374/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 375/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 376/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 377/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 378/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 379/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 380/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 381/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 382/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 383/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 384/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 385/500
313/313 [==============================] - 1s 4ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 386/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 387/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 388/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 389/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 390/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 391/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 392/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 393/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 394/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 395/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 396/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 397/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 398/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 399/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 400/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 401/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 402/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 403/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 404/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 405/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 406/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 407/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 408/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 409/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 410/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 411/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 412/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 413/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 414/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 415/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 416/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 417/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 418/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 419/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 420/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 421/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 422/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 423/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 424/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 425/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 426/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 427/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 428/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 429/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 430/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 431/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 432/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 433/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 434/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 435/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 436/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 437/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 438/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 439/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 440/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 441/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 442/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 443/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 444/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 445/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 446/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 447/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 448/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 449/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 450/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 451/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 452/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 453/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 454/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 455/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 456/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 457/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 458/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 459/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 460/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 461/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 462/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 463/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 464/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 465/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 466/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 467/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 468/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 469/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 470/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 471/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 472/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 473/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 474/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 475/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 476/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 477/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 478/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 479/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 480/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 481/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 482/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 483/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 484/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 485/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 486/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 487/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 488/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 489/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 490/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 491/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 492/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 493/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 494/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 495/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 496/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 497/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 498/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 499/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Epoch 500/500
313/313 [==============================] - 1s 3ms/step - loss: 1.5546 - accuracy: 0.4424 - val_loss: 1.6383 - val_accuracy: 0.4158
Out[5]:
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy', 'lr'])
In [6]:
import matplotlib.pyplot as plt

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)
#epochs = range(1, len(loss) + 1)

# "bo" is for "blue dot"
plt.plot(epochs, loss, 'bo', label='Training loss')
# b is for "solid blue line"
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()

plt.show()
In [7]:
plt.clf()   # clear figure
acc_values = history_dict['accuracy']
val_acc_values = history_dict['val_accuracy']

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()

plt.show()

Improvements to the Parameter Update Equation

In the next few sections we present a number of modifications to the base parameter update equation $w\leftarrow w - \eta\frac{\partial {\mathcal L}}{\partial w}$, which help to improve the performance of the Gradient Descent algorithm. Some of these algorithms automatically adapt the effective Learning Rate as the training progresses (for example the ADAGRAD, RMSPROP and Adam algorithms), while others improve the speed of convergence (for example the Momentum, Nesterov Momentum and Adam algorithms).

Momentum

Momentum is one of the most popular techniques used to improve the speed of convergence of the Gradient Descent algorithm. The basic idea behind Momentum is the following: Some Loss Funtions are characterized by the figure shown on LHS of Figure TNN4. In this case the gradient along one of the dimensions is very large, while along the other dimension it is small. If we do the Gradient Descent iteration for this system then the parameter on the steep side fluctuates from one side of the "canyon" to the other, while the parameter on the shallow side progresses very slowly down the canyon. This behavior slows down the speed of convergence quite a lot. An ingenious but simple technique that can counteract this behavior is as follows: Replace the Gradient Descent iteration by the following:

At the end of the $n^{th}$ iteration of the Backprop algorithm, define a sequence $v(n)$ by

$$ v(n) = \rho\; v(n-1) - \eta \; g(n) $$

with

$$ v(0) = -\eta \; g(0) $$

where $\rho$ is new hyper-parameter called the "momentum" parameter, and $g(n)$ is the gradient evaluated at parameters value $w(n)$, defined by

$$ g(n) = \frac{\partial {\mathcal L(n)}}{\partial w} $$

for Stochastic Gradient Descent and

$$ g(n) = {\eta\over B}\sum_{m=nB}^{(n+1)B}\frac{\partial {\mathcal L(m)}}{\partial w} $$

for Batch Stochastic Gradient Descent (note that in this case $n$ is an index into the batch number). The change in parameter values on each iteration is now defined as

\begin{equation} w(n+1) = w(n) + v(n) \quad \quad (**Wn1**) \end{equation}

It can be shown from these equations that $v(n)$ can be written as

\begin{equation} v(n) = - \eta\sum_{i=0}^n \rho^{n-i} g(i) \quad \quad (**Wn2**) \end{equation}

so that

\begin{equation} w(n+1) = w(n) - \eta\sum_{i=0}^n \rho^{n-i} g(i) \quad \quad (**Wn3**) \end{equation}

When the momentum parameter $\rho = 0$, then this equation reduces to the usual Stochastic Gradient Descent iteration. On the other hand, when $\rho > 0$, then we get some interesting behaviors:

  • If the gradients $g(i)$ are such that they change sign frequently (as in the steep side of Figure TNN4), then the stepsize $\sum_{i=0}^n \rho^{n-i}g(i)$ will be small. Thus the change in these parameters with the number of iterations will limited.

  • If the gradients $g(i)$ are such that they maintain their sign (as in the shallow portion of Figure TNN4), then the stepsize $\sum_{i=0}^n \rho^{n-i}g(i)$ will be large. This means that if the gradients maintain their sign then the corresponding parameters will take bigger and bigger steps as the algorithm progresses, even though the individual gradients may be small.

In [8]:
#TNN5
nb_setup.images_hconcat(["DL_images/TNN5.png"], width=2000)
Out[8]:

The Momentum algorithm thus accelerates parameter convergence for parameters whose gradients consistently point in the same direction, and slows parameter change for parameters whose gradient changes sign frequently, thus resulting in faster convergence (this is shown on the RHS of Figure TNN4). The variable $v(n)$ is analogous to velocity in a dynamical system, while the parameter $1-\rho$ plays the role of the co-efficient of friction. The value of $\rho$ determines the degree of momentum, with the momentum becoming stronger as $\rho$ approaches $1$. Note that

$$ \sum_{i=0}^{n} \rho^{n-i}g(i) \le {g_{max}\over 1-\rho} $$

$\rho$ is usually set to the neighborhood of $0.9$ and from the above equation it follows that $\sum_{i=0}^n \rho^{n-i}g(i)\approx 10g$ assuming all the $g(i)$ are approximately equation to $g$. Hence the effective gradient in Equation (Wn3) is ten times the value of the actual gradient. This results in an "overshoot" where the value of the parameter shoots past the minimum point to the other side of the bowl, and then reverses itself. This is a desirable behavior since it prevents the algorithm from getting stuck at a saddle point or a local minima, since the momentum carries it out from these areas (see Figure TNN5).

Nesterov Momentum

Nesterov Momentum is a variation on the plain Momentum method described above. Note that the Momentum parameter update equations can be written as:

$$ v(n) = \rho\; v(n-1) - \eta \; g(w(n)) $$$$ w(n+1) = w(n) + v(n) $$

In the first equation we have explicitly written out the fact that the gradient $g$ is computed for parameter value $w(n)$. These equations can be improved by evaluation of the gradient at parameter value $w(n+1)$ instead. This may seem like circular reasoning since in order to compute $w(n+1)$ we first need to compute $g(w(n))$. However note that $w(n+1)\approx w(n) + \rho v(n-1)$. This leads to the velocity update equation for Nesterov Momentum

$$ v(n) = \rho\; v(n-1) - \eta \; g(w(n)+\rho v(n-1)) $$

where $g(w(n)+\rho v(n-1))$ denotes the gradient computed at parameter values $w(n) + \rho v(n-1)$. By using a slightly more accurate estimate of the gradient in each step, it has been observed in practice that the Gradient Descent process speeds up considerably when compared to the plain Momentum method.

The ADAGRAD Algorithm

The Momentum and Nesterov Momentum algorithms help to improve the speed of convergence, however we still have the issue of optimally varying the Learning Rate parameter (see Section LearningRateSelection). It would be nice if this could be done automatically as part of the parameter update equation and this is precisely what the ADAGRAD algorithm does. This algorithm replaces the parameter update rule with the following equation:

\begin{equation} w(n+1) = w(n) - \frac{\eta}{\sqrt{\sum_{i=1}^n g(n)^2+\epsilon}}\; g(n) \quad \quad (**Wng**) \end{equation}

The constant $\epsilon$ has been added to better condition the denominator and is usually set to a small number such $10^{-7}$.

Equation (Wng) leads to the following benefits: Each parameter gets its own adaptive Learning Rate, such that large gradients have smaller learning rates and small gradients have larger learning rates ($\eta$ is usually defaulted to $0.01$). As a result the progress along each dimension evens out over time, which helps the training process. This is a type of Learning Rate annealing, but it is more powerful since:

  • Each parameter gets its own customized rate,
  • The change in rates happens automatically as part of the parameter update equation.

Also, the accumulation of gradients in the denominator leads to smaller Learning Rates over time, which has the same effect as annealing. This is a double edged sword, since the continuous decrease in Learning Rates can lead to a halt of training in large networks that require a greater number of iterations. This problem is addressed by the RMSPROP algorithm, which is described next.

The RMSPROP Algorithm

The RMSPROP algorithm accumulates the sum of gradients using a sliding window, using the following formula:

$$ E[g^2]_n = \rho E[g^2]_{n-1} + (1-\rho) g(n)^2 $$

where $\rho$ is a decay constant (usually set to $0.9$). This operation (called a Low Pass Filter) has a windowing effect, since it forgets gradients that are far back in time. The quantity $RMS[g]_n$ defined by

$$ RMS[g]_n = \sqrt{E[g^2]_n + \epsilon} $$

is used in the denominator of equation (Wng), resulting in following the parameter update equation:

\begin{equation} w(n+1) = w(n) - \frac{\eta}{RMS[g]_n}\; g(n) \quad \quad (**WRMS**) \end{equation}

Note that $$ E[g^2]_n = (1-\rho)\sum_{i=0}^n \rho^{n-i} g(i)^2 \le \frac{g_{max}}{1-\rho} $$ which shows that the parameter $\rho$ prevents the sum from blowing up, and a large value of $\rho$ is equivalent to using a larger window of previous gradients in computing the sum. Hence RMSPROP retains the benefits of ADAGRAD while avoiding the decay of the Learning Rate to zero.

The Adam Algorithm

The Adaptive Moment Estimation (Adam) algorithm combines the best of algorithms such as Momentum that speed up the training process, with algorithms such as RMSPROP that adaptively vary the effective Learning Rate. The update equtions for Adam are as follows:

$$ \Lambda(n) = \beta\Lambda(n-1) +(1-\beta)g(n),\ \ \ {\hat\Gamma}(n) = \frac{\Gamma(n)}{1-\beta^n} $$$$ \Delta(n) = \alpha\Delta(n-1) + (1-\alpha) g(n)^2,\ \ \ {\hat\Delta}(n) = \frac{\Delta(n)}{1-\alpha^n} $$$$ w(n+1) = w(n) - \eta\frac{\hat\Lambda(n)}{\sqrt{\hat\Delta(n) + \epsilon}} $$

The definition of the sequence $\Delta(n)$ is identical to that of $E[g^2]_n$ in the RMSPROP, and it serves an identical purpose, i.e., it is used to customize the effective Learning Rate on a per parameter basis, so that the rates for parameters with larger gradients are equalized with those for parameters with smaller gradients.

The sequence $\Lambda(n)$ is used to provide "Momentum" to the updates, and works in a fashion similar to the velocity sequence $v(n)$ in the Momentum algorithm. It is easy to show that

$$ \Lambda(n) = (1-\rho)\sum_{i=0}^n \rho^{n-i} g(i) \le \frac{g_{max}}{1-\rho} $$

which shows that $\Lambda(n)$ is the weighted sum of the previous $n$ gradients (compare this with the expression for $v(n)$ in Equation (Wn2)). Since $\Lambda(n)$ and $\Delta(n)$ are initialized as vectors of 0s, they are biased towards 0 at the start of the iteration. These biases are counteracted by computing the estimates $\hat\Lambda(n)$ and $\hat\Delta(n)$. The parameters $\alpha$ and $\beta$ are usually defaulted to $10^{-8}$ and $0.999$ respectively.

Adam serves as the default choice for the parameter update rule, since it combines the best features of the other update algorithms.

Specifying Optimizers in Keras

Optimizers can either be instantiated before being invoked as in:

model = Sequential()
model.add(Dense(64, kernel_initializer='uniform', input_shape=(10,)))
model.add(Activation('softmax'))

sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='mean_squared_error', optimizer=sgd)

Or they can be called by name, in which case default parameters are used, as in:

model.compile(loss='mean_squared_error', optimizer='sgd')

The following optimizers are available in Keras:

# Stochastic Gradient Descent
keras.optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False)

# RMSProp
keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)

#AdaGrad
keras.optimizers.Adagrad(learning_rate=0.01)

#Adadelta
keras.optimizers.Adadelta(learning_rate=1.0, rho=0.95)

#Adam
keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)

If any of the optimizers are invoked by name, then Keras supplied default values for all the relevant parameters, which usually work quite well in practice.

Keras has added a new feature called the KerasTuner, which as the name implies, can be used to automatically tune and find the best parameters. More information about the KerasTuner, along with a short tutorial, can be found at the following webpages: https://www.tensorflow.org/tutorials/keras/keras_tuner, https://keras.io/keras_tuner/.

Choice of Activation Functions

In [16]:
#AF1
nb_setup.images_hconcat(["DL_images/AF1.png"], width=600)
Out[16]:

The choice of Activation Function has a major influence on the training and operation of DLN systems. When DLNs were first proposed, the first choice for the activation was the sigmoid, probably as a result of what was then known about biological neurons. This turned out to be an unfortunate choice, as illustrated in Figure AF1. This picture shows a single neuron with a sigmoid activation. Using the Gradient Flow rules from Chapter TrainingNNsBackprop, we can see that the backpropagation of the gradient $\frac{\partial\mathcal L}{\partial z}$ results in

$$ \frac{\partial\mathcal L}{\partial a} = z(1-z)\frac{\partial\mathcal L}{\partial z} $$

If the neuron is saturated, i.e., $a$ lies significantly away from the origin, then from the shape of the sigmoid it follows that either $z$ or $(1-z)$ is zero, which implies that $\frac{\partial\mathcal L}{\partial a} \approx 0$. As a result the gradients flowing back to the next layer of nodes will also be zero. The neuron in this state is said to be "dead". Once a neuron is dead, it stays dead, since in order to get get it back into the active state, the inputs weights (shown on the LHS of Figure AF1) need to change. However the weights cannot change since the gradient with respect to the weights is given by $\frac{\partial\mathcal L}{\partial w} = z'\delta$ and $\delta = 0$.

Thus the choice of the sigmoid function contributed to the Vanishing Gradient problem that plagued the first DLN systems. Interestingly enough, a suitable replacement for the sigmoid (the ReLU function) was not proposed until 2010. But once it was in place, it contributed to the rapid advances in the field since then. Our objective in this section is to survey some of the many Activation Functions that have been proposed in the last few years.

The tanh Function

In [17]:
#AF2
nb_setup.images_hconcat(["DL_images/AF2.png"], width=600)
Out[17]: