16.1. Net input#

16.2. Examples of Different Types of Neurons#

16.3. ReLU (restricted linear unit)#

16.4. TanH (hyperbolic tangent)#

16.5. Output Layer#

Image("NLP_images/Softmax.png", width=700)

#The Softmax function
#Assume 10 output nodes with randomly generated values
z = randn(32)  #inputs from last hidden layer of 32 nodes to the output layer
w = rand(32*10).reshape((10,32))  #weights for the output layer
b = rand(10)   #bias terms at output later
a = w.dot(z) + b  #Net input at output layer
e = exp(a)
softmax_output = e/sum(e)
print(softmax_output.round(3))
print('final tag =',where(softmax_output==softmax_output.max())[0][0])

[0.057 0.075 0.612 0.004 0.006 0.002 0.069 0.037 0.036 0.104]
final tag = 2

Image("NLP_images/Loss_function.png", width=600)

16.6. More on Cross Entropy#

https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/

Notation from the previous slides:

• $y_i$: actual probability of the correct class $i$, i.e., $1$ or $0$.

• $a_i$ or $\widehat{y_i}$: predicted probability of the correct class.

• $n$: number of observations in the data.

Log Loss is a special case for binary outcomes of Cross Entropy; pdf

16.7. Entropy#

Entropy is a measure of disorder in the universe. It has taken on many flavors and interpretations from origins in physics to information theory and now AI and ML. A great introduction to entropy is in a recent article: https://www.quantamagazine.org/what-is-entropy-a-measure-of-just-how-little-we-really-know-20241213/

y = [0.33, 0.33, 0.34]
bits = log2(y)
print(bits)
entropy = -sum(y*bits)
print(entropy)

[-1.59946207 -1.59946207 -1.55639335]
1.58481870497303

y = [0.2, 0.3, 0.5]
entropy = -sum(y*log2(y))
print(entropy)

1.4854752972273344

y = [0.1, 0.1, 0.8]
print(log2(y))
entropy = -sum(y*log2(y))
print(entropy)

[-3.32192809 -3.32192809 -0.32192809]
0.9219280948873623

16.8. Cross-entropy:#

where $a_i={\hat{y}}_i$.

Note that $C>E$, always.

#Correct prediction
y = [0, 0, 1]
yhat = [0.1, 0.1, 0.8]
crossentropy = -sum(y*log2(yhat))
print(crossentropy)

0.3219280948873623

#Wrong prediction
yhat = [0.1, 0.6, 0.3]
crossentropy = -sum(y*log2(yhat))
print(crossentropy)

1.7369655941662063

#Correct prediction
y = [0, 0, 1.0]
yhat = [0.1, 0.1, 0.8]
KL = sum(y[2]*log2(y[2]/yhat[2]))
print(KL)

#Wrong prediction
yhat = [0.1, 0.6, 0.3]
KL = sum(y[2]*log2(y[2]/yhat[2]))
print(KL)

0.32192809488736235
1.7369655941662063

16.9. Mutual Information (MI)#

Another popular distance measure is MI. The idea is that for two random variables $X$ and $Y$, MI quantifies how much information about any random variable (rv) can be gleaned from the other rv.

• One way to think about this is to note that linear correlation (or regression) quantifies the relationship of two rvs. MI aims to do the same by looking at the distance between the joint statistical distribution of $X,Y$ and the product of the marginals. Clearly, this is nonlinear, unlike correlation.

• MI is related to entropy because the distance between $X$ and $Y$ distributions is quantified using Kullback-Leibler (KL) distance, which is based on Shannon entropy. The following defines MI as the KL divergence between the joint distribution and the product of the marginals for $X,Y$.

• Opening this up using the formula for KL divergence, we get

16.10. Fitting the DL NN#

Image("NLP_images/Gradient_descent.png", width=600)

Image("NLP_images/Chain_rule.png", width=600)

Image("NLP_images/Delta_values.png", width=600)

Image("NLP_images/Output_layer.png", width=600)

Image("NLP_images/Feedforward_Backprop.png", width=600)

Image("NLP_images/Recap.png", width=600)

Image("NLP_images/Backprop_one_slide.png", width=600)

16.11. SoftMax Function#

16.12. Delta of Softmax#

Given that $\frac{\partial D}{\partial a_j}=e^{a_j}$,

16.13. Batch Stochastic Gradient#

Image("NLP_images/batch_stochastic_gradient.png", width=700)

16.14. Fitting the NN#

1. Initialize all the weight and bias parameters $(w_{ij}^{(r)},b_i^{(r)})$ (this is a critical step).

2. For $q=0,...,\frac{M}{B}-1$, repeat the following steps (2a) - (2f):

a. For the training inputs $X_i(m),qB\le m\le (q+1)B$, compute the model predictions $y(m)$ given by

and for $r=1$,

The logits and classification probabilities are computed using

and

This step constitutes the forward pass of the algorithm.

b. Evaluate the gradients ${\delta }_k^{(R+1)}(m)$ for the logit layer nodes using

This step and the following one constitute the start of the backward pass of the algorithm, in which we compute the gradients ${\delta }_k^{(r)},1\le k\le K,1\le r\le R$ for all the hidden nodes.

c. Back-propagate the $\delta$s using the following equation to obtain the ${\delta }_j^{(r)}(m),1\le r\le R,1\le j\le P^r$ for each hidden node in the network.

d. Compute the gradients of the Cross Entropy Function $\mathcal{L}(m)$ for the $m$-th training vector $(X(m),T(m))$ with respect to all the weight and bias parameters using the following equation.

and

e. Change the model weights according to the formula

f. Increment $q\leftarrow ((q+1)\mathrm{mod}B)$, and go back to step $(a)$.

Compute the Loss Function $L$ over the Validation Dataset given by

If $L$ has dropped below some threshold, then stop. Otherwise go back to Step 2.

16.15. Gradient Descent Example#

Image("NLP_images/Gradient_Descent_Scheme.png", width=600)

def f(x):
    return 3*x**2 -5*x + 10

x = linspace(-4,4,100)
plot(x,f(x))
grid()

dx = 0.001
eta = 0.05  #learning rate
x = -3
for j in range(20):
    df_dx = (f(x+dx)-f(x))/dx
    x = x - eta*df_dx
    print(x,f(x))

-1.8501500000001698 29.519915067502733
-1.0452550000002532 18.503949045077853
-0.4818285000003115 13.105618610239208
-0.08742995000019249 10.460081738472072
0.18864903499989083 9.163520200219716
0.3819043244999847 8.528031116715445
0.5171830271499616 8.216519714966186
0.6118781190049631 8.06379390252634
0.6781646833034642 7.988898596522943
0.7245652783124417 7.95215813604575
0.7570456948186948 7.934126078037087
0.7797819863730542 7.925269906950447
0.7956973904611502 7.920916059254301
0.8068381733227685 7.918772647178622
0.8146367213259147 7.917715356568335
0.8200957049281836 7.917192371084045
0.8239169934497053 7.916932669037079
0.8265918954147882 7.9168030076222955
0.8284643267903373 7.916737788340814
0.8297750287532573 7.916704651261121

16.16. Vanishing Gradients#

• In large problems, gradients simply vanish too soon before training has reached an acceptable level of accuracy.

• There are several issues with gradient descent that need handling, and to solve this, there are several fixes that may be applied.

• Learning rate $\eta$ may be too large or too small.

Image("NLP_images/LearningRate_Matters.png", width=600)

16.17. Gradients in Multiple Dimensions#

Image("NLP_images/GD_MultipleDimensions.png", width=600)

16.18. Saddle Points#

Image("NLP_images/GD_saddle.png", width=400)

16.19. Effect of the Learning Rate#

The idea is to start with a high learning rate and then adaptively reduce it as we get closer to the minimum of the loss function.

Image("NLP_images/LearningRateAnnealing.png", width=500)

16.20. Annealing#

Adjust the learning rate as a step function when the reduction in the loss function begins to plateau.

Image("NLP_images/LearningRateAnnealing2.png", width=500)

16.21. Learning Rate Algorithms#

• Improve the speed of convergence (for example the Momentum, Nesterov Momentum, and Adam algorithms).

• Adapt the effective Learning Rate as the training progresses (for example the ADAGRAD, RMSPROP and Adam algorithms).

16.22. Momentum#

Fixes the problem where the gradient is multidimensional and has a fast gradient on some axes and a slow one on the others.

At the end of the $n^{th}$ iteration of the Backprop algorithm, define a sequence $v(n)$ by

where $\rho$ is new hyper-parameter called the “momentum” parameter, and $g(n)$ is the gradient evaluated at parameters value $w(n)$.

$g(n)$ is defined by

for Stochastic Gradient Descent and

for Batch Stochastic Gradient Descent (note that in this case $n$ is an index into the batch number).

The change in parameter values on each iteration is now defined as

It can be shown from these equations that $v(n)$ can be written as

so that

16.23. Behavior of Momentum Parameter#

When the momentum parameter $\rho =0$, then this equation reduces to the usual Stochastic Gradient Descent iteration. On the other hand, when $\rho >0$, then we get some interesting behaviors:

• If the gradients $g(i)$ are such that they change sign frequently (as in the steep side of the loss surface), then the stepsize $\sum _{i=0}^n{\rho }^{n-i}g(i)$ will be small. Thus the change in these parameters with the number of iterations will limited.

• If the gradients $g(i)$ are such that they maintain their sign (as in the shallow portion of the loss surface), then the stepsize $\sum _{i=0}^n{\rho }^{n-i}g(i)$ will be large. This means that if the gradients maintain their sign then the corresponding parameters will take bigger and bigger steps as the algorithm progresses, even though the individual gradients may be small.

16.24. Properties of the Momentum algorithm#

• The Momentum algorithm thus accelerates parameter convergence for parameters whose gradients consistently point in the same direction, and slows parameter change for parameters whose gradient changes sign frequently, thus resulting in faster convergence.

• The variable $v(n)$ is analogous to velocity in a dynamical system, while the parameter $(1-\rho )$ plays the role of the co-efficient of friction.

• The value of $\rho$ determines the degree of momentum, with the momentum becoming stronger as $\rho$ approaches $1$.

Note that

$\rho$ is usually set to the neighborhood of $0.9$ and from the above equation it follows that $\sum _{i=0}^n{\rho }^{n-i}g(i)\approx 10g$ assuming all the $g(i)$ are approximately equal to $g$. Hence the effective gradient is ten times the value of the actual gradient. This results in an “overshoot” where the value of the parameter shoots past the minimum point to the other side of the bowl, and then reverses itself. This is a desirable behavior since it prevents the algorithm from getting stuck at a saddle point or a local minima, since the momentum carries it out from these areas.

16.25. Nesterov Momentum#

1. Recall that the Momentum parameter update equations can be written as: $$v(n)=\rho v(n-1)-\eta g(w(n))$$$$w(n+1)=w(n)+v(n)$$

2. These equations can be improved by evaluation of the gradient at parameter value $w(n+1)$ instead.

Circular? in order to compute $w(n+1)$ we first need to compute $g(w(n))$.

4. Gives the velocity update equation for Nesterov Momentum

where $g(w(n)+\rho v(n-1))$ denotes the gradient computed at parameter values $w(n)+\rho v(n-1)$.

5. Gradient Descent process speeds up considerably when compared to the plain Momentum method.

16.26. The ADAGRAD Algorithm#

1. Parameter update rule: $$w(n+1)=w(n)-\frac{\eta }{\sqrt{\sum _{i=1}^{n}g(n{)}^2+ϵ}}g(n)$$

2. Constant $ϵ$ has been added to better condition the denominator and is usually set to a small number such ${10}^{-7}$.

3. Each parameter gets its own adaptive Learning Rate, such that large gradients have smaller learning rates and small gradients have larger learning rates ($\eta$ is usually defaulted to $0.01$). As a result the progress along each dimension evens out over time, which helps the training process.

4. The change in rates happens automatically as part of the parameter update equation.

5. Downside: accumulation of gradients in the denominator leads to the continuous decrease in Learning Rates which can lead to a halt of training in large networks that require a greater number of iterations.

16.27. The RMSPROP Algorithm#

1. Accumulates the sum of gradients using a sliding window: $$E[g^2{]}_n=\rho E[g^2{]}_{n-1}+(1-\rho )g(n{)}^2$$where$\rho$isadecayconstant(usuallysetto$0.9$). This operation (called a Low Pass Filter) has a windowing effect, since it forgets gradients that are far back in time.

2. The quantity $RMS[g{]}_n$ defined by (RMS = Root Mean Square) $$RMS[g{]}_n=\sqrt{E[g^2{]}_n+ϵ}$$ \begin{equation} w(n+1) = w(n) - \frac{\eta}{RMS[g]_n}; g(n)
   \end{equation}

Note that $$E[g^2{]}_n=(1-\rho )\sum _{i=0}^n{\rho }^{n-i}g(i{)}^2\le \frac{g_{max}}{1-\rho }$$whichshowsthattheparameter$\rho$preventsthesumfromblowingup,andalargevalueof$\rho$ is equivalent to using a larger window of previous gradients in computing the sum.

16.28. The ADAM Algorithm#

1. The Adaptive Moment Estimation (Adam) algorithm combines the best of algorithms such as Momentum that speed up the training process, with algorithms such as RMSPROP that adaptively vary the effective Learning Rate.

2. $\mathrm{\Delta }(n)$ is identical to that of $E[g^2{]}_n$ in the RMSPROP, and it serves an identical purpose, i.e., rates for parameters with larger gradients are equalized with those for parameters with smaller gradients.

3. The sequence $\mathrm{\Lambda }(n)$ is used to provide “Momentum” to the updates, and works in a fashion similar to the velocity sequence $v(n)$ in the Momentum algorithm.

4. The parameters $\alpha$ and $\beta$ are usually defaulted to ${10}^{-8}$ and $0.999$ respectively.

For a terrific collection of all the variations of gradient descent, see https://ruder.io/optimizing-gradient-descent/

16.29. Back to Activation Functions (Vanishing Gradients)#

Image("NLP_images/sigmoid_activation.png", width=700)

16.30. tanh Activation#

Image("NLP_images/tanh_activation.png", width=500)

1. Unless the input is in the neghborhood of zero, the function enters its saturated regime.

2. It is superior to the sigmoid in one respect, i.e., its output is zero centered. This speeds up the training process.

3. The $\tanh$ function is rarely used in modern DLNs, the exception being a type DLN called LSTM.

16.31. ReLU Activation#

Image("NLP_images/relu_activation.png", width=400)

1. No saturation problem.

2. Gradients $\frac{\partial L}{\partial w}$ propagate undiminished through the network, provided all the nodes are active.

16.32. Dead ReLU Problem#

Image("NLP_images/dead_relu.png", width=400)

1. The dotted line in this figure shows a case in which the weight parameters $w_i$ are such that the hyperplane $\sum w_i{z}_i$ does not intersect the “data cloud” of possible input activations. There does not exist any possible input values that can lead to $\sum w_i{z}_i>0$. The neuron’s output activation will always be zero, and it will kill all gradients backpropagating down from higher layers.

2. Vary initialization to correct this.

16.33. Leaky ReLU#

Image("NLP_images/leaky_relu.png", width=600)

• Fixes the dead ReLU problem.

16.34. PreLU#

Image("NLP_images/prelu.png", width=500)

• Instead of deciding on the value of $c$ through experimentation, why not determine it using Backpropagation as well. This is the idea behind the Pre-ReLU or PReLU function.

Note that each neuron $i$ now has its own parameter ${\beta }_i,1\le i\le S$, where $S$ is the number of nodes in the network. These parameters are iteratively estimated using Backprop.

Substituting the value for $\frac{\partial z_i}{\partial {\beta }_i}$ we obtain $$\frac{\partial L}{\partial {\beta }_i}=a_i\frac{\partial L}{\partial z_i}if{a}_i\le 0\text{and}0\text{otherwise}$$

which is then used to update ${\beta }_i$ using ${\beta }_i\to {\beta }_i-\eta \frac{\partial L}{\partial {\beta }_i}$.

Once training is complete, the PreLU based DLN network ends up with a different value of ${\beta }_i$ at each neuron, which increases the flexibility of the network at the cost of an increase in the number of parameters.

16.35. Maxout#

Image("NLP_images/maxout.png", width=600)

Generalizes Leaky ReLU.

We may allow the two hyperplanes to be independent with their own set of parameters, as shown in the Figure above.

16.36. Initializing Weights#

In practice, the DLN weight parameters are initialized with random values drawn from Gaussian or Uniform distributions and the following rules are used:

• Guassian Initialization: If the weight is between layers with $n_{in}$ input neurons and $n_{out}$ output neurons, then they are initialized using a Gaussian random distribution with mean zero and standard deviation $\sqrt{\frac{2}{n_{in}+n_{out}}}$.

• Uniform Initialization: In the same configuration as above, the weights should be initialized using an Uniform distribution between $-r$ and $r$, where $r=\sqrt{\frac{6}{n_{in}+n_{out}}}$.

When using the ReLU or its variants, these rules have to be modified slightly:

• Guassian Initialization: If the weight is between layers with $n_{in}$ input neurons and $n_{out}$ output neurons, then they are initialized using a Gaussian random distribution with mean zero and standard deviation $\sqrt{\frac{4}{n_{in}+n_{out}}}$.

• Uniform Initialization: In the same configuration as above, the weights should be initialized using an Uniform distribution between $-r$ and $r$, where $r=\sqrt{\frac{12}{n_{in}+n_{out}}}$.

The reasoning behind scaling down the initialization values as the number of incident weights increases is to prevent saturation of the node activations during the forward pass of the Backprop algorithm, as well as large values of the gradients during backward pass.

16.37. Data Preprocessing#

Image("NLP_images/data_preprocessing.png", width=600)

Centering: This is also sometimes called Mean Subtraction, and is the most common form of preprocessing. Given an input dataset consisting of $M$ vectors $X(m)=(x_1(m),...,x_N(m)),m=1,...,M$, it consists of subtracting the mean across each individual input component $x_i,1\le i\le N$ such that $$x_i(m)\leftarrow x_i(m)-\frac{\sum _{s=1}^M{x}_i(s)}{M},1\le i\le N,1\le m\le M$$

Scaling: After the data has been centered, it can be scaled in one of two ways:

• By dividing by the standard deviation, once again along each dimension, so that the overall transform is

• By Normalizing each dimension so that the min and max along each axis are -1 and +1 repectively.

• In general Scaling helps optimization because it balances out the rate at which the weights connected to the input nodes learn.

16.38. Zero-Centering#

Recall that for a K-ary Linear Classifier, the parameter update equation is given by:

If the training sample is such that $t_q=1$ and $t_k=0,j\ne q$, then the update becomes:

and

Lets assume that the input data is not centered so that $x_j\ge 0,j=1,...,N$. Since $0\le y_k\le 1$ it follows that

and

i.e. the update results in all the weights moving in the same direction, except for one. This is shown graphically in the Figure above, in which the system is trying move in the direction of the blue arrow which is the quickest path to the minimum. However if the input data is not centered, then it is forced to move in a zig-zag fashion as shown in the red-curve. The zig-zag motion is caused due to the fact that all the parameters move in the same direction at each step due to the lact of zero-centering in the input data.

16.39. Zero Centering Helps#

Image("NLP_images/zero_centering_helps.png", width=400)

16.40. Batch Normalization#

Normalization applied to the hidden layers.

https://towardsdatascience.com/batch-norm-explained-visually-how-it-works-and-why-neural-networks-need-it-b18919692739

Image("NLP_images/batch_normalization.png", width=600)

• Higher learning rates: In a non-normalized network, a large learning rate can lead to oscillations and cause the loss function increase rather than decrease.

• Better Gradient Propagation through the network, enabling DLNs with more hidden layers.

• Reduces strong dependencies on the parameter initialization values.

• Helps to regularize the model.

16.41. Under and Over-fitting#

Image("NLP_images/underoverfitting.png", width=600)

16.42. Regularization#

• Early Stopping

• L1 Regularization

• L2 Regularization

• Dropout Regularization

• Training Data Augmentation

• Batch Normalization

16.43. Early Stopping#

Image("NLP_images/early_stopping.png", width=600)

16.44. L2 Regularization#

L2 Regularization is a commonly used technique in ML systems is also sometimes referred to as “Weight Decay”. It works by adding a quadratic term to the Cross Entropy Loss Function $L$, called the Regularization Term, which results in a new Loss Function $L_R$ given by:

\begin{equation} L_R = {L} + \frac{\lambda}{2} \sum_{r=1}^{R+1} \sum_{j=1}^{P^{r-1}} \sum_{i=1}^{P^r} (w_{ij}^{(r)})^2
\end{equation}

L2 Regularization also leads to more “diffuse” weight parameters, in other words, it encourages the network to use all its inputs a little rather than some of its inputs a lot.

16.45. L1 Regularization#

L1 Regularization uses a Regularization Function which is the sum of the absolute value of all the weights in DLN, resulting in the following loss function ($L$ is the usual Cross Entropy loss):

At a high level L1 Regularization is similar to L2 Regularization since it leads to smaller weights.

1. Both L1 and L2 Regularizations lead to a reduction in the weights with each iteration. However the way the weights drop is different:

2. In L2 Regularization the weight reduction is multiplicative and proportional to the value of the weight, so it is faster for large weights and de-accelerates as the weights get smaller.

3. In L1 Regularization on the other hand, the weights are reduced by a fixed amount in every iteration, irrespective of the value of the weight. Hence for larger weights L2 Regularization is faster than L1, while for smaller weights the reverse is true.

4. As a result L1 Regularization leads to DLNs in which the weight of most of the connections tends towards zero, with a few connections with larger weights left over. This type of DLN that results after the application of L1 Regularization is said to be “sparse”.

16.46. Dropout regularization#

Image("NLP_images/dropout.png", width=600)

The basic idea behind Dropout is to run each iteration of the Backprop algorithm on randomly modified versions of the original DLN. The random modifications are carried out to the topology of the DLN using the following rules:

• Assign probability values $p^{(r)},0\le r\le R$, which is defined as the probability that a node is present in the model, and use these to generate $\{0,1\}$-valued Bernoulli random variables $e_j^{(r)}$:

• Modify the input vector as follows:

\begin{equation} \hat x_j = e_j^{(0)} x_j, \quad 1 \leq j \leq N
\end{equation}

• Modify the activations $z_j^{(r)}$ of the hidden layer r as follows:

\begin{equation} \hat z_j^{(r)} = e_j^{(r)} z_j^{(r)}, \quad 1 \leq r \leq R,\ \ 1 \leq j \leq P^r
\end{equation}

1. After the Backprop is complete, we have effectively trained a collection of up to $2^s$ thinned DLNs all of which share the same weights, where $s$ is the total number of hidden nodes in the DLN.

2. In order to test the network, strictly speaking we should be averaging the results from all these thinned models, however a simple approximate averaging method works quite well.

3. The main idea is to use the complete DLN as the test network.

16.47. Bagging (Ensemble Learning)#

Image("NLP_images/Bagging.png", width=500)

16.48. TensorFlow Playground#

Image("NLP_images/TensorFlow_playground.png", width=600)

http://playground.tensorflow.org/

16.49. Pattern Recognition: Cancer#

%pylab inline
import pandas as pd

Populating the interactive namespace from numpy and matplotlib

/usr/local/lib/python3.11/dist-packages/IPython/core/magics/pylab.py:159: UserWarning: pylab import has clobbered these variables: ['e', 'f']
`%matplotlib` prevents importing * from pylab and numpy
  warn("pylab import has clobbered these variables: %s"  % clobbered +

## Read in the data set
data = pd.read_csv("NLP_data/BreastCancer.csv")
data.head()

x = data.loc[:,'Cl.thickness':'Mitoses']
print(x.head())
y = data.loc[:,'Class']
print(y.head())

   Cl.thickness  Cell.size  Cell.shape  Marg.adhesion  Epith.c.size  \
0             5          1           1              1             2   
1             5          4           4              5             7   
2             3          1           1              1             2   
3             6          8           8              1             3   
4             4          1           1              3             2   

   Bare.nuclei  Bl.cromatin  Normal.nucleoli  Mitoses  
0            1            3                1        1  
1           10            3                2        1  
2            2            3                1        1  
3            4            3                7        1  
4            1            3                1        1  
0    benign
1    benign
2    benign
3    benign
4    benign
Name: Class, dtype: object

16.50. One-Hot Encoding#

## Convert the class variable into binary numeric
ynum = zeros((len(x),1))
for j in arange(len(y)):
    if y[j]=="malignant":
        ynum[j]=1
ynum[:10]

array([[0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [1.],
       [0.],
       [0.],
       [0.],
       [0.]])

! conda install -c conda-forge keras -y
! conda install -c conda-forge tensorflow -y

/bin/bash: line 1: conda: command not found
/bin/bash: line 1: conda: command not found

## Make label data have 1-shape, 1=malignant
from keras.utils import to_categorical
y.labels = to_categorical(ynum, num_classes=2)
#x = x.as_matrix()
print(y.labels[:10])
print(shape(x))
print(shape(y.labels))
print(shape(ynum))

[[1. 0.]
 [1. 0.]
 [1. 0.]
 [1. 0.]
 [1. 0.]
 [0. 1.]
 [1. 0.]
 [1. 0.]
 [1. 0.]
 [1. 0.]]
(683, 9)
(683, 2)
(683, 1)

16.51. Keras to define DLN#

## Define the neural net and compile it
from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=9))
model.add(Dense(32, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

16.52. Train the Model#

## Fit/train the model (x,y need to be matrices)
model.fit(x, ynum, epochs=25, batch_size=32, verbose=2, validation_split=0.3)

Epoch 1/25
15/15 - 3s - 219ms/step - accuracy: 0.6569 - loss: 0.5819 - val_accuracy: 0.8098 - val_loss: 0.5369
Epoch 2/25
15/15 - 0s - 7ms/step - accuracy: 0.8703 - loss: 0.4287 - val_accuracy: 0.9463 - val_loss: 0.3267
Epoch 3/25
15/15 - 0s - 10ms/step - accuracy: 0.8766 - loss: 0.3622 - val_accuracy: 0.9415 - val_loss: 0.2815
Epoch 4/25
15/15 - 0s - 10ms/step - accuracy: 0.8996 - loss: 0.3258 - val_accuracy: 0.9512 - val_loss: 0.2461
Epoch 5/25
15/15 - 0s - 7ms/step - accuracy: 0.8912 - loss: 0.2946 - val_accuracy: 0.9512 - val_loss: 0.2230
Epoch 6/25
15/15 - 0s - 7ms/step - accuracy: 0.9142 - loss: 0.2702 - val_accuracy: 0.9512 - val_loss: 0.1833
Epoch 7/25
15/15 - 0s - 9ms/step - accuracy: 0.9247 - loss: 0.2512 - val_accuracy: 0.9561 - val_loss: 0.1677
Epoch 8/25
15/15 - 0s - 18ms/step - accuracy: 0.9247 - loss: 0.2346 - val_accuracy: 0.9707 - val_loss: 0.1535
Epoch 9/25
15/15 - 0s - 9ms/step - accuracy: 0.9331 - loss: 0.2172 - val_accuracy: 0.9463 - val_loss: 0.1531
Epoch 10/25
15/15 - 0s - 8ms/step - accuracy: 0.9289 - loss: 0.2101 - val_accuracy: 0.9659 - val_loss: 0.1413
Epoch 11/25
15/15 - 0s - 7ms/step - accuracy: 0.9477 - loss: 0.1885 - val_accuracy: 0.9659 - val_loss: 0.1374
Epoch 12/25
15/15 - 0s - 10ms/step - accuracy: 0.9477 - loss: 0.1753 - val_accuracy: 0.9707 - val_loss: 0.1208
Epoch 13/25
15/15 - 0s - 7ms/step - accuracy: 0.9456 - loss: 0.1675 - val_accuracy: 0.9805 - val_loss: 0.1065
Epoch 14/25
15/15 - 0s - 9ms/step - accuracy: 0.9519 - loss: 0.1606 - val_accuracy: 0.9805 - val_loss: 0.0990
Epoch 15/25
15/15 - 0s - 9ms/step - accuracy: 0.9582 - loss: 0.1538 - val_accuracy: 0.9805 - val_loss: 0.0975
Epoch 16/25
15/15 - 0s - 9ms/step - accuracy: 0.9540 - loss: 0.1421 - val_accuracy: 0.9854 - val_loss: 0.0925
Epoch 17/25
15/15 - 0s - 9ms/step - accuracy: 0.9644 - loss: 0.1369 - val_accuracy: 0.9756 - val_loss: 0.0816
Epoch 18/25
15/15 - 0s - 7ms/step - accuracy: 0.9707 - loss: 0.1357 - val_accuracy: 0.9707 - val_loss: 0.0884
Epoch 19/25
15/15 - 0s - 9ms/step - accuracy: 0.9686 - loss: 0.1207 - val_accuracy: 0.9659 - val_loss: 0.0976
Epoch 20/25
15/15 - 0s - 7ms/step - accuracy: 0.9644 - loss: 0.1194 - val_accuracy: 0.9756 - val_loss: 0.0826
Epoch 21/25
15/15 - 0s - 7ms/step - accuracy: 0.9686 - loss: 0.1159 - val_accuracy: 0.9756 - val_loss: 0.0771
Epoch 22/25
15/15 - 0s - 7ms/step - accuracy: 0.9707 - loss: 0.1120 - val_accuracy: 0.9756 - val_loss: 0.0770
Epoch 23/25
15/15 - 0s - 9ms/step - accuracy: 0.9707 - loss: 0.1060 - val_accuracy: 0.9756 - val_loss: 0.0738
Epoch 24/25
15/15 - 0s - 9ms/step - accuracy: 0.9686 - loss: 0.0975 - val_accuracy: 0.9756 - val_loss: 0.0629
Epoch 25/25
15/15 - 0s - 10ms/step - accuracy: 0.9728 - loss: 0.0967 - val_accuracy: 0.9756 - val_loss: 0.0603

<keras.src.callbacks.history.History at 0x7adbe3172750>

## Accuracy
yhat = model.predict(x, batch_size=32).round(0).astype(int).T[0]
ynum = ynum.T[0]
acc = sum(yhat==ynum)
print("Accuracy = ",acc/len(ynum))

## Confusion matrix
from sklearn.metrics import confusion_matrix
confusion_matrix(yhat,ynum)

22/22 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step
Accuracy =  0.9736456808199122

array([[429,   3],
       [ 15, 236]])

16.53. Another Canonical Example: Digit Recognition (MNIST)#

• Extensible to many finance prediction problems.

• Information set: 784 (28 x 28) pixels for category prediction.

• Would you run a multinomial regression on these 784 columns.

https://www.cs.toronto.edu/~kriz/cifar.html

Image("NLP_images/MNIST.png", width=800)

## Read in the data set
train = pd.read_csv("NLP_data/train.csv", header=None)
test = pd.read_csv("NLP_data/test.csv", header=None)
print(shape(train))
print(shape(test))

(60000, 785)
(10000, 785)

## Reformat the data
X_train = train.loc[:,:783]
Y_train = train.loc[:,784]
print(shape(X_train))
print(shape(Y_train))
X_test = test.loc[:,:783]
Y_test = test.loc[:,784]
print(shape(X_test))
print(shape(Y_test))
y.labels = to_categorical(Y_train, num_classes=10)
print(shape(y.labels))
print(y.labels[1:5,:])
print(Y_train[1:5])

(60000, 784)
(60000,)
(10000, 784)
(10000,)
(60000, 10)
[[0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]]
1    3
2    0
3    0
4    2
Name: 784, dtype: int64

## Define the neural net and compile it
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
# from keras.optimizers import SGD
from tensorflow.keras.utils import plot_model

data_dim = shape(X_train)[1]

model = Sequential([
    Dense(100, input_shape=(784,)),
    Activation('sigmoid'),
    Dense(100),
    Activation('sigmoid'),
    Dense(100),
    Activation('sigmoid'),
    Dense(100),
    Activation('sigmoid'),
    Dense(10),
    Activation('softmax'),
])

#model = Sequential()
#model.add(Dense(100, activation='sigmoid', input_dim=data_dim))
#model.add(Dropout(0.25))
#model.add(Dense(100, activation='sigmoid'))
#model.add(Dropout(0.25))
#model.add(Dense(100, activation='sigmoid'))
#model.add(Dropout(0.25))
#model.add(Dense(100, activation='sigmoid'))
#model.add(Dropout(0.25))
#model.add(Dense(10, activation='softmax'))

model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.summary()

/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

Model: "sequential_1"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense_4 (Dense)                      │ (None, 100)                 │          78,500 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ activation (Activation)              │ (None, 100)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_5 (Dense)                      │ (None, 100)                 │          10,100 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ activation_1 (Activation)            │ (None, 100)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_6 (Dense)                      │ (None, 100)                 │          10,100 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ activation_2 (Activation)            │ (None, 100)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_7 (Dense)                      │ (None, 100)                 │          10,100 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ activation_3 (Activation)            │ (None, 100)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_8 (Dense)                      │ (None, 10)                  │           1,010 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ activation_4 (Activation)            │ (None, 10)                  │               0 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 109,810 (428.95 KB)

 Trainable params: 109,810 (428.95 KB)

 Non-trainable params: 0 (0.00 B)

%%capture
!pip install pydot
!pip install graphviz

plot_model(model)

## Fit/train the model (x,y need to be matrices)
model.fit(X_train, y.labels, epochs=10, batch_size=32, verbose=2, validation_split=0.2)

Epoch 1/10
1500/1500 - 15s - 10ms/step - accuracy: 0.7166 - loss: 0.8949 - val_accuracy: 0.8577 - val_loss: 0.4873
Epoch 2/10
1500/1500 - 10s - 7ms/step - accuracy: 0.8733 - loss: 0.4287 - val_accuracy: 0.8904 - val_loss: 0.3820
Epoch 3/10
1500/1500 - 14s - 10ms/step - accuracy: 0.8944 - loss: 0.3581 - val_accuracy: 0.9078 - val_loss: 0.3181
Epoch 4/10
1500/1500 - 6s - 4ms/step - accuracy: 0.9140 - loss: 0.2899 - val_accuracy: 0.9189 - val_loss: 0.2722
Epoch 5/10
1500/1500 - 5s - 3ms/step - accuracy: 0.9235 - loss: 0.2557 - val_accuracy: 0.9259 - val_loss: 0.2508
Epoch 6/10
1500/1500 - 4s - 3ms/step - accuracy: 0.9274 - loss: 0.2403 - val_accuracy: 0.9276 - val_loss: 0.2445
Epoch 7/10
1500/1500 - 5s - 3ms/step - accuracy: 0.9310 - loss: 0.2250 - val_accuracy: 0.9283 - val_loss: 0.2394
Epoch 8/10
1500/1500 - 5s - 3ms/step - accuracy: 0.9350 - loss: 0.2142 - val_accuracy: 0.9326 - val_loss: 0.2275
Epoch 9/10
1500/1500 - 4s - 3ms/step - accuracy: 0.9371 - loss: 0.2035 - val_accuracy: 0.9367 - val_loss: 0.2161
Epoch 10/10
1500/1500 - 6s - 4ms/step - accuracy: 0.9402 - loss: 0.1972 - val_accuracy: 0.9388 - val_loss: 0.2056

<keras.src.callbacks.history.History at 0x7adbe013af50>

## In Sample
yhat = model.predict(X_train, batch_size=32)
yhat = [argmax(j) for j in yhat]

## Confusion matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(yhat,Y_train)
print(" ")
print(cm)

##
acc = sum(diag(cm))/len(Y_train)
print("Accuracy = ",acc)

1875/1875 ━━━━━━━━━━━━━━━━━━━━ 4s 2ms/step
 
[[5776    1   42    9   29   56   49    8   36   45]
 [   0 6513   16    7    9    5   13   20   39    5]
 [  23   63 5662  119   35   20   24   77   62    4]
 [  12   35   53 5730    0  169    3   44  123   63]
 [   6    7   41    3 5317   17   24   22   22  102]
 [  18    3   14   96    3 4900   33    2   55   29]
 [  40    9   40   13   47   92 5738    1   58    5]
 [   3   16   29   55   15   23    0 5972   10  105]
 [  36   71   45   68   24   87   34   18 5374   40]
 [   9   24   16   31  363   52    0  101   72 5551]]
Accuracy =  0.9422166666666667

## Out of Sample
yhat = model.predict(X_test, batch_size=32)
yhat = [argmax(j) for j in yhat]

## Confusion matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(yhat,Y_test)
print(" ")
print(cm)

##
acc = sum(diag(cm))/len(Y_test)
print("Accuracy = ",acc)

313/313 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step
 
[[ 964    0   10    1    3   11   13    1    9    7]
 [   0 1099    2    0    0    3    3    5    2    4]
 [   3    9  981   19   10    4    7   24    8    0]
 [   1    8    5  953    0   33    1   11   16   12]
 [   0    0    7    0  908    2    5    2    4   18]
 [   4    0    1   15    0  796    5    0    8    5]
 [   4    6   10    0    8   13  921    0   13    0]
 [   2    1    6   12    2    2    0  959    7   11]
 [   2   12    7    6    5   22    3    1  899    4]
 [   0    0    3    4   46    6    0   25    8  948]]
Accuracy =  0.9428

16.54. Adversarial Models#

To see adversarial models, try: https://kennysong.github.io/adversarial.js/

16.55. Image Processing, Transfer Learning#

• Slides on Image Processing using Deep Learning by Subir Varma: https://drive.google.com/file/d/19xPCf2M66Dws06XxXLgEhuqJFiKmUEJ1/view?usp=sharing

• Image processing with transfer learning: https://drive.google.com/file/d/1D3Cg288wVY-e5BHuDNHec3wd-LStflMr/view?usp=sharing. From: Practical Deep Learning for Cloud and Mobile (O’Reilly) by Anirudh Koul, Siddha Ganju & Meher Kasam.

• Recognizing Images using NNs: https://drive.google.com/file/d/1OMQOZuEmnw0Kxvo5O1C9kf6mbQnPXYvj/view?usp=sharing. (Build your first Convolutional Neural Network to recognize images: A step-by-step guide to building your own image recognition software with Convolutional Neural Networks using Keras on CIFAR-10 images! by Joseph Lee Wei En.): https://medium.com/intuitive-deep-learning/build-your-first-convolutional-neural-network-to-recognize-images-84b9c78fe0ce

16.56. Learning the Black-Scholes Equation#

See : Hutchinson, Lo, Poggio (1994)

from scipy.stats import norm
def BSM(S,K,T,sig,rf,dv,cp):  #cp = {+1.0 (calls), -1.0 (puts)}
    d1 = (math.log(S/K)+(rf-dv+0.5*sig**2)*T)/(sig*math.sqrt(T))
    d2 = d1 - sig*math.sqrt(T)
    return cp*S*math.exp(-dv*T)*norm.cdf(d1*cp) - cp*K*math.exp(-rf*T)*norm.cdf(d2*cp)

df = pd.read_csv('NLP_data/BS_training.csv')

16.57. Normalizing spot and call prices#

$C$ is homogeneous degree one, so $$aC(S,K)=C(aS,aK)$$Thismeanswecannormalizespotandcallpricesandremoveavariablebydividingby$K$.$$\frac{C(S,K)}{K}=C(S/K,1)$$

df['Stock Price'] = df['Stock Price']/df['Strike Price']
df['Call Price']  = df['Call Price'] /df['Strike Price']

16.58. Data, libraries, activation functions#

n = 300000
n_train =  (int)(0.8 * n)
train = df[0:n_train]
X_train = train[['Stock Price', 'Maturity', 'Dividends', 'Volatility', 'Risk-free']].values
y_train = train['Call Price'].values
test = df[n_train+1:n]
X_test = test[['Stock Price', 'Maturity', 'Dividends', 'Volatility', 'Risk-free']].values
y_test = test['Call Price'].values

#Import libraries
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, LeakyReLU
import tensorflow as tf

def custom_activation(x):
    return tf.math.exp(x)

16.59. Set up, compile and fit the model#

nodes = 120
model = Sequential()

model.add(Dense(nodes, input_dim=X_train.shape[1]))
#model.add("relu")
model.add(Dropout(0.25))

model.add(Dense(nodes, activation='elu'))
model.add(Dropout(0.25))

model.add(Dense(nodes, activation='relu'))
model.add(Dropout(0.25))

model.add(Dense(nodes, activation='elu'))
model.add(Dropout(0.25))

model.add(Dense(1))
model.add(Activation(custom_activation))

model.compile(loss='mse',optimizer='rmsprop')

model.fit(X_train, y_train, batch_size=64, epochs=10, validation_split=0.1, verbose=2)

Epoch 1/10

/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

3375/3375 - 15s - 4ms/step - loss: 0.0056 - val_loss: 4.6593e-04
Epoch 2/10
3375/3375 - 9s - 3ms/step - loss: 0.0015 - val_loss: 9.6565e-04
Epoch 3/10
3375/3375 - 9s - 3ms/step - loss: 0.0012 - val_loss: 9.3496e-04
Epoch 4/10
3375/3375 - 9s - 3ms/step - loss: 0.0010 - val_loss: 2.4749e-04
Epoch 5/10
3375/3375 - 10s - 3ms/step - loss: 9.4946e-04 - val_loss: 2.5724e-04
Epoch 6/10
3375/3375 - 9s - 3ms/step - loss: 8.7600e-04 - val_loss: 2.5109e-04
Epoch 7/10
3375/3375 - 10s - 3ms/step - loss: 8.2751e-04 - val_loss: 1.6569e-04
Epoch 8/10
3375/3375 - 11s - 3ms/step - loss: 7.9041e-04 - val_loss: 2.5504e-04
Epoch 9/10
3375/3375 - 10s - 3ms/step - loss: 7.7280e-04 - val_loss: 1.6397e-04
Epoch 10/10
3375/3375 - 9s - 3ms/step - loss: 7.3299e-04 - val_loss: 1.9175e-04

<keras.src.callbacks.history.History at 0x7adbb98ae990>

model.summary()

Model: "sequential_2"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense_9 (Dense)                      │ (None, 120)                 │             720 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout (Dropout)                    │ (None, 120)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_10 (Dense)                     │ (None, 120)                 │          14,520 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_1 (Dropout)                  │ (None, 120)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_11 (Dense)                     │ (None, 120)                 │          14,520 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_2 (Dropout)                  │ (None, 120)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_12 (Dense)                     │ (None, 120)                 │          14,520 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_3 (Dropout)                  │ (None, 120)                 │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_13 (Dense)                     │ (None, 1)                   │             121 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ activation_5 (Activation)            │ (None, 1)                   │               0 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 88,804 (346.89 KB)

 Trainable params: 44,401 (173.44 KB)

 Non-trainable params: 0 (0.00 B)

 Optimizer params: 44,403 (173.45 KB)

def CheckAccuracy(y,y_hat):
    stats = dict()

    stats['diff'] = y - y_hat

    stats['mse'] = mean(stats['diff']**2)
    print("Mean Squared Error:      ", stats['mse'])

    stats['rmse'] = sqrt(stats['mse'])
    print("Root Mean Squared Error: ", stats['rmse'])

    stats['mae'] = mean(abs(stats['diff']))
    print("Mean Absolute Error:     ", stats['mae'])

    stats['mpe'] = sqrt(stats['mse'])/mean(y)
    print("Mean Percent Error:      ", stats['mpe'])

    #plots
    mpl.rcParams['agg.path.chunksize'] = 100000
    figure(figsize=(10,3))
    plt.scatter(y, y_hat,color='black',linewidth=0.3,alpha=0.4, s=0.5)
    plt.xlabel('Actual Price',fontsize=20,fontname='Times New Roman')
    plt.ylabel('Predicted Price',fontsize=20,fontname='Times New Roman')
    plt.show()

    figure(figsize=(10,3))
    plt.hist(stats['diff'], bins=50,edgecolor='black',color='white')
    plt.xlabel('Diff')
    plt.ylabel('Density')
    plt.show()

    return stats