Chapter 15 Glossary
For quick reference, we present the entire set of notation that is used in this monograph.
\(X^{(i)} \in {\cal R}^n\): input vector for observation \(i\). Think of each observation as a row (example) in the training or testing data.
\(n\): size of the feature set, i.e., the number of input variables per observation.
\(M\): number of observations in the data (sample size).
\(T^{(i)}\): true class (category, value) of observation \(i\).
\(Y^{(i)} = h_W(X^{(i)})\): function that generates the predicted (estimated) value of the ouput \(Y^{(i)}\).
\(L(Y^{(i)},T^{(i)})\): loss function that penalizes the algorithm for the error between the predicted value \(Y^{(i)}\) and the true value \(T^{(i)}\) for observation \(i\). Therefore, this is also known as the “error” function.
\(\sigma(a) = \frac{e^a}{1+e^a}\): sigmoid function that generates an output value in \((0,1)\). The sigmoid function is applied at the output layer in networks with a single output node.
\(Y = \sigma(\sum_{i=1}^n W_i X_i + b)\): generic form of the sigmoid function applied to weighted input data.
\(W_{i j}^{(r)}\) parameter weights on input \(j\) at the \(i\)-th node in hidden layer \(r\) of the neural network.
\(b_i^{(r)}\): bias parameter at the \(i\)-th node in layer \(r\).
\(Y_k, k = 1,\ldots,K\): outputs for a multiclass network with \(K\) outputs.
\(K\): number of nodes at the output layer.
\(Y_k = \frac{e^{a_k}}{\sum_{i=1}^K e^{a_i}}, k=1,2,\ldots,K\): “softmax” function for the \(K\) outputs at each node of the multiclass network output layer. This is also known as the “normalized exponential” function.
\(\Phi(X_i)\): Basis function of input \(i\). This is used to modify the sigmoid and introduce nonlinearity. The modified sigmoid function is as follows:
\[ \sigma \left( \sum_{i=1}^n W_i \Phi(X_i) + b \right) \]
- \(Z_j^{(r)} = f(a_j^{(r)})\): “activation” function \(f\), at node \(j\) in hidden layer \(r\), where
\[ a_j^{(r)} = \sum_{i=1}^n W_{ji}^{(r)} X_i + b_j^{(r)}, \quad 1 \leq j \leq n_r \]
\(n_r\): number of nodes at hidden layer \(r\).
\(R\): number of hidden layers.
\(\min_{{\bf W,b}} \sum_{m=1}^M L_m \left[h_W(X^{(m)}),T^{(m)} \right]\): optimization required to fit the deep learning network.
\(s \ll M\): smaller random batch size for Stochastic Batch Gradient Descent.
\(\frac{\partial L_m}{\partial W_{ij}^{(r+1)}} = \delta_i^{(r+1)} \cdot Z_j^{(r)}\): gradient of \({\bf W}\) for optimization of the model.
\(\frac{\partial L_m}{\partial b_{i}^{(r+1)}} = \delta_i^{(r+1)}\): gradient of \({\bf b}\) for optimization of the model.
\(\delta_i^{(r)} = \frac{\partial L_m}{\partial a_{i}^{(r)}}\): link derivative for the Backprop procedure. This is the key to making the deep learning net optimization facile and analytical. Without backpropogation, the computations on a deep learning net grow exponentially in the number of hidden layers \(R\).
\(\left[ n \cdot n_1 + \sum_{i=1}^{R-1} n_i \cdot n_{i+1} + n_R \cdot K \right] + \left[ \sum_{i=1}^R n_i + K \right]\): number of parameters to be estimated in a deep learning net that has \(n\) input variables, \(R\) hidden layers, and \(K\) output nodes. The first bracket is the total number of \({\bf W}\) weights and the second square bracket counts the number of bias parameters. This is also the number of gradients that need to be computed for Gradient Descent.