32. Generalized Language Models#
Good reference: https://lilianweng.github.io/lil-log/2019/01/31/generalized-language-models.html
Slides: https://docs.google.com/presentation/d/1xhOocjNJ-6YU_jXPJb_Yi0Vloen65QCyM51S7-Dadfk/edit?usp=sharing
Hands On LLMs (book) GitHub: HandsOnLLM/Hands-On-Large-Language-Models
Interviews on the history of the transformer: https://www.quantamagazine.org/when-chatgpt-broke-an-entire-field-an-oral-history-20250430/
32.1. Classification with Embeddings and BERT#
We can use many approaches as seen earlier. A good summary of classification approaches in various NLP libraries is discussed here: https://towardsdatascience.com/which-is-the-best-nlp-d7965c71ec5f
from google.colab import drive
drive.mount('/content/drive') # Add My Drive/<>
import os
os.chdir('drive/My Drive')
os.chdir('Books_Writings/NLPBook/')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
%%capture
# %pylab inline
import pandas as pd
import os
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import Image
# # Use pytorch kernel and install TF, as needed
# !pip install --upgrade torch
# !pip install --upgrade tensorflow
# !pip install --upgrade transformers
32.2. Sequence of Classification Approaches#
Various forms of input are possible:
Use a single vector from the TDM for classification of each document. Easy to construct, but lacking context. Input size is fixed. Vocab size is large.
Use a single TFIDF vector. Same as TDM vectors.
Word2Vec. Convert each word in the document into a fixed length vector. Combine vectors into a matrix for the document and this is the input into the classifier. Requires a package like gensim to make the word embeddings, some compute effort required. Fixed input size, 100-300, not huge as in TDM, TFIDF.
Doc2Vec. Each document is converted into a vector, which is input into the classifier (also needs gensim). Input size is fixed.
MPN. Use a standard NN to create word embeddings. Only takes a few tokens and truncates the document/sentence. Enlarging the window results in an explosion in parameters. No context.
RNN. Keeps track of word sequences and generates one embedding for a sequence of words. Can take any sequence length. Same weight matrix for all inputs. Keeps context. But slow, loses track of words further back in the sequence, so may be giving greater weight to words at the end. Vanishing gradients.
LSTMs. Same as RNN, but tries to fix the problem of vanishing gradients for RNNs. Goes in only one direction, so full context is missed. For example, in translation, words before and after current word matter.
CNN. Faster than RNNs as they do not have to wait to process tokens sequentially. Parallelization possible. Not fixed input, so padding is required.
Attention. These are bidirectional, so work better for all tasks as they have greater context. Also computationally better than LSTMs. No fixed input, limited maximum sequence length.
This historical sequence is also presented in these slides from NVIDIA:
32.3. Read in the data#
Datasets:
Reddit news with Dow sign, https://www.kaggle.com/aaron7sun/stocknews
Movie reviews, https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews
Financial Phrase Bank, https://www.researchgate.net/publication/251231364_FinancialPhraseBank-v10
# Read data
# df = pd.read_csv('NLP_data/Combined_News_DJIA.csv') # Reddit News vs Dow data
# df = pd.read_csv('NLP_data/movie_review.csv', parse_dates=True, index_col=0) # Movie Reviews data
df = pd.read_csv('NLP_data/Sentences_AllAgree.txt', sep=".@", header=None, encoding = "ISO-8859-1") # Finbert data
print(df.shape)
# df.columns = ["Label","Text"] # for movie reviews
df.columns = ["Text","Label"]
df.head()
(2264, 2)
/tmp/ipykernel_12285/2819765382.py:4: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.
df = pd.read_csv('NLP_data/Sentences_AllAgree.txt', sep=".@", header=None, encoding = "ISO-8859-1") # Finbert data
| Text | Label | |
|---|---|---|
| 0 | According to Gran , the company has no plans t... | neutral |
| 1 | For the last quarter of 2010 , Componenta 's n... | positive |
| 2 | In the third quarter of 2010 , net sales incre... | positive |
| 3 | Operating profit rose to EUR 13.1 mn from EUR ... | positive |
| 4 | Operating profit totalled EUR 21.1 mn , up fro... | positive |
# # Remove all the b-prefixes (for DJIA dataset)
# for k in range(1,26):
# colname = "Top"+str(k)
# df[colname] = df[colname].str[2:]
# # Prepare the data
# columns = ['Top' + str(i+1) for i in range(25)]
# df['Text'] = df[columns].apply(lambda x: ' '.join(x.astype(str)), axis=1)
# df = df[['Label', 'Text']]
# df.head()
# Plot class distribution
import seaborn as sns
sns.countplot(x='Label', data=df)
<Axes: xlabel='Label', ylabel='count'>
32.4. Now install raw text tools#
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.metrics import roc_curve,auc
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
# See Transformers from Hugging Face: https://huggingface.co/transformers/
# Simple Transformers: https://github.com/ThilinaRajapakse/simpletransformers
!pip install gensim
Requirement already satisfied: gensim in /usr/local/lib/python3.13/dist-packages (4.4.0)
Requirement already satisfied: numpy>=1.18.5 in /usr/local/lib/python3.13/dist-packages (from gensim) (2.1.3)
Requirement already satisfied: scipy>=1.7.0 in /usr/local/lib/python3.13/dist-packages (from gensim) (1.16.3)
Requirement already satisfied: smart_open>=1.8.1 in /usr/local/lib/python3.13/dist-packages (from gensim) (8.0.1)
Requirement already satisfied: wrapt in /usr/local/lib/python3.13/dist-packages (from smart_open>=1.8.1->gensim) (2.4.0)
import json
from sklearn import feature_extraction, feature_selection, metrics
from sklearn import model_selection, naive_bayes, pipeline, manifold, preprocessing
import gensim
import gensim.downloader as gensim_api
# from tensorflow.keras import models, layers, preprocessing as kprocessing
# from tensorflow.keras import backend as K
# from tensorflow.keras.utils import plot_model
import transformers
import nltk
nltk.download("stopwords")
nltk.download("wordnet")
nltk.download('omw-1.4')
stopwords = nltk.corpus.stopwords.words("english")
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data] Package wordnet is already up-to-date!
[nltk_data] Downloading package omw-1.4 to /root/nltk_data...
[nltk_data] Package omw-1.4 is already up-to-date!
# Use texthero as an alternative text cleaner, instead of the code below
import re # regex
def removeNumbersStr(s):
for c in range(10):
n = str(c)
s = s.replace(n," ")
return s
def cleanText(text, stem=False, lemm=True, stop=True):
text = re.sub(r'[^\w\s]', '', str(text).lower().strip()) # remove stuff
text = removeNumbersStr(text)
text = text.split() # tokenize
if stop is not None: # remove stopwords
text = [word for word in text if word not in stopwords]
if stem == True: # stemming
ps = nltk.stem.porter.PorterStemmer()
text = [ps.stem(word) for word in text]
if lemm == True:
lem = nltk.stem.wordnet.WordNetLemmatizer()
text = [lem.lemmatize(word) for word in text]
text = " ".join(text)
return text
df["cleanTxt"] = [cleanText(df.Text[j]) for j in range(len(df.Label))]
print(df.shape)
df.head()
(2264, 3)
| Text | Label | cleanTxt | |
|---|---|---|---|
| 0 | According to Gran , the company has no plans t... | neutral | according gran company plan move production ru... |
| 1 | For the last quarter of 2010 , Componenta 's n... | positive | last quarter componenta net sale doubled eur e... |
| 2 | In the third quarter of 2010 , net sales incre... | positive | third quarter net sale increased eur mn operat... |
| 3 | Operating profit rose to EUR 13.1 mn from EUR ... | positive | operating profit rose eur mn eur mn correspond... |
| 4 | Operating profit totalled EUR 21.1 mn , up fro... | positive | operating profit totalled eur mn eur mn repres... |
df_train, df_test = model_selection.train_test_split(df, test_size=0.2)
y_train = df_train["Label"].values
y_test = df_test["Label"].values
# Choose BOW or TFIDF in NLTK
# vectorizer = feature_extraction.text.CountVectorizer(max_features=10000, ngram_range=(1,2)) # BOW
vectorizer = feature_extraction.text.TfidfVectorizer(max_features=10000, ngram_range=(1,2)) # TFIDF
corpus = df_train["cleanTxt"]
vectorizer.fit(corpus)
X_train = vectorizer.transform(corpus)
X_train
<Compressed Sparse Row sparse matrix of dtype 'float64'
with 29654 stored elements and shape (1811, 10000)>
vocab = vectorizer.vocabulary_ # is a dict
list(vocab.keys())[:30]
['delivery',
'scheduled',
'summer',
'autumn',
'delivery scheduled',
'adp',
'news',
'finnish',
'security',
'privacy',
'software',
'solution',
'developer',
'stonesoft',
'oyj',
'said',
'today',
'usd',
'million',
'eur',
'order',
'deliver',
'stonegate',
'network',
'product',
'unnamed',
'russian',
'adp news',
'finnish security',
'security privacy']
X_train.shape
(1811, 10000)
32.5. Visualize the DTM#
plt.figure(figsize=(15,7))
sns.heatmap(X_train.todense() [:,np.random.randint(0,X_train.shape[1],2000)]==0,
vmin=0, vmax=1, cbar=False).set_title('Document Term Matrix (DTM)')
plt.xlabel('Terms'); plt.ylabel('Documents')
Text(158.22222222222223, 0.5, 'Documents')
32.6. Reduce the dimension of the vocabulary#
# Feature reduction using feature selection in sklearn
# This can also be done using TextHero
y = df_train["Label"]
X_names = vectorizer.get_feature_names_out()
p_value_limit = 0.75
df_features = pd.DataFrame()
for cat in np.unique(y):
chi2, p = feature_selection.chi2(X_train, y==cat)
df_features = pd.concat([df_features, pd.DataFrame({"feature":X_names, "score":1-p, "y":cat})])
df_features = df_features.sort_values(["y","score"],ascending=[True,False])
df_features = df_features[df_features["score"]>p_value_limit]
X_names = df_features["feature"].unique().tolist()
print(type(X_names)); print(X_names[:10])
print("# features =",len(X_names))
<class 'list'>
['decreased', 'decreased eur', 'fell', 'eur mn', 'mn', 'fell eur', 'compared profit', 'sale decreased', 'profit decreased', 'eur']
# features = 1317
# !conda install -c conda-forge xgboost -y
32.7. TFIDF Transform Classification#
# Define Vectorizer
vectorizer = feature_extraction.text.TfidfVectorizer(vocabulary=X_names)
vectorizer.fit(corpus)
X_train = vectorizer.transform(corpus)
vocab = vectorizer.vocabulary_
print("Check vocab length:", len(vocab))
tmp = np.zeros(X_train.shape[0])
for j in range(len(tmp)):
if y_train[j]=='negative':
tmp[j] = 1
elif y_train[j]=='positive':
tmp[j] = 2
y_train = tmp
# Define Classifier
import xgboost as xgb
# classifier = xgb.XGBClassifier(objective="binary:logistic") # for 2 classes
classifier = xgb.XGBClassifier(objective="multi:softmax") # for multiclass
Check vocab length: 1317
# Pipeline using sklearn
model = pipeline.Pipeline([("vectorizer", vectorizer),
("classifier", classifier)])
model["classifier"].fit(X_train, y_train)
X_test = df_test["cleanTxt"].values
# Accessing the classifier directly when making prediction
predicted = model["classifier"].predict(vectorizer.transform(X_test)) # Use transform here
predicted_prob = model["classifier"].predict_proba(vectorizer.transform(X_test)) # Use transform here
tmp = np.zeros(X_test.shape[0])
for j in range(len(tmp)):
if y_test[j]=='negative':
tmp[j] = 1
elif y_test[j]=='positive':
tmp[j] = 2
y_test = tmp
accuracy = metrics.accuracy_score(y_test, predicted)
# auc = metrics.roc_auc_score(y_test, predicted_prob[:,1]) # only for binary classification
print("Accuracy:", round(accuracy,2))
# print("Auc:", round(auc,2))
print("Detail:")
print(metrics.classification_report(y_test, predicted))
cm = metrics.confusion_matrix(y_test, predicted)
print(cm)
Accuracy: 0.87
Detail:
precision recall f1-score support
0.0 0.92 0.97 0.94 280
1.0 0.72 0.67 0.70 58
2.0 0.80 0.71 0.75 115
accuracy 0.87 453
macro avg 0.81 0.78 0.80 453
weighted avg 0.86 0.87 0.86 453
[[271 3 6]
[ 4 39 15]
[ 21 12 82]]
32.8. Using Embeddings#
Ideally, we want an embedding model which gives us the smallest embedding vector and works great for the task. The smaller the embedding size, the lesser the compute required for training as well as inference.
Instead of TFIDF representations of text, we can use embeddings based on Word2Vec, Doc2Vec, etc. Much of these approaches have been superceded now by Transformer generated embeddings in the class of BERT models.
Slides: https://docs.google.com/presentation/d/1xhOocjNJ-6YU_jXPJb_Yi0Vloen65QCyM51S7-Dadfk/edit?usp=sharing
Image("NLP_images/BERT (0).png", width=900)
Image("NLP_images/BERT (1).png", width=900)
Image("NLP_images/models_timeline.png", width=900)
https://lifearchitect.ai/timeline/
Image("NLP_images/BERT (2).png", width=900)
32.9. Transformers#
Read this amazing short book for a complete introduction to transformers.
Image("NLP_images/BERT (3).png", width=900)
Image("NLP_images/BERT (4).png", width=900)
Image("NLP_images/BERT (5).png", width=900)
Image("NLP_images/BERT (6).png", width=900)
Image("NLP_images/BERT (7).png", width=900)
# Image("NLP_images/BERT (8).png", width=900)
Image("NLP_images/BERT (11).png", width=900)
Image("NLP_images/BERT (10).png", width=900)
The main idea of Attention is to modify word vectors so that they have more context.
There is some controversy about when the term “attention” first entered the literature, going way back to the 1990s. For a full and very balanced discussion, see: https://www.turingpost.com/p/attention
Image("NLP_images/BERT (13).png", width=900)
Image("NLP_images/BERT (14).png", width=900)
Ref: https://jalammar.github.io/illustrated-transformer
Image("NLP_images/BERT (15).png", width=900)
Image("NLP_images/BERT (16).png", width=900)
See the BertViz library for visualizing attention.
Image("NLP_images/Positional_encoding.png", width=900)
Image("NLP_images/Normalizations.png", width=900)
Image("NLP_images/BERT (12).png", width=900)
Image("NLP_images/BERT (17).png", width=900)
Image("NLP_images/BERT (9).png", width=900)
32.10. Transformers - Toy Example#
See also Layer Normalization: https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html
# Code to exemplify transformers
# 2 tokens, embedding dimension of 4
# Embedding matrix
x = np.random.rand(2,4)
print(f"Input Embeddings:\n {x}")
# Set up weight matrices for query (Q), key (K), and value (V) vectors
wQ = np.random.rand(4,3)
wK = np.random.rand(4,3)
wV = np.random.rand(4,3)
# Generate the queries, keys, and value vectors
q = x.dot(wQ)
k = x.dot(wK)
v = x.dot(wV)
print(f"Query:\n {q}")
print(f"Key:\n {k}")
print(f"Value:\n {v}")
# Self attention for each word
score = q.dot(k.T)
print(f"Score:\n {score}") # each row is self attention for a word
# Divide by Sqrt of query dimension
sqrt_dk = np.sqrt(len(q[0]))
score = score/sqrt_dk
print(f"Sqrt dk: {sqrt_dk}\nRevised score:\n {score}")
# Softmax
softmax = np.exp(score) / np.sum(np.exp(score), axis=1, keepdims=True)
print(f"Softmax:\n {softmax}") # This gives the self-attention values
# z = Softmax x value
z = softmax.dot(v)
print(f"Z:\n {z}")
# Rescale back to original dimension using multiple heads (in this example we have just one head)
w0 = np.random.rand(3,4)
z = z.dot(w0)
print(f"Output:\n {z}") # Each row to be fed into the separate FFNNs to complete the encoder
Input Embeddings:
[[0.9443169 0.75786491 0.79335266 0.80572671]
[0.48401044 0.18324881 0.3939482 0.85550841]]
Query:
[[2.1940294 2.64374723 1.37295209]
[1.34829991 1.61841313 0.85603735]]
Key:
[[1.49112166 1.95609418 1.9325009 ]
[0.88634721 1.14016688 0.88531887]]
Value:
[[1.50000053 1.16514665 1.65953483]
[0.86047467 0.83414695 1.08663392]]
Score:
[[11.09621447 6.17448527]
[ 6.83054063 3.79818893]]
Sqrt dk: 1.7320508075688772
Revised score:
[[6.40640241 3.56484073]
[3.94361447 2.1928854 ]]
Softmax:
[[0.94488085 0.05511915]
[0.85204474 0.14795526]]
Z:
[[1.46475041 1.14690223 1.62795702]
[1.40537931 1.1161735 1.57477112]]
Output:
[[1.91111169 1.25455453 2.60698992 2.67189818]
[1.838373 1.20914842 2.52760706 2.57621807]]
# Multiheads for Attention
# Suppose we have 6 of these Attention results and we put them side by side
# Then we get a matrix of size 2 x 24, lets create a dummy one here:
z = np.random.rand(2,24)
# We want to bring this down to a matrix of 2 x 4 in size
# Which means we need to multiply it by a matrix of dimension 24 x 4
wO = np.random.rand(24,4)
z = z.dot(wO)
print(f"Output:\n {z}") # Each row to be fed into the separate FFNNs to complete the encoder
Output:
[[5.16046335 5.78532964 6.64986222 5.95012318]
[3.9047244 4.95412075 6.23709722 6.10038884]]
Image("NLP_images/BERT (18).png", width=900)
Image("NLP_images/BERT (19).png", width=900)
Both schemes shown above are examples of “Self-supervised Learning”.
Image("NLP_images/BERT (20).png", width=900)
Image("NLP_images/BERT (21).png", width=900)
Image("NLP_images/BERT (22).png", width=900)
Image("NLP_images/BERT (23).png", width=900)
Image("NLP_images/BERT (24).png", width=900)
Image("NLP_images/BERT (25).png", width=900)
32.11. Recap of Transformers: https://drive.google.com/file/d/1LrzHTZoXP-PQ2Gzv1IOd4wJOfhtkKkzW/view?usp=sharing#
32.12. Transformers – Other Useful Links#
Video on transformers: https://www.youtube.com/watch?v=bCz4OMemCcA
3Blue1Brown: https://www.youtube.com/watch?v=wjZofJX0v4M (Transformers); https://www.youtube.com/watch?v=eMlx5fFNoYc (Attention)
Build a transformer from scratch: https://medium.com/towards-data-science/build-your-own-transformer-from-scratch-using-pytorch-84c850470dcb
Detailed view of transformers: https://e2eml.school/transformers.html
Understanding and coding attention: https://drive.google.com/file/d/1wzQ8q0gno23v8zoyP6_lCsa-DXg5xbqS/view?usp=sharing
Attnetion in Transformers: Concepts and Code in PyTorch (from DeepLearning.ai: https://learn.deeplearning.ai/courses/attention-in-transformers-concepts-and-code-in-pytorch/
32.13. BERT (Summary)#
BERT piggybacks on Transformer models, for a technical overview, see: https://drive.google.com/file/d/1G4tEu0SQrYglVIvgRqY17Khlbhvuf-4s/view?usp=sharing
BERT handles context better than word embeddings. Therefore it takes care of polysemy, i.e., same word meaning different things in different context.
BERT is trained using a denoising objective (masked language modeling), where it aims to reconstruct a noisy version of a sentence back into its original version. The concept is similar to autoencoders.
The original BERT uses a next-sentence prediction objective, but it was shown in the RoBERTa paper that this training objective doesn’t help that much. In this way, BERT is trained on gigabytes of data from various sources (much of Wikipedia) in an unsupervised fashion.
Google Research and Toyota Technological Institute jointly released a much smaller/smarter Lite Bert called ALBERT. (“ALBERT: A Lite BERT for Self-supervised Learning of Language Representations”). BERT x-large has 1.27 Billion parameters, vs ALBERT x-large with 59 Million parameters! The core architecture of ALBERT is BERT-like in that it uses a transformer encoder architecture, along with GELU activation. It also uses the identical vocabulary size of 30K as used in the original BERT. (V=30,000).
The downside of BERT is compute: you definitely need a GPU.
Will Transformers take over everything in NLP and computer vision? https://www.quantamagazine.org/will-transformers-take-over-artificial-intelligence-20220310/
32.14. BERT with transfer learning#
BERT input has a special structure. Take the sequence of words - “The paycheck protection program” and it gets encoded as
CLS | The | paycheck | protection | program | SEP | PAD | PAD | PAD
There are 3 vectors that are generated by this:
(1) Token IDs: these are integers that refer to the vocab index. Some have fixed IDs such as
CLS = 101 (start id)
UNK = 100 (unknown id)
SEP = 102 (end/separator id)
PAD = 0 (padding slots id)
The actual words get their ids from the vocab.
(2) Mask = 1 from CLS through SEP, 0 thereafter. It delineates the text from its padding.
(3) Segment = 1 for SEP, zero elsewhere. It delineates sentences.
Steps:
tokenize + transform + create embedding
See the first element of the embedding that is generated, it is what is passed to the NN.
# !pip install --upgrade transformers
import torch
from transformers import AutoTokenizer, AutoModel
# 1. Load the pre-trained BERT tokenizer and model
model_name = "google-bert/bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
# 2. Prepare input text
txt = "love the show"
# 3. Tokenize input and convert to PyTorch tensors
# return_tensors='pt' ensures the output is a PyTorch tensor
input_ids = tokenizer(txt, padding=True, truncation=True, return_tensors="pt")
# 4. Pass inputs through the BERT model (disable gradient calculations for speed)
with torch.no_grad():
embedding = model(**input_ids)
print(input_ids)
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] BertModel LOAD REPORT from: google-bert/bert-base-uncased
Key | Status | |
-------------------------------------------+------------+--+-
cls.predictions.transform.LayerNorm.bias | UNEXPECTED | |
cls.predictions.transform.dense.bias | UNEXPECTED | |
cls.seq_relationship.weight | UNEXPECTED | |
cls.predictions.transform.dense.weight | UNEXPECTED | |
cls.predictions.bias | UNEXPECTED | |
cls.predictions.transform.LayerNorm.weight | UNEXPECTED | |
cls.seq_relationship.bias | UNEXPECTED | |
Notes:
- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
{'input_ids': tensor([[ 101, 2293, 1996, 2265, 102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}
print("Structure of BERT input (size of text + 2):", input_ids)
print("Length of embedding structure:", len(embedding))
print("Shape of first element of embedding:", embedding[0][0].shape) # size of input ids, BERT input vector size
print("Shape of second element of embedding:", embedding[1][0].shape) #
Structure of BERT input (size of text + 2): {'input_ids': tensor([[ 101, 2293, 1996, 2265, 102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}
Length of embedding structure: 2
Shape of first element of embedding: torch.Size([5, 768])
Shape of second element of embedding: torch.Size([768])
# Reuse the data from the TFIDF classification problem
corpus = df_train["Text"] # use Text not cleanTxt as we want context and need to keep the sentences as they are
len(corpus)
1811
## Prepare BERT input for the training dataset
max_seq_length = 160 # The longest sentences in the dataset are less than this, adjust as needed
# Process each text in the corpus
input_ids_list = []
attention_masks_list = []
for txt in corpus:
# BERT tokenizers handle raw text directly, so removing manual cleaning.
# Tokenize and encode using the tokenizer, ensuring fixed length
encoded_input = tokenizer(
txt, # Use raw text directly
add_special_tokens=True, # Add '[CLS]' and '[SEP]'
max_length=max_seq_length, # Pad/truncate to max_seq_length
padding='max_length', # Pad to max_length
truncation=True, # Truncate if sequence is longer than max_seq_length
return_attention_mask=True,
return_token_type_ids=False, # DistilBERT does not use token_type_ids
return_tensors='pt' # Return PyTorch tensors
)
input_ids_list.append(encoded_input['input_ids'])
attention_masks_list.append(encoded_input['attention_mask'])
# Concatenate all tensors and convert to numpy arrays for X_train structure
X_train = [
torch.cat(input_ids_list, dim=0).numpy(),
torch.cat(attention_masks_list, dim=0).numpy()
]
# X_train is a 3 dimension tensor
print(len(X_train)) # one each for Token ID, Mask, Segment arrays
print(len(X_train[0])) # Size of the training set
print(len(X_train[0][0])) # max sequence length + 2 (for CLS and SEP)
2
1811
160
df_train
| Text | Label | cleanTxt | |
|---|---|---|---|
| 1071 | The deliveries are scheduled for the summer an... | neutral | delivery scheduled summer autumn |
| 436 | ( ADP News ) - Sep 30 , 2008 - Finnish securit... | positive | adp news sep finnish security privacy software... |
| 523 | The Swedish subsidiary holds 1.0 % net smelter... | neutral | swedish subsidiary hold net smelter return nsr... |
| 501 | In the method the smelt spouts 2 are separated... | neutral | method smelt spout separated working area shie... |
| 1101 | The parties have agreed not to disclose the tr... | neutral | party agreed disclose transaction value |
| ... | ... | ... | ... |
| 1268 | Investments in product development stood at 6.... | neutral | investment product development stood mln euro mln |
| 168 | Operating profit rose to EUR 9.2 mn from EUR 6... | positive | operating profit rose eur mn eur mn correspond... |
| 1090 | The mill will have capacity to produce 500,000... | neutral | mill capacity produce tonne pulp per year |
| 371 | Finnish power supply solutions and systems pro... | negative | finnish power supply solution system provider ... |
| 1014 | Raute is listed on the Nordic exchange in Hels... | neutral | raute listed nordic exchange helsinki |
1811 rows × 3 columns
k = np.random.randint(len(X_train[0])) # Pick a random sentence, try 302
# print(df_train["Text"][k])
print(len(X_train[0][k]))
print(X_train[0][k]) # Token ids
print(X_train[1][k]) # mask
# print(X_train[2][k]) # segment
160
[ 101 1996 6705 8778 2003 2764 2429 2000 1996 4781 29464 24475
14645 3486 2620 2620 1998 29464 24475 14645 3486 23499 102 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0]
[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0]
32.15. Fine-Tuning#
We are using BERT below, which only requires the token IDs and masks (not segments).
import torch
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
from transformers import DistilBertForSequenceClassification, get_linear_schedule_with_warmup # Changed from BertForSequenceClassification
from torch.optim import AdamW
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np
import os
import random
import re # For tokenization cleaning
# Set the device for training
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
# Convert X_train numpy arrays to PyTorch tensors
# X_train was previously defined as [token_ids, attention_masks] for DistilBERT
input_ids_train = torch.tensor(X_train[0], dtype=torch.long)
attention_masks_train = torch.tensor(X_train[1], dtype=torch.long)
# Removed token_type_ids_train as DistilBERT does not use them
# Define inverse_dic mapping string labels to numerical labels (moved here for execution flow)
inverse_dic = {'neutral': 0, 'negative': 1, 'positive': 2}
labels_train = torch.tensor([inverse_dic[label] for label in df_train["Label"].values], dtype=torch.long)
# Create the DataLoader for our training set (removed token_type_ids_train)
train_dataset = TensorDataset(input_ids_train, attention_masks_train, labels_train)
train_dataloader = DataLoader(
train_dataset,
sampler=RandomSampler(train_dataset), # Select batches randomly
batch_size=32 # BERT training recommended batch size
)
# Determine the number of unique labels for the classification head
num_labels = len(np.unique(df_train["Label"].values))
# Load DistilBertForSequenceClassification model (changed from BertForSequenceClassification and model name)
model = DistilBertForSequenceClassification.from_pretrained(
"distilbert-base-uncased", # Use the DistilBERT model.
num_labels=num_labels, # The number of output labels.
output_attentions=False, # Whether the model returns attentions weights.
output_hidden_states=False, # Whether the model returns all hidden-states.
)
# Tell pytorch to run this model on the GPU if available.
model.to(device)
# Optimizer for fine-tuning
optimizer = AdamW(model.parameters(),
lr=2e-5, # Learning rate - default is 5e-5 to 1e-4
eps=1e-8 # Adam Epsilon - default is 1e-8
)
epochs = 4 # Number of training epochs. BERT authors recommend between 2 and 4.
# Total number of training steps is number of batches * number of epochs.
total_steps = len(train_dataloader) * epochs
# Create the learning rate scheduler.
scheduler = get_linear_schedule_with_warmup(optimizer,
num_warmup_steps=0, # Default value in run_glue.py
num_training_steps=total_steps)
Using device: cuda
[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key | Status |
------------------------+------------+-
vocab_layer_norm.weight | UNEXPECTED |
vocab_transform.weight | UNEXPECTED |
vocab_layer_norm.bias | UNEXPECTED |
vocab_projector.bias | UNEXPECTED |
vocab_transform.bias | UNEXPECTED |
pre_classifier.weight | MISSING |
classifier.bias | MISSING |
classifier.weight | MISSING |
pre_classifier.bias | MISSING |
Notes:
- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING: those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
# Create label y
dic_y_mapping = {0:'neutral', 1:'negative', 2:'positive'}
# Create inverse_dic mapping string labels to numerical labels
inverse_dic = {'neutral': 0, 'negative': 1, 'positive': 2}
print(len(X_train[1]))
print(np.unique(y_train))
1811
[0. 1. 2.]
import random # Added import for random
# Set the seed for reproducibility
seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)
# Store the average loss after each epoch so we can plot them.
loss_values = []
# Training loop
for epoch_i in range(0, epochs):
print(f'======== Epoch {epoch_i + 1} / {epochs} ========')
print('Training...')
total_loss = 0
model.train() # Put the model into training mode.
for step, batch in enumerate(train_dataloader):
# Progress update every 40 batches.
if step % 40 == 0 and not step == 0:
print(f' Batch {step:>5,} of {len(train_dataloader):>5,}.')
# Unpack the batch from our dataloader and copy to the device.
# Removed b_token_type_ids as DistilBERT does not use them
b_input_ids = batch[0].to(device)
b_input_mask = batch[1].to(device)
b_labels = batch[2].to(device) # Labels are now at index 2
model.zero_grad()
# Perform a forward pass (evaluate the model on this training batch).
# The `outputs` object contains the loss, logits, and any other outputs returned by the model.
# Removed token_type_ids from model call
outputs = model(b_input_ids,
attention_mask=b_input_mask,
labels=b_labels)
loss = outputs.loss
total_loss += loss.item()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # Clip the norm of the gradients to 1.0 to prevent "exploding gradients"
optimizer.step()
scheduler.step()
# Calculate the average loss over the training data.
avg_train_loss = total_loss / len(train_dataloader)
loss_values.append(avg_train_loss)
print(f" Average training loss: {avg_train_loss:.2f}")
print("\nTraining complete!")
======== Epoch 1 / 4 ========
Training...
Batch 40 of 57.
Average training loss: 0.64
======== Epoch 2 / 4 ========
Training...
Batch 40 of 57.
Average training loss: 0.21
======== Epoch 3 / 4 ========
Training...
Batch 40 of 57.
Average training loss: 0.09
======== Epoch 4 / 4 ========
Training...
Batch 40 of 57.
Average training loss: 0.06
Training complete!
32.15.1. PyTorch Model Summary#
The BertForSequenceClassification model is now set up. The training will proceed to fine-tune this model.
# Prepare BERT input for the test dataset (similar to train_dataloader preparation)
# Reuse max_seq_length from previous cells (160) and tokenizer.
corpus_test = df_test["Text"]
input_ids_list_test = []
attention_masks_list_test = []
for txt in corpus_test:
# BERT tokenizers handle raw text directly, so removing manual cleaning.
# Tokenize and encode using the tokenizer, ensuring fixed length
encoded_input = tokenizer(
txt, # Use raw text directly
add_special_tokens=True, # Add '[CLS]' and '[SEP]'
max_length=max_seq_length, # Pad/truncate to max_seq_length
padding='max_length', # Pad to max_length
truncation=True, # Truncate if sequence is longer than max_seq_length
return_attention_mask=True,
return_token_type_ids=False, # DistilBERT does not use token_type_ids
return_tensors='pt' # Return PyTorch tensors
)
input_ids_list_test.append(encoded_input['input_ids'])
attention_masks_list_test.append(encoded_input['attention_mask'])
# Convert to PyTorch tensors (removed test_token_type_ids)
test_input_ids = torch.cat(input_ids_list_test, dim=0)
test_attention_masks = torch.cat(attention_masks_list_test, dim=0)
# Define inverse_dic mapping string labels to numerical labels (moved here for execution flow)
inverse_dic = {'neutral': 0, 'negative': 1, 'positive': 2}
test_labels_numerical = torch.tensor([inverse_dic[label] for label in df_test["Label"].values], dtype=torch.long)
# Create the DataLoader for our test set (removed test_token_type_ids)
test_dataset = TensorDataset(test_input_ids, test_attention_masks, test_labels_numerical)
test_dataloader = DataLoader(
test_dataset,
sampler=SequentialSampler(test_dataset), # Pull out batches sequentially.
batch_size=32
)
print('Predicting on test set...')
model.eval() # Put the model in evaluation mode
predictions = []
true_labels = []
# Predict
for batch in test_dataloader:
batch = tuple(t.to(device) for t in batch)
# Unpack the batch with no token_type_ids
b_input_ids, b_input_mask, b_labels = batch
with torch.no_grad():
# Removed token_type_ids from model call
outputs = model(b_input_ids,
attention_mask=b_input_mask
)
logits = outputs.logits
logits = logits.detach().cpu().numpy()
label_ids = b_labels.to('cpu').numpy()
predictions.append(logits)
true_labels.append(label_ids)
# Concatenate the predictions and true labels
flat_predictions = np.concatenate(predictions, axis=0)
flat_predictions = np.argmax(flat_predictions, axis=1).flatten() # Get predicted class
flat_true_labels = np.concatenate(true_labels, axis=0)
# Evaluate performance
print("\nClassification Report:")
target_names = list(inverse_dic.keys()) # ['neutral', 'negative', 'positive']
print(classification_report(flat_true_labels, flat_predictions, target_names=target_names))
print("\nConfusion Matrix:")
print(confusion_matrix(flat_true_labels, flat_predictions))
Predicting on test set...
Classification Report:
precision recall f1-score support
neutral 0.99 0.97 0.98 280
negative 0.95 0.97 0.96 58
positive 0.92 0.96 0.94 115
accuracy 0.97 453
macro avg 0.95 0.96 0.96 453
weighted avg 0.97 0.97 0.97 453
Confusion Matrix:
[[272 1 7]
[ 0 56 2]
[ 3 2 110]]
TFIDF accuracy = 80-85%
Word2Vec accuracy = 70-75%
BERT accuracy = >90%
32.16. REFERENCES#
Using BERT for the first time (by J Alammar): http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/; code: https://colab.research.google.com/github/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb
Stanford Sentiment Treebank: https://nlp.stanford.edu/sentiment/index.html
32.17. From the reference above, using the SST dataset (movie reviews)#
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import cross_val_score
import torch
import transformers as ppb
import warnings
warnings.filterwarnings('ignore')
df_large = pd.read_csv('https://github.com/clairett/pytorch-sentiment-classification/raw/master/data/SST2/train.tsv', delimiter='\t', header=None)
print(df_large.shape)
df_large.head()
(6920, 2)
| 0 | 1 | |
|---|---|---|
| 0 | a stirring , funny and finally transporting re... | 1 |
| 1 | apparently reassembled from the cutting room f... | 0 |
| 2 | they presume their audience wo n't sit still f... | 0 |
| 3 | this is a visually stunning rumination on love... | 1 |
| 4 | jonathan parker 's bartleby should have been t... | 1 |
# Is the dataset balanced in labels?
df = df_large[:1500] # take a small subset
df[1].value_counts()
| count | |
|---|---|
| 1 | |
| 1 | 782 |
| 0 | 718 |
32.18. Get the pre-trained model#
# For DistilBERT:
model_class, tokenizer_class, pretrained_weights = (ppb.DistilBertModel, ppb.DistilBertTokenizer, 'distilbert-base-uncased')
## Want BERT instead of distilBERT? Uncomment the following line:
#model_class, tokenizer_class, pretrained_weights = (ppb.BertModel, ppb.BertTokenizer, 'bert-base-uncased')
# Load pretrained model/tokenizer
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)
[transformers] DistilBertModel LOAD REPORT from: distilbert-base-uncased
Key | Status | |
------------------------+------------+--+-
vocab_layer_norm.weight | UNEXPECTED | |
vocab_transform.weight | UNEXPECTED | |
vocab_layer_norm.bias | UNEXPECTED | |
vocab_projector.bias | UNEXPECTED | |
vocab_transform.bias | UNEXPECTED | |
Notes:
- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
# Get tokenized version
tokenized = df[0].apply((lambda x: tokenizer.encode(x, add_special_tokens=True)))
print(tokenized.shape)
tokenized[0]
(1500,)
[101,
1037,
18385,
1010,
6057,
1998,
2633,
18276,
2128,
16603,
1997,
5053,
1998,
1996,
6841,
1998,
5687,
5469,
3152,
102]
32.19. Construct token IDs and masks#
# Add padding and set max len to the longest entry in the dataset
max_len = 0
for i in tokenized.values:
if len(i) > max_len:
max_len = len(i)
padded = np.array([i + [0]*(max_len-len(i)) for i in tokenized.values])
print(np.array(padded).shape)
padded[:3]
(1500, 59)
array([[ 101, 1037, 18385, 1010, 6057, 1998, 2633, 18276, 2128,
16603, 1997, 5053, 1998, 1996, 6841, 1998, 5687, 5469,
3152, 102, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0],
[ 101, 4593, 2128, 27241, 23931, 2013, 1996, 6276, 2282,
2723, 1997, 2151, 2445, 12217, 7815, 102, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0],
[ 101, 2027, 3653, 23545, 2037, 4378, 24185, 1050, 1005,
1056, 4133, 2145, 2005, 1037, 11507, 10800, 1010, 2174,
14036, 2135, 3591, 1010, 2061, 2027, 19817, 4140, 2041,
1996, 7511, 2671, 4349, 3787, 1997, 11829, 7168, 9219,
1998, 28971, 2308, 1999, 8301, 8737, 2100, 4253, 102,
0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0]])
# Add a mask to let BERT know where the real tokens are and not the padding
# Essentially we can use zero for the padding mask so those tokens do not compute
attention_mask = np.where(padded != 0, 1, 0)
print(attention_mask.shape)
attention_mask[:3]
(1500, 59)
array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
32.20. Run token IDs and attention masks through BERT to get embeddings#
%%time
input_ids = torch.tensor(padded)
attention_mask = torch.tensor(attention_mask)
with torch.no_grad():
last_hidden_states = model(input_ids, attention_mask=attention_mask) #embeddings
CPU times: user 1min 16s, sys: 17 s, total: 1min 33s
Wall time: 17.1 s
# Embeddings from BERT
print(len(last_hidden_states[0][:,0,:]))
last_hidden_states[0][:,0,:]
1500
tensor([[-0.2159, -0.1403, 0.0083, ..., -0.1369, 0.5867, 0.2011],
[-0.1726, -0.1448, 0.0022, ..., -0.1744, 0.2139, 0.3720],
[-0.0506, 0.0720, -0.0296, ..., -0.0715, 0.7185, 0.2623],
...,
[ 0.0062, 0.0426, -0.1080, ..., -0.0417, 0.6836, 0.3451],
[ 0.0087, 0.0605, -0.3309, ..., -0.2005, 0.6268, 0.1546],
[-0.2395, -0.1362, 0.0463, ..., -0.0285, 0.2219, 0.3242]])
# Collect the CLS embedding and labels to set up the classification task
features = last_hidden_states[0][:,0,:].numpy()
labels = df[1]
print(features.shape)
(1500, 768)
32.21. Use the BERT transformed dataset for machine learning as usual#
train_features, test_features, train_labels, test_labels = train_test_split(features, labels)
lr_clf = LogisticRegression()
lr_clf.fit(train_features, train_labels)
lr_clf.score(test_features, test_labels)
0.8346666666666667
32.22. Speed#
As you can see, BERT runs slow, so it is good to use a machine with GPUs. For more on computation speed, see: https://blog.inten.to/speeding-up-bert-5528e18bb4ea