### Back to Basic: Fine Tuning BERT for Sentiment Analysis

Today’s post continues on from yesterday. It will cover the training and evaluation function as well as test set prediction. Ideally, we would have another post, covering the same process except using PyTorch Lightning!

2. Tokenisation and data processing
4. Bert classifier
5. Initialise optimizer, loss function, and scheduler
6. Training and Eval
7. Eval on test set

### 7. Training and Eval

In [30]:
import random
import time

model.eval()

val_accuracy = []
val_loss = []

b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

loss = loss_fn(logits, b_labels)
val_loss.append(loss.item())

# Get the predictions
preds = torch.argmax(logits, dim=1).flatten()

# Calculate the accuracy rate
accuracy = (preds == b_labels).cpu().numpy().mean() * 100
val_accuracy.append(accuracy)

val_loss = np.mean(val_loss)
val_accuracy = np.mean(val_accuracy)

return val_loss, val_accuracy
In [52]:
for epoch_i in range(epochs):
print("EPOCH: %s" % epoch_i)
t_epoch, t_batch = time.time(), time.time()

total_loss = 0

model.train()

b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

loss = loss_fn(logits, b_labels)
total_loss += loss.item()

loss.backward()

optimizer.step()
scheduler.step()

if evaluation:

print('Training Completed!')
In [53]:
seed_value = 21
random.seed(seed_value)
np.random.seed(seed_value)
torch.manual_seed(seed_value)
torch.cuda.manual_seed_all(seed_value)
In [54]:
EPOCH: 0
EPOCH: 1
EPOCH: 2
EPOCH: 3
Training Completed!

### 8. Predictions on Test Set

In [56]:
# Preparing the test data

# Create the DataLoader for our test set
test_sampler = SequentialSampler(test_dataset)
In [59]:
import torch.nn.functional as F

model.eval()

all_logits = []

b_input_ids, b_attn_mask = tuple(t.to(device) for t in batch)[:2]

# Compute logits
all_logits.append(logits)

all_logits = torch.cat(all_logits, dim = 0)

probs = F.softmax(all_logits, dim = 1).cpu().numpy()

return probs
In [60]:
# Compute predicted probabilities on the test set

# Get predictions from the probabilities
threshold = 0.9
preds = np.where(test_probs[:, 1] > threshold, 1, 0)

# Number of tweets predicted non-negative
print("Number of tweets predicted non-negative: ", preds.sum())
Number of tweets predicted non-negative:  947
In [65]:
test_data['sentiment'] = preds
In [66]:
Out[66]:
id tweet sentiment
0 33 @SouthwestAir get your damn act together. Don’… 0
1 58 @AmericanAir horrible at responding to emails…. 0
2 135 @AmericanAir hey where is your crew? Flight aa… 0
3 159 Ok come on we are late let’s goooo @united 0
4 182 @AmericanAir since you are now affiliated with… 0
In [67]:
test_data['sentiment'].value_counts()
Out[67]:
0    3608
1     947
Name: sentiment, dtype: int64

Data Scientist