Transformers

German to English Translation using seq2seq RNN Encoder-Decoder Model

The above seq2seq model is converting a German phrase to its English counterpart. Let’s break it down:

• Both Encoder and Decoder are RNNs.
• At every time step in the Encoder, the RNN takes a word vector (xi) from the input sequence and a hidden state (Hi) from the previous time step.
• The hidden state is updated at each time step.
• The hidden state from the last unit is known as the context vector. This contains information about the input sequence.
• This context vector is then passed to the decoder and it is then used to generate the target sequence (English phrase).
• If we use the Attention mechanism, then the weighted sum of the hidden states are passed as the context vector to the decoder.

Challenges:

Despite being so good at what it does, there are certain limitations of seq-2-seq models with attention:

1. Dealing with long-range dependencies is still challenging.
2. The sequential nature of the model architecture prevents parallelization. These challenges are addressed by Google Brain’s Transformer concept.

Transformer Architecture:

Once you have summed the Token Embeddings and Positional Embeddings , you pass the resultant vector to the Self-Attention layer:

The Self-Attention layer allows the model to analyze the relationships between inividial input tokens to better capture the CONTEXTUAL dependencies between the words:

The self-attention weights learnt during training and stored in these layers reflect the importance of each word in that input sequence to all other layers in the sequence.

But this does not happen once! Infact, multiple self-attention weights are learnt in parallel:

Intuition is that different attemtion maps will learn different aspects of language in the input sequence:

Now that all of the attention weights are applied to your input data, The output is passed to a fully connected FFN network , the output is a probability score for each token in your vocbulary

The data that leaves the eoncoder is a deep representation of the structure and the meaning of the sequence.

This deep representation is inserted into the niddle of the Decoder to influence the decoder’s self-attention mechanism.

Then, a start-of-sequence special token is added to the input of the Decoder which triggers the Decoder to predict the next token which it does based on the contextual understanding that has been provided from the Encoder,

The output of the Decoder self-attention layer is passed through the Decoder’s FFN and through the final SOFTMAX output layer. At this point, we have our first token.

You continue this loop by passing the output token of the model back to the input to trigger the next token untill end-of-sequence token is received.

Illustrated Transformer:

# # Load pre-trained BERT model and tokenizer
model_name = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)

When you load a pre-trained BERT model, it includes several components:

1. Embedding Layer: This is the first layer of the model, and it’s responsible for converting input tokens into vectors. This layer includes pre-trained embeddings for each token in the BERT tokenizer’s vocabulary.

2. Transformer Layers: These are the main body of the model. They take the vectors from the embedding layer and transform them through a series of self-attention mechanisms and feed-forward networks. The weights in these layers are also pre-trained.

3. Classification Layer: This is the final layer of the model, and it’s responsible for making predictions based on the output of the Transformer layers. In the BertForSequenceClassification model, this is a simple linear layer that outputs a logit for each class in your classification task.

So, when you load a pre-trained BERT model, you’re actually loading pre-trained embeddings along with pre-trained Transformer layers. These components are then fine-tuned on your specific task (sentiment analysis in this case).