RNN and LSTMs

While FeedForward netweorks can accomodate arbitrary sized sequences through the use of vector addition and concatenation (CBOW), such representations are quite limited and disregard the order of the features. Recurrent Neural Networks allow representing arbitrarily sized sequential inputs in a fixed-size vectors, paying attention to the structured properties of the inputs.

RNNs, particularly ones with gated architectures such as the LSTM and the GRU, are very powerful at capturing statistical regularities in sequential inputs.

RNNs

• One of the most promising theories was suggested by Jordan(1986).
• Jordan described a network containing recurrent connections.
• The recurrent connections allow the network’s hidden* units to see its own previous output, so that the subsequent behaviour can be shaped by previous responses.
• There recurrent connections are what give the Network “Memory”.

Hidden These units are called “hidden” in the sense that they interact exclusively with other nodes internal to the network, and not the outside world!.

Imagine that there is a sequential input to be processed, and some clock which regulates presentation of the input to the network. Processing would then consist of the following sequence of events:

1. At time t, the input units receive the first input in the sequence.
2. Both the input units and context units activate the hidden units.
3. The hidden units then feed forward to activate the output units.
4. The hidden units also feed back to activate the context units.
5. Output is compared with a teacher input, and backpropogation of error is used to adjust the connection stengths incrementally.

6. At time step, t+1, the above sequence is repeated.

   

   

   

RNN as a Generator

RNN as a Conditional Generator

• The generation framework discussed above, generates the next token based on the previously generated tokens.
• In the conditioned generation framework, the next token is generated based on the previously generated tokens, and an additional conditioning context ‘c’.

What kind of information can be encoded in the context c?

• Pretty much any data we can put our hands on during training, and that we find useful.
  • For example, if we have a large corpus of news items categorized into different topics, we can treat the topic as a conditioning context.
  • Our language model will then be able to generate texts conditioned on the topic.
  • If we are interested in movie reviews, we can condition the generation on the genre of the movie, the rating of the review, and perhaps the geographic region of the author.
  • We can then control these aspects when generating text.

RNN as an ENCODER-DECODER

Another popular approach takes c to be itself a sequence, most commonly a piece of text. This gives rise to the sequence to sequence conditioned generation framework, also called the encoder-decoder framework

a) The encoder summarizes the source sentence as a vector c.

b) The decoder RNN is then used to predict (using a language modeling objective) the target sequence words conditioned on the previously predicted words as well as the encoded sentence c.

c) The encoder and decoder RNNs are trained jointly.

Problems with RNN Encoder-Decoder Network

https://arxiv.org/pdf/1409.1259.pdf

The encoder extracts a fixed-length vector representation from a variable-length input sentence, and from this representation the decoder generates a correct, variable-length target translation. However, this mechanism runs into problems with long source sentences.

• At the core of all these recent works lies an encoder–decoder architecture.
• The encoder processes a variable-length input (source sentence) and builds a fixed-length vector representation.
• Conditioned on the encoded representation, the decoder generates a variable-length sequence (target sentence).

Solution NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE

More info on the attention mechanism.

Weight Matrix

The concept of weight matrix sharing in Recurrent Neural Networks (RNNs) primarily revolves around the idea of using the same set of weights across different time steps or iterations of the network.

In traditional feedforward neural networks, each layer has its unique set of weights connecting the neurons in one layer to the next. However, in RNNs, weight matrix sharing refers to the reusability of the same weights across multiple time steps. This sharing of weights allows the network to maintain memory and capture temporal dependencies in sequential data.

Here’s what makes a RNN recurrent: it uses the same weights for each step. More specifically, a typical vanilla RNN uses only 3 sets of weights to perform its calculations:

• W_xh, used for all x_t -> h_t links.
• W_hh, used for all h_t-1 -> h_t links.
• W_hy, used for all h_t -> y_t links.

Solving Vanishing Gradient problem using LSTMs

• LSTMs are a variant of RNNs that try to improve this problem by introducing a more complex architecture that enables additive relationships between gradients instead of multiplicative ones, which are the culprit of the vanishing gradient problem.
• LSTMs also have a gating mechanism that allows them to selectively forget or remember information from previous time steps, which helps them to capture long-term dependencies more effectively.