Attention mechanism

A Textual similarity network:

1) In 1^st stage, the goal is to calculate pair-wise word similarities. 2) Each word in sentence a can be similar to several words in sentence b, and vice versa.

i) For each word w_i^a in sentence a we compute a l_b-dimensional vector of its similarities to words in sentence b, normalized via softmax so that all similarities are positive and sum to one.

ii) This is called allignment vector for the word.

3) We similarly compute an alignment vector for each word in ‘b’:

4) Now, for every word w_i^a in sentence a we compute a vector w_i^b by performing weighted-sum of the words in b that are aligned to w_i^a.

5) Again, for every word w_j^b in sentence b we compute a vector w_j^a by performing weighted-sum of the words in a that are aligned to w_j^b.

Such weighted sum representation of a sequence of vectors, where the weights are computed by the softmax over scores such as described above are often referred to as ATTENTION MECHANISM.