Hello, and welcome to this video on embeddings. In this video, we will discuss how to extract features from text and we will introduce embedding layers. We've previously mentioned that neural networks can work with text data, but we have not really explained how to go from a sequence of words to numerical features. The simplest way is to compile a dictionary of words and assign to each word a numerical index. In this example, word number four is the word 'of', while word number two is the word 'be'. Now that we have a sequence of numbers, we can feed it, for example, to a recurring neural net. However, since we have tens of thousands of words in our dictionary, the numbers appearing in the sequence will be very large, which is not a good start for a neural network. 

Besides, since the index in the dictionary can be arbitrarily chosen, there is no relation between two words within your bi-indices. We could binarise the index to a huge factor with as many entries as the words in the dictionary, setting zero everywhere and a single one at the index corresponding to that word. This approach will also present some problems, because of the huge size and the sparsity of the input vector. The solution to all these problems is what's called an embedding layer. An embedding layer takes the huge sparse vector that we generated with the vocabulary and fully connects it to a much smaller set of nodes, for example ten or a hundred, or even two. This has the effect of mapping each word onto a vector space with few co-ordinates, making the space of words pretty dense. 

Also, this method carries an additional value that it will encapsulate semantic meaning in the denser space. Since we are training the embedding layer at the same time as the classification layers, the embedding will try to map words with close meaning to vectors that are close in the embedding space. This will produce semantic clouds of vectors and allow for very interesting consequences. One such consequence is that we will be able to perform mathematical operations with words, because the distance between the related concepts will be preserved. For example, the vector going from the word 'man' to the word 'woman' will encode a change of gender, and so one can expect that translating the word 'king' by such a vector, one should end up near the vector representing the word 'queen'. In this video, we've introduced the embedding layer and showed how they reduced the hugest vocabulary space to a more manageable dense space with semantic meaning. We've also showed that the vectors in this space can be added and subtracted in an interesting way. Thank you for watching and see you in the next video.