Sentence Autocomplete Using Pytorch

Natural Language Processing(NLP) is one of the most flourishing parts of deep learning. Several applications of NLP are being used continuously in daily life. In this article, we are going to see how we can use NLP to autocomplete half-written sentences using deep learning methods. We will also see how we can generate clean data for training our NLP model. We will cover the following steps in this article

1. Cleaning the text data for training the NLP model
2. Loading the dataset using PyTorch
3. Creating the LSTM model
4. Training an NLP model
5. Making inferences from the trained model

We have seen applications like google keyboard where Google recommends what to type next based on the words which we have already written in the chatbox draft. However, to recommend the next term application like Google has been trained on billions of written sentences. In our model, we will use Wikipedia sentences that are freely available on the internet to download and that we can use for training our model.

Dataset for Sentence Autocomplete Model

We will use a Wikipedia dataset that we can download from here. The one main problem with the Wikipedia dataset is that it has special characters, non-meaningful words, and unknown words we can not use directly in our model that is why before using the dataset for training our model we must have to clean it. Since our dataset is an ms-word document we will use the python-docx library for reading the dataset document. We can use the following command for installing the library.

!pip install python-docx

Python Code for Cleaning the Dataset

We will use the Pythonre-module for removing special characters and words between them. Also since Python is a case-sensitive language we will convert all the words to lower cases. As we are developing these models for only English-speaking audiences, we will remove non-English words.

Output:

Text data cleaned and saved to: /content/output.csv

This code will output a CSV file which will be a cleaned dataset document that we can use for training our model. We can download this cleaned CSV file from here.

Class For Loading The Dataset Using Pytorch

We will define a custom Python class having a torch.utils.data.Dataset as metaclass for loading the dataset. In this class, we will use the load_words method for reading the CSV file. Also, we will use the get_unique_words method for counting the frequency of unique words in the dataset. The __len__ method will determine the length of the dataset. Whereas the __getitems__ method will create a tensor for each word.

LSTM Model For Sentence Autocompletion

​We will use a long-short-term memory network (LSTM) for our model. As we know LSTM network has an edge over simple RNNs since they have three extra gates which prevents gradient vanishing problem in the neural networks. Let us see the parameters and methods we are using in this model one by one.

• __init__(self, dataset): This is the initialization method of the class. It sets up the architecture and parameters of the LSTM model. Here are the key components that are being used within this method: self.lstm_size: It specifies the size (number of units) in the LSTM hidden state. self.embedding_dim: It Specifies the dimensionality of the word embeddings. self.num_layers: It specifies the number of layers in the LSTM. Since we don't want our model to be computationally expensive we will use only 3 layers. self.embedding: It defines an embedding layer that converts input indices to dense word embeddings. self.lstm: This argument defines the LSTM layer with the specified input size, hidden size, number of layers, and dropout rate. self.fc: Here we are using a fully connected later at the end of the network Definesar layer that maps the LSTM output to the vocabulary size to generate logits.
• forward(self, x, prev_state): This method performs the forward pass of the model. It takes an input tensor x and the previous state of the LSTM prev_state as input. Here are the steps within this method: self.embedding(x): The embed argument passes the input tensor x through the embedding layer to get the word embeddings. self.lstm(embed, prev_state): The output state passes the embeddings and previous state through the LSTM layer to get the output and updated state from the network. self.fc(output): Whereas the fully connected layer produces the logits value (the model's predictions) and state (the updated LSTM state) for each incomplete sentence.
• init_state(self, sequence_length): This method initializes the LSTM state with all zeros. It takes the length of the input sequence as input and returns an initial state tensor with zeros. The state tensor has a shape of (num_layers, sequence_length, lstm_size)

Initiating Hyperparameters and DataLoader for Training Sentence Autocomplete Model

In the model, we will define hyperparameters basically hyperparameters are the configurable parameters that determine the behavior of the model during training. In this code, the hyperparameters are defined as follows:

• sequence_length: The length of the input sequence.
• batch_size: The number of samples processed in each iteration.
• learning_rate: The step size at which the optimizer adjusts the model's parameters.
• num_epochs: The number of times the entire dataset is passed through the model during training.

We divided our dataset into two parts for training and validation using Pytorch. We used PyTorch's DataLoader to create iterators for efficient loading of data during training and validation. The training and validation datasets are passed to the respective data loaders, specifying the batch size and whether shuffling is required for the training data.

For training the model we have used a training loop that iterates over the specified number of epochs. For each epoch, the model is put in training mode using(model. train()), and the total loss is initialized. The loop iterates over batches of data from the training data loader. In every loop a sequence of task happen these are as:

• For each batch, the optimizer's gradients are reset using the Pytorch function (optimizer.zero_grad()),
• The initial hidden state for the LSTM model is obtained using the equation (hidden = model.init_state(sequence_length))
• The model is called with the inputs and hidden state using equation (outputs, _ = model(inputs, hidden)).
• The loss is then computed using the predicted outputs and the targets and the gradients are calculated using the Pytorch (loss.backwards ()) function.

The optimizer then performs a parameter update using the optimizer.step(), and the loss is added to the total loss for the epoch. After the loop, the average loss for the epoch is calculated and printed.

Output:

Epoch [1/10], Average Loss: 6.8103
Epoch [1/10], Validation Loss: 6.5937
Epoch [2/10], Average Loss: 6.4668
Epoch [2/10], Validation Loss: 6.3104
Epoch [3/10], Average Loss: 6.2176
Epoch [3/10], Validation Loss: 6.1290
Epoch [4/10], Average Loss: 6.0840
Epoch [4/10], Validation Loss: 6.0208
Epoch [5/10], Average Loss: 5.9850
Epoch [5/10], Validation Loss: 5.9312
Epoch [6/10], Average Loss: 5.8937
Epoch [6/10], Validation Loss: 5.8397
Epoch [7/10], Average Loss: 5.8056
Epoch [7/10], Validation Loss: 5.7547
Epoch [8/10], Average Loss: 5.7232
Epoch [8/10], Validation Loss: 5.6692
Epoch [9/10], Average Loss: 5.6379
Epoch [9/10], Validation Loss: 5.5762
Epoch [10/10], Average Loss: 5.5473
Epoch [10/10], Validation Loss: 5.4821

Making Inference From The Model

For making inferences from the model. We will take an incomplete input string sentence from the author then the input sentence is preprocessed by splitting it into individual words and converting each word to its corresponding index in the dataset. This is done using a list comprehension that iterates over the words in the input sentence and retrieves their corresponding indexes from the dataset using a word_to_index dictionary. The resulting list of indexes is stored in the input_indexes variable. We convert this value into a tensor that will be passed through the model for predicting an output word.

Output:

Input Sentence: he is your 
Predicted Next Word: mayor