NLP Gensim Tutorial

Gensim is an open-source Python library used for efficient text processing, topic modeling and vector-space modeling in NLP. It is designed for speed and memory efficiency, making it suitable for handling large text datasets.

• Supports Word2Vec, Doc2Vec and LDA
• Handles large datasets through streaming and incremental training
• Widely used for semantic analysis, topic discovery and text similarity tasks

1. Common Terminologies

Let us understand what some of the below mentioned terms mean before moving forward.

• Corpus: A corpus is a large and structured collection of text documents used for training or analyzing language models.
• Vector: A vector is the numerical representation of text data that helps machines process and understand language.
• Model: A model is a machine learning or statistical algorithm that learns patterns from data for prediction or analysis.
• Topic Modelling: Topic modeling is an unsupervised learning technique used to discover hidden topics within a collection of documents.
• Topic: A topic is a group of related words that frequently appear together and represent a common subject or idea.

2. Installation of NLP Gensim Library

First Install the library using the following command 

pip install gensim

Now, import the library and check the version to verify installation.

3. Create a Corpus from a given Dataset 

You need to follow these steps to create your corpus: 

• Load your Dataset
• Preprocess the Dataset
• Create a Dictionary
• Create Bag of Words Corpus

3.1 Load your Dataset

You can have a .txt file as your dataset or you can also load datasets using the Gensim Downloader API. Here, we have loaded a text file.

Gensim Downloader API: This is a module available in the Gensim library which is an API for downloading, getting information and loading datasets/models.

3.2 Preprocess the Dataset

Text preprocessing is used to clean and prepare text data for NLP tasks. The simple_preprocess() function tokenizes and normalizes the text by converting it into lowercase tokens and removing unwanted characters.

• Cleans and normalizes text data
• Converts text into tokens
• Removes punctuation and special characters
• Returns processed text as a list of tokens

Output

3.3 Create a Dictionary

Now we have our preprocessed data which can be converted into a dictionary by using the corpora.Dictionary( ) function. This dictionary is a map for unique tokens.

Output

Saving Dictionary on Disk or as Text File: You can save/load your dictionary on the disk as well as a text file as mentioned below.

3.4 Create Bag of Words Corpus

After creating the dictionary, the doc2bow() function is used to create a Bag of Words corpus. It converts words into IDs and counts how many times each word appears in a document.

Output

Saving Corpus on Disk: Now, save/load the corpus 

4. Create a TFIDF matrix in Gensim

TF-IDF stands for Term Frequency – Inverse Document Frequency. It is used to identify important words in documents by reducing the importance of commonly occurring words across the corpus.

• Highlights important words in documents
• Reduces the weight of frequently occurring common words
• Helps identify meaningful keywords
• Commonly used in text analysis and information retrieval

4.1 Building a BOW corpus

You can build a TFIDF model using Gensim and the corpus you developed previously as:

Output

4.2 Applying TF-IDF Model

Output

5. Creating Bigrams and Trigrams with Gensim

Some words frequently appear together and form a different meaning compared to their individual words. Gensim can identify these word combinations using bigrams and trigrams.

• Bigrams: Combination of two words
• Trigrams: Combination of three words

5.1 Loading Dataset using Gensim Downloader API

We will be building bigrams and trigrams using the text8 dataset here which can be downloaded using the Gensim downloader API.

5.2 Building Bigram using Phraser Model

Here, we are building a bigram using Phraser Model.

5.3 Building Trigram using Phraser Model

To create a Trigram we simply pass the above obtained bigram model to the same function. 

Output

👁 Output for trigram

6. Create Word2Vec model using Gensim

Machine learning models cannot process text directly, so words must be converted into numerical representations called word embeddings. Unlike CountVectorizer and TF-IDF, Word2Vec preserves relationships between words by mapping them into a vector space.

• Converts words into numerical vectors
• Preserves semantic relationships between words
• Supports mathematical operations on word vectors
• Pre-trained models like Word2Vec, GloVe and FastText can be loaded using Gensim
• Custom Word2Vec models can also be trained on datasets

6.1 Train the model

Output

You can also use the most_similar( ) function to find similar words to a given word.

Output

6.2 Update the model

Output

7. Create Doc2Vec model using Gensim

Doc2Vec extends Word2Vec by generating vector representations for entire documents instead of individual words. It helps identify relationships and similarities between documents based on their content.

7.1 Train the model

Load the dataset, Define a function to list the tagged documents, and train the dataset.

Output

7.2 Update the model Code

Initialize the model, build the vocabulary, Train the Doc2Vec model and Analyze the output.

Output

8. Topic Modelling

8.1 Illustration of NLP Topic-based Categorization

Topic modeling groups related words into meaningful topics based on their occurrence patterns in documents.

• Topic 1 represents words related to liquids or containers
• Topic 2 represents sports-related words
• Topic 3 represents vehicle-related words

Some of the Topic Modelling Techniques are:

• Latent Dirichlet Allocation (LDA)
• Latent Semantic Indexing (LSI)

8.2 Topic Modelling using LDA

Latent Dirichlet Allocation (LDA) is a topic modeling technique that treats each document as a mixture of multiple topics. The quality of generated topics depends on text preprocessing, selecting the optimal number of topics and tuning model parameters.

• Identifies hidden topics in documents
• Represents documents as combinations of topics
• Topic quality depends on preprocessing and parameter tuning

8.2.1 Prepare the Data
Data preparation includes removing stopwords and performing lemmatization before training the LDA model.

• Removes unnecessary stopwords
• Applies lemmatization for word normalization
• Uses Gensim and NLTK for preprocessing tasks

pip install pattern

Now, we will import nltk and key components.

Here, we have pre-processed the data by removing stopwords and lemmatization.

Output

8.2.2 Create Dictionary and Corpus
The processed data will now be used to create the dictionary and corpus. 

8.2.3 Train LDA model
We will be training the LDA model with 5 topics using the dictionary and corpus created previously. Here the LdaModel( ) function is used but you can also use the LdaMulticore( ) function as it allows parallel processing. 

Output

8.2.4 Interpret the Output
The LDA model majorly gives us information regarding 3 things: 

• Topics in the document
• What topic each word belongs to
• Phi value: Probability of a word to lie in a particular topic. For a given word, sum of the phi values give the number of times that word occurred in the document.

8.3 Topic Modelling using LSI

To create the model with LSI just follow the steps same as with LDA. The only difference will be while training the model. Use the LsiModel( ) function instead of the LdaMulticore( ) or LdaModel( ). We trained the model using LSI and then printed the topics.

9. Compute Similarity Matrices

Similarity matrices are used in NLP to measure how closely related two text documents or vectors are. Cosine similarity compares vectors based on the angle between them, while soft cosine similarity also considers relationships between similar words using word embeddings.

• Soft cosine similarity considers semantic similarity between words
• Values closer to 1 indicate higher similarity
• Word embeddings like Word2Vec are used for soft cosine similarity
• Commonly used in text analysis and information retrieval

Output

100%|██████████| 14/14 [00:11<00:00, 1.23it/s]Similarity between s1 and document 1: 1.0000

Similarity between s1 and document 2: 0.8372

Similarity between s1 and document 3: 0.7568

Some of the similarity and distance metrics which can be calculated for this word embedding model are mentioned below: 

10. Text Summarization using Gensim

Gensim provides the summarize() function for automatic text summarization using the TextRank algorithm. It extracts the most important sentences from a document to generate a shorter summary.

• Uses the TextRank summarization technique
• Automatically identifies important sentences
• No need for manual tokenization or sentence splitting
• Useful for summarizing large text documents

Output

11. Extracting Important Keywords from Text

You can get the Important keywords from the paragraph.

Output

Gensim library comes most handy while working on language processing.

Related Articles:

• Word2Vec using Gensim
• Topic Modelling using Latent Dirichlet Allocation (LDA)
• Word Embedding and Cosine Similarity using Gensim