Sentiment Analysis using NLTK

Sentiment Analysis using NLTK involves analyzing text data to determine whether the expressed opinion is positive, negative or neutral. NLTK provides essential tools for text preprocessing, tokenization, and sentiment scoring, making it a popular choice for basic NLP sentiment classification tasks.

• Performs text preprocessing like tokenization and stop‑word removal
• Uses sentiment lexicons such as VADER for polarity scoring
• Commonly applied in reviews, social media analysis, and feedback systems

Implementing Sentimental Analysis using NLTK

In this section, we are going to perform sentiment analysis with NLTK using Twitter samples dataset using the following steps:

Step 1: Installing NLTK

We need to install NLTK and support packages for our model.

NLTK packages:

Step 2: Loading and Preprocessing Data

The sample documents can be downloaded from here.

For sentiment analysis:

• Loads positive and negative sentence lists from JSON files using Python’s json.load() for structured access.
• Combines and shuffles data for randomized training and testing splits, preventing bias in model training.

Output:

Total samples loaded: 196

Step 3: Explore Data Analysis (EDA)

Quick look at data distribution:

• Uses pandas to visualize and count label distribution for data balance.
• Displays sample entries to inspect text format and sentiment labeling.

Output:

Step 4: Text Preprocessing Pipeline

• Defines a function to tokenize sentences into words with word_tokenize.
• Converts words to lowercase, removes stop words and punctuation for noise reduction.
• Lemmatizes tokens using WordNetLemmatizer to reduce inflectional forms to base words.
• Applies preprocessing to all sentences, producing cleaned text for feature extraction.

Output:

Step 5: Feature Extraction

• Transforms processed text into numerical vectors using CountVectorizer creating a word occurrence matrix also known as Bag of Words.
• Maps sentiment labels to binary values (positive = 1, negative = 0) for model compatibility.
• Outputs matrix shape for verifying feature dimensionality.

Output:

Feature matrix shape: (196, 347)

Step 6: Train-Test Split

We will:

• Splits features and labels into training and test sets with train_test_split, preserving class proportions (stratify).
• Enables objective model evaluation.

Step 7: Model Training

Use Multinomial Naive Bayes:

• Trains a MultinomialNB classifier, which is efficient for word frequency-based text data.
• Fits the model on training data to learn text-sentiment relationships.

Output:

Step 8: Model Evaluation

• Predicts sentiments on the test set and computes accuracy score.
• Prints classification report detailing precision, recall and F1-score for each class.
• Generates confusion matrix to visualize prediction correctness and misclassifications.

Output:

Step 9: Visualization of Confusion Matrix

We will visualize the confusion matrix:

• Use matplotlib and seaborn to plot a heatmap for the confusion matrix.
• Visualizes true vs predicted labels for diagnostic insights.

Output:

Step 10: Use NLTK’s VADER Sentiment Analyzer (Lexicon-based)

Here:

• Instantiates SentimentIntensityAnalyzer to compute sentiment polarity.
• Defines a function returning 'positive', 'negative', or 'neutral' based on the compound score.
• Applies VADER to example and dataset sentences to compare lexicon-based vs ML-based approaches.

Output:

Positive

Step 11: Compare VADER and ML Model Predictions

• Maps VADER sentiment outputs to binary for accuracy comparison against labels.
• Calculates and prints VADER-based accuracy for benchmark evaluation.

Output:

VADER Sentiment Accuracy compared to labels:
0.9548387096774194

Step 12: Predict Results on New Sentences

We will predict the results of more sentences using the model,

• Preprocesses new input sentences and predicts sentiment using the trained model.
• Demonstrates real-world use for unseen text.

Output:

Sentence: This is an amazing product with great quality.
=> Sentiment: positive

Sentence: I did not like the service at all, very poor.
=> Sentiment: negative

As we can see the model was able to predict whether it was positive or negative emotion.

Step 13: Save the Model and Vectorizer

Here:

• Serializes the trained model and vectorizer with joblib.dump, enabling future predictions without retraining.
• Facilitates deployment and reproducibility.

Output:

['count_vectorizer.joblib']