Complement Naive Bayes (CNB) Algorithm

Complement Naive Bayes (CNB) is a variant of the Naive Bayes algorithm that is specifically designed to improve classification performance on imbalanced datasets and text classification tasks. It modifies the way probabilities are estimated to reduce bias towards majority classes, making it more suitable than the standard Multinomial Naive Bayes in many cases.

Challenge of Unbalanced Datasets

An unbalanced dataset means one type of data appears much more often than the other. This often happens in spam filtering (more normal emails than spam) or medical diagnosis (more healthy cases than disease cases).

Example:

If 95% of cases are "not fraud" and only 5% are "fraud," a model that always predicts "not fraud" will be 95% accurate but will miss all fraud cases. This shows why special methods are needed to deal with such uneven data.

Working

1. For each class, compute the complement frequency: the frequency of features in all other classes combined.
2. Estimate the conditional probabilities using these complement frequencies.
3. Normalize the values to ensure they form valid probability distributions.
4. Classify a sample by choosing the class with the maximum posterior probability.

For a class C and feature F:

• = count of feature f in the complement of class c
• = smoothing parameter (Laplace smoothing)
• = vocabulary size

Example

Suppose classifying sentences as Apples or Bananas using word frequencies, To classify a new sentence (Round=1, Red=1, Soft=1):

• MNB would estimate probabilities for Apples using only Apples data
• CNB estimates probabilities for Apples using Bananas' data (complement) and vice versa

Solving by CNB: We classify a new sentence with features {Round =1, Red =1, Soft =1} and vocabulary {Round, Red, Soft}.

Step 1: Complement counts

• For Apples, use Bananas’ counts -> {Round:5, Red:1, Soft:3}
• For Bananas, use Apples’ counts -> {Round:3, Red:4, Soft:1}

Step 2: Probabilities (using Laplace smoothing, α =1)

For Apples:

• Round = (5+1)/(5+1+3+3) = 6/12 = 0.5
• Red = (1+1)/12 = 0.167
• Soft = (3+1)/12 = 0.333

For Bananas:

• Round = (3+1)/(3+1+4+1) = 4/11 ≈ 0.364
• Red = (4+1)/11 = 0.455
• Soft = (1+1)/11 = 0.182

Step 3: Scores, Multiply feature probabilities:

• Apples = 0.5 × 0.167 × 0.333 ≈ 0.0278
• Bananas = 0.364 × 0.455 × 0.182 ≈ 0.0301

Final Result -> Bananas

Implementing CNB

We can implement CNB using scikit-learn on the wine dataset (for demonstration purposes).

1. Import libraries and load data

We will import and load the required libraries

• Import load_wine for dataset loading from sklearn.
• Use train_test_split to divide data into training and test sets.
• Import ComplementNB as the classifier.
• Import evaluation metrics: classification_report and accuracy_score.

2. Split into training and test sets

We will split the dataset into training and test sets:

• Split the dataset into 70% training and 30% testing data.
• Set random_state=42 for reproducibility.

3. Train the CNB classifier

We will train the Complement Naive Bayes classifier

• Create a ComplementNB instance.
• Fit the classifier on the training data.

4. Evaluate the model

We will now evaluate the trained model:

• Predict class labels for the test set using predict().
• Print the accuracy score and the classification report for detailed metrics.

Note: CNB is better suited for discrete data like text. For continuous features (as in this dataset), Gaussian Naive Bayes might perform better.

When to Use CNB

Limitations

• Assumes feature independence, which may reduce accuracy in real data.
• Works best with discrete features like word counts; continuous data needs preprocessing.
• May introduce bias in already balanced datasets, reducing its advantage.

Related articles

• Naive Bayes Classifiers
• Gaussian Naive Bayes
• Multinomial Naive Bayes