Lung Cancer Detection using Convolutional Neural Network (CNN)

Computer Vision is one of the applications of deep neural networks and one such use case is in predicting the presence of cancerous cells. In this article, we will learn how to build a classifier using Convolution Neural Network which can classify normal lung tissues from cancerous tissues.

The following process will be followed to build this classifier:

Implementation

Step 1: Importing Libraries

We use libraries for data handling, visualization and deep learning model development.

• Pandas, NumPy, Matplotlib and Scikit-learn for data handling and analysis.
• OpenCV for image processing.
• TensorFlow to build and train machine learning models, including CNNs.

Step 2: Importing Dataset

The dataset used is a custom subset (~2400images) containing three lung cancer categories

• Normal (lung_n)
• Adenocarcinoma (lung_aca)
• Squamous cell carcinoma (lung_scc)

Output:

The data set has been extracted.

Step 3: Visualizing the Data

We visualize random images from each category to understand the dataset.

Output:

Step 4: Preparing the Dataset

Instead of loading all images into memory, we use ImageDataGenerator to load images batch-by-batch.

• Rescales pixel values
• Splits data into training (80%) and validation (20%)
• Prevents memory overflow

Output:

Step 5: Model Development

Now we start building the CNN. Here, we use TensorFlow and Keras to define the architecture of our CNN model.

• Sequential(): Builds a linear stack of layers.
• Conv2D(): Applies convolution with specified filters, kernel size, ReLU activation and padding.
• MaxPooling2D(): Downsamples feature maps by taking max values over pool size.
• Flatten(): Converts 2D feature maps into 1D vector.
• Dense(): Fully connected layer with given units and activation.
• BatchNormalization(): Normalizes activations to speed up training.
• Dropout(): Randomly drops neurons to reduce overfitting.
• model.summary(): Displays model architecture details.

Output:

Step 6: Model Compilation

After defining the model architecture we will compile the model with an optimizer, loss function and evaluation metric then train it using the training data.

• We use the Adam optimizer which adjusts the learning rate during training to speed up convergence.
• Categorical cross entropy loss is appropriate as loss function for multi-class classification problems as it measures the difference between the predicted and actual probability distributions.
• EarlyStopping:Stops training if validation accuracy doesn’t improve for a set number of epochs (patience).
• ReduceLROnPlateau: Reduces learning rate when validation loss plateaus, controlled by patience and factor.
• Custom myCallback class: Stops training early when validation accuracy exceeds 90%.
• self.model.stop_training = True: Signals to stop training inside the callback.

Step 7: Model Training

The model is trained using generator-based data.

Output:

Step 8: Visualizing

Let's visualize the training and validation accuracy with each epoch.

• pd.DataFrame(history.history) converts training history into a DataFrame.
• history_df.loc[:, ['accuracy', 'val_accuracy']].plot() plots training and validation accuracy.

Output:

This graph shows the training and validation accuracy of the model over epochs. The training accuracy increases steadily reaching near 1.0 indicating the model is learning well from the training data. Although it's the case of overfitting

Step 9: Model Evaluation

We evaluate the model using predictions from validation data.

Output:

Download full code from here