Segment Anything : A Foundation Model for Image Segmentation

In computer vision, segmenting an image into separate segments or regions is a crucial operation. The article "Segment Anything – A Foundation Model for Image Segmentation" provides an introduction to Attention Res-UNet which is an essential model for making separate aspects visible through images.

In this article, we explore the idea of a foundation model designed for image segmentation, which includes its structure and how to execute it in several stages such as data preparation, creation, learning as well as outcome forecasts, also talk about performance evaluation measures of the product and offers some examples for a better understanding of its use across different fields too.

Overview of Image Segmentation

The significance of image segmentation extends beyond mere visual understanding, permeating into diverse domains and industries. In medical imaging, for instance, segmentation plays a pivotal role in delineating anatomical structures, identifying lesions, and assisting in disease diagnosis and treatment planning. Similar to this, segmentation helps with urban planning, environmental monitoring, and land cover classification in satellite imaging analysis. Furthermore, precise environment segmentation is essential for path planning, obstacle detection, and scene comprehension in the context of autonomous driving.

There has been an unbroken link between the development of deep learning techniques and the growth of image segmentation techniques ever since the introduction of convolutional neural networks (CNNs). With their extraordinary ability to capture complex spatial relationships and hierarchical representations found in images, these deep-learning architectures have completely changed the field of image segmentation. Researchers and professionals have been able to accomplish previously unheard-of levels of precision and effectiveness in segmentation jobs across numerous areas because of CNNs.

What is Attention Res-UNet

Attention ResUNet is an advanced neural network architecture for high-precision image segmentation, particularly in medical imaging. It integrates the strengths of UNet's encoder-decoder structure, ResNet's residual learning, and attention mechanisms to enhance segmentation accuracy and efficiency. The residual blocks facilitate training deeper networks by maintaining gradient flow, while attention gates focus on relevant image regions, improving feature representation. This combination allows Attention ResUNet to deliver superior performance in tasks like tumour detection, organ segmentation, and retinal vessel segmentation, making it a powerful tool for complex segmentation challenges.

Image Segmentation Stepwise Implementation

Putting into practice a foundation model for picture segmentation requires a methodical process that includes multiple crucial components. A thorough explanation of the implementation procedure is provided below:

Step 1: Import necessary libraries

Step 2: Download and Extract Mask Images

Step 3: Load Mask Images and Image Dataset

Load Mask Images

Output:

Mask Image Dataset Size: 10015
 image_id
856 ISIC_0026955_segmentation.png
8481 ISIC_0029327_segmentation.png
3663 ISIC_0031816_segmentation.png
5760 ISIC_0029516_segmentation.png
141 ISIC_0030870_segmentation.png

Load Image Dataset

Output:

Image Dataset Size: 5000
 image_id
658 ISIC_0027306.jpg
3046 ISIC_0027958.jpg
153 ISIC_0026017.jpg
2846 ISIC_0027431.jpg
2056 ISIC_0026607.jpg

Step 4: Preprocessing

Load and Resize Images and Masks

Output:

Loaded Images: 1000
Images Not Found: []

Display Image and Mask

Output:

Step 5: Model Building

• Model Architecture: Consists of residual connections and attention processes combined with the U-Net topology.
• Encoder Block: This four-layer block combines max pooling and a residual convolutional block for downsampling.
• Decoder block: The four-layer decoder block combines an attention block, a residual convolutional block, and upsampling.
• 1x1 convolutions (3-layers) are used to get the final output, which is then batch normalized and sigmoid activated.Using the training dataset and validation, the Attention Residual U-Net model is trained.

Define Helper Functions

Attention block

Attention block for enhancing feature maps from the encoder using gating signal from the decoder

Encoder block

Encoder block consisting of a residual convolutional block and a max pooling layer

Decoder block

Define Attention ResUNet Model

Model Summary

Output:

Model: "AttentionResUNet"
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ input_layer_2 │ (None, 128, 128, │ 0 │ - │
│ (InputLayer) │ 3) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_96 (Conv2D) │ (None, 128, 128, │ 1,792 │ input_layer_2[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_96[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_62 │ (None, 128, 128, │ 0 │ batch_normalizat… │
│ (Activation) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_98 (Conv2D) │ (None, 128, 128, │ 256 │ input_layer_2[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_97 (Conv2D) │ (None, 128, 128, │ 36,928 │ activation_62[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_98[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_97[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_26 (Add) │ (None, 128, 128, │ 0 │ batch_normalizat… │
│ │ 64) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_63 │ (None, 128, 128, │ 0 │ add_26[0][0] │
│ (Activation) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ max_pooling2d_8 │ (None, 64, 64, │ 0 │ activation_63[0]… │
│ (MaxPooling2D) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_99 (Conv2D) │ (None, 64, 64, │ 73,856 │ max_pooling2d_8[… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_99[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_64 │ (None, 64, 64, │ 0 │ batch_normalizat… │
│ (Activation) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_101 (Conv2D) │ (None, 64, 64, │ 8,320 │ max_pooling2d_8[… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_100 (Conv2D) │ (None, 64, 64, │ 147,584 │ activation_64[0]… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_101[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_100[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_27 (Add) │ (None, 64, 64, │ 0 │ batch_normalizat… │
│ │ 128) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_65 │ (None, 64, 64, │ 0 │ add_27[0][0] │
│ (Activation) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ max_pooling2d_9 │ (None, 32, 32, │ 0 │ activation_65[0]… │
│ (MaxPooling2D) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_102 (Conv2D) │ (None, 32, 32, │ 295,168 │ max_pooling2d_9[… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_102[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_66 │ (None, 32, 32, │ 0 │ batch_normalizat… │
│ (Activation) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_104 (Conv2D) │ (None, 32, 32, │ 33,024 │ max_pooling2d_9[… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_103 (Conv2D) │ (None, 32, 32, │ 590,080 │ activation_66[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_104[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_103[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_28 (Add) │ (None, 32, 32, │ 0 │ batch_normalizat… │
│ │ 256) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_67 │ (None, 32, 32, │ 0 │ add_28[0][0] │
│ (Activation) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ max_pooling2d_10 │ (None, 16, 16, │ 0 │ activation_67[0]… │
│ (MaxPooling2D) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_105 (Conv2D) │ (None, 16, 16, │ 1,180,160 │ max_pooling2d_10… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_105[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_68 │ (None, 16, 16, │ 0 │ batch_normalizat… │
│ (Activation) │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_107 (Conv2D) │ (None, 16, 16, │ 131,584 │ max_pooling2d_10… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_106 (Conv2D) │ (None, 16, 16, │ 2,359,808 │ activation_68[0]… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_107[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_106[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_29 (Add) │ (None, 16, 16, │ 0 │ batch_normalizat… │
│ │ 512) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_69 │ (None, 16, 16, │ 0 │ add_29[0][0] │
│ (Activation) │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ max_pooling2d_11 │ (None, 8, 8, 512) │ 0 │ activation_69[0]… │
│ (MaxPooling2D) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_108 (Conv2D) │ (None, 8, 8, │ 4,719,616 │ max_pooling2d_11… │
│ │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 8, 8, │ 4,096 │ conv2d_108[0][0] │
│ (BatchNormalizatio… │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_70 │ (None, 8, 8, │ 0 │ batch_normalizat… │
│ (Activation) │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_110 (Conv2D) │ (None, 8, 8, │ 525,312 │ max_pooling2d_11… │
│ │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_109 (Conv2D) │ (None, 8, 8, │ 9,438,208 │ activation_70[0]… │
│ │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 8, 8, │ 4,096 │ conv2d_110[0][0] │
│ (BatchNormalizatio… │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 8, 8, │ 4,096 │ conv2d_109[0][0] │
│ (BatchNormalizatio… │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_30 (Add) │ (None, 8, 8, │ 0 │ batch_normalizat… │
│ │ 1024) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_71 │ (None, 8, 8, │ 0 │ add_30[0][0] │
│ (Activation) │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_111 (Conv2D) │ (None, 8, 8, 512) │ 524,800 │ activation_71[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 8, 8, 512) │ 2,048 │ conv2d_111[0][0] │
│ (BatchNormalizatio… │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_72 │ (None, 8, 8, 512) │ 0 │ batch_normalizat… │
│ (Activation) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_113 (Conv2D) │ (None, 8, 8, 512) │ 262,656 │ activation_72[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_transpose_8 │ (None, 8, 8, 512) │ 2,359,808 │ conv2d_113[0][0] │
│ (Conv2DTranspose) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_112 (Conv2D) │ (None, 8, 8, 512) │ 1,049,088 │ activation_69[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_31 (Add) │ (None, 8, 8, 512) │ 0 │ conv2d_transpose… │
│ │ │ │ conv2d_112[0][0] │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_73 │ (None, 8, 8, 512) │ 0 │ add_31[0][0] │
│ (Activation) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_114 (Conv2D) │ (None, 8, 8, 1) │ 513 │ activation_73[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_74 │ (None, 8, 8, 1) │ 0 │ conv2d_114[0][0] │
│ (Activation) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_16 │ (None, 16, 16, 1) │ 0 │ activation_74[0]… │
│ (UpSampling2D) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ lambda_8 (Lambda) │ (None, 16, 16, │ 0 │ up_sampling2d_16… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ multiply_8 │ (None, 16, 16, │ 0 │ lambda_8[0][0], │
│ (Multiply) │ 512) │ │ activation_69[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_115 (Conv2D) │ (None, 16, 16, │ 262,656 │ multiply_8[0][0] │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_17 │ (None, 16, 16, │ 0 │ activation_71[0]… │
│ (UpSampling2D) │ 1024) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_115[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ concatenate_8 │ (None, 16, 16, │ 0 │ up_sampling2d_17… │
│ (Concatenate) │ 1536) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_116 (Conv2D) │ (None, 16, 16, │ 7,078,400 │ concatenate_8[0]… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_116[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_75 │ (None, 16, 16, │ 0 │ batch_normalizat… │
│ (Activation) │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_118 (Conv2D) │ (None, 16, 16, │ 786,944 │ concatenate_8[0]… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_117 (Conv2D) │ (None, 16, 16, │ 2,359,808 │ activation_75[0]… │
│ │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_118[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 2,048 │ conv2d_117[0][0] │
│ (BatchNormalizatio… │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_32 (Add) │ (None, 16, 16, │ 0 │ batch_normalizat… │
│ │ 512) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_76 │ (None, 16, 16, │ 0 │ add_32[0][0] │
│ (Activation) │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_119 (Conv2D) │ (None, 16, 16, │ 131,328 │ activation_76[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 16, 16, │ 1,024 │ conv2d_119[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_77 │ (None, 16, 16, │ 0 │ batch_normalizat… │
│ (Activation) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_121 (Conv2D) │ (None, 16, 16, │ 65,792 │ activation_77[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_transpose_9 │ (None, 16, 16, │ 590,080 │ conv2d_121[0][0] │
│ (Conv2DTranspose) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_120 (Conv2D) │ (None, 16, 16, │ 262,400 │ activation_67[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_33 (Add) │ (None, 16, 16, │ 0 │ conv2d_transpose… │
│ │ 256) │ │ conv2d_120[0][0] │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_78 │ (None, 16, 16, │ 0 │ add_33[0][0] │
│ (Activation) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_122 (Conv2D) │ (None, 16, 16, 1) │ 257 │ activation_78[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_79 │ (None, 16, 16, 1) │ 0 │ conv2d_122[0][0] │
│ (Activation) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_18 │ (None, 32, 32, 1) │ 0 │ activation_79[0]… │
│ (UpSampling2D) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ lambda_9 (Lambda) │ (None, 32, 32, │ 0 │ up_sampling2d_18… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ multiply_9 │ (None, 32, 32, │ 0 │ lambda_9[0][0], │
│ (Multiply) │ 256) │ │ activation_67[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_123 (Conv2D) │ (None, 32, 32, │ 65,792 │ multiply_9[0][0] │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_19 │ (None, 32, 32, │ 0 │ activation_76[0]… │
│ (UpSampling2D) │ 512) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_123[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ concatenate_9 │ (None, 32, 32, │ 0 │ up_sampling2d_19… │
│ (Concatenate) │ 768) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_124 (Conv2D) │ (None, 32, 32, │ 1,769,728 │ concatenate_9[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_124[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_80 │ (None, 32, 32, │ 0 │ batch_normalizat… │
│ (Activation) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_126 (Conv2D) │ (None, 32, 32, │ 196,864 │ concatenate_9[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_125 (Conv2D) │ (None, 32, 32, │ 590,080 │ activation_80[0]… │
│ │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_126[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 1,024 │ conv2d_125[0][0] │
│ (BatchNormalizatio… │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_34 (Add) │ (None, 32, 32, │ 0 │ batch_normalizat… │
│ │ 256) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_81 │ (None, 32, 32, │ 0 │ add_34[0][0] │
│ (Activation) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_127 (Conv2D) │ (None, 32, 32, │ 32,896 │ activation_81[0]… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 32, 32, │ 512 │ conv2d_127[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_82 │ (None, 32, 32, │ 0 │ batch_normalizat… │
│ (Activation) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_129 (Conv2D) │ (None, 32, 32, │ 16,512 │ activation_82[0]… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_transpose_10 │ (None, 32, 32, │ 147,584 │ conv2d_129[0][0] │
│ (Conv2DTranspose) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_128 (Conv2D) │ (None, 32, 32, │ 65,664 │ activation_65[0]… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_35 (Add) │ (None, 32, 32, │ 0 │ conv2d_transpose… │
│ │ 128) │ │ conv2d_128[0][0] │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_83 │ (None, 32, 32, │ 0 │ add_35[0][0] │
│ (Activation) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_130 (Conv2D) │ (None, 32, 32, 1) │ 129 │ activation_83[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_84 │ (None, 32, 32, 1) │ 0 │ conv2d_130[0][0] │
│ (Activation) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_20 │ (None, 64, 64, 1) │ 0 │ activation_84[0]… │
│ (UpSampling2D) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ lambda_10 (Lambda) │ (None, 64, 64, │ 0 │ up_sampling2d_20… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ multiply_10 │ (None, 64, 64, │ 0 │ lambda_10[0][0], │
│ (Multiply) │ 128) │ │ activation_65[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_131 (Conv2D) │ (None, 64, 64, │ 16,512 │ multiply_10[0][0] │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_21 │ (None, 64, 64, │ 0 │ activation_81[0]… │
│ (UpSampling2D) │ 256) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_131[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ concatenate_10 │ (None, 64, 64, │ 0 │ up_sampling2d_21… │
│ (Concatenate) │ 384) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_132 (Conv2D) │ (None, 64, 64, │ 442,496 │ concatenate_10[0… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_132[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_85 │ (None, 64, 64, │ 0 │ batch_normalizat… │
│ (Activation) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_134 (Conv2D) │ (None, 64, 64, │ 49,280 │ concatenate_10[0… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_133 (Conv2D) │ (None, 64, 64, │ 147,584 │ activation_85[0]… │
│ │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_134[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 512 │ conv2d_133[0][0] │
│ (BatchNormalizatio… │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_36 (Add) │ (None, 64, 64, │ 0 │ batch_normalizat… │
│ │ 128) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_86 │ (None, 64, 64, │ 0 │ add_36[0][0] │
│ (Activation) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_135 (Conv2D) │ (None, 64, 64, │ 8,256 │ activation_86[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 64, 64, │ 256 │ conv2d_135[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_87 │ (None, 64, 64, │ 0 │ batch_normalizat… │
│ (Activation) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_137 (Conv2D) │ (None, 64, 64, │ 4,160 │ activation_87[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_transpose_11 │ (None, 64, 64, │ 36,928 │ conv2d_137[0][0] │
│ (Conv2DTranspose) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_136 (Conv2D) │ (None, 64, 64, │ 16,448 │ activation_63[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_37 (Add) │ (None, 64, 64, │ 0 │ conv2d_transpose… │
│ │ 64) │ │ conv2d_136[0][0] │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_88 │ (None, 64, 64, │ 0 │ add_37[0][0] │
│ (Activation) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_138 (Conv2D) │ (None, 64, 64, 1) │ 65 │ activation_88[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_89 │ (None, 64, 64, 1) │ 0 │ conv2d_138[0][0] │
│ (Activation) │ │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_22 │ (None, 128, 128, │ 0 │ activation_89[0]… │
│ (UpSampling2D) │ 1) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ lambda_11 (Lambda) │ (None, 128, 128, │ 0 │ up_sampling2d_22… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ multiply_11 │ (None, 128, 128, │ 0 │ lambda_11[0][0], │
│ (Multiply) │ 64) │ │ activation_63[0]… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_139 (Conv2D) │ (None, 128, 128, │ 4,160 │ multiply_11[0][0] │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ up_sampling2d_23 │ (None, 128, 128, │ 0 │ activation_86[0]… │
│ (UpSampling2D) │ 128) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_139[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ concatenate_11 │ (None, 128, 128, │ 0 │ up_sampling2d_23… │
│ (Concatenate) │ 192) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_140 (Conv2D) │ (None, 128, 128, │ 110,656 │ concatenate_11[0… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_140[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_90 │ (None, 128, 128, │ 0 │ batch_normalizat… │
│ (Activation) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_142 (Conv2D) │ (None, 128, 128, │ 12,352 │ concatenate_11[0… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_141 (Conv2D) │ (None, 128, 128, │ 36,928 │ activation_90[0]… │
│ │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_142[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 256 │ conv2d_141[0][0] │
│ (BatchNormalizatio… │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ add_38 (Add) │ (None, 128, 128, │ 0 │ batch_normalizat… │
│ │ 64) │ │ batch_normalizat… │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_91 │ (None, 128, 128, │ 0 │ add_38[0][0] │
│ (Activation) │ 64) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ conv2d_143 (Conv2D) │ (None, 128, 128, │ 65 │ activation_91[0]… │
│ │ 1) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ batch_normalizatio… │ (None, 128, 128, │ 4 │ conv2d_143[0][0] │
│ (BatchNormalizatio… │ 1) │ │ │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ activation_92 │ (None, 128, 128, │ 0 │ batch_normalizat… │
│ (Activation) │ 1) │ │ │
└─────────────────────┴───────────────────┴────────────┴───────────────────┘
 Total params: 39,090,377 (149.12 MB)
 Trainable params: 39,068,871 (149.04 MB)
 Non-trainable params: 21,506 (84.01 KB)

Step 6: Model Training

Prepare Data for Training

Output:

(1000, 128, 128, 1)

Define Callbacks and Compile Model

Train Model

Output:

Epoch 1/50
32/32 [==============================] - 89s 962ms/step - loss: 0.4492 - accuracy: 0.8589
Epoch 2/50
32/32 [==============================] - 22s 676ms/step - loss: 0.3339 - accuracy: 0.9085
Epoch 3/50
32/32 [==============================] - 22s 678ms/step - loss: 0.2804 - accuracy: 0.9188
Epoch 4/50
32/32 [==============================] - 22s 679ms/step - loss: 0.2390 - accuracy: 0.9282
Epoch 5/50
32/32 [==============================] - 22s 684ms/step - loss: 0.2261 - accuracy: 0.9261
Epoch 6/50
32/32 [==============================] - 22s 690ms/step - loss: 0.2024 - accuracy: 0.9345
Epoch 7/50
32/32 [==============================] - 22s 692ms/step - loss: 0.1948 - accuracy: 0.9338
Epoch 8/50
32/32 [==============================] - 22s 693ms/step - loss: 0.1869 - accuracy: 0.9359
Epoch 9/50
32/32 [==============================] - 22s 694ms/step - loss: 0.1766 - accuracy: 0.9385
Epoch 10/50
32/32 [==============================] - 22s 697ms/step - loss: 0.1765 - accuracy: 0.9373
Epoch 11/50
32/32 [==============================] - 22s 700ms/step - loss: 0.1702 - accuracy: 0.9388
Epoch 12/50
32/32 [==============================] - 22s 702ms/step - loss: 0.1637 - accuracy: 0.9403
Epoch 13/50
32/32 [==============================] - 23s 705ms/step - loss: 0.1560 - accuracy: 0.9432
Epoch 14/50
32/32 [==============================] - 22s 699ms/step - loss: 0.1505 - accuracy: 0.9458
Epoch 15/50
32/32 [==============================] - 22s 700ms/step - loss: 0.1451 - accuracy: 0.9467
Epoch 16/50
32/32 [==============================] - 22s 702ms/step - loss: 0.1444 - accuracy: 0.9465
Epoch 17/50
32/32 [==============================] - 22s 700ms/step - loss: 0.1419 - accuracy: 0.9474
Epoch 18/50
32/32 [==============================] - 22s 698ms/step - loss: 0.1366 - accuracy: 0.9488
Epoch 19/50
32/32 [==============================] - 22s 701ms/step - loss: 0.1317 - accuracy: 0.9502
Epoch 20/50
32/32 [==============================] - 22s 701ms/step - loss: 0.1356 - accuracy: 0.9475
Epoch 21/50
32/32 [==============================] - 22s 700ms/step - loss: 0.1271 - accuracy: 0.9518
Epoch 22/50
32/32 [==============================] - 22s 699ms/step - loss: 0.1206 - accuracy: 0.9541
Epoch 23/50
32/32 [==============================] - 22s 700ms/step - loss: 0.1293 - accuracy: 0.9506
Epoch 24/50
32/32 [==============================] - 22s 701ms/step - loss: 0.1226 - accuracy: 0.9532
Epoch 25/50
32/32 [==============================] - 22s 701ms/step - loss: 0.1232 - accuracy: 0.9527
Epoch 26/50
32/32 [==============================] - 22s 699ms/step - loss: 0.1362 - accuracy: 0.9481
Epoch 27/50
32/32 [==============================] - 22s 699ms/step - loss: 0.1158 - accuracy: 0.9550
Epoch 28/50
32/32 [==============================] - 22s 699ms/step - loss: 0.1057 - accuracy: 0.9595
Epoch 29/50
32/32 [==============================] - 22s 700ms/step - loss: 0.1132 - accuracy: 0.9555
Epoch 30/50
32/32 [==============================] - 22s 702ms/step - loss: 0.1097 - accuracy: 0.9577
Epoch 31/50
32/32 [==============================] - 22s 702ms/step - loss: 0.0975 - accuracy: 0.9621
Epoch 32/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0986 - accuracy: 0.9617
Epoch 33/50
32/32 [==============================] - 22s 698ms/step - loss: 0.1057 - accuracy: 0.9579
Epoch 34/50
32/32 [==============================] - 22s 701ms/step - loss: 0.0950 - accuracy: 0.9627
Epoch 35/50
32/32 [==============================] - 22s 703ms/step - loss: 0.0931 - accuracy: 0.9634
Epoch 36/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0878 - accuracy: 0.9655
Epoch 37/50
32/32 [==============================] - 22s 699ms/step - loss: 0.1033 - accuracy: 0.9596
Epoch 38/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0928 - accuracy: 0.9638
Epoch 39/50
32/32 [==============================] - 23s 704ms/step - loss: 0.0972 - accuracy: 0.9618
Epoch 40/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0953 - accuracy: 0.9623
Epoch 41/50
32/32 [==============================] - 22s 698ms/step - loss: 0.0808 - accuracy: 0.9679
Epoch 42/50
32/32 [==============================] - 22s 701ms/step - loss: 0.0744 - accuracy: 0.9708
Epoch 43/50
32/32 [==============================] - 22s 703ms/step - loss: 0.0691 - accuracy: 0.9732
Epoch 44/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0736 - accuracy: 0.9708
Epoch 45/50
32/32 [==============================] - 22s 698ms/step - loss: 0.0671 - accuracy: 0.9738
Epoch 46/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0681 - accuracy: 0.9733
Epoch 47/50
32/32 [==============================] - 22s 702ms/step - loss: 0.0685 - accuracy: 0.9728
Epoch 48/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0880 - accuracy: 0.9650
Epoch 49/50
32/32 [==============================] - 22s 699ms/step - loss: 0.0734 - accuracy: 0.9711
Epoch 50/50
32/32 [==============================] - 22s 700ms/step - loss: 0.0623 - accuracy: 0.9752

Step 7: Predictions

Helper Functions to Calculate Area

Predict and Display Results

Output:

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step

Text(0.5, 1.0, 'Ground Truth Mask')

Application of Attention Res-UNet in Image Segmentation

Applying foundation models to image segmentation provides an efficient way to handle challenging segmentation jobs in many fields. Using these pre-trained models has a number of advantages, including promoting creativity, increasing performance, and enabling effective model development. The following are some salient features that illustrate the application of foundation models to picture segmentation:

• Effective Model Development: By offering pre-trained networks, foundation models expedite the development process and save time and resources that would otherwise be required for initial training. Model deployment for segmentation tasks is accelerated as a result.
• Benefits of Transfer Learning: By utilizing transfer learning, foundation models apply pre-training knowledge to particular segmentation tasks. This improves model performance and generalization by making it easier to collect generic visual properties and spatial relationships.
• Improved Performance: A foundation model's accuracy and efficiency are increased when it is adjusted on a target dataset to better suit the specifics of the segmentation task. This flexibility guarantees that the model operates at its best across a variety of settings and datasets.
• Versatility Across Domains: Foundation models exhibit versatility, since they can effectively tackle segmentation problems in a range of domains like as item identification, medical imaging, and environmental monitoring. Their versatility enables customisation to satisfy certain objectives and use scenarios.
• Real-world Applications and Seamless Integration: These models allow for quick prototype and deployment in real-world applications by integrating smoothly into current workflows and frameworks. With their advanced segmentation capabilities, foundation models enable practitioners to tackle a wide range of problems, from medical diagnosis to urban planning.

Performance Evaluation and Case Studies of Attention Res-UNet

When evaluating the efficacy and practicality of foundation models for image segmentation, case studies and performance evaluation are essential tools. Let's examine these features in more detail:

• Metrics Assessment: Metrics like Mean Average Precision (mAP), Dice Coefficient, and Intersection over Union (IoU) are used to assess foundation models. These measures put segmentation robustness, accuracy, and consistency into numerical form.
• Benchmarking Against Ground Truth: To verify the effectiveness of the model, segmentation results are compared to annotations from the ground truth. This guarantees that objects and regions inside photos are correctly delineated by the model.
• Validation on Test Datasets: To confirm the model's generalization ability, extensive validation is carried out on various test datasets. To ensure reliability, this entails evaluating performance across many datasets and circumstances.
• Accuracy: The primary metric for evaluating the performance of Attention ResUNet models is accuracy. This involves comparing the model's segmentation results against ground truth labels. Higher accuracy indicates better performance in accurately identifying and delineating objects in images.
• Speed and Efficiency: Apart from accuracy metrics, the computational efficiency of the Attention ResUNet model is also crucial, especially in real-time applications. Evaluation should include metrics such as inference time and model size to assess efficiency.

Case Studies:

• Medical Imaging: In medical imaging, foundation models are used to perform tasks including organ segmentation, tumor detection, and illness diagnosis. Case examples illustrate how these models enhance the precision of diagnoses and support clinicians in making treatment decisions.
• Autonomous Driving: Foundation models provide for robust scene analysis, obstacle recognition, and path planning in the field of autonomous driving. Case examples demonstrate how these models improve self-driving car safety and navigation.
• Satellite Imagery Analysis: Applications of foundation models include land cover classification, urban planning, and environmental monitoring in satellite imaging analysis. Case studies demonstrate how useful these models are for gleaning insightful information from remote sensing data.
• Object Detection and Recognition: Foundation models are used for tasks like object detection and recognition across a variety of domains, going beyond segmentation. Case examples show how adaptable and effective these models are at identifying and categorizing objects of interest.
• Industrial Applications: Process optimization, defect detection, and quality control are all achieved in industrial settings through the use of foundation models. Case studies demonstrate how these models enhance productivity and dependability in production and manufacturing settings.

Conclusion

To sum up, foundation models offer a reliable and effective framework for model creation and implementation, marking a paradigm shift in the field of image segmentation. Researchers and practitioners can use transfer learning to efficiently and accurately handle a variety of segmentation problems by utilizing pre-trained CNN architectures. Foundation models will become more and more important as the industry develops since they will push innovation, improve image segmentation, and promote cooperation amongst many sectors and domains.