lung segmentation

using a machine learning project to look at different dieases and analyze what part of the lung is damaged
Anika Rastogi
Grade 10

Problem

in our world, many people are affected by pulmonary diseases, with more than 4 million individuals dying prematurely from these conditions each year. this issue is particularly severe in low- to middle-income countries. did you know that lung diseases are the third leading cause of death in the united states? this raises an important question: How can we reduce wait times and improve the accuracy of diagnostic devices?

one promising approach is lung segmentation, a technique that scans the lungs to identify damaged tissue or lesions. This leads us to a critical question:

how can lung segmentation enhance the accuracy of diagnosing pulmonary diseases such as tuberculosis and viral pneumonia?

Method

the method that i am using for this project is a mix of image recognition, and image segmentation, both using machine learning. 

 image recognition: https://www.mathworks.com/discovery/image-recognition-matlab.html#:~:text=Image%20recognition%20is%20the%20process,medical%20imaging%2C%20and%20security%20surveillance.: 

image recognition, also known as computer vision, is an expanding field within artificial intelligence that enables machines to interpret and recognize visual images.

this capability relies on advancements in algorithms and models that allow computers to detect patterns and objects in visual data. 

image recognition involves various algorithm designs that aim to replicate human visual systems to some limit.

the process begins by collecting a range of images and labeling the training data, or by using an existing dataset. this step is crucial as it enables the algorithm to learn to recognize patterns.

during the training phase, the code adjusts its parameters and features to minimize the differences among the images. once the ai is trained, the model can recognize unseen images, which is where the testing data comes into play. we can input a set of photos that the ai has not encountered before, and based on that specific group, the ai will make predictions and provide an accuracy percentage.

image segmentation is the process of dividing an image into segments, to make it easier to analyze. -> it starts by categorizing regions that have the same similarities. (shape, colour, and more)  

https://www.ibm.com/topics/image-segmentation#:~:text=Image%20segmentation%20is%20a%20computer,faster%2C%20more%20advanced%20image%20processing. 

simple version of how it works: (learned from junio tech)

1. preprocess the data -> like image recognition first thing we have to do is adjust the parameters. 

2. feature extraction -> it finds objects or shapes that are found in different parts of the image. 

3. segmentation ->  divides it based on the feature extraction. 

4. post-process -> refined the images. 

 

 

 

 

Research

research

preliminary result: i did the PREQUEL of this project last year, which was "classification of lung diseases", in which i took the viral pneumonia, bacterial pneumonia, COVID-19, tuberculosis, and normal lung datasets, and trained a model to classify them without a label. which i receive a 75% percent accuracy.

this year i am doing lung segmentation, which is finding the damaged tissue in the lungs, and "coloring" it.

segmentation processes (detailed) :

a) find a dataset:

https://data.mendeley.com/datasets/8gf9vpkhgy/1:

this has 3 files: darwin ( reveals the lung opacity behind/near the heart, the severity of viral pneumonia), montgomery (pulmonary tuberculosis), shenzhen (also pulmonary tuberculosis).

|_>  acquired by the departments of health and services. all of the datasets contain normal and abnormal images.

b) preprocess the data:

1.  gather all the data onto the code

  1. group the necessary files together

  2. change the size and color, so they match each other

  3. change the image into an array

  4. load the data so it's able to be used in the model model

c) train the datasets:

  1. using the files, each file gives a lung CT and a mask identical to the lung; 20 % of the portion is reserved for the validation proportion.

d) implicating the U Net model

  1. which specifies the shape of the image

2.  you have the initial layer (raw material) -> in the size of 256 x 256

  1. it starts by extracting features from all the photos, there are 64 filters at the beginning of my code. and 3x3 kernels, with kernels, determine the regions of the input process at each step.

  2. padding ensures the output is the same size as the input. and relu makes sure the ReLU function is activated -> creates the output if the input is positive.

  3. maxpool2D

6.  it create a larger number of filters ( 128) to ensure it gets everything.  and changes it kernel to 1x1.

** added more dimension that filters the photos a bit more.

  1. returnes the model.

  2. build the model

9. compile the model. in the code there should be these 3 things:

  • binary_crossentropy is the loss function, which is used to measure the difference between the predicted and actual values.

  • adam is the optimizer, learning rate.

  • accuracy is the metric used to evaluate the model and add percentages as averages.

10.  epoch is the last thing for the u-NET model, when it's the training session for the ai, at the end of each session, it gives the accuracy, loss, val_accurary, and val_loss. (val is the validation test)

e) create a plot and accuracy chart, to know when things are not working.

f) add a refined model, that fills in the black spaces, and more the image sharper.

f) visualize the models, one with just the prediction mask, and one with damaged tissues.

  1. using matplotlib. *****

g) save the model!!

whats the mask: a mask is an array with the same dimersion as the original image. each pixel values determines which pixel belongs in a specific region.

there are 2 values;

white(value 1) , represent the backgroud

black(value 0), represent the area of interst, in this case the damages tissues.

**real-life application *****:

a) medical imaging -> this could help the doctors, and residents, who are the backbone of the hospitals, to double check the certain part of the lung.

b) preprocessing the images for manual review: can preprocess it and mark high interst areas for review.

c) can be used to explained diagnosed treatment to med students.

d) tracking the disease in the human body over time. the ability to scan lungs over time, could lead to track changes. such as the progessions in covid-19 or pnemonia.

diseases

viral pneumonia is an infection in your lungs that is caused by a virus. the most common cause is the flu. the germs that come from the virus usually stick to the upper part of your lungs. -> when they go to the lower part of the lungs the air sacs get infected and they start filling up fluid. (which in the CT scans is the white "smoke") https://www.webmd.com/lung/viral-pneumonia

symptoms of viral pneumonia:

  • fever
  • dry cough
  • headache
  • sore throat
  • loss of appetite
  • sore muscles

how to prevent viral pneumonia: hygiene, taking pulmonary disease vaccines all of them will decrease your chances of any damage to the lungs (like the COVID-19 vaccine), lastly, smoking is a no-no, because smoke inhalation can not only the lung tissue but also the lungs skill/ability to defend itself from other infections.

risk factors:

  • age:

infants and young children

elderly individuals (age>65)

  • weak immune system:

HIV/AIDS, cancer, or use of the immunosupprestive drugs

chronic illness like diabetes or kidney diease.

  • chronic respiratory condiditon:

asthma, COPD, or cystic fibrosis

  • Smoking:
    • Damages lung tissue and impairs clearance of pathogens.
  • Crowded Living Conditions:
    • Nursing homes, prisons, or shelters where viral spread is higher.
  • Recent Respiratory Infection:
    • Cold or flu can increase susceptibility to viral pneumonia.
  • Exposure to Viruses:
    • Close contact with infected individuals (e.g., flu, RSV, or COVID-19).
  • Environmental Factors:
    • Pollution or exposure to irritants that weaken lung defenses.

Diagnosis:

  • Medical History and Symptoms:
    • Fever, cough, shortness of breath, fatigue, and chest discomfort.
  • Physical Examination:
    • Crackling or wheezing sounds heard with a stethoscope.
  • Chest X-ray:
    • Identifies lung inflammation or infiltrates typical of pneumonia.
  • Blood Tests:
    • Measures white blood cell count and detects viral infections.
  • Viral Testing:
    • Nasal swabs or throat swabs to identify specific viruses (e.g., influenza, RSV, or SARS-CoV-2).
  • Pulse Oximetry:
    • Assesses oxygen levels in the blood to detect hypoxemia.
  • Sputum Analysis:
    • Examines mucus for signs of infection or to rule out bacterial causes.

tuberculosis is a disease that affects meaningly the lungs, caused by a bacteria called mycobacterium. this disease can spread from when a person with the illness coughs or sneezes. this puts tiny droplets in the air with certain germs, which brings it to other people when they breathe air, infecting others. https://www.mayoclinic.org/diseases-conditions/tuberculosis/diagnosis-treatment/drc-20351256

there are 3 stages of tuberculosis:

primary tuberculosis: where the defense system captures most of the bacteria, but in many cases the bacteria may get away, and multiply.

latent tuberculosis infection;  right after the primary stage. in this stage the immune system creates a wall around the lung tissue with TB, so no further germs can spread

active tuberculosis stage: happens when the immune system defenses let down, and can no longer control the disease

symptoms of tuberculosis:

  • cough
  • coughing out blood
  • chest pain
  • pain with breathing
  • fever
  • chills
  • night sweats
  • weight loss
  • loss of appetite
  • fatigue

a quarter of the world is affected by tuberculosis, and 1/4 of that quarter can be cured by tuberculosis with antibiotics

Tuberculosis (TB)

Risk Factors:

  • Weakened Immune System:
    • HIV/AIDS (major risk factor).
    • Use of corticosteroids, chemotherapy, or biologic agents.
  • Close Contact:
    • Exposure to someone with active TB disease.
  • Living Conditions:
    • Crowded areas like prisons, shelters, or refugee camps.
  • Geography:
    • Living in or traveling to areas with high TB prevalence (e.g., sub-Saharan Africa, Southeast Asia).
  • Age:
    • Infants and elderly individuals are at higher risk.
  • Chronic Illnesses:
    • Diabetes, kidney disease, or malnutrition.
  • Substance Use:
    • Alcoholism or intravenous drug use.
  • Smoking:
    • Increases risk of TB infection and active disease.
  • Healthcare Workers:
    • Increased exposure to infected individuals.

Diagnosis:

  • Medical History and Symptoms:
    • Persistent cough (≥3 weeks), fever, night sweats, weight loss, and hemoptysis (coughing up blood).
  • Tuberculin Skin Test (TST):
    • A small amount of purified protein derivative (PPD) is injected under the skin, and the reaction is assessed after 48-72 hours.
  • Interferon-Gamma Release Assays (IGRA):
    • Blood test to detect TB infection (preferred for those vaccinated with BCG).
  • Chest X-ray or CT Scan:
    • Identifies lung abnormalities such as cavitations or nodules.
  • Sputum Microscopy:
    • Acid-fast bacilli (AFB) staining to detect TB bacteria.
  • Sputum Culture:
    • Confirms diagnosis and assesses drug sensitivity (gold standard, but takes weeks).
  • Molecular Testing:
    • PCR-based tests (e.g., GeneXpert) for rapid TB detection and resistance to rifampin.
  • Bronchoscopy:
    • May be used if sputum samples are inconclusive.

pneumothorax; medical condition where the air enters a plueral space, the area between lungs and chest wall. This stops the negative pressure that keeps the lung inflated, leading to a partial or complete lung collapse. → arrives without a obvious cause, result of trauma, or a symptom of another disease. → can be life threaten

types of pneumothorax:

  • spontaneous pneomothorax: happens without injure. can be

                  - primary: comes in healthy individual, often in tall thin young adult or smokers.

              - secondary: linked to previous or existing lung diseases

  • traumatic pneumothorax: can be caused by bunt or sharp chest trauma, like rib fracture, or accidents. can also be from medical procedures like central line insertions.

  • tension pnemothorax: a condition where air continues to collect in the pleural space, which causes high pressure that shrinks the lung, heart, and vessels.

symptoms:

  • sudden sharp breath or stabbing chest pain usually on one side
  • shortness of breath
  • fat heartbeat
  • tired or dizziness
  • dis coloration of the skin or lips

causes:

  • spontaneous rupture of small air blisters on the surface of the lung
  • chest trauma from accidents, or previous lung injuries
  • chronic lung disease like tuberculosis or pneumonia
  • medical procedure, like medical ventilation with high pressure

risk factors:

  • smoking, weakens the lungs’=
  • being young and thin, especially in young males
  • family history
  • pre-existing lung dieases ‘
  • activities that rapid pressure changes.

diagnosis:

  • physical exam: doctors may detected reduced breathing, on chest percussion
  • imaging:
  • chest x-ray: common diagnosis tool to confirm the presences of air
  • ct scans: provides a detailed view in complex.

 

Data

data

this was my code the first time:

import os
import numpy as np
import matplotlib.pyplot as plt #plot the image 
from tensorflow.keras.utils import load_img, img_to_array # type: ignore # load the images 
from tensorflow.keras import layers, models # type: ignore
from sklearn.model_selection import train_test_split # type: ignore # split the data.

darwin_img_dir = "/users/skquirtr/desktop/lungsegmentation/darwin/img" # the documents
darwin_mask_dir = "/users/skquirtr/desktop/lungsegmentation/darwin/mask"

montgomery_image_dir = "/users/skquirtr/desktop/lungsegmentation/montgomery/img"
montgomery_mask_dir = "/users/skquirtr/desktop/lungsegmentation/montgomery/mask"

shenzhen_image_dir = "/users/skquirtr/desktop/lungsegmentation/shenzhen/img"
shenzhen_mask_dir = "/users/skquirtr/desktop/lungsegmentation/shenzhen/mask"

def preprocess_data(image_dirs, mask_dirs, target_size =(256, 256)):
    images = []
    masks = []
    for image_dir, mask_dir in zip(image_dirs, mask_dirs): # combines the files so it'll be processed tghther. so they can be processed togther
        image_files = sorted(os.listdir(image_dir)) # serpeate the image and mask files/sorts them
        mask_files = sorted(os.listdir(mask_dir))
        for img_file, mask_file in zip(image_files, mask_files):
            # start to load and resize images
            img = load_img(os.path.join(image_dir, img_file), target_size= target_size, color_mode="grayscale")# make sure the image are in grayscale
            mask = load_img(os.path.join(mask_dir, mask_file), target_size= target_size, color_mode="grayscale")

            images.append(img_to_array(img) / 255.0) # converts image into an array.
            masks.append(img_to_array(mask) / 255.0)

    return np.array(images), np.array(masks)

# combine the directories for all three files
image_dirs = [darwin_img_dir, montgomery_image_dir, shenzhen_image_dir]
mask_dirs = [darwin_mask_dir, montgomery_mask_dir,shenzhen_mask_dir]

# load images and masks from all datasets
images, masks = preprocess_data(image_dirs, mask_dirs)

X_train, X_val, y_train, y_val = train_test_split(images, masks, test_size=0.2, random_state=42) # splits the data into 20% test and 80% train

# build the U-Net model
def unet_model(input_size=(256, 256, 1)):
    inputs = layers.Input(input_size) # only accepts 256 by 256 images
    conv1 = layers.Conv2D(64, 3, activation='relu', padding='same')(inputs) # extract features from the input. the 64 is the number of filters. and 3 is the kernel size, applies to the corners, and edges
    c1 = layers.Conv2D(64, 3, activation='relu', padding='same')(conv1) # padding ensures that the output is the same size as the input. relu is activation for the reLU function 
    c1 = layers.MaxPooling2D(pool_size=(2, 2))(c1) # reduce the dimesions of the data
    b = layers.Conv2D(128, 3, activation='relu', padding='same')(c1) # 128 is the number of filters 
    c2 = layers.Conv2DTranspose(64, 2, strides=(2, 2), activation='relu', padding='same')(b)# strides is the size of the stride
    outputs = layers.Conv2D(1, 1, activation='sigmoid')(c2) # final layer, 1 is the kernel size (1x1). and sigmoid is the activation function
    return models.Model(inputs, outputs) # returns the model 

model = unet_model() # build the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy']) # compile the model. binary_crossentropy is the loss function, which is used to measure the difference between the predicted and actual values. adam is the optimizer, learning rate. accuracy is the metric used to evaluate the model and add percentages as averages. 
history = model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=16) # number of "training sessions"

val_loss, val_accuracy = model.evaluate(X_val, y_val)
print(f"Validation Accuracy: {val_accuracy:.2f}, Loss: {val_loss:.2f}")
0    
def display_sample(X, y_true, y_pred, index):    
    plt.figure(figsize=(12, 4))
    plt.subplot(1, 3, 1)
    plt.title("Input Image")
    plt.imshow(X[index].squeeze(), cmap='gray')
    plt.subplot(1, 3, 2)
    plt.title("True Mask")
    plt.imshow(y_true[index].squeeze(), cmap='gray')
    plt.subplot(1, 3, 3)
    plt.title("Predicted Mask")
    plt.imshow(y_pred[index].squeeze(), cmap='gray')
    plt.show()

in this code, i ran 10 epochs (”training sessions”), getting the result of:

ending up to have an accuracy of 85%

janurary 15: the result of my model, for the school science fair

import osimport numpy as npimport randomimport matplotlib.pyplot as plt #plot the image from tensorflow.keras.utils import load_img, img_to_array # type: ignore # load the images from tensorflow.keras import layers, models # type: ignorefrom sklearn.model_selection import train_test_split # type: ignore # split the datafrom tensorflow.keras.preprocessing.image import ImageDataGenerator # type: ignore # data augmentation

darwin_img_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/darwin/darwin/img" # the documentsdarwin_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/darwin/darwin/mask"
montgomery_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/montgomery/montgomery/img"montgomery_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/montgomery/montgomery/mask"
shenzhen_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/shenzhen/shenzhen/img"shenzhen_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/shenzhen/shenzhen/mask"
pnemothorax_image_dir = "/home/anikimport os
import numpy as np
import random
import matplotlib.pyplot as plt #plot the image 
from tensorflow.keras.utils import load_img, img_to_array # type: ignore # load the images 
from tensorflow.keras import layers, models # type: ignore
from sklearn.model_selection import train_test_split # type: ignore # split the data
from tensorflow.keras.preprocessing.image import ImageDataGenerator # type: ignore # data augmentation

darwin_img_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/darwin/darwin/img" # the documents
darwin_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/darwin/darwin/mask"

montgomery_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/montgomery/montgomery/img"
montgomery_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/montgomery/montgomery/mask"

shenzhen_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/shenzhen/shenzhen/img"
shenzhen_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/shenzhen/shenzhen/mask"

pnemothorax_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/pneumothorax/siim-acr-pneumothorax/img"
pnemothorax_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/pneumothorax/siim-acr-pneumothorax/mask"

def preprocess_data(image_dirs, mask_dirs, target_size =(256, 256)):
    images = []
    masks = []
    for image_dir, mask_dir in zip(image_dirs, mask_dirs): # combines the files so it'll be processed tghther. so they can be processed togther
        image_files = sorted(os.listdir(image_dir)) # serpeate the image and mask files/sorts them
        mask_files = sorted(os.listdir(mask_dir))
        for img_file, mask_file in zip(image_files, mask_files):
            # start to load and resize images
            img = load_img(os.path.join(image_dir, img_file), target_size= target_size, color_mode="grayscale")# make sure the image are in grayscale
            mask = load_img(os.path.join(mask_dir, mask_file), target_size= target_size, color_mode="grayscale")

            images.append(img_to_array(img) / 255.0) # converts image into an array.
            masks.append(img_to_array(mask) / 255.0)

    return np.array(images), np.array(masks)

# combine the directories for all three files
image_dirs = [pnemothorax_image_dir,montgomery_image_dir,darwin_img_dir,shenzhen_image_dir]
mask_dirs = [pnemothorax_mask_dir, montgomery_mask_dir, darwin_mask_dir, shenzhen_mask_dir]

# load images and masks from all datasets
images, masks = preprocess_data(image_dirs, mask_dirs)
# data augmentation parameters
img_gen_args = dict(rotation_range=20,  # rotate the image within a range of 20 degrees
                    width_shift_range=0.1,  # Shift the image horizontally within a range of 0.1
                    height_shift_range=0.1,  # Shift the image vertically within a range of 0.1
                    shear_range=0.1,  # Shear the image within a range of 0.1
                    zoom_range=0.1,  # Zoom the image within a range of 0.1
                    horizontal_flip=True,  # Flip the image horizontally
                    fill_mode='nearest')  # Fill the image with the nearest pixel

X_train, X_val, y_train, y_val = train_test_split(images, masks, test_size=0.2, random_state=42) # splits the data into 20% test and 80% train

# Create ImageDataGenerator instances for images and masks
image_datagen = ImageDataGenerator(**img_gen_args)
mask_datagen = ImageDataGenerator(**img_gen_args)

# Fit the data generators to the training data
image_datagen.fit(X_train, augment=True)
mask_datagen.fit(y_train, augment=True)

# Create generators for training data
image_generator = image_datagen.flow(X_train, batch_size=16, seed=42)
mask_generator = mask_datagen.flow(y_train, batch_size=16, seed=42)

# Combine generators into one that yields image and mask pairs
train_generator = zip(image_generator, mask_generator)

# build the U-Net model
def unet_model(input_size=(256, 256, 1)):
    inputs = layers.Input(input_size)
    
    # Encoder
    c1 = layers.Conv2D(64, 3, activation='relu', padding='same')(inputs)# convolutional layer
    c1 = layers.Conv2D(64, 3, activation='relu', padding='same')(c1)#64 convolutional layers
    p1 = layers.MaxPooling2D(pool_size=(2, 2))(c1) # downsizing the feature map
    
    c2 = layers.Conv2D(128, 3, activation='relu', padding='same')(p1) # convolutional layer
    c2 = layers.Conv2D(128, 3, activation='relu', padding='same')(c2)#128 convolutional layers
    p2 = layers.MaxPooling2D(pool_size=(2, 2))(c2) # downsizing the feature map using max pooling layer
    
    c3 = layers.Conv2D(256, 3, activation='relu', padding='same')(p2)#
    c3 = layers.Conv2D(256, 3, activation='relu', padding='same')(c3)
    p3 = layers.MaxPooling2D(pool_size=(2, 2))(c3)
    
    c4 = layers.Conv2D(512, 3, activation='relu', padding='same')(p3)
    c4 = layers.Conv2D(512, 3, activation='relu', padding='same')(c4)
    p4 = layers.MaxPooling2D(pool_size=(2, 2))(c4)
    
    # Bottleneck
    c5 = layers.Conv2D(1024, 3, activation='relu', padding='same')(p4)
    c5 = layers.Conv2D(1024, 3, activation='relu', padding='same')(c5)
    
    # Decoder
    u6 = layers.Conv2DTranspose(512, 2, strides=(2, 2), padding='same')(c5)
    u6 = layers.concatenate([u6, c4])
    c6 = layers.Conv2D(512, 3, activation='relu', padding='same')(u6)
    c6 = layers.Conv2D(512, 3, activation='relu', padding='same')(c6)
    
    u7 = layers.Conv2DTranspose(256, 2, strides=(2, 2), padding='same')(c6)
    u7 = layers.concatenate([u7, c3])
    c7 = layers.Conv2D(256, 3, activation='relu', padding='same')(u7)
    c7 = layers.Conv2D(256, 3, activation='relu', padding='same')(c7)
    
    u8 = layers.Conv2DTranspose(128, 2, strides=(2, 2), padding='same')(c7)
    u8 = layers.concatenate([u8, c2])
    c8 = layers.Conv2D(128, 3, activation='relu', padding='same')(u8)
    c8 = layers.Conv2D(128, 3, activation='relu', padding='same')(c8)
    
    u9 = layers.Conv2DTranspose(64, 2, strides=(2, 2), padding='same')(c8)
    u9 = layers.concatenate([u9, c1])
    c9 = layers.Conv2D(64, 3, activation='relu', padding='same')(u9)
    c9 = layers.Conv2D(64, 3, activation='relu', padding='same')(c9)
    
    outputs = layers.Conv2D(1, 1, activation='sigmoid')(c9)
    
    return models.Model(inputs, outputs)

model = unet_model()
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
history = model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=3, batch_size=16)
    
def display_sample(X, y_true, y_pred):
    index = random.randint(0, len(X) - 1)
    plt.figure(figsize=(12, 4))

    # Input Image
    plt.subplot(1, 3, 1)
    plt.title("Input Image")
    plt.imshow(X[index].squeeze(), cmap='gray')
    plt.axis('off')

    # True Mask
    plt.subplot(1, 3, 2)
    plt.title("True Mask")
    plt.imshow(y_true[index].squeeze(), cmap='gray')
    plt.axis('off')

    # Predicted Mask
    plt.subplot(1, 3, 3)
    plt.title("Predicted Mask")
    plt.imshow(y_pred[index].squeeze(), cmap='gray')
    plt.axis('off')

    plt.tight_layout()
    plt.show()

# Plot training & validation accuracy values
plt.figure(figsize=(12,4))

# Accuracy plot
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='training accuracy')
plt.plot(history.history['val_accuracy'], label='validation accuracy')
plt.title('Model Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')

# Loss plot
plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='training loss')
plt.plot(history.history['val_loss'], label='validation loss')
plt.title('Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend(loc='upper right')

# Show the plots
plt.tight_layout()
plt.show()

# Make predictions on the validation set
y_pred = model.predict(X_val)

# Display a sample of the input image, true mask, and predicted mask
display_sample(X_val, y_val, y_pred)

def detect_damaged_tissue(y_pred, threshold=0.5):
    """
    Detect and highlight damaged tissues from predicted masks. # explains the function
    Args:
    - y_pred (numpy array): Predicted masks from the model.
    - threshold (float): Threshold to classify pixels as damaged tissue.
    Returns:
    - damaged_masks (numpy array): Binary masks highlighting damaged tissue.
    """
    # apply thresholding to identify potential damaged areas
    damaged_masks = (y_pred > threshold).astype(np.uint8)
    return damaged_masks

# detect damaged tissue from predicted masks
damaged_masks = detect_damaged_tissue(y_pred, threshold=0.5)

# visualize the results for the first few samples
for i in range(3):  # Display 3 samples
    plt.figure(figsize=(15, 5))

    # input Image
    plt.subplot(1, 4, 1)
    plt.title("Input Image")
    plt.imshow(X_val[i].squeeze(), cmap='gray')
    plt.axis('off')

    # true Mask
    plt.subplot(1, 4, 2)
    plt.title("True Mask")
    plt.imshow(y_val[i].squeeze(), cmap='gray')
    plt.axis('off')

    # predicted Mask
    plt.subplot(1, 4, 3)
    plt.title("Predicted Mask")
    plt.imshow(y_pred[i].squeeze(), cmap='gray')
    plt.axis('off')

    # damaged Tissue
    plt.subplot(1, 4, 4)
    plt.title("Detected Damaged Tissue")
    plt.imshow(damaged_masks[i].squeeze(), cmap='gray')
    plt.axis('off')

    plt.tight_layout()
    plt.show()

model.save('/home/anika-rastogi/Documents/Github/lung-segmentation/lungsegementation.keras') # save the modela-rastogi/Documents/Github/lung-segmentation/pneumothorax/siim-acr-pneumothorax/img"pnemothorax_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/pneumothorax/siim-acr-pneumothorax/mask"

ended up with 96~ percent accuracy.

predicted mask #5.png

this is in the mask my model predicts. the mask somewhat aligns with the mask, but there are some things wrong with this mask (for example the blurriness indicates the model struggles with sharp edges.)

damged tissue#10.png

this is my first picture, that detects the damaged tissue. so far it’s doing a fine job with it, but the main problem in this is that it also takes in the things that aren't supposed to be there, like the ribs.but according to the doctor, it is accurate. black is damagged

MORE DATA TO COME****

import os
import numpy as np
import random
import matplotlib.pyplot as plt #plot the image 
from tensorflow.keras.utils import load_img, img_to_array # type: ignore # load the images 
from tensorflow.keras import layers, models # type: ignore
from sklearn.model_selection import train_test_split # type: ignore # split the data
import cv2 # type: ignore   

darwin_img_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/darwin/darwin/img" # the documents
darwin_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/darwin/darwin/mask"

montgomery_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/montgomery/montgomery/img"
montgomery_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/montgomery/montgomery/mask"

shenzhen_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/shenzhen/shenzhen/img"
shenzhen_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/shenzhen/shenzhen/mask"

pnemothorax_image_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/pneumothorax/siim-acr-pneumothorax/img"
pnemothorax_mask_dir = "/home/anika-rastogi/Documents/Github/lung-segmentation/pneumothorax/siim-acr-pneumothorax/mask"

def preprocess_data(image_dirs, mask_dirs, target_size =(256, 256)):
    images = []
    masks = []
    for image_dir, mask_dir in zip(image_dirs, mask_dirs): # combines the files so it'll be processed tghther. so they can be processed togther
        image_files = sorted(os.listdir(image_dir)) # serpeate the image and mask files/sorts them
        mask_files = sorted(os.listdir(mask_dir))
        for img_file, mask_file in zip(image_files, mask_files):
            # start to load and resize images
            img = load_img(os.path.join(image_dir, img_file), target_size= target_size, color_mode="grayscale")# make sure the image are in grayscale
            mask = load_img(os.path.join(mask_dir, mask_file), target_size= target_size, color_mode="grayscale")

            images.append(img_to_array(img) / 255.0) # converts image into an array.
            masks.append(img_to_array(mask) / 255.0)

    return np.array(images), np.array(masks)

# combine the directories for all three files
image_dirs = [pnemothorax_image_dir,montgomery_image_dir,darwin_img_dir,shenzhen_image_dir]
mask_dirs = [pnemothorax_mask_dir, montgomery_mask_dir, darwin_mask_dir, shenzhen_mask_dir]

# load images and masks from all datasets
images, masks = preprocess_data(image_dirs, mask_dirs)
# data augmentation parameters
img_gen_args = dict(rotation_range=20,  # rotate the image within a range of 20 degrees
                    width_shift_range=0.1,  # Shift the image horizontally within a range of 0.1
                    height_shift_range=0.1,  # Shift the image vertically within a range of 0.1
                    shear_range=0.1,  # Shear the image within a range of 0.1
                    zoom_range=0.1,  # Zoom the image within a range of 0.1
                    horizontal_flip=True,  # Flip the image horizontally
                    fill_mode='nearest')  # Fill the image with the nearest pixel

X_train, X_val, y_train, y_val = train_test_split(images, masks, test_size=0.2, random_state=42) # splits the data into 20% test and 80% train

# build the U-Net model
def unet_model(input_size=(256, 256, 1)):
    inputs = layers.Input(input_size)
    
    # Encoder
    c1 = layers.Conv2D(64, 3, activation='relu', padding='same')(inputs)# convolutional layer
    c1 = layers.Conv2D(64, 3, activation='relu', padding='same')(c1)#64 convolutional layers
    p1 = layers.MaxPooling2D(pool_size=(2, 2))(c1) # downsizing the feature map
    
    c2 = layers.Conv2D(128, 3, activation='relu', padding='same')(p1) # convolutional layer
    c2 = layers.Conv2D(128, 3, activation='relu', padding='same')(c2)#128 convolutional layers
    p2 = layers.MaxPooling2D(pool_size=(2, 2))(c2) # downsizing the feature map using max pooling layer
    
    c3 = layers.Conv2D(256, 3, activation='relu', padding='same')(p2)#
    c3 = layers.Conv2D(256, 3, activation='relu', padding='same')(c3)
    p3 = layers.MaxPooling2D(pool_size=(2, 2))(c3)
    
    c4 = layers.Conv2D(512, 3, activation='relu', padding='same')(p3)
    c4 = layers.Conv2D(512, 3, activation='relu', padding='same')(c4)
    p4 = layers.MaxPooling2D(pool_size=(2, 2))(c4)
    
    # Bottleneck
    c5 = layers.Conv2D(1024, 3, activation='relu', padding='same')(p4)
    c5 = layers.Conv2D(1024, 3, activation='relu', padding='same')(c5)
    
    # Decoder
    u6 = layers.Conv2DTranspose(512, 2, strides=(2, 2), padding='same')(c5)
    u6 = layers.concatenate([u6, c4])
    c6 = layers.Conv2D(512, 3, activation='relu', padding='same')(u6)
    c6 = layers.Conv2D(512, 3, activation='relu', padding='same')(c6)
    
    u7 = layers.Conv2DTranspose(256, 2, strides=(2, 2), padding='same')(c6)
    u7 = layers.concatenate([u7, c3])
    c7 = layers.Conv2D(256, 3, activation='relu', padding='same')(u7)
    c7 = layers.Conv2D(256, 3, activation='relu', padding='same')(c7)
    
    u8 = layers.Conv2DTranspose(128, 2, strides=(2, 2), padding='same')(c7)
    u8 = layers.concatenate([u8, c2])
    c8 = layers.Conv2D(128, 3, activation='relu', padding='same')(u8)
    c8 = layers.Conv2D(128, 3, activation='relu', padding='same')(c8)
    
    u9 = layers.Conv2DTranspose(64, 2, strides=(2, 2), padding='same')(c8)
    u9 = layers.concatenate([u9, c1])
    c9 = layers.Conv2D(64, 3, activation='relu', padding='same')(u9)
    c9 = layers.Conv2D(64, 3, activation='relu', padding='same')(c9)
    
    outputs = layers.Conv2D(1, 1, activation='sigmoid')(c9)
    
    return models.Model(inputs, outputs)

model = unet_model()
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
history = model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=16)
    
def display_sample(X, y_true, y_pred):
    index = random.randint(0, len(X) - 1)
    plt.figure(figsize=(12, 4))

    # Input Image
    plt.subplot(1, 3, 1)
    plt.title("Input Image")
    plt.imshow(X[index].squeeze(), cmap='gray')
    plt.axis('off')

    # True Mask
    plt.subplot(1, 3, 2)
    plt.title("True Mask")
    plt.imshow(y_true[index].squeeze(), cmap='gray')
    plt.axis('off')

    # Predicted Mask
    plt.subplot(1, 3, 3)
    plt.title("Predicted Mask")
    plt.imshow(y_pred[index].squeeze(), cmap='gray')
    plt.axis('off')

    plt.tight_layout()
    plt.show()

# Plot training & validation accuracy values
plt.figure(figsize=(12,4))

# Accuracy plot
plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='training accuracy')
plt.plot(history.history['val_accuracy'], label='validation accuracy')
plt.title('Model Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')

# Loss plot
plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='training loss')
plt.plot(history.history['val_loss'], label='validation loss')
plt.title('Model Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend(loc='upper right')

# Show the plots
plt.tight_layout()
plt.show()

# Make predictions on the validation set
y_pred = model.predict(X_val)

# Display a sample of the input image, true mask, and predicted mask
display_sample(X_val, y_val, y_pred)

def detect_damaged_tissue(y_pred, threshold=0.5):
    """
    Detect and highlight damaged tissues from predicted masks.
    Args:
    - y_pred (numpy array): Predicted masks from the model.
    - threshold (float): Threshold to classify pixels as damaged tissue.
    Returns:
    - damaged_masks (numpy array): Binary masks highlighting damaged tissue.
    """
    # Apply thresholding to identify potential damaged areas
    damaged_masks = (y_pred > threshold).astype(np.uint8)
    # Invert the mask colors
    damaged_masks = 1 - damaged_masks
    return damaged_masks

# detect damaged tissue from predicted masks
damaged_masks = detect_damaged_tissue(y_pred, threshold=0.5)

def refine_damaged_tissue(damaged_masks, kernel_size=5, threshold=0.6):
    """
    Refine the damaged tissue masks using morphological operations.
    Args:
        damaged_masks (numpy array): Predicted damaged tissue masks.
        kernel_size (int): Size of the morphological kernel.
        threshold (float): Threshold to binarize the mask.
    Returns:
        refined_masks (numpy array): Refined binary masks.
    """
    refined_masks = []
    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (kernel_size, kernel_size)) #establish the size of kernel, creates an elliptical kernel

    for mask in damaged_masks:
        # Apply thresholding
        binary_mask = (mask.squeeze() > threshold).astype(np.uint8) # in turn, the mask is converted into a binary mask binary mask means it is either 0 or 1

        # Morphological operations
        refined = cv2.morphologyEx(binary_mask, cv2.MORPH_OPEN, kernel)  # remove small noise
        refined = cv2.morphologyEx(refined, cv2.MORPH_CLOSE, kernel)     # close small gaps

        refined_masks.append(refined)

    return np.array(refined_masks)

# Refine the detected damaged tissue masks
refined_damaged_masks = refine_damaged_tissue(damaged_masks, kernel_size=5, threshold=0.6) # refine the damaged tissue masks

# visualize the results for the first few samples
for i in range(3):  # Display 3 samples
    plt.figure(figsize=(15, 5))

    # input Image
    plt.subplot(1, 4, 1)
    plt.title("Input Image")
    plt.imshow(X_val[i].squeeze(), cmap='gray')
    plt.axis('off')

    # true Mask
    plt.subplot(1, 4, 2)
    plt.title("True Mask")
    plt.imshow(y_val[i].squeeze(), cmap='gray')
    plt.axis('off')

    # predicted Mask
    plt.subplot(1, 4, 3)
    plt.title("Predicted Mask")
    plt.imshow(y_pred[i].squeeze(), cmap='gray')
    plt.axis('off')

    # damaged Tissue
    plt.subplot(1, 4, 4)
    plt.title("Detected Damaged Tissue")
    plt.imshow(damaged_masks[i].squeeze(), cmap='gray')
    plt.axis('off')

    plt.tight_layout()
    plt.show()

model.save('/home/anika-rastogi/Documents/Github/lung-segmentation/lungsegementation.keras') # save the model

loss point:

figure _for school.png

this had an accuracy of 97%!!!

predicted mask

predictedmask_schoolresult.png

this is the predicted mask; you can still see the blurrineed in the picture, but it is a bit more accurate.

damaged tissue;

damaged tissue #9.png

the black portions, in the damaged portion.

figur gdam.png

this is the unsure portion, but I inverted the colors because then I will be able to see the outline. but this is the black damaged portion.

Conclusion

This project focuses on segmenting lungs to identify 3 important conditions; tuberculosis, viral pneumonia, and pneumthorax. By using advanced image analysis techniques, I was able to highlight the damaged areas of the lungs, making it easier to study and understand them.

I believe this work is important because it can help doctors make faster and more accurate decisions, especially for people who have a serious stage of these diseases. This project showed that technology can play a big role in improving healthcare, BUT in the case of my project, it needs much work. Future improvements could include adding more classes and adding a transformer. And using larger datasets.

In the end, this project shows how combining science, technology, and medicine can make a big difference in diagnosing and treating lung diseases.

Citations

Danilov, Viacheslav. “Chest X-Ray Dataset for Lung Segmentation.” Mendeley Data, vol. 1, 31 Jan. 2022, data.mendeley.com/datasets/8gf9vpkhgy/1, https://doi.org/10.17632/8gf9vpkhgy.1. Accessed 1 Dec. 2024.

“What Is Viral Pneumonia?” WebMD, www.webmd.com/lung/viral-pneumonia.

Mayo Clinic. “Tuberculosis - Diagnosis and Treatment - Mayo Clinic.” Mayoclinic.org, Mayo clinic, 22 Mar. 2023, www.mayoclinic.org/diseases-conditions/tuberculosis/diagnosis-treatment/drc-20351256.\

kaggle2007. “Simple NN with Accuracy and Loss Plotting.” Kaggle.com, Kaggle, 10 Mar. 2019, www.kaggle.com/code/kaggle2007/simple-nn-with-accuracy-and-loss-plotting. Accessed 26 Jan. 2025.

‌Mayo Clinic. “Pneumothorax - Symptoms and Causes.” Mayo Clinic, 8 Aug. 2024, www.mayoclinic.org/diseases-conditions/pneumothorax/symptoms-causes/syc-20350367.

‌Association, American Lung. “Pneumothorax (Collapsed Lungs).” Www.lung.org, www.lung.org/lung-health-diseases/lung-disease-lookup/pneumothorax.

vbookshelf. “Chest X-Ray Images with Pneumothorax Masks.” Kaggle.com, 2020, www.kaggle.com/datasets/vbookshelf/pneumothorax-chest-xray-images-and-masks. Accessed 26 Jan. 2025.

‌

‌

Acknowledgement

I acknowledge my mother, my father, and my brother. 

I acknowledge the people from juniotech, who helped me along the way, especially Irada Shamilova,  and Tim Gubski, from Junio Tech

I acknowledge Ms.Lai for giving me this opportunity 

And lastly I acknowledge Izabela Sztukowski, from the children hospital, and Paul Mceachern from the University of Calgary. for helping me with my research on lungs.