Open-access COVID-19 Severity Prediction Using Combined Machine Learning and Transfer Learning Approaches

Abstract

The global spread of Coronavirus Disease 2019 (COVID-19) has resulted in an extensive pandemic, with the virus rapidly transmitting through interactions among infected individuals, presenting a substantial threat to healthcare professionals. In response, computer scientists have employed artificial intelligence methodologies to identify and address COVID-19. This study utilizes machine learning and transfer learning techniques to forecast the severity of the coronavirus, aiding healthcare providers in determining the progression of the illness in patients. Prediction of disease severity occurs in two stages. Initially, blood parameter values are utilized for preliminary screening of coronavirus infection through machine learning methods. The first stage employs the proposed Probabilistic Stacked Ensemble Classifier, employing optimal features selected using the proposed Modified Mutual Information feature selection algorithm, to detect the presence or absence of the virus. Following that, the subsequent phase employs proposed mResNet-50, a transfer learning approach, which utilizes Computed Tomography (CT)-scan images to predict the stage of infection in affected individuals. Experimental results indicate that the model achieves a 97.79% accuracy rate in forecasting infection stages and demonstrates the generalizability of the proposed model across benchmark datasets.

Keywords:
Machine Learning; Transfer Learning; COVID-19; clinical data; CT-Scan

HIGHLIGHTS

Proposed a hybrid feature selection method.

Proposed a stacked ensemble classifier.

Proposed a transfer learning approach for image classification.

Developed a model with the proposed techniques and measured its efficacy.

INTRODUCTION

In late December 2019, an infectious virus named Corona was initially identified in an individual in Wuhan, China, sparking a widespread health crisis. The virus rapidly spread, causing a considerable number of individuals who had contact with the infected person to contract the illness. Various screening methods, including laboratory tests, Reverse Transcription-Polymerase Chain Reaction (RT-PCR), X-rays, and CT scans, are utilized to detect the disease. RT-PCR is commonly employed as the primary method for early detection, despite occasional inaccuracies and its time-consuming and costly nature [1]. Consequently, healthcare professionals have proposed an alternative approach: analysing blood samples from affected individuals. Upon entry into the body, the virus triggers specific changes that can be exploited to determine if someone is infected with the coronavirus. Blood tests provide rapid and cost-effective results, making them an appealing initial screening method. CT scans are recommended for coronavirus-affected patients since X-ray images, despite their advantages, are less sensitive and may lead to false predictions in cases of early or mild infection.

Machine learning algorithms play a vital role in developing a decision support system aimed at predicting the progression of coronavirus and identifying its stages using both clinical data and CT scan images [2]. By leveraging clinical data such as patient demographics and laboratory test results, machine learning algorithms can estimate the likelihood of disease progression. Furthermore, transfer learning, a machine learning approach, plays a significant role as pre-trained Convolutional Neural Network (CNN) architectures can extract features from images and classify them using a limited dataset. The functionality of both techniques is combined for a holistic approach that enables early detection and precise staging. This provides valuable assistance for healthcare providers in personalized treatment planning, thereby enhancing patient care and optimizing healthcare resource allocation amid the COVID-19 pandemic.

The objective of this research paper is to predict the severity of COVID-19 in two phases. The first phase conducts an early screening to detect the presence or absence of COVID-19. The disease severity is predicted using CT scan lung images, involving the proposed feature selection method, Modified Mutual Information (MMI), to identify relevant features from the clinical dataset. Additionally, a proposed classifier, the Probabilistic Stacked Ensemble Classifier (PSEC), is utilized to identify the presence or absence of coronavirus. Moreover, a modified ResNet-50 transfer learning approach will be employed to extract features from CT scan images, aiding in the prediction of COVID-19 stages.

The main contributions of this paper include: (i) Introducing a hybrid feature selection algorithm that integrates wrapper and filter-based methods, enhanced with a backtracking process. (ii) Proposing a probabilistic stacked ensemble classifier for detecting the presence or absence of COVID-19. (iii) Developing a transfer learning approach for predicting disease severity using CT scan lung images. (iv)Validating the proposed model through experimental analysis and benchmarking its performance against state-of-the-art models. (v) Demonstrating the effectiveness, generalizability, and robustness of the proposed methods across various benchmark datasets through comprehensive study results.

Related Works

Houssein and coauthors [3], devised a hybrid quantum-classical convolutional neural network (HQ-CNN) model to examine chest X-ray images for COVID-19 detection, leveraging a dataset containing 5445 images. The model achieved impressive performance metrics, including 98.6% accuracy and 99% recall for distinguishing COVID-19 from normal cases, 98.2% accuracy and 99.5% recall for distinguishing COVID-19 from viral pneumonia cases, and 98% accuracy and 98.8% recall for distinguishing COVID-19 from bacterial pneumonia cases. The authors of [4] implemented a model to forecast COVID-19 severity based on chest X-ray images. They trained the model using pre-trained CheXNet features, alongside Extra Tree and SVM classifiers with Recursive Feature Elimination (RFE), resulting in an impressive accuracy of 99.6%.

Alaiad and coauthors [5], introduced a deep learning model designed to diagnose severity across four levels-normal, mild, and severe-utilizing ResNet101, achieving an accuracy of 99.5%. The model was trained using 2205 CT images. Albataineh and coauthors [6], formulated a model employing SVM to discern the severity of COVID-19, categorizing it as mild, moderate, or severe, through analysis of CT-Scan lung images. Remarkably, they attained an accuracy rate of 99.12%. Farahat and coauthors [7], introduced a system utilizing a neural network-based fusion system to determine the required support level for each patient. It integrates estimates from three, Markov-Gibbs random field based models using 307 CT-Scan images of COVID-19-infected patients. The system categorizes patients into three levels of support: Level 0, Level 1, and Level 2, achieving an accuracy of 97.72%.

Zoabi and coauthors [8] developed a model trained on a dataset of 51,831 individuals to predict COVID-19 test results using only eight binary features, such as age, sex, and specific clinical symptoms. Remarkably, their model achieved a remarkable 95% Area under the Curve (AUC) score. The authors of [9] have constructed a hybrid deep learning model, CNN-Long Short-Term Memory (LSTM), integrated with Synthetic Minority Oversampling Technique (SMOTE), to address the temporal aspects of progression-based symptoms within the COVID-19 dataset. Impressively, their model achieved an accuracy of 96.34%. A model incorporating Random Forest (RF), Extreme Gradient Boosting (XGBoost), Logistic Regression (LR), K-Nearest Neighbor (KNN), and Support Vector Machine (SVM) classifiers, coupled with the SMOTE for classification was developed by Chadaga and coauthors [10]. Furthermore, they utilized the Shapley Additive Explanations (SHAP) method to determine the significance of each feature in predicting virus diagnosis using clinical data, achieving an accuracy rate of 91%. Gök and Olgun [11] constructed a model utilizing a random forest classifier, achieving an accuracy rate of 98% when analyzing blood sample data. Laatifi and coauthors [12], introduced a novel feature engineering technique, Uniform Manifold Approximation and Projection (UMAP), which exhibited outstanding performance. It achieved 100% accuracy, specificity, sensitivity, and Receiver Operating Characteristic Curve (ROC) curve in prognostic prediction across various machine learning classifiers including XGBoost, AdaBoost, Random Forest, and ExtraTrees, utilizing clinical data.

Fang and coauthors [13] developed seasonal ARIMA and XGBoost models to forecast COVID-19 cases 14 days ahead, with XGBoost performing particularly well. Cihan [14] used an ARIMA model to forecast the number of people to be vaccinated in ten days across various countries. Raman and coauthors [15] used artificial intelligence to predict COVID-19 severity from initial laboratory values helping clinicians identify patients at risk of disease progression, morbidity and mortality. Angulakshmi and coauthors [16] employed the VGG-16 transfer learning model to predict COVID-19 using chest x-ray images, achieving an accuracy of 99%.

A comprehensive literature survey reveals that existing research has predominantly focused on predicting COVID-19 severity using individual modalities such as x-rays, CT-scans, laboratory tests, or symptoms, as well as forecasting disease severity and vaccination rates across countries. This research introduces a novel approach by first predicting the presence or absence of COVID-19 using clinical data and then assessing disease severity using CT-scan lung images. In the first phase, machine learning algorithms such as SVM, Naïve Bayes, and logistic regression are employed to enhance predictive precision and reliability. In the second phase, a transfer learning approach is utilized to predict disease severity. This two-phase architecture represents a significant progression beyond the current state-of-the-art, offering novel contributions to the field of COVID-19 prediction.

Motivation and Justification

The onset of the COVID-19 pandemic led to a significant transformation in healthcare screening methods, with blood tests becoming the preferred choice over CT scans and X-rays due to concerns about side effects and radiation exposure. In this landscape, machine learning emerges as an indispensable tool, offering diverse feature selection and classification algorithms crucial for disease prediction. Accurately identifying relevant features from datasets is pivotal for precise predictions. An approach is introduced in this study with the implementation of a hybrid feature selection algorithm known as modified mutual information, seamlessly integrating a classifier to overcome the challenge of determining an optimal threshold value. Machine learning provides various classification algorithms to work on the selected features. Among various classification algorithms, SVM excel in classification, their lack of probabilistic explanation is addressed in the ensemble model by integrating probabilistic estimates through probabilistic classifiers. This enhances diagnostic accuracy and provides insights into uncertainty and risk levels, crucial for optimal patient care. Leveraging historical data and prior knowledge, the Bayesian classifier anticipates outcome probabilities, while LR offers interpretability and suitability for binary classification, serving as a valuable meta-classifier. The two-tier stacked ensemble classifier comprises SVM, Naïve Bayes in the first layer, and Logistic Regression in the second layer, thus improving the prediction accuracy of COVID-19 presence or absence based on selected features. Disease severity for infected patients is predicted utilizing their CT-Scan lung images.

In the realm of machine learning, transfer learning and deep learning play pivotal roles in enhancing image data classification accuracy. Transfer learning simplifies model-building by leveraging pre-trained deep learning network models, which are trained on extensive datasets. This study modifies the ResNet-50 architecture by adding residual units and dropout layers. This modified ResNet-50 predicts the severity of the disease. Thus, the proposed framework first detects COVID-19 presence and then predicts disease severity for infected patients. Experimental results validate the framework's superiority over conventional methods and pre-trained CNN algorithms, emphasizing its exceptional predictive accuracy across benchmark datasets. These findings underscore the importance of integrating advanced machine learning techniques into healthcare practices, particularly amid the challenges posed by the COVID-19 pandemic, paving the way for more effective disease detection and management strategies.

Outline of the Work

Figure 1
Outline of the Work

The workflow depicted in Figure 1 of this research paper involves several key steps. Initially, the raw data in the clinical dataset undergoes preprocessing to handle missing values, eliminate redundancies, and convert categorical values into numerical ones. Following preprocessing, the dataset is examined for outliers, which are subsequently removed, and the dataset is balanced using the SMOTE balancing technique. Significant features in the preprocessed dataset are identified using the proposed modified mutual information (MMI) feature selection algorithm. The selected features are then inputted into the proposed probabilistic stacked ensemble classifier (PSEC) to determine the presence or absence of COVID-19. Additionally, CT-Scan images of COVID-19 affected patients are preprocessed, and the stage of infection is identified using mResNet-50. Finally, the model's performance is evaluated using metrics such as accuracy, precision, recall, and F1-Score.

MATERIAL AND METHODS

This section provides an overview of the methodology employed by the proposed model for predicting the severity of COVID-19. Initially, clinical data undergoes preprocessing to ensure its suitability for analysis. Subsequently, the preprocessed data is inputted into the Modified Mutual Information feature selection algorithm, which identifies the most crucial features for analysis. These selected features serve as input for a stacked ensemble heterogeneous classifier, which aims to detect the presence or absence of COVID-19. Additionally, CT-Scan images of COVID-19 positive patients undergo preprocessing using image preprocessing techniques. These preprocessed images are then fed into the mResNet-50 model to predict the severity of the disease. The performance of the model is evaluated using various metrics, including precision, accuracy, recall, and F1-Score.

Data Preprocessing

In this study, a COVID-19 predictive model is developed using a clinical dataset. During the pre-processing phase, the dataset is carefully examined for null values and redundant data. Fortunately, the clinical dataset used in this study is free from any null values or redundant entries. The dataset consists of one categorical feature and 26 numerical features. As machine learning algorithms can only work with numerical data, the categorical feature "gender" is transformed into a numerical format using one-hot code encoding.

One-hot Encoding

The conversion from categorical to numerical data is accomplished using a technique known as one-hot encoding. This process enhances the algorithms' predictive capabilities [17]. In the current research study, the categorical feature 'Gender' is transformed into a numerical feature using the one-hot encoding method. This involves creating two new columns, namely Gender_1 and Gender_2, to represent the different categories within the 'Gender' feature numerically.

iForest

Anomalies in a dataset significantly differ from typical records in both quantity and quality, impacting classification model effectiveness. Employing the Isolation Forest (iForest) technique [18], the outliers are detected and eliminated in a COVID-19 clinical dataset. iForest assesses average path lengths within its tree structures, identifying instances with notably shorter paths as outliers. It operates efficiently with a small subsample size and an appropriate number of trees, utilizing the 'contamination' parameter to specify outlier proportions. In this study, 24 outliers were detected using the chosen parameter configuration. Following their removal, SMOTE addresses dataset imbalance, contributing to improved data integrity for analysis.

SMOTE

SMOTE, or Synthetic Minority Over-sampling Technique, [19] is a widely used method to address imbalanced classification challenges. By creating synthetic samples for the minority class through interpolation among existing minority samples, SMOTE effectively amplifies its representation in the dataset. This process involves selecting a minority sample and identifying its k nearest neighbors, then generating new samples along the line segments connecting them. By balancing class distribution, SMOTE mitigates bias during model training, enhancing classifier performance, especially in scenarios where the minority class is underrepresented. In the case of a heavily skewed COVID-19 dataset, SMOTE was employed post-outlier removal to rebalance it, ensuring a fair representation of both classes in training and testing subsets.

Proposed Modified Mutual Information Algorithm (MMI)

In contrast to the conventional Mutual Information algorithm [20], which relies on a predetermined threshold value for feature selection, the proposed MMI algorithm considers both classification performance and the capacity to capture feature interactions. This algorithm integrates a backtracking mechanism to identify the most optimal set of features for the model. The Modified Mutual Information algorithm initiates by computing the mutual information score for each feature concerning the target variable, providing insights into their relevance. Features with a score of zero are flagged as irrelevant and subsequently removed from the feature set. Subsequently, the algorithm selects the feature with the highest information score and presents it to the classifier, monitoring and storing its accuracy. Then, a feature subset is formed by iteratively adding the next feature with the highest information score, with each addition followed by accuracy evaluation. This process continues until all features in the set have been individually included, and their accuracies recorded. The algorithm incorporates a backtracking procedure, whereby if the accuracy of the current feature set is inferior to that of the previous set, adjustments are made accordingly. This iterative approach ultimately leads to the identification of the optimal feature subset.

The Proposed MMI hybrid method signifies a progression in feature selection techniques by amalgamating aspects from both filter and wrapper methods. Key modifications include:

(i) Dynamic Feature Selection: Unlike traditional filter methods that often rely on fixed thresholds or predetermined feature counts, the proposed MMI method interacts directly with the classifier. It dynamically selects feature subsets based on real-time performance feedback from the classifier, allowing for more adaptive and informed feature selection, potentially enhancing classification accuracy.

(ii) Backtracking Mechanism: In contrast to traditional forward selection wrapper methods, which proceed linearly without revisiting previous decisions, the proposed MMI method integrates a backtracking mechanism into the optimal subset selection process. This feature enables the algorithm to reassess and adjust prior decisions based on current performance indicators, enhancing its ability to identify the most relevant and discriminative features for classification tasks.

Pseudocode for proposed Modified Mutual Information Algorithm (MMI)

Input :

D ← Dataset, F ← Features {f1,f2,f3,….fn}, C ← Target variable, n1 ← number of features in the set F.

Output : S ← selected feature subset.

Begin

Initialisation: S ← {}, pre_acc←0 and cur_acc ← 0

//computing the Mutual Information for all the features in the feature set F

For i = 1 to n1 // n1 is the total number of features

Calculate the mutual information IG(fi, C).

End for

Remove the features in F whose mutual information = zero

//Selecting Best features from feature set F

While (F = {})

fnew← feature fi which has maximum mutual information score

S← S U { fnew }

F← F-{ fnew }

pre_acc ← cur_acc

Input the features in S to a classifier.

cur_acc ← accuracy for the feature subset in S

// Backtracking

If pre_acc > cur_acc then

S= S - { fnew }

End if

End While

Feature subset ‘S’ contains the optimal features.

return S

End

Proposed Probabilistic Stacked Ensemble Classifier (PSEC)

The proposed classifier functions within a two-tier architecture, employing various base classifiers such as SVM [21] with linear and radial bias functions, alongside Naïve Bayes [22]. The meta-classifier, Logistic Regression [23], combines predictions from these base classifiers to generate the final decision. While SVM operates as a binary classifier using a margin-based approach for class separation, it lacks the ability to provide probability estimates. By integrating SVM with a probabilistic classifier, this method enhances the capability to assess the likelihood of disease presence. Predictions from the base classifiers serve as inputs for the logistic regression classifier, acting as a combiner to yield the ultimate predicted outcome.

The functionality of the proposed classifier is depicted in Figure 2. Initially, the dataset undergoes splitting into training and testing sets at an 80:20 ratio. Within the training set, an additional partition occurs at an 80:20 ratio, with 80% of the data used to train the base classifiers (SVM-Linear, SVM-RBF, NB) in the first layer, while the remaining 20% forms the validation set. This validation set serves as input for the second layer process.

In the second layer, the well-trained base classifiers receive features from the validation set, generating three predictions: P1, P2, and P3. P1 corresponds to the output of the trained SVM-Linear, P2 represents the output of the trained SVM-RBF, and P3 reflects the output of the trained Naïve Bayes. Subsequently, a new dataset is constructed to train the meta-classifier (LR), with P1, P2, and P3 as input features, and the class label 'y' from the validation set as the target variable. The meta-classifier (Logistic Regression) learns the relevance of these input features with respect to the class label 'y'. Finally, the proposed classifier is evaluated using the test data to predict the presence of COVID-19 in individuals.

Pseudocode for Probabilistic Stacked Ensemble Classifier (PSEC)

Input :

D ← Dataset, F ← Features {f1, f2, f3,….fn}, Y ← Target variable, n1 ← number of features in the set

F, T1 ← Training Dataset , T2 ← Testing Dataset, T← training subset, V ← validation subset, D1 ←

dataset, B1,B2, B3←base classifier1, M← Meta classifier. P1, P2, P3 ← predicted output of the base

classifiers

Output : P ← predicted output.

Begin

// Dataset Division

Split the D into T1 and T2 in 80 : 20 ratio

//Secondary Division

Split T1 into T and V in 80:20 ratio

// Base Classifier Selection

Assign B1←SVM-Linear, B2 ← SVM-RBF, B3←Naïve Bayes

// Meta-Classifier Integration

Assign M ← Logistic Regression

//Layer 1 Training

Train classifiers B1, B2 and B3 with the data in T

//Layer 1 validation

Test trained B1, B2, B3 using data in V and their predicted output are P1, P2, P3 respectively.

// Layer 2 Processing

N←{P1,P2,P3, Y} // N is new dataset formed using P1,P2,P3 as input data and Y (Target Variable of V)

//Meta Classifier Training

Train the meta-classifier M with the dataset N

//Prediction Generation

Test the trained meta classifier with input data in dataset T2 and P is the predicted output

End

Image Preprocessing

Preprocessing of images is conducted to harmonize image attributes with the model's requirements, thereby augmenting the accuracy and predictive prowess of the system. Given the dataset's inclusion of images of diverse sizes, standardization becomes imperative. To accomplish this, all images are resized to a uniform 224 x 224 dimensions, aligning with the input specifications of pre-trained CNN models. The Gaussian smoothing technique is applied to eliminate noise, resulting in cleaner images, while the adaptive histogram equalization method is employed to enhance image contrast.

Gaussian smoothing

This study utilizes Gaussian smoothing [24] as a significant method for noise reduction and image enhancement. The Gaussian distribution (GSx1), with a standard deviation (σ), is distinguished based on whether it is for single estimation (x1) or double estimation (x1, y1). In the former, (x1) denotes an image point, while in the latter, (x1, y1) represents a pixel. This distinction is clarified through two equations Equation1 and Equation 2 [15], as follows:

G S ( x 1 ) = 1 ν 2 π σ e x 1 2 2 2 (1)

G S ( x 1, y 1 ) = 1 2 π σ 2 e ( x 1 3 y 1 2 ) 2 ( 2 ) (2)

This approach efficiently diminishes high-frequency noise, resulting in a clearer and visually enhanced depiction of the underlying image. Incorporating Gaussian smoothing in the study serves as a foundational procedure in preprocessing CT-scan lung images, enabling a more lucid examination of crucial details and assisting in more precise diagnostic evaluations.

Adaptive Histogram Equalisation (AHE)

In this study, an Adaptive Histogram Equalization (AHE) [25], a powerful approach for enhancing image contrast and refining visual quality across various applications, is utilized. Unlike conventional histogram equalization methods that treat the entire image uniformly, AHE divides the image into smaller regions or tiles. By independently equalizing the histogram of each tile, AHE effectively adapts to local contrast variations, preserving intricate details and enhancing image clarity. This adaptive approach is particularly beneficial in situations where images exhibit significant fluctuations in lighting conditions or contrast across different regions.

Proposed mResNet-50

A method called mRestNet-50 is introduced for predicting the severity of coronavirus disease using CT-Scan images. The enhanced ResNet-50 architecture, specifically designed for disease severity detection, integrates two additional residual units at the end. Furthermore, a dropout layer is introduced to facilitate the learning of robust and generalized features, aiming to improve performance on unseen data.

To handle potential variations in CT-scan image resolutions and dimensions, a standardization procedure is applied. This involves resizing the training dataset images to a consistent size of 224 x 224 x 3, followed by image enhancement and augmentation to enrich the dataset before feature extraction and classification.

The network architecture comprises multiple layers with diverse characteristics such as output sizes, strides, padding, output channels, and activation functions. Initially, the input image undergoes convolution (Conv1), resulting in reduced dimensions and increased channels in the output. Subsequent MaxPooling (MaxPool1) operations further decrease the output size.

The classical ResNet-50 architecture includes 16 residual blocks distributed across various stages, each utilizing ReLU activation [26]. The newly proposed mResNet-50 enhances the architecture by incorporating two residual units (Res5d, Res5e) with 2048 output channels in stage 5, resulting in a total of 5 blocks with ReLU activation. Each block comprises multiple convolutional layers with shortcut connections, enhancing the network's depth and addressing the vanishing gradient problem.

In summary, the deployment of the mResNet-50 architecture comprises three main steps: (i) integrating two residual units, (ii) incorporating a dropout layer preceding the FC layer, and (iii) adjusting the FC layer to support three output neurons.

Pseudocode for Proposed mResNet-50

Input: X← CT-Scan Image dataset

Output : C ← Target variable

Begin

// convolution operation on the input image to extract features

Conv1_output ← convolution(X)

Conv1output ← ReLU(Conv1_output)

// max pooling operation to reduce the output size

Maxpool_output ← MaxPooling(Conv1_output)

ResNet50_output ← classicalResNet50_output(Maxpool_output)

//Enhancement with mResNet-50

Res5d_output ←AddResidualUnit(ResNet50_output, 2048)

Res5e_output ←AddResidualUnit(ResNet50_output, 2048)

// Final Layers

AvgPool_output ← AveragePooling(Res5e_output)

Dropout_output ← Dropout(AvgPool_output, rate=0.5)

FC_output ← FullyConnectedLayer(Dropout_output, output_neurons=3)

Y_pred ← SoftMax(FC_output)

End

RESULTS

In this section, the results of developing a COVID-19 disease severity prediction model are presented. By utilizing clinical data as input, the model distinguishes the presence or absence of COVID-19 and accurately predicts Normal, severe, moderate, and mild cases through analysis of Ct-Scan Lung images. Performance evaluation encompasses metrics such as precision, recall, accuracy, and F1-score. Furthermore, comparisons with state-of-the-art models or benchmarks are conducted to showcase the effectiveness of the proposed model.

Dataset Description

Clinical Dataset

This research study utilizes a clinical dataset containing diverse information obtained from blood samples of patients. A new clinical dataset was compiled, incorporating data from 2000 patients who underwent blood tests due to exhibiting COVID-19 symptoms between August 2020 and August 2021 at a private hospital in Thoothukudi, Tamil Nadu, India. The blood test dataset comprises 27 features, and based on the values of these features, the final diagnosis was recorded as either COVID-19 positive or negative. The dataset is balanced, consisting of 1000 blood test results evenly split between COVID-19 positive and negative patients.

Data Availability

The data supporting the findings of this study will be openly accessible on GitHub at https://github.com/merviname/COVID-19 following the acceptance of this paper.

Clinical Data Pre-processing

The dataset was checked for outliers using the iForest method, and the detected outliers were removed. This process resulted in an imbalanced dataset. To address this, the dataset was rebalanced using SMOTE.

CT-Scan COVID-19 Dataset

To assess the severity of COVID-19, this research collected data from 1252 COVID-19 positive patients and 828 COVID-19 negative patients available in the Kaggle open repository [27]. These data originated from actual patients treated in hospitals in Sao Paulo, Brazil. A radiologist evaluated the gathered data and categorized them as Normal, mild, moderate, or severe based on the extent of infection. Following categorization, the dataset comprises 828 images which are normal, 816 images with mild infection, 263 images with moderate infection, and 173 images with severe infection. Figure 2(a) displays the CT-Scan image of a patient with mild infection, while Figure 2(b) presents the CT-Scan image of a patient with moderate infection. In Figure 2(c), the CT-Scan image of a patient with severe infection is depicted, and Figure 2(d) showcases the CT-Scan image of a COVID-19 negative patient (Normal).

Figure 2
CT Lung Images: Mild, Moderate, Severe and Normal

Image Pre-processing

The dataset comprised images of different sizes. To ensure consistency, all images were resized to 224x224, the standard format compatible with the pre-trained CNN architecture. Figure 3 exhibits a selection of images in diverse sizes, while Figure 4 demonstrates the resized image.

Figure 3
Original image

Figure 4
Image resized to 224 x 224

Noise Removal

To ensure precise diagnosis and efficient treatment planning in CT-scan lung images, noise removal is essential. Various techniques are employed to enhance image quality and reduce noise interference. In this study, Gaussian smoothing filters were utilized to diminish noise while preserving critical structures. Figure 5 depicts the image before noise removal, while Figure 6 demonstrates the same image after applying the Gaussian smoothing method, successfully eliminating noise.

Figure 5
Original Image (Before Noise Removal)

Figure 6
Original Image (After Noise Removal)

Enhancing the Image Contrast

Following noise removal in CT-scan lung images, enhancing image contrast is pivotal for improving diagnostic clarity. This study integrates Adaptive Histogram Equalization (AHE), a technique that redistributes pixel intensities to expand the dynamic range and enhance local contrast. AHE accentuates subtle variations in lung tissue density, particularly benefiting regions with low contrast, thereby improving the visibility of critical details for accurate diagnosis. Figure 7 illustrates the CT-lung image before AHE, while Figure 8 displays the same image after AHE, demonstrating the enhancement achieved through this technique.

Figure 7
Before Enhancing Image

Figure 8
After Enhancing Image contrast

Image Augmentation

To address the issue of dataset imbalance and to prevent data overfitting, data augmentation techniques were employed to balance the size of each class within the dataset. Figure 9 showcases various augmentation techniques, such as horizontal flip, rotation, shear, and vertical flip, applied to the dataset to achieve balance. Subsequently, Table 1 presents the balanced dataset post-data augmentation, illustrating the effectiveness of this approach.

Figure 9
Image Augmentation

Table 1
Distribution of train, validation and test sets of Dataset of COVID-19 CT scans (after augmentation)

Performance Metrics

The evaluation of the proposed model involves the utilization of various performance metrics, elaborated in Table 2.

Table 2
Metrics used for Calculating Performance

In the table above, TP signifies that the predicted class matches the actual class label. TN denotes the total of all rows and columns, excluding those associated with the class for which TN is being calculated. FP is obtained by adding all values in the column related to that class, excluding the TP. FN for a particular class is calculated by summing all values in the row linked to that class, except for the TP value.

Ground Truth versus Predicted Output

This section utilizes clinical data to detect COVID-19 presence with the Probabilistic Stacked Ensemble Classifier, featuring features selected by the Modified Mutual Information algorithm which includes CRP, ESR(60 MIN), DC:Neutrophils, Lymphocytes, LDH, Billirubin T, Direct Billirubin, RDW-CV, PC, MCV, Urea, RBS and D-DIMER. Upon confirming COVID-19, the patient's CT-scan lung image is analyzed by the mResNet-50 transfer learning model. Table 3 presents the input data, along with their actual and predicted outputs.

Table 3
Predicting severity of COVID-19 using Existing Classifiers and Proposed COVID-19 Severity Prediction model

The predicted classes are represented as follows: 0 for Normal, 1 for Mild, 2 for Moderate, and 3 for Severe. From the provided table, it can be deduced that the proposed classifier exhibits two instances of misclassification. Firstly, for patient number 4, the actual class is identified as 1 (COVID-19 Positive), whereas the predicted class is 0 (COVID-19 Negative) during phase 1. Consequently, this individual is not evaluated for disease severity. Secondly, for patient number 7, the actual class is recorded as 0 (COVID-19 Negative), yet the predicted class is 1 (COVID-19 Positive) during phase 1. Consequently, the CT-scan of this individual is inputted into the proposed mResNet-50 model, which classifies it as normal.

Experiment 1: Analysing the Performance of proposed feature selection method with state-of-the-art feature selection and classification methods

To identify influential features crucial for predicting COVID-19, various conventional feature selection methods along with the Proposed MMI are employed on clinical dataset. The top 14 features selected by various methods are as follows: ANOVA-F selected TC, monocytes, RBS, Direct Bilirubin, DC: Neutrophils, lymphocytes, basophils, ESR (60 min), MCH, D-Dimer, CRP, LDH, eosinophils, and Bilirubin T. The chi-square method selected MCV, MCH, eosinophils, age, monocytes, urea, ESR (60 min), RBS, CRP, D-Dimer, LDH, lymphocytes, DC: Neutrophils, and TC. Mutual Information identified DC: Neutrophils, lymphocytes, CRP, Bilirubin T, ESR (60 min), Direct Bilirubin, D-Dimer, LDH, MCV, MCH, RBS, RBC, urea, and eosinophils. Pearson Correlation selected DC: Neutrophils, eosinophils, ESR (60 min), monocytes, basophils, MCV, MCH, RBS, Direct Bilirubin, CRP, RDW-CV, Bilirubin T, LDH, and SGOT. SFS selected Gender_1, Gender_2, HB, TC, DC: Neutrophils, lymphocytes, eosinophils, monocytes, basophils, ESR (60 min), PC, PCV, MCV, and CRP. RFE identified lymphocytes, DC: Neutrophils, eosinophils, monocytes, basophils, ESR (60 min), PC, PCV, MCV, RBS, RDW-CV, urea, creatinine, CRP, and D-Dimer. The proposed MMI method selected CRP, ESR (60 min), DC: Neutrophils, lymphocytes, LDH, Bilirubin T, Direct Bilirubin, RDW-CV, PC, MCV, urea, RBS, and D-Dimer. These features were then utilized with SVM, NB, KNN, and LR classifiers, with performance analysis results summarized in Table 4.

Table 4
Proposed MMI Algorithm with State-of-the-Art Feature Selection and Classification

The table summarizes the performance metrics of various classifiers applied after feature selection using different algorithms. Across all feature selection methods, SVM consistently demonstrates strong performance, achieving accuracy ranging from 90.4% to 95.5%. Naïve Bayes and LR classifiers also exhibit competitive accuracy rates, generally ranging between 87.5% and 94.2%. Notably, the proposed MMI method consistently yields improved performance across all classifiers, with SVM achieving the highest accuracy of 95.5%. This suggests that the proposed feature selection approach effectively enhances the predictive capabilities of the classifiers, indicating its potential utility in COVID-19 severity prediction tasks.

Experiment 2: Analysing the Performance of Proposed Classifier with State-of-the-art Classifier

This experiment is held to analyse the performance of the proposed Probabilistic Stacked Ensemble Classifier (PSEC). For that the selected features from the proposed Modified Mutual Information algorithm are fed into the various state-of-the-art classifiers and the proposed PSEC. The results of the experiment are shown in Figure: 10.

Figure 10
Performance Analysis of the Proposed Classifier with other State-of-art Classifiers

The graph compares the performance of a proposed probabilistic stacked ensemble classifier against SVM, NB, k-NN, and LR using Accuracy, Precision, Recall, F1-Score, and AUC metrics. The proposed classifier outperforms all others across all metrics, achieving the highest values in accuracy, precision, recall, F1-Score, and AUC. SVM is the next best performer, while NB, k-NN, and LR show lower performance. This indicates the superior effectiveness of the proposed classifier.

Experiment 3: Performance of Proposed mResNet-50 with State-of-the-art Transfer Learning Model

The CT-Scan images of patients classified as COVID-19 positive by the proposed PSEC are fed into the proposed mResNet-50 model to predict disease severity. Experiments are conducted to analyse the performance of the proposed mResNet-50 alongside state-of-the-art transfer learning models. The results are presented in Figure 11.

Figure 11
Performance Analysis of the proposed mResNet-50 with other pre-trained models

The graph compares various pre-trained CNN models, including AlexNet, VGG16, VGG19, Inception V3, Inception V2, ResNet18, ResNet50, ResNet101, and ResNet152, with the proposed mResNet-50 model across multiple metrics: Accuracy, Precision, Recall, F1-Score, and AUC. The proposed mResNet-50 consistently achieves the highest values, indicating superior performance.

Experiment 4: Analysis of the proposed Model with Other Benchmark Dataset

The proposed model exhibits superior performance, showcasing its potential. Its applicability extends to heart disease and kidney stone datasets. Initially, it diagnoses and predicts types of heart failure using heart failure data. Similarly, it identifies affected individuals in the kidney disease dataset. Dataset details are in Table 5, with performance analysis in Figure 12.

Table 5
Description of the Benchmark Dataset

Figure 12
Evaluating the Proposed Model across Various Benchmark Datasets

The graph presents the performance metrics of the proposed model when applied to three different datasets: Congenital Heart Disease, Kidney Stone, and COVID-19. The metrics evaluated include Accuracy, Precision, Recall, F1-Score, and AUC (Area Under the Curve). The COVID-19 dataset consistently achieves the highest performance across all metrics, with notable scores such as 97.79% in Accuracy and 95.9% in AUC. The Kidney Stone dataset also performs well, particularly in Precision (95.7%) and F1-Score (96.1%). In contrast, the Congenital Heart Disease dataset shows the lowest performance, with Accuracy at 86.54% and AUC at 84.4%. Overall, the proposed model demonstrates superior effectiveness in predicting COVID-19 cases compared to the other two medical conditions.

Experiment 5: Analysis of the proposed Model with Other state-of-the-art models

The following table presents a comparative analysis of various state-of-the-art models and the proposed model for COVID-19 detection. The metrics used for evaluation include accuracy, precision, recall, and F1-score, showcasing the performance of each model. Table 6 highlights how the proposed model fares against existing architectures such as COVID-NET, Deformable ResNet-50, Severity Prediction System, and a COVID-19 detection hybrid model.

Table 6
Analysis of proposed model with state-of-the-art models

The table demonstrates that the proposed model outperforms other state-of-the-art models in accuracy and F1-score. It achieves the highest accuracy of 97.79%, exceeding the Deformable ResNet-50's 97.6%. Additionally, the proposed model shows strong performance in precision (95.2%) and recall (96.5%), resulting in an F1-score of 95.9%. Although the Deformable ResNet-50 matches the proposed model in recall, the proposed model's overall metrics indicate its superior effectiveness and robustness in COVID-19 detection.

DISCUSSION

This research introduces a new model for predicting COVID-19 severity using machine learning and transfer learning techniques. Most of the research literature focuses on predicting the number of COVID-19 cases or forecasting vaccination rates. While some studies assess COVID-19 severity using CT scans, symptoms, laboratory tests, or X-rays, this study integrates both blood test results and CT scan images. Initially, it uses blood sample data to detect the presence or absence of COVID-19 and then evaluates the severity in infected individuals. This dual approach helps reduce misclassification.The data presented in the table 4 indicates that the proposed MMI algorithm surpasses other state-of-the-art feature selection methods. Among the filter methods (ANOVA-F, Chi-Square, Mutual Information, and Pearson Correlation), comparable performance is observed, with accuracies ranging from 93.7% to 94.5%. The wrapper methods (RFE and SFS) exhibit slightly lower performance compared to the filter methods, achieving accuracies of 92.7% and 91.6%, respectively. In contrast, the proposed MMI method outperforms all other techniques, achieving the highest accuracy of 95.5%. These findings suggest that the MMI method delivers superior results in feature selection, leading to improved classification performance compared to the other methods evaluated.

Based on Figure 10, it's clear that the proposed PSEC exhibits superior performance compared to other state-of-the-art classifiers, achieving an accuracy of 98.5%. The accuracy scores of the other state-of-the-art classifiers, SVM, NB, k-NN, and LR, are 95.5%, 94.2%, 93.5%, and 92.5%, respectively. From Figure 11, it is clear that the proposed mResNet-50 model outperforms others across all metrics, achieving nearly 95% in Accuracy, Precision, and F1-Score. Traditional models like AlexNet and VGG16/19 lag behind with metrics in the 80-85% range. Inception models (V2 and V3) offer moderate improvements, reaching 85-90%, while ResNet variants, particularly ResNet-50 and ResNet-152, show significant performance enhancements, nearing 90-95%.

From Figure 12, it's apparent that the proposed model exhibits optimal performance across various datasets. Surprisingly, the model's performance surpasses that on the COVID-19 dataset. Table 6 further confirms this observation, indicating that the congenital heart disease dataset, characterized by a low number of instances, experiences a performance decline compared to other datasets. Thus, the results highlight the significant impact of instance count on model performance. This experiment effectively demonstrates the model's ability to generalize to diverse datasets.

Future iterations could incorporate suggestions for medication based on disease severity, thereby providing a more comprehensive approach to patient care and treatment planning. Overall, the research represents a significant step forward in leveraging machine learning and transfer learning techniques for COVID-19 severity prediction, with promising implications for improving patient outcomes.

CONCLUSION

This research article presents a novel approach for predicting the severity of COVID-19 using a stacked ensemble approach combining machine learning classifiers and a pre-trained CNN transfer learning model. The model's performance was evaluated using both clinical datasets for early screening and CT-scan image datasets for predicting disease stages, achieving an impressive accuracy of 97.79% compared to state-of-the-art methods. One of the main strengths of this research lies in its comprehensive approach to severity prediction. By integrating both clinical data and CT-scan images, the model offers a robust method for assessing disease severity. Moreover, the model's performance was validated on benchmark datasets, demonstrating its generalizability and effectiveness across different scenarios. The proposed prediction model has significant implications for healthcare practitioners. It can serve as a valuable tool for early identification of severity risks, enabling healthcare professionals to develop personalized patient care strategies and allocate resources such as ICU beds and ventilators more effectively. Additionally, by incorporating healthcare providers' experience and learning from past patient data, the model enhances its predictive capabilities and reliability. However, it's important to acknowledge the limitations of the model. It is specifically designed for predicting disease severity at two levels: initial screening using clinical data and subsequent severity prediction using images. Therefore, its applicability may be limited to diseases with similar prediction frameworks.

REFERENCES

  • 1 Banerjee A, Ray S, Vorselaars B, Kitson J, Mamalakis M, Weeks S, et al. Use of Machine Learning and Artificial Intelligence to predict SARS-CoV-2 infection from Full Blood Counts in a population. Int. Immunopharmacol. 2020 Sep; 86:106705.
  • 2 Browning L, Colling R, Rakha E, Rajpoot N, Rittscher J, James JA, et al. Digital pathology and artificial intelligence will be key to supporting clinical and academic cellular pathology through COVID-19 and future crises: the PathLAKE consortium perspective. J Clin Pathol. 2021 Jul; 74(7):443-7.
  • 3 Houssein EH, Abohashima Z, Elhoseny M, Mohamed WM. Hybrid quantum-classical convolutional neural network model for COVID-19 prediction using chest X-ray images. J. Comput. Des. Eng. 2022 Apr;9(2):343-63.
  • 4 Sayed SA, Elkorany AM, Sayed Mohammad S. Applying Different Machine Learning Techniques for Prediction of COVID-19 Severity. IEEE Access. 2021 Sep; 9:135697-707. Doi: 10.1109/ACCESS.2021.3116067.
    » https://doi.org/10.1109/ACCESS.2021.3116067.
  • 5 Alaiad AI, Mugdadi EA, Hmeidi II, Obeidat N, Abualigah L. Predicting the Severity of COVID-19 from Lung CT Images Using Novel Deep Learning. J Med Biol Eng. 2023 Apr;43(2):135-46. doi: 10.1007/s40846-023-00783-2.
    » https://doi.org/10.1007/s40846-023-00783-2.
  • 6 Albataineh Z, Aldrweesh F, Alzubaidi MA. COVID-19 CT-images diagnosis and severity assessment using machine learning algorithm. Cluster Comput. 2024 Feb; 27:547-62.
  • 7 Farahat IS, Sharafeldeen A, Ghazal M, Alghamdi NS, Mahmoud A, Connelly J, et al. An AI-based novel system for predicting respiratory support in COVID-19 patients through CT imaging analysis. Sci. Rep. 2024 Jan; 14(1):1-13.
  • 8 Zoabi Y, Deri-Rozov S, Shomron N. Machine learning-based prediction of COVID-19 diagnosis based on symptoms. npj Digit. Med. 2021 Jan; 4(3):1-5.
  • 9 Solayman S, Aumi A, Mery CS, Mubassir M, Khan R. Automatic COVID-19 prediction using explainable machine learning techniques. Int J. Cogn. Comput. Eng. 2023 Jun; 4: 36-46.
  • 10 Chadaga K, Chakraborty C, Prabhu S, Umakanth S, Bhat V, Sampathila N. Clinical and Laboratory Approach to Diagnose COVID-19 Using Machine Learning. Interdiscip Sci. 2022 Jun;14(2):452-70. doi: 10.1007/s12539-021-00499-4.
    » https://doi.org/10.1007/s12539-021-00499-4.
  • 11 Gök EC, Olgun MO. SMOTE-NC and gradient boosting imputation based random forest classifier for predicting severity level of covid-19 patients with blood samples. Neural Comput Appl. 2021 Jun; 33(22):15693-707.
  • 12 Laatifi M, Douzi S, Bouklouz A, Ezzine H, Jaafari J, Zaid Y, et al. Machine learning approaches in Covid-19 severity risk prediction in Morocco. J Big Data. 2022 Jan; 9(5):1-21.
  • 13 Fang ZG, Yang SQ, Lv CX, An SY, Wu W. Application of a data-driven XGBoost model for the prediction of COVID-19 in the USA: a time-series study. BMJ Open. 2022 Jul; 12(7):1-8. DOI: 10.1136/bmjopen-2021-056685.
    » https://doi.org/10.1136/bmjopen-2021-056685.
  • 14 Cihan P. Forecasting fully vaccinated people against COVID-19 and examining future vaccination rate for herd immunity in the US, Asia, Europe, Africa, South America, and the World. Appl Soft Comput. 2021 Nov; 111:107708. DOI: 10.1016/j.asoc.2021.107708.
    » https://doi.org/10.1016/j.asoc.2021.107708.
  • 15 Raman G, Ashraf B, Demir YK, Kershaw CD, Cheruku S, Atis M, et al. Machine learning prediction for COVID-19 disease severity at hospital admission. BMC Med. Inform. Decis. Mak. 2023 Mar; 23(1):1-6.
  • 16 Angulakshmi M, Deepa M, Anand M, Vanitha M, Thaseen S, Karthieyan P. COVID -19 Predictions using Transfer Learning based Deep Learning Model with Medical Internet of Things. Int. J. Recent Innov. Trends Comput. Commun. 2023 Mar;11(3):43-50.
  • 17 Govender P, Fashoto SG, Maharaj L, Adeleke MA, Mbunge E, Olamijuwon J, et al. Machine learning algorithms with one hot encoding, BoW and without PCA using Python. PLoS ONE. 2022 Feb; 17(2):1-19.
  • 18 Farizi WSA, Hidayah I, Rizal MN. Isolation Forest Based Anomaly Detection: A Systematic Literature Review. 2021 8th International Conference on Information Technology, Computer and Electrical Engineering (ICITACEE), Semarang, Indonesia, 2021, p.118-122, doi: 10.1109/ICITACEE53184.2021.9617498.
    » https://doi.org/10.1109/ICITACEE53184.2021.9617498.
  • 19 Joloudari JH, Marefat A, Nematollahi MA, Oyelere SS, Hussain S. Effective Class-Imbalance Learning Based on SMOTE and Convolutional Neural Networks. Appl. Sci. 2023 Mar; 13(6):1-34.
  • 20 Kumar S, Sharma A, Tsunoda T. An improved discriminative filter bank selection approach for motor imagery EEG signal classification using mutual information. In Proceedings of the 16th International Conference on Bioinformatics (InCoB)-Bioinformatics, Shenzhen, China, 2017, p.20-22.
  • 21 Huang S, Cai N, Pacheco PP, Narrandes S, Wang Y, Xu W. Applications of Support Vector Machine (SVM) Learning in Cancer Genomics. Cancer Genomics Proteomics. 2018 Jan; 15(1):41-51. doi: 10.21873/cgp.20063.
    » https://doi.org/10.21873/cgp.20063.
  • 22 Chen H, Hu S, Hua R, Zaho X. Improved naive Bayes classification algorithm for traffic risk management. EURASIP J. Adv. Signal Process. 2021 Jun; 30(2021):1-12.
  • 23 Sarmas E, Spiliotis E, Marinakis V, Koutselis T, Doukas H. A Meta-learning classification model for decisions on energy efficiency investments. Energy and Buildings. 2022 Mar; 258:111836.
  • 24 Xie S, Lin H, Cheng C, Chen Y, Wang Y, Zhao Y, Yong W. Shear strength model of joints based on Gaussian smoothing method and macro-micro roughness. Comput. Geotech. 2022 Mar; 143 (3):104605.
  • 25 Kansal S, Tripathi RK. New adaptive histogram equalization heuristic approach for contrast enhancement. IET Image Processing. 2020 Apr; 14(6):1110-9.
  • 26 Ma X, Li Z, Zhang L. An improved ResNet-50 for garbage image classification. Technical Gazette. 2022; 29(5):1552-9.
  • 27 Soares, Angelov. SARS-COV-2 Ct-Scan Dataset [Internet]. Kaggle: 2020. [Updated 2020 Apr 25; cited 2024 Mar 28]. Available from: http://www.kaggle.com/plameneduardo/sarscov2-ctscan-dataset
    » http://www.kaggle.com/plameneduardo/sarscov2-ctscan-dataset
  • 28 Chauhan A. Predicting Heart Failure [Internet]. Kaggle: 2022. [Updated: 2020 Jul 13; cited 2024 Mar 28]. Available from: https://www.kaggle.com/datasets/whenamancodes/heart-failure-clinical-records
    » https://www.kaggle.com/datasets/whenamancodes/heart-failure-clinical-records
  • 29 Iqbal M. Chronic Kidney Disease [Internet]. Kaggle: 2016. [Updated: 2018 Mar 23; cited 2024 Mar 28]. Available from: https://www.kaggle.com/datasets/mansoordaku/ckdisease
    » https://www.kaggle.com/datasets/mansoordaku/ckdisease
  • 30 Xu X, Wang T, Yuan H, Jia Q, Ceng J, Dong Y, et al. ImageCHD: A 3D Computed Tomography Image Dataset for Classification of Congenital Heart Disease. In Proc. Medical Image Computing and Computer Assisted Interventions (MICCAI), Lima, Peru. 2020. p.1-13.
  • 31 Yildirim K, Bozdag PG, Talo M, Yildirim O, Karabatak M, Acharya UR. Deep learning model for automated kidney stone detection using coronal CT images. Comput. Biol. Med. 2021 Aug; 135:104589.
  • 32 Wang L, Lin ZQ, Wong A. COVID-net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images. Sci. Rep 2020 Nov; 10(1):1-12.
  • 33 Foysal M, Hossain ABM, Yassine A, Hossain MS. Detection of COVID-19 Case from Chest CT Images Using Deformable Deep Convolutional Neural Network. J. Healthc. Eng, 2023 Feb; 2023:1-12.
  • 34 Aswathy A.L, Anand HS, Chandra SSV. COVID-19 severity detection using machine learning techniques from CT-images. Evol. Intel. 2023 Aug; 16:1423-31.
  • 35 Oğuz Ç, Yağanoğlu M. Detection of COVID-19 using deep learning techniques and classification methods. Inf Process Manag. 2022 Sep; 59(5):103025. doi: 10.1016/j.ipm.2022.103025.
    » https://doi.org/10.1016/j.ipm.2022.103025.
  • Funding:
    This research received no external funding.

Edited by

  • Editor-in-Chief:
    Alexandre Rasi Aoki
  • Associate Editor:
    Alexandre Rasi Aoki

Publication Dates

  • Publication in this collection
    30 Sept 2024
  • Date of issue
    2024

History

  • Received
    30 Mar 2024
  • Accepted
    25 June 2024
location_on
Instituto de Tecnologia do Paraná - Tecpar Rua Prof. Algacyr Munhoz Mader, 3775 - CIC, 81350-010 , Tel: +55 41 3316-3054 - Curitiba - PR - Brazil
E-mail: babt@tecpar.br
rss_feed Acompanhe os números deste periódico no seu leitor de RSS
Ir para o topo Reportar erro