Ensemble-based Heart Disease Diagnosis (EHDD) using Feature Selection and PCA Extraction Methods

1.1. The Organization of the Paper is as Follows

Section 2 presents a comprehensive literature review. Section 3 details the proposed EHDD model. Section 4 outlines the experimental methodology. Section 5 presents and discusses the research findings. Finally, Section 6 summarizes the study's conclusions and directions for future research.

2.1. Research Gap

• Accurate classification of the UCI repository continues to be a major machine-learning problem.
• The efficient selection of dataset extraction and feature selection algorithms will improve heart disease diagnosis and prediction.
• A decision-making system is inclusive in that it not only diagnoses but also controls and recommends treatment for heart disease.
• Conventional techniques such as neural networks, linear regression, support vector machines, Naive Bayes, and decision trees have been employed for prediction tasks. However, more advanced classifiers typically offer higher accuracy.
• The difficulties in utilizing limited attributes have yet to be reduced in order to reduce the procedure on patients for future predictions.

3.1. Rationale of the Proposed Technique

The classification problem has n features, namely C = {y1,y2,… yn}, where few may be redundant [18]. To reduce the computation cost, a function FS transforms the M dimensional dataset into low dimension FS’ where M < N.

Linear Discriminant Analysis (LDA), a filter-based technique is applied on FS’, where the top features are selected from FS’. Principal Component Analysis (PCA) feature extraction technique is applied on FS’ [19].

Choosing the appropriate feature enhances the proposed model's accuracy. The following are the benefits of using the filter-based LDA approach over the other methods:

• It enables faster training of the machine learning system.
• It reduces the model's complexity and computational requirements and improves interpretability.
• The appropriate subset leads to a more accurate model.
• It decreases the chance of overfitting (and the “curse of dimensionality”).

3.2. Feature Selection

The Linear Discriminant Analysis (LDA) filter-based feature selection algorithm is developed to decrease the dimensionality of high-dimensional datasets while maintaining class separability. The algorithm begins by computing the mean vectors for each class within the dataset, which serve as the reference points for class distinctions. Next, scatter matrices are calculated to measure the variance within each class (within-class scatter matrix) and between different classes (between-class scatter matrix). The mean vector is then derived to quantify similarities and differences in the scatter matrices.

For feature selection optimization, the eigenvalues and eigenvectors of the scatter matrices are computed. Here, the importance of the eigenvectors is reflected by their corresponding eigenvalues. For correct projection onto the new axes, the signs of the eigenvalues are flipped. The eigenvalues are sorted in descending order, and the most important features are selected based on the corresponding eigenvalues. The dataset is then projected onto a new feature space based on the selected eigenvectors. This leaves us with a dataset of reduced dimensionality that preserves the main distinctions between classes.

This professional implementation of LDA ensures faster training, reduced computational complexity, and enhanced model interpretability while addressing challenges like overfitting. This approach is especially useful for datasets with well-defined class separations, as it aims to maximize the variance between classes while minimizing the variance within each class.

One of the most significant parts of data mining is feature selection. It is very much needed while handling high dimensional data. Removing unnecessary characteristics provides many advantages, including reduced memory and computational cost, increased accuracy, and avoidance of overfitting. The importance of characteristics is measured by their correlation with the dependent variable using filter approaches [20]. The dataset's dimension is reduced to M features, where M<N without trailing originality. Filter techniques are much faster than embedded and wrapper methods because they do not require model training [21]. Linear Discriminant Analysis (LDA) takes the input features and produces a linear set of samples with comparable points close to each other, provided it determines the new samples' position on the projection points. The process for selecting LDA features is given in pseudocode 1. The classes are classified into i={1, 2,….c} depending on the scores of the scatter matrix Sc. The mean vector (m) is derived from the similarities between Sc and Sa and is used to project points over a certain distance [22]. The eigenvalues are assigned, and the Sc matrix's signs are reversed in the next step. As a result, the sorting of eigenvalues in decreasing order, Q, yields the new sample Z [23].

Pseudocode 1 explains the procedure for LDA feature selection. Using the scores of scatter matrix Sc, the classes are categorized into i={1, 2,….c} depending on the classes. Points are projected in a distance by mean vector (m) similarities from Sc and Sa [24]. In the next step, the eigenvalues are assigned, and the signs of the Sc matrix are reversed. Thus, the new sample Z is formed by the sorting of eigen values in decreasing order Q.

Algorithm 1- Linear Discriminant Analysis filter-based feature selection

Pseudocode 1- Linear Discriminant Analysis filter-based feature selection

Input: Dataset Y with n features, m samples, and c classes.

Output: Z = Original dataset * Top k eigenvectors

Procedure:

For each class i:

Compute mean vector m_i = Average (all samples in class i)

Initialize S_w = 0

For each class i:

For each sample y in class i:

Compute S_w = S_w + (y - m_i) * (y - m_i)^T

Compute overall mean vector m = Average (all samples in the dataset)

Initialize S_b = 0

For each class i:

Compute S_b = S_b + n_i * (m_i - m) * (m_i - m)^T

Compute eigenvalues & eigenvectors of S_w^-1 * S_b

Sort eigenvalues in descending order

Select the top k eigenvalues and their corresponding eigenvectors

Project original dataset onto the top k eigenvectors.

3.3. Feature Extraction

The Principal Component Analysis (PCA) feature extraction algorithm reduces high-dimensional data to a lower-dimensional space by identifying the directions with the greatest variance. The process, outlined in pseudocode 2, begins by standardizing the dataset to ensure each feature contributes equally to the analysis. Calculation of the mean and standard deviation for each feature features adjustment of the data.

Computing then follows for the covariance matrix in determining the relationships in the feature set. Its eigenvalues and eigenvectors are found. The resulting eigenvalues denote the total variance described by each eigenvector. Consequently, from these eigenvalues, one picks the set that describes most of the observed patterns by their respective biggest size in this eigenvalue context.

The dataset is projected onto these principal components, giving a lower-dimensional representation that contains the most important information of the data. It is then ensured that the cumulative variance accounts for a predefined percentage (e.g., 95%) of the total variance to ensure high accuracy while losing the least amount of information.

This is one of the professional PCA implementations to overcome such high-dimensional data challenges by reducing both the complexity and the computation overhead with enhanced interpretability and performance of the machine learning models.

The features are extracted from the FS' through PCA [25]. The input values will now be standardized by statistical methods such as Mean (M), Standard Deviation (SD), and Covariance (Cv).

, where X_m ϵ Sc is the points match, and X ϵ Sc is the total number of points

, where y_i is the value from the points, M is the mean, and N is the size.

The covariance between x and y features is represented as standard Cv, then the eigenvector ev₁, ev_{2, ….,} ev_n, and the eigenvalues λ₁, λ₂,…, λ_n .

Algorithm -2 PCA feature Extraction Algorithm

Pseudocode 2: Dimensionality Reduction Using PCA

Input: Dataset Y (m samples, n features)

Output: Transformed dataset Z with K dimensions.

Procedure:

For each feature in Y:

a. Compute the mean (M) of the feature.

b. Compute the standard deviation (SD) of the feature.

c. Normalize each feature value: y_norm = (y - M) / SD.

Compute the covariance matrix Cv of the normalized dataset Y_norm.

Perform eigendecomposition on Cv:

a. Extract eigenvalues (λ1, λ2, ..., λn).

b. Extract corresponding eigenvectors (ev1, ev2, ..., evn).

Sort eigenvalues in descending order and reorder eigenvectors accordingly.

Calculate cumulative variance explained by eigenvalues.

Determine the smallest K such that cumulative variance > 95%.

Select the top K eigenvectors to form matrix VK.

Transform the dataset: Z = Y_norm * VK.

3.4. Classification

SVM is a classification technique that divides data points into two distinct categories using a hyperplane. Typically, the dataset is split into training (about 10%) and testing (about 90%) sets. The training set is often further divided for model tuning, where hyperparameters are adjusted to optimize performance on a validation subset before evaluating the model on the full test set [26]. The points may stretch out on either the positive or negative axis because the projection is scattered. SVM classifier works effectively on positive points (-1), but the negative points are again labeled as positive (1) by SVM. As a result, the Random Forest (RF) classifier combines with SVM, and an ensemble technique is used.

3.5. Ensemble Technique

The following are the reasons why ensemble is better than many other classifiers:

• Negative points do not produce adequate results, and SVM is computationally expensive.
• The SVM method almost correctly classifies the points on 0.
• The SVM algorithm almost correctly classifies the points on negative points (-1).
• The Random Forest algorithm almost correctly classifies the positive point (1) class.

A majority vote process is utilized to improve accuracy and appropriately identify negative and positive points.

3.6. Innovative Aspects of the EHDD Model

The Ensemble-based Heart Disease Diagnosis (EHDD) model offers a unique approach by combining Linear Discriminant Analysis (LDA) for feature selection, Principal Component Analysis (PCA) for feature extraction, and ensemble classifiers for the final classification step. While each of these methods has been explored individually in machine learning applications, their combined application in diagnosing heart disease offers a significant advancement over existing methodologies.

The use of LDA as a filter based feature selection method reduces data complexity by isolating the most relevant features while maintaining class separability. This optimizes the input data for further processing. Subsequently, PCA extracts the most important patterns by transforming the high dimensional dataset into a reduced-dimension space, retaining only the most critical components. This sequential reduction of dimensionality ensures that the dataset is not only computationally efficient but also highly informative, overcoming the “curse of dimensionality” commonly associated with medical datasets.

The integration of ensemble learning methods, specifically combining SVM and RF, further enhances the model's predictive capability. SVM effectively handles linear boundaries, while Random Forest introduces non-linear decision-making capabilities. By leveraging the strengths of both classifiers through majority voting, the EHDD model achieves robust performance and minimizes errors.

The innovative combination of LDA, PCA, and ensemble techniques results in an impressive accuracy of 98%, outperforming other models that are tested on the Cleveland dataset. This approach enhances classification accuracy and ensures computational efficiency, making it a promising solution for real-world diagnostics of HD. A comparative analysis shows the superiority of the EHDD model over traditional approaches, and further validates its novelty and effectiveness.

3.7. Technical Contributions

4.1. Innovative Integration

Sequential use of PCA for dimensionality reduction and LDA for classification has a special advantage.

• PCA ensures that only the most informative features are passed to the classifiers, while LDA utilizes these features to optimize the class separability for heart disease.
• This integration ensures a balanced trade-off between data compression and accuracy of diagnosis, one common challenge in medical datasets.

6.1. Hyperparameter Selection

There are two hyperparameters for SVM to classify, i.e., Subject diagnosed with Heart Disease (SHD) and Healthy Subject (HS). The scale is set to (0,1), where 0 indicates a penalty and 1 indicates correct classification [27].

6.2. Evaluation Criteria

Accuracy, specificity, and sensitivity qualitative measurements are used to evaluate the performance of the proposed system [28].

where TP = True Positive, FP = False Positive, TN = True Negative, and FN = False Negative, respectively.

6.3. Dataset

The Cleveland dataset from the UCI repository is used [29]. The original dataset has thirteen features [30]. Fig. (2) represents the first 10 rows of the heart dataset of Cleveland. The inference made from the dataset is that the total number of people without heart disease is 138, which is labeled as healthy, and people with the chance for heart disease is 165, which is labeled as sick. It has 14 attributes. Table 1 depicts the features of the dataset. Age, sex, and chest pain are taken with 4 values. Other features include: blood pressure at rest, cholesterol (serum), blood sugar (fasting) greater than 120, electrocardiogram results at rest, maximum heart rate, exercise-induced angina, the peak slope during exercise, and the presence of major vessel defects (normal to reversible). The target variable consists of class 0 and class 1.

6.4. Generalization of Dataset

The EHDD model demonstrates exceptional accuracy on the Cleveland dataset, achieving 98% accuracy through its integration of LDA, PCA, and ensemble classifiers. However, ensuring the model's robustness and applicability across diverse datasets is critical for its broader adoption. Validating the model's performance and adaptability in real-world scenarios requires generalizing it to other datasets with diverse demographic and medical profiles.

Testing the EHDD model on datasets beyond Cleveland, such as those from other geographic regions or medical institutions, will provide a comprehensive evaluation of its effectiveness. Preliminary experiments with datasets whose attribute types differ, say by age groups, ethnic groups, and medical history, would point out some possible weaknesses in the model. Those experiments could also be fine-tuned in feature selection and classification for different kinds of data distributions.

One potential limitation in generalization is the demographic specificity of the Cleveland dataset, which may not fully represent global patient populations. Variability in medical equipment, data collection methods, and patient characteristics could affect the model's accuracy when applied to different settings. The model could be fine-tuned or re-trained on datasets with more diverse attributes. Techniques like transfer learning and domain adaptation could further enhance its generalization capability.

Future work will focus on validating the EHDD model across multiple datasets, incorporating additional features where necessary, and addressing challenges in bias reduction and representational diversity. By doing so, the EHDD model can evolve into a universally applicable tool for HD diagnostics.

7.1. Data Points

Each patient record in the data set has 13 features or attributes. The attributes/ features capture aspects of patient health, including demographics, medical history, vital signs, and results from diagnostic tests. The target column indicates whether the patient has heart disease (1) or not (0). The well-structured data set was useful for training the machine learning models to predict the possibilities of heart disease in regard to various characteristics of a patient.

Fig. (2) depicts the association between different variables. Target: it indicates healthy is 1 and sick is 0. The data set has 14 attributes: sex, blood pressure, age, cholesterol, etc., along with other medical test results. A correlation matrix visually depicts relationships between these attributes. Darker red hues signify strong positive correlations, indicating variables increase together. Conversely, darker blue shades represent strong negative correlations, where one variable rises as the other falls. Lighter colors denote weak or negligible correlations.

Fig. (3) depicts the correlation between features and value. It is represented in the heat map. With higher the, there is more correlation. With lower values and negative values, there is less or zero correlation. Observations from correlations are as follows:

7.1.1. Age and Chest Pain (cp)

Older individuals are more likely to experience chest pain.

7.2. Advantage of the Proposed Model

7.3. Limitations of the EHDD Model

The EHDD model achieves 98% accuracy on the Cleveland dataset but faces limitations in dataset dependency, computational overhead from ensemble techniques, and potential latency in real-time applications. These challenges may hinder generalization to diverse datasets and resource constrained environments. Future work should focus on optimizing computational efficiency and testing on varied datasets for broader applicability in healthcare.

7.4. Ethical Considerations in the EHDD Model

The EHDD model raises ethical concerns, including the risk of misdiagnosis, bias in training data, and the need for interpretability. Misdiagnoses, such as false negatives or positives, can lead to serious health consequences. Bias in datasets, especially demographic under-representation, could impact fairness. Robust validation, diverse data inclusion, and model interpretability through explainable AI (XAI) are critical for responsible deployment and trust in healthcare.

7.5. Integrating EHDD into Clinical Workflows

The EHDD model improves clinical workflows through the simplification of heart disease diagnosis, improved accuracy, and a decrease in diagnostic errors. Dimensionality reduction techniques are used, thus ensuring efficiency, which makes the model valuable in busy clinical environments. The model may also reduce diagnostic costs because it automates initial stages and optimizes resource allocation. In critical care, it aids in triaging patients and supporting timely, effective treatment, which improves healthcare outcomes.