1. Logistic Regression
The binary logistic regression model is a classification model represented by the conditional probability distribution P(Y|X), in the form of a parameterized logistic distribution. Here, the random variable X takes real values, and the random variable Y takes values of 1 or 0. The model parameters can be estimated using a supervised method.
Advantages:
1. Low computational cost, easy to understand and implement;
2. Suitable for scenarios where classification probabilities are needed;
3. Robust to small data noise and not significantly affected by slight multicollinearity.
Disadvantages:
1. Prone to underfitting, classification accuracy may not be high;
2. Performs poorly when data is missing or features are very large.
2. Support Vector Machine (SVM)
For linearly separable learning tasks of two classes, SVM finds a hyperplane that maximizes the margin to separate the two classes, ensuring the hyperplane has the best generalization ability.
Advantages:
1. Can solve ML problems in small sample situations;
2. Can improve generalization performance;
3. Can handle high-dimensional problems, avoiding the curse of dimensionality;
4. Can solve nonlinear problems;
5. Can avoid issues with neural network structure selection and local minima.
The choice of parameters C and gamma affects classification performance:
C is the penalty coefficient; the larger C is, the higher the cross-validation score, but it may lead to overfitting;
Gamma is the rate at which the kernel function approaches 0; a smaller gamma results in a faster decrease of the function, leading to a higher cross-validation score, which can also cause overfitting.
Disadvantages:
1. Sensitive to missing data;
2. No universal solution for nonlinear problems, careful selection of the kernel function is necessary.
Key Advantages of SVM:
1) Effectively solves classification and regression problems with high-dimensional features, maintaining good performance even when the feature dimension exceeds the number of samples.
2) Only a portion of support vectors are used to make decisions about the hyperplane, not relying on all data.
3) A large variety of kernel functions can be used, allowing for flexible solutions to various nonlinear classification and regression problems.
4) When the sample size is not massive, classification accuracy is high and generalization ability is strong.
Key Disadvantages of SVM:
1) If the feature dimension is much larger than the number of samples, SVM performs generally.
2) SVM is computationally intensive when the sample size is very large and the kernel mapping dimension is very high (not suitable for large datasets).
3) There are no universal standards for choosing the kernel function for nonlinear problems, making it difficult to select an appropriate kernel.
4) SVM is sensitive to missing data.
1) It is generally recommended to normalize the data before training, and of course, the data in the test set also needs to be normalized.
2) In cases with a very large number of features, or when the number of samples is far less than the number of features, using a linear kernel can yield good results, and only the penalty coefficient C needs to be selected.
3) When choosing a kernel function, if linear fitting is poor, it is generally recommended to use the default Gaussian kernel ‘rbf’. At this point, we mainly need to painstakingly tune the penalty coefficient C and the kernel function parameter gamma through multiple rounds of cross-validation to select appropriate values.
4) Theoretically, the Gaussian kernel should not perform worse than the linear kernel, but this theory is based on spending more time tuning parameters. Therefore, whenever possible, we should use the linear kernel to solve problems.
3. Decision Tree
A heuristic algorithm, the core is to apply criteria such as information gain at each node of the decision tree to select features, recursively constructing the decision tree.
Advantages:
1. Low computational complexity, easy to understand and explain, making the meaning expressed by the decision tree easy to grasp;
2. The data preprocessing stage is relatively simple and can handle missing data;
3. Can handle both numerical and categorical attributes simultaneously, and can construct decision trees for datasets with many attributes;
4. It is a white-box model; given an observation model, it is easy to infer the corresponding logical expression based on the generated decision tree.
5. Can produce feasible and effective classification results for large datasets in a relatively short time.
6. Can construct decision trees for datasets with many attributes.
Disadvantages:
1. For datasets with imbalanced samples across categories, the results of information gain tend to favor attributes with more values;
2. Sensitive to noisy data;
3. Prone to overfitting;
4. Ignores the correlation between attributes in the dataset;
5. Difficulties in handling missing data.
Advantages of Decision Trees:
1) Simple and intuitive; the generated decision tree is very straightforward.
2) Requires almost no preprocessing, no need for prior normalization, and can handle missing values.
3) The cost of prediction using decision trees is O(log_2m), where m is the number of samples.
4) Can handle both discrete and continuous values; many algorithms focus only on either discrete or continuous values.
5) Can handle multi-dimensional output classification problems.
6) Compared to black-box classification models like neural networks, decision trees can provide good logical explanations.
7) Can use cross-validation pruning to select models, thereby improving generalization ability.
8) Good fault tolerance to outliers, with high robustness.
Disadvantages of Decision Tree Algorithms:
1) Decision tree algorithms are very prone to overfitting, leading to weak generalization ability. This can be improved by setting the minimum sample number for nodes and limiting the depth of the decision tree.
2) Decision trees can undergo drastic changes in structure due to slight modifications in samples. This can be resolved using ensemble learning methods.
3) Finding the optimal decision tree is an NP-hard problem, typically solved using heuristic methods, which can easily fall into local optima. Ensemble learning methods can improve this.
4) Some complex relationships are difficult for decision trees to learn, such as XOR. In such cases, neural network classification methods can be used instead.
5) If certain features have a disproportionately large sample ratio, the generated decision tree may be biased towards these features. This can be improved by adjusting sample weights.
4. KNN Algorithm
A lazy classification method that identifies the k nearest training objects to the test object from the training set, and assigns the dominant category from these k training objects to the test object.
Advantages:
1. Simple and effective, easy to understand and implement;
2. The cost of retraining is low (due to changes in the category system and training set);
3. Computational time and space are linear with respect to the size of the training set;
4. The error rate converges asymptotically to the Bayesian error rate, serving as an approximation to Bayesian methods;
5. Suitable for handling multi-modal classification and multi-label classification problems;
6. More suitable for test samples with a lot of overlap or intersection in the class domains;
Disadvantages:
1. A lazy learning method, slower than some active learning algorithms;
2. High computational cost, needing to edit sample points;
3. Performs poorly on imbalanced datasets; weighted voting methods can be used to improve performance;
4. The choice of k greatly influences classification results; a small k is sensitive to noise, requiring estimation of the optimal k value.
5. Limited interpretability, high computational load.
Main Advantages of KNN:
1) The theory is mature and simple, can be used for both classification and regression.
2) Can be used for nonlinear classification.
3) Training time complexity is lower than that of algorithms like support vector machines, being only O(n).
4) Compared to algorithms like naive Bayes, it makes no assumptions about the data, achieving high accuracy and being insensitive to outliers.
5) Since KNN primarily relies on nearby limited samples rather than the discriminative class domain, it is more suitable for test samples with a lot of overlap or intersection in class domains.
6) This algorithm is particularly suitable for automatic classification of class domains with large sample sizes, while small sample sizes can lead to misclassification.
Main Disadvantages of KNN:
1) High computational cost, especially when the number of features is very high.
2) When samples are imbalanced, the prediction accuracy for rare classes is low.
3) Models like KD trees and ball trees require a lot of memory to establish.
4) Using a lazy learning method, which does not learn much, leads to slower prediction speeds compared to algorithms like logistic regression.
5) Compared to decision tree models, KNN models have limited interpretability.
5. Naive Bayes Algorithm
The classification principle of the Bayesian classifier is to use the prior probabilities of each category, and then use Bayes’ theorem and the independence assumption to calculate the probability of attributes belonging to a category and the posterior probability of the object, which is the probability that the object belongs to a certain category, selecting the category with the maximum posterior probability as the category to which the object belongs.
Advantages:
1. Solid mathematical foundation, stable classification efficiency, easy to interpret;
2. Requires very few estimated parameters and is not very sensitive to missing data;
3. Does not require a complex iterative solving framework, suitable for large-scale datasets.
Disadvantages:
1. The independence assumption between attributes often does not hold (consider using clustering algorithms to cluster attributes with high correlation);
2. Requires knowledge of prior probabilities, leading to classification decisions with error rates.
Main Advantages of Naive Bayes:
1) The Naive Bayes model originates from classical mathematical theory and has stable classification efficiency.
2) Performs well on small-scale data, can handle multi-class tasks, suitable for incremental training, especially when the data volume exceeds memory, allowing for batch incremental training.
3) Not very sensitive to missing data, the algorithm is also relatively simple, commonly used for text classification.
Main Disadvantages of Naive Bayes:
1) Theoretically, the Naive Bayes model has the lowest error rate compared to other classification methods. However, this is not always the case in practice, as the Naive Bayes model assumes independence among attributes given the output category, which often does not hold in real applications, leading to poor classification performance when the number of attributes is large or when there is significant correlation among attributes. In cases of low attribute correlation, Naive Bayes performs best. Algorithms like semi-Naive Bayes have been developed to consider partial correlations for moderate improvement.
2) Requires knowledge of prior probabilities, which often depend on assumptions; the assumed model can take many forms, leading to poor predictive performance in some cases due to the assumptions of the prior model.
3) Since we determine the posterior probability based on the prior and the data, classification decisions carry a certain error rate.
4) Sensitive to the expression form of input data.
6. Random Forest Algorithm
Main Advantages of RF:
1) Training can be highly parallelized, providing advantages in training speed for large samples in the big data era. This is the main advantage.
2) By randomly selecting decision tree nodes to partition features, it can still efficiently train models even when the sample feature dimensions are very high.
3) After training, it can provide the importance of each feature for the output.
4) Due to random sampling, the variance of the trained model is small, and its generalization ability is strong.
5) Compared to boosting series like Adaboost and GBDT, RF is relatively simple to implement.
6) Insensitive to missing features.
Main Disadvantages of RF:
1) On certain noisy sample sets, RF models are prone to overfitting.
2) Features with many value partitions can greatly influence RF’s decisions, affecting the fitting of the model.
7. AdaBoost Algorithm
Boosting methods start from weak learning algorithms, repeatedly learning to obtain a series of weak classifiers (i.e., base classifiers), and then combine these weak classifiers to form a strong classifier. Most boosting methods change the probability distribution of the training dataset (the weight distribution of the training data), calling weak learning algorithms to learn a series of weak classifiers based on different training data distributions.
Advantages:
1. High classification accuracy;
2. Can use various methods to construct sub-classifiers; the AdaBoost algorithm provides a framework;
3. Simple and does not require feature selection;
4. Does not lead to overfitting.
Disadvantages:
1. Samples that are misclassified multiple times may receive excessive weight, affecting the choice of classifier and causing degradation issues (the weight update method needs improvement);
2. Imbalanced data issues lead to a sharp decline in classification accuracy;
3. Training the algorithm takes time and is difficult to expand;
4. Issues like overfitting and lack of robustness may arise.
Key Advantages of AdaBoost:
1) AdaBoost achieves high classification accuracy as a classifier.
2) Within the AdaBoost framework, various regression classification models can be used to build weak learners, making it very flexible.
3) As a simple binary classifier, it is easy to construct and the results are understandable.
4) Not prone to overfitting.
Main Disadvantages of AdaBoost:
1) Sensitive to outliers; outliers may receive higher weights in iterations, affecting the final strong learner’s prediction accuracy.
8. GBDT
Main Advantages of GBDT:
1) Can flexibly handle various types of data, including continuous and discrete values.
2) Achieves high prediction accuracy with relatively little tuning time, especially compared to SVM.
3) Uses robust loss functions, demonstrating strong robustness to outliers, such as Huber loss functions and quantile loss functions.
Main Disadvantages of GBDT:
1) Due to the dependency between weak learners, it is difficult to train the data in parallel. However, partial parallelization can be achieved through self-sampling SGBT.
9. XGBoost Algorithm
1. Advantages of XGBoost compared to GBDT:
Adds the complexity of the tree model to the regularization term to avoid overfitting, thus outperforming GBDT in generalization performance.
Uses Taylor expansion to express the loss function, utilizing both first and second derivatives to accelerate optimization speed.
GBDT only supports CART as the base learner, while XGBoost also supports linear classifiers as base learners.
Introduces feature subsampling, similar to random forests, to avoid overfitting and reduce computation.
When searching for optimal split points, it implements an approximate greedy algorithm to speed up and reduce memory overhead, while also considering the handling of missing values in sparse datasets.
XGBoost supports parallel processing. The parallelization in XGBoost is not during model generation, but rather on features, storing features in memory in blocks after sorting them, and reusing this structure in subsequent iterations. This block also enables parallelization, and during node splitting, the gain for each feature is computed, allowing for multi-threaded computation of the feature with the highest gain for splitting.
2. Disadvantages compared to LightGBM:
XGBoosting uses pre-sorting, which traverses and selects the optimal split point before iterations; this greedy method is time-consuming when the data volume is large. In contrast, LightGBM uses a histogram algorithm, which occupies less memory and has lower complexity for data splitting.
XGBoosting generates decision trees level-wise, splitting leaves in the same layer, optimizing for multi-threading, which is less prone to overfitting. However, many leaf node splits may have low gain, making further splitting unnecessary, leading to unnecessary overhead. LightGBM adopts a depth-optimized, leaf-wise growth strategy, selecting the node with the highest gain from the current leaf for splitting, iterating to grow deeper decision trees, which may lead to overfitting. Therefore, a threshold is introduced to limit this and prevent overfitting.
10. Artificial Neural Networks
Advantages:
1. High classification accuracy, strong parallel distributed processing capability, and strong distributed storage and learning ability;
2. Strong robustness and fault tolerance to noise, capable of approximating complex nonlinear relationships and possessing associative memory functions.
Disadvantages:
1. Neural networks require a large number of parameters, such as network topology, weight, and threshold initial values;
2. The learning process cannot be observed; output results are difficult to explain, which may affect the credibility and acceptability of the results;
3. Learning time is excessively long and may not achieve learning objectives.