Source: Zhihu Abner says AI
This article is approximately 4500 words long and suggests a reading time of 9 minutes.
This article summarizes the pros and cons of the top 10 machine learning algorithms.
1. Logistic Regression
The binary logistic regression model is a classification model represented by the conditional probability distributionP(Y|X), which takes the form of a parameterized logistic distribution. Here, the random variable X takes real values, and the random variable Y takes values of 1 or 0. The model parameters can be estimated using supervised methods.
Advantages:
1. Low computational cost, easy to understand and implement;
2. Suitable for scenarios where classification probabilities are needed;
3. Good robustness to small data noise, not easily affected by slight multicollinearity.
Disadvantages:
1. Prone to underfitting, classification accuracy may not be high;
2. Performs poorly when data is missing or when features have large variations.
2. Support Vector Machine
For linearly separable learning tasks between two classes, SVM finds a hyperplane with the maximum margin to separate the two classes, ensuring the best generalization ability of the hyperplane.
Advantages:
1. Can solve ML problems with small samples;
2. Can improve generalization performance;
3. Can handle high-dimensional problems, avoiding the curse of dimensionality;
4. Can solve nonlinear problems;
5. Avoids issues with neural network structure selection and local minima;
The choice of parameters C and g affects classification performance:
C is the penalty coefficient; the larger C is, the higher the cross-validation accuracy, but it is prone to overfitting;
g is the rate at which the kernel function approaches 0; the smaller g is, the faster the function decreases, leading to high cross-validation accuracy, but also prone to overfitting.
Disadvantages:
1. Sensitive to missing data;
2. No universal solution for nonlinear problems; careful selection of the kernel function is required.
The main advantages of the SVM algorithm include:
1) Effectively solves classification and regression problems with high-dimensional features, maintaining good performance even when the feature dimension exceeds the sample size.
2) Only a portion of the support vectors are used to make decisions about the hyperplane, without relying on the entire dataset.
3) A wide variety of kernel functions are available, allowing for flexible solutions to various nonlinear classification and regression problems.
4) When the sample size is not massive, classification accuracy is high, and generalization ability is strong.
The main disadvantages of the SVM algorithm include:
1) If the feature dimension is much larger than the sample size, SVM performs poorly.
2) When the sample size is very large and the kernel function’s mapping dimension is very high, the computational load is excessive, making it unsuitable for large datasets.
3) There is no universal standard for selecting kernel functions for nonlinear problems, making it challenging to choose an appropriate kernel function.
4) SVM is sensitive to missing data.
1) It is generally recommended to normalize data before training, and the data in the test set also needs normalization.
2) When the number of features is very high or the sample size is much smaller than the number of features, using a linear kernel can yield good results, requiring only the selection of the penalty coefficient C.
3) When selecting a kernel function, if linear fitting is poor, it is generally recommended to use the default Gaussian kernel ‘rbf’. In this case, careful tuning of the penalty coefficient C and the kernel function parameter γ is necessary through multiple rounds of cross-validation to choose suitable values.
4) Theoretically, the Gaussian kernel should not perform worse than the linear kernel; however, this theory is based on spending more time tuning parameters. Therefore, we should use the linear kernel whenever possible to solve problems.
3. Decision Tree
A heuristic algorithm that applies criteria such as information gain at each node of the decision tree to select features, recursively constructing the decision tree.
Advantages:
1. Low computational complexity, easy to understand and interpret, allowing for a clear understanding of the decision tree’s meaning;
2. Simple data preprocessing stage, capable of handling missing data;
3. Can handle both numerical and categorical attributes simultaneously, and can construct decision trees for datasets with many attributes;
4. It is a white-box model; given an observation model, it is easy to infer the corresponding logical expression based on the generated decision tree;
5. Can provide feasible and effective classification results for large datasets within a relatively short time.
6. Capable of constructing decision trees for datasets with many attributes.
Disadvantages:
1. For datasets with imbalanced sample sizes among categories, the results of information gain tend to favor attributes with more values;
2. Sensitive to noisy data;
3. Prone to overfitting;
4. Ignores the correlations between attributes in the dataset;
5. Difficulties in handling missing data.
Advantages of Decision Trees:
1) Simple and intuitive; the generated decision tree is very clear.
2) Requires minimal preprocessing; no need for prior normalization or handling of missing values.
3) The cost of using decision trees for prediction is O(log_2m), where m is the number of samples.
4) Can handle both discrete and continuous values, whereas many algorithms focus only on one.
5) Can handle multi-dimensional output classification problems.
6) Compared to black-box classification models like neural networks, decision trees provide good logical explanations.
7) Can use cross-validation pruning to select models, thereby improving generalization ability.
8) Good fault tolerance for outliers, with high robustness.
Disadvantages of Decision Tree Algorithms:
1) Decision tree algorithms are very prone to overfitting, leading to weak generalization ability. This can be improved by setting a minimum sample number for nodes and limiting the depth of the decision tree.
2) A small change in samples can lead to drastic changes in the tree structure. This can be addressed through ensemble learning methods.
3) Finding the optimal decision tree is an NP-hard problem, typically solved through heuristic methods that can easily get stuck in local optima. Ensemble learning methods can help improve this.
4) Decision trees find it difficult to learn some complex relationships, such as XOR. For such relationships, neural network classification methods are generally used.
5) If certain features have a disproportionately large sample ratio, the generated decision tree may be biased towards those features. This can be improved by adjusting sample weights.
4. KNN Algorithm
A lazy classification method that finds the k closest training objects to the test object from the training set, then identifies the dominant category among these k training objects and assigns it to the test object.
Advantages:
1. Simple and effective, easy to understand and implement;
2. Low cost for re-training (changes in category systems and training set);
3. Computational time and space are linear with respect to the size of the training set;
4. The error rate gradually converges to the Bayesian error rate, serving as an approximation to Bayesian methods;
5. Suitable for multi-modal classification and multi-label classification problems;
6. Particularly suitable for classification samples with a lot of overlap or intersection in class domains.
Disadvantages:
1. It is a lazy learning method, slower than some active learning algorithms;
2. High computational load, requiring clipping of sample points;
3. Poor performance on imbalanced datasets; weighted voting methods can be used for improvement;
4. The choice of k significantly impacts classification performance; a small k is sensitive to noise, requiring estimation of the optimal k;
5. Limited interpretability and high computational demand.
Key Advantages of KNN:
1) The theory is mature and simple, applicable for both classification and regression;
2) Can be used for nonlinear classification;
3) Training time complexity is lower than algorithms like SVM, only O(n);
4) Compared to algorithms like Naive Bayes, it makes no assumptions about the data, achieving high accuracy and being insensitive to outliers;
5) Since KNN primarily relies on a limited number of nearby samples rather than discriminative class domain methods, it is more suitable for classification samples with a lot of overlap or intersection in class domains;
6) This algorithm is more suitable for automatic classification of class domains with larger sample sizes, while smaller class domains may lead to misclassification.
Main Disadvantages of KNN:
1) High computational load, especially with many features;
2) Low prediction accuracy for rare classes in imbalanced datasets;
3) Models like KD-trees and ball trees require significant memory;
4) Using a lazy learning method, with minimal learning, results in slower prediction speeds compared to algorithms like logistic regression;
5) Compared to decision tree models, KNN models have limited interpretability.
5. Naive Bayes Algorithm
The classification principle of the Bayesian classifier is to use the prior probabilities of each category, then apply Bayes’ theorem and the independence assumption to calculate the probabilities of attributes and the posterior probability of an object, which is the probability that the object belongs to a certain class, selecting the class with the maximum posterior probability as the class to which the object belongs.
Advantages:
1. Strong mathematical foundation, stable classification efficiency, and easy to explain;
2. Requires very few estimated parameters and is less sensitive to missing data;
3. No complex iterative solving framework is needed, making it suitable for large-scale datasets.
Disadvantages:
1. The independence assumption between attributes is often not valid (consider using clustering algorithms to cluster highly correlated attributes first);
2. Requires knowledge of prior probabilities, leading to errors in classification decisions.
Main Advantages of Naive Bayes:
1) The Naive Bayes model originates from classical mathematical theory and has stable classification efficiency.
2) Performs well on small-scale data, can handle multi-class tasks, suitable for incremental training, especially when data volume exceeds memory, allowing for batch incremental training.
3) Less sensitive to missing data, and the algorithm is relatively simple, commonly used in text classification.
Main Disadvantages of Naive Bayes:
1) Theoretically, the Naive Bayes model has the lowest error rate compared to other classification methods. However, this is not always the case in practice because the Naive Bayes model assumes independence between attributes given the output category, which often does not hold in practical applications, leading to poor classification performance when the number of attributes is large or when there is significant correlation between attributes. In contrast, it performs best when attribute correlations are low. Algorithms like semi-Naive Bayes can moderately improve this by considering partial correlations.
2) Requires knowledge of prior probabilities, which often depend on assumptions; various models of assumptions can lead to poor prediction performance.
3) The classification decision involves a certain error rate due to reliance on prior and data to determine posterior probabilities.
4) Sensitive to the representation of input data.
6. Random Forest Algorithm
Main Advantages of RF:
1) Training can be highly parallelized, offering speed advantages for large sample training in the big data era. This is the primary advantage.
2) Due to the random selection of decision tree node splitting features, it can still efficiently train models even when the sample feature dimensions are very high.
3) After training, it can provide the importance of each feature for the output.
4) Due to random sampling, the variance of the trained models is low, and the generalization ability is strong.
5) Compared to boosting series methods like Adaboost and GBDT, RF is relatively simple to implement.
6) Insensitive to missing values in some features.
Main Disadvantages of RF:
1) In some datasets with significant noise, the RF model is prone to overfitting.
2) Features with many possible values can have a greater influence on RF decisions, affecting the fitting of the model.
7. AdaBoost Algorithm
The boosting method starts from weak learning algorithms, repeatedly learning to obtain a series of weak classifiers (basic classifiers), and then combines these weak classifiers to form a strong classifier. Most boosting methods modify the probability distribution of the training dataset (the weight distribution of the training data), calling weak learning algorithms to learn a series of weak classifiers based on different training data distributions.
Advantages:
1. High classification accuracy;
2. Can use various methods to construct sub-classifiers; the Adaboost algorithm provides a framework;
3. Simple and does not require feature selection;
4. Does not lead to overfitting.
Disadvantages:
1. If misclassified samples are repeatedly weighted too heavily, it can lead to deterioration in classifier selection due to excessive weights (improvements in weight updating methods are necessary);
2. Imbalanced data can lead to a sharp decline in classification accuracy;
3. The algorithm requires significant training time and is difficult to extend;
4. Issues such as overfitting and low robustness may arise.
Main Advantages of Adaboost:
1) Adaboost has high classification accuracy as a classifier.
2) Under the Adaboost framework, various regression classification models can be used to construct weak learners, making it very flexible.
3) As a simple binary classifier, it is easy to construct and the results are understandable.
4) Not prone to overfitting.
Main Disadvantages of Adaboost:
1) Sensitive to outliers; anomalous samples may receive high weights during iterations, affecting the prediction accuracy of the final strong learner.
8. GBDT
Main Advantages of GBDT:
1) Can flexibly handle various types of data, including continuous and discrete values.
2) Achieves high prediction accuracy with relatively little tuning time, especially compared to SVM.
3) Uses robust loss functions, providing strong robustness to outliers, such as the Huber loss function and Quantile loss function.
Main Disadvantages of GBDT:
1) Due to dependencies between weak learners, parallel training of data is challenging. However, self-sampling SGBT can achieve partial parallelization.
9. XGBoost Algorithm
1. Advantages of XGBoost Compared to GBDT:
Incorporates the complexity of the tree model into the regularization term to avoid overfitting, thus achieving better generalization performance than GBDT.
The loss function is expanded using Taylor series, utilizing both first and second derivatives to accelerate optimization speed.
GBDT only supports CART as the base learner, while XGBoost also supports linear classifiers as base learners.
Introduces feature subsampling, like random forests, which helps avoid overfitting and reduces computation.
When searching for the optimal split point, XGBoost implements an approximate greedy algorithm to improve efficiency and reduce memory usage, while also considering the handling of missing values in sparse datasets.
XGBoost supports parallel processing. The parallelism in XGBoost is not model generation but rather in features. Features are sorted and stored in memory in blocks, which can be reused in subsequent iterations. This block structure enables parallelization, allowing for multi-threaded calculations of the gain for each feature when splitting nodes, ultimately selecting the feature with the maximum gain for splitting.
2. Disadvantages Compared to LightGBM:
XGBoost uses pre-sorting; before iterations, it pre-sorts the features of nodes to find the optimal split points. When data volume is large, the greedy method is time-consuming. LightGBM uses a histogram algorithm, occupying less memory and reducing the complexity of data splitting.
XGBoost generates decision trees level-wise, splitting leaves at the same level, allowing for multi-threaded optimization, which reduces the risk of overfitting. However, many leaf node splits yield low gains, making further splits unnecessary, leading to unnecessary overhead. LightGBM adopts a depth-optimized, leaf-wise growth strategy, selecting the node with the highest gain from the current leaf for splitting in each iteration, which can result in deeper decision trees and overfitting, thus introducing a threshold to prevent overfitting.
10. Artificial Neural Networks
Advantages:
1. High classification accuracy, strong parallel distributed processing capability, and robust storage and learning abilities;
2. Strong robustness and fault tolerance to noisy neurons, capable of approximating complex nonlinear relationships, and possessing associative memory functions.
Disadvantages:
1. Neural networks require a large number of parameters, such as network topology, weights, and initial threshold values;
2. Learning processes are not observable, making output results difficult to interpret, which can affect the credibility and acceptability of the results;
3. Learning time is prolonged, and may not achieve the learning objectives.
