Source: Turing Artificial Intelligence, Aotu Data
This article is about 3600 words long and suggests a reading time of 7 minutes.
This article introduces the 10 most common machine learning algorithms in an illustrated manner.
In the field of machine learning, there is a saying that “there is no free lunch in the world”, which simply means that no single algorithm can perform best on every problem. This theory is particularly important in supervised learning.
For example, you cannot say that neural networks are always better than decision trees, and vice versa. Model performance is influenced by many factors, such as the size and structure of the dataset. Therefore, you should try many different algorithms based on your problem, while using a data test set to evaluate performance and select the best one.
Of course, the algorithms you try must be relevant to your problem, and understanding this is the main task of machine learning. For instance, if you want to clean a house, you might use a vacuum cleaner, broom, or mop, but you certainly wouldn’t start digging a hole with a shovel.
For those eager to understand the basics of machine learning, here is a list of the top ten machine learning algorithms used by data scientists, introducing the characteristics of these ten algorithms in a way that is easy for everyone to understand and apply.
1. Linear Regression
Linear regression is possibly one of the most well-known and easily understood algorithms in statistics and machine learning.
Since predictive modeling primarily focuses on minimizing model error, or making the most accurate predictions at the expense of interpretability, we borrow, reuse, and steal algorithms from many different fields, involving some statistical knowledge.
Linear regression is represented by an equation that describes the linear relationship between input variables (x) and output variables (y) by finding specific weights (B) for the input variables.
Example: y = B0 + B1 * x
Given input x, we will predict y, and the goal of the linear regression learning algorithm is to find the values of coefficients B0 and B1.
Different techniques can be used to learn the linear regression model from data, such as linear algebra solutions for ordinary least squares and gradient descent optimization.
Linear regression has been around for over 200 years and has been extensively studied. If possible, some rules of thumb when using this technique are to remove very similar (correlated) variables and to eliminate noise from the data. It is a quick and simple technique and a good first algorithm.
2. Logistic Regression
Logistic regression is another technique borrowed from the statistical field into machine learning. It is a dedicated method for binary classification problems (problems with two class values). Although it has the word regression in its name, it actually deals with classification problems.
Logistic regression is similar to linear regression, as both aim to find the weight values for each input variable. Unlike linear regression, the predicted output values are transformed using a nonlinear function known as the logistic function.
The logistic function looks like a large S and can convert any value into the range of 0 to 1. This is useful because we can apply corresponding rules to the output of the logistic function to classify values as 0 and 1 (for example, if IF less than 0.5, then output 1) and predict class values.
Due to the unique learning method of the model, predictions made through logistic regression can also be used to compute the probability of belonging to class 0 or class 1. This is very useful for problems that require providing many basic principles.
Like linear regression, logistic regression performs better when you remove attributes that are unrelated to the output variable and very similar (correlated) attributes. It is a quick and effective model for handling binary classification problems.
3. Linear Discriminant Analysis
Traditional logistic regression is limited to binary classification problems. If you have more than two classes, then Linear Discriminant Analysis (LDA) is the preferred linear classification technique.
LDA is represented very simply. It consists of the statistical properties of your data calculated for each class. For a single input variable, this includes:
-
Mean for each class.
-
Variance calculated across all classes.
LDA predicts by calculating the discriminative value for each class and predicting the class with the maximum value. This technique assumes that the data has a Gaussian distribution (bell curve), so it is best to manually remove outliers from the data first. It is a simple yet powerful method for classification prediction modeling problems.
4. Decision Tree
Decision trees are an important algorithm in machine learning.
The decision tree model can be represented as a binary tree. Yes, it is a binary tree from algorithms and data structures, nothing special. Each node represents a single input variable (x) and the left and right children on that variable (assuming the variable is numeric).
The leaf nodes of the tree contain the output variable (y) used for making predictions. Predictions are made by traversing the tree, stopping when a leaf node is reached, and outputting the class value of that leaf node.
Decision trees learn quickly and predict quickly. They often predict accurately for many problems, and you do not need to make any special preparations for the data.
5. Naive Bayes
Naive Bayes is a simple yet extremely powerful predictive modeling algorithm.
The model consists of two types of probabilities that can be calculated directly from your training data: 1) the probability of each class; 2) the conditional probability of each x value given a class. Once calculated, the probability model can be used to predict new data using Bayes’ theorem. When your data is numerical, it is usually assumed to follow a Gaussian distribution (bell curve) for easy estimation of these probabilities.
Naive Bayes is called naive because it assumes that each input variable is independent. This is a strong assumption and is unrealistic for real data, but the technique is still very effective for a wide range of complex problems.
6. K-Nearest Neighbors
The KNN algorithm is very simple and very effective. The KNN model represents the entire training dataset.
It predicts new data points by searching for K most similar instances (neighbors) within the entire training set and summarizing the output variable of these K instances. For regression problems, the new point may be the average output variable, while for classification problems, the new point may be the mode class value.
The key to success lies in how to determine the similarity between data instances. If your attributes are all on the same scale, the simplest method is to use Euclidean distance, which can be calculated directly based on the distance between each input variable.
KNN may require a lot of memory or space to store all the data, but calculations (or learning) are only performed when predictions are needed. You can also update and manage your training set at any time to maintain prediction accuracy.
The concept of distance or closeness may break down in high-dimensional environments (with many input variables), negatively affecting the algorithm. Such events are referred to as the curse of dimensionality. This also implies that you should only use those input variables that are most relevant to the predicted output variable.
7. Learning Vector Quantization
The disadvantage of K-Nearest Neighbors is that you need to maintain the entire training dataset. Learning Vector Quantization (or LVQ) is an artificial neural network algorithm that allows you to suspend any number of training instances and learn them accurately.
LVQ is represented by a set of codebook vectors. Initially, vectors are randomly selected, and then iterated multiple times to adapt to the training dataset.
After learning, the codebook vectors can be used to predict in the same way as K-Nearest Neighbors. By calculating the distance between each codebook vector and new data instances, the most similar neighbors (best matches) can be found, and the class value of the best matching unit or the actual value in the case of regression is returned as the prediction. You can achieve the best results if you restrict the data to the same range (e.g., between 0 and 1).
If you find that KNN gives good results on your dataset, try using LVQ to reduce the memory requirements of storing the entire training dataset.
8. Support Vector Machine
Support Vector Machine is perhaps one of the most popular and discussed machine learning algorithms.
A hyperplane is a line that separates the input variable space. In SVM, a hyperplane is selected to separate the points in the input variable space by their class (class 0 or class 1). In two-dimensional space, it can be viewed as a line, where all input points can be completely separated by this line. The SVM learning algorithm aims to find the coefficient values that allow the hyperplane to have the best separation of classes.
The distance between the hyperplane and the nearest data points is called the margin, and the hyperplane with the maximum margin is the best choice. At the same time, only those data points that are close to the hyperplane are related to the definition of the hyperplane and the construction of the classifier; these points are called support vectors, which support or define the hyperplane. In practice, we use optimization algorithms to find the coefficient values that maximize the margin.
SVM may be one of the most powerful out-of-the-box classifiers, and it is worth trying on your dataset.
9. Random Forest
Random Forest is one of the most popular and powerful machine learning algorithms. It is an ensemble machine learning algorithm known as Bootstrap Aggregation or Bagging.
Bootstrap is a powerful statistical method used to estimate a quantity, such as a mean, from data samples. It draws a large number of sample data, calculates the mean, and averages all the means to more accurately estimate the true mean.
In bagging, the same method is used, but the most commonly used model is the decision tree, rather than estimating the entire statistical model.
It trains data through multiple sampling, then builds a model for each data sample. When you need to predict new data, each model makes predictions, and the prediction results are averaged to better estimate the true output value.
Random Forest is an adjustment of decision trees; instead of selecting the best split point, Random Forest achieves suboptimal splits by introducing randomness.
Therefore, the differences between the models created for each data sample will be greater, but still accurate in their own right. Combining the prediction results can better estimate the correct potential output values.
If you obtain good results with high variance algorithms (like decision trees), then adding this algorithm will improve the results.
10. Boosting and AdaBoost
Boosting is an ensemble technique that creates a strong classifier from several weak classifiers. It first builds a model from the training data, then creates a second model to try to correct the errors of the first model. Models are added continuously until the training set is perfectly predicted or the maximum number has been added.
AdaBoost is the first truly successful Boosting algorithm developed for binary classification and is also the best starting point for understanding Boosting.Currently, the most famous algorithms based on AdaBoost are random gradient boosting.
AdaBoost is often used with short decision trees. After creating the first tree, the performance of each training instance on the tree determines how much attention the next tree needs to give to this training instance. Hard-to-predict training data is given more weight, while easy-to-predict instances are given less weight.
Models are created sequentially, and the update of each model affects the learning effect of the next tree in the sequence. After all trees are built, the algorithm predicts new data and weights the performance of each tree based on the accuracy of the training data.
Because the algorithm pays great attention to error correction, having clean data without outliers is very important.
A typical question raised by beginners when faced with various machine learning algorithms is, “Which algorithm should I use?” The answer to this question depends on many factors, including:
-
Size, quality, and nature of the data
-
Available computation time
-
Urgency of the task
-
What you want to do with the data
Even an experienced data scientist cannot know which algorithm will perform best without trying different algorithms. While there are many other machine learning algorithms, these algorithms are the most popular. If you are a beginner in machine learning, this is a great starting point for learning.
