Bagging is also known as bootstrap aggregating. This video explains it well: https://youtu.be/2Mg8QD0F1dQ?si=X8kwhQk7Q3cN-OGS
https://youtu.be/GM3CDQfQ4sw?si=uUFry1AWUbX6AlJc
https://youtu.be/MIPkK5ZAsms?si=AroeA6nbDYYVskNt
- Training Process: Bagging trains models in parallel, while boosting trains models sequentially.
- Purpose: Bagging aims to reduce variance, and boosting aims to reduce bias and variance.
- Data Handling: Bagging uses bootstrapped replicas of the data, and boosting reweights the original data based on the previous models’ errors.
- Model Combination: Bagging often uses simple averaging or majority voting, whereas boosting uses a weighted average.
What are the main differences between adaboost and gradient boosting?
AdaBoost (Adaptive Boosting) and Gradient Boosting are both boosting techniques that build a series of weak learners (typically decision trees) and combine them into a strong learner. However, they differ in how they create and combine the weak learners:
AdaBoost:
-
Weighted Instances: AdaBoost starts by training a weak learner on the initial dataset and then increases the weights of misclassified instances. Each subsequent weak learner is then forced to concentrate more on the examples that previous learners misclassified.
-
Error-Based Learning: The weight of each weak learner in the final model is determined by its accuracy, with more accurate learners given more weight. Misclassification error directly influences the weights.
-
Learning Rate: AdaBoost does not typically use a learning rate (also called a shrinkage parameter). However, it adjusts the weights of the classifiers and the training samples after each iteration.
-
Sensitivity to Noisy Data and Outliers: AdaBoost can be sensitive to noisy data and outliers because it tries to correct all misclassifications with equal zeal, which can lead to overemphasis on outliers.
-
Weak Learners: AdaBoost usually works with weak learners that are just slightly better than random guessing. The most common choice is decision trees with a single split, known as decision stumps.
Gradient Boosting:
-
Residual Correction: Gradient Boosting builds each tree to predict the residuals or errors of the previous trees combined and thus works by fitting new models to provide a more accurate estimate of the response variable.
-
Gradient Descent: It uses a gradient descent algorithm to minimize the loss function. The gradients of the loss function are used to indicate the direction in which the next models should be built to minimize the error.
-
Learning Rate: Gradient Boosting often includes a learning rate to shrink the contribution of each tree to prevent overfitting. It is a key parameter that can be tuned for better performance.
-
Handling of Noisy Data and Outliers: Gradient Boosting is generally more robust to noise and outliers because it optimizes a loss function directly, rather than focusing on correcting misclassifications.
-
Flexibility with Weak Learners: Gradient Boosting can be used with more complex trees, and the depth of the trees can be a hyperparameter to tune. It’s not limited to using decision stumps.
Key Differences Summarized:
- Approach to Training: AdaBoost alters the weights of misclassified points and of the weak learners, while Gradient Boosting builds each tree to improve on the remaining errors (residuals).
- Weighting Strategy: AdaBoost determines weights of learners based on their error rates, while Gradient Boosting uses gradients to minimize a loss function.
- Complexity of Learners: AdaBoost typically uses very simple models, while Gradient Boosting can use more complex models.
- Adjustment for Overfitting: Gradient Boosting uses a learning rate to control for overfitting, whereas AdaBoost’s primary mechanism is through adjusting the weights of instances and models.
Each algorithm has its own strengths and weaknesses, and the choice between them would depend on the specific nature of the problem and the data. Gradient Boosting, due to its flexibility and robustness, is often preferred for many problems. However, AdaBoost can be faster and simpler, which can be advantageous in certain scenarios.