About Ensemble Techniques in ML.

Have you conceptually understood ML ensemble techniques?

6 min readFeb 20, 2022

This post talks about different ensemble techniques that an ML enthusiast should know to improve the results of their models. This is a read-along post focused only on conceptual understanding (saving the codes for later).

If you have an interview or a presentation coming up, or if you are here to understand ensemble techniques within a few minutes, then this post is ideal for you. So here goes…

The “Ensemble” Buzz

In the past 2 decades, “Ensemble” has been a buzzword, not because of its semantic meaning — “a group of musicians”, but because of Data/ML scientists. This has been a go-to solution for most scientists, as they reap the benefits of these techniques.

What are ensemble techniques?

As a data scientist, you have a few techniques handy which can be used to improve the performance of your models. Namely bagging, boosting, stacking, and cascading. Using “combined” results from multiple baseline models, we can achieve higher “generalized” performance with the final model.

How do ensemble techniques improve the model performance?

Here’s a good analogy that forms the answer:

Consider yourself working for an investment firm and that you had to make a decision on whether or not to invest in a particular company A. Here are few things that you can do:

Go to the best stock market advisor in your firm, and get an opinion on whether or not you should invest in company A. Suppose he says

I worked the fundamental analysis for this company. Chances that this company will be a success is 80%.

Because 80% confidence isn’t enough for a huge investment, here’s what you could do next:

Go to another stock market advisor in the firm, get her opinion on the investment. Suppose she says,

I performed technical analysis on the company, chances that this company will be a success is 85%.

Does this increase your confidence?

Yes! Because we got two of the best advisors to say that the company is worthwhile investing with a chance of at least 80%.

Now, if you have the time and resources to get a third/fourth/fifth… opinion, you will definitely choose to pursue it.

Moving this analogy to the ML domain, your advisors are trained baseline models, time, and resources with the time and computational power for training models. And you are the final model which makes the decision to whether or not to invest.

What does the “improve” performance of a model mean?

Broadly speaking, it increases the accuracy of the model, reduces the bias and variance of the model.

Before moving on to the techniques, note the below points:

Ensemble techniques involve multiple machine learning models
Data fed into each of these models can be the same/different, which leads us to the two sampling methods for data.

Random Sampling / Mixing Training Data

The conventional way is to use the entire training data for training/validation on a single model.

The ensemble way of sampling data can be of the following types:

Sampling with replacement

Allows random sampling of a subset from the entire train data for each model independently.
Preferred for large datasets, in which case the resulting covariance of data between models will be less.

2. Sampling without replacement

Allows data to be evenly distributed across the baseline models, such that no data is common for any of the models.
Preferred for smaller datasets, in which case the resulting covariance of data between models is 0.

3. Bernoulli sampling:

Each data point is subjected to Bernoulli trials, which decide whether or not the data point must be a part of the sample. Not particularly useful if random sampling is used.

Brief the techniques already!

Here’s an overview diagram of all the ensemble techniques:

Bagging (Bootstrap Aggregation)

Here’s a point about bootstrap sampling step:

Random sample with replacement for a subset of data used as input for the models.

Here are two ways of doing the aggregation step:

Hard voting: Outputs the mode of all predicted labels. Useful for classification tasks.
Aggregation: Outputs the aggregate of results from all models. Useful for regression. Additionally, we can also perform a weighted average for the results from different models.

Here are few points to note about bagging:

Bagging can reduce variance without affecting the bias of a model. This is because of the bootstrap sampling and aggregation steps in the technique.
Choosing low bias and high variance models as learners are preferred in bagging. Given the previous point, the bias remains low, and variance is reduced after bagging.
Low bias models include Decision trees, KNN and SVM.

Boosting

Boosting involves creating a strong classifier with a number of weak classifiers. Primarily used for reducing bias.

Get Srikanth Kadaba’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The models are trained in a sequential manner, hard to parallelize training, which means the training time complexity is high. The common boosting techniques are briefed below:

Adaptive Boosting (AdaBoost)
To increase the performance of a model, we increase the weight of the misclassified/wrongly predicted data. This weighted data is fed as input to the next learner in the next iteration.

Because the incorrectly classified data is now weighted more, the model will now give more importance to these data points, and the probability that its prediction is right will increase.

Finally, a voting mechanism is used to decide the output from all the models combined (weak and strong learners).

Gradient Boosting
In this case, we train the consecutive learners based on the residual errors of the previous learners. This is achieved by setting the residual error as the target label for training new models.

In the case of classification, the residuals are calculated based on the probabilities of the classes. Probabilities are calculated based on odds. (Reference)

Xtreme Gradient Boosting (XGB)

XGBoost is a state-of-the-art algorithm in residual gradient boosting. It has a few advantages over the traditional boosting techniques:

Parallel Processing: XGBoost supports GPU and Spark compatibility.
Train upon existing models: XGBoost allows saving the results of training and building on those results at any other instance.
Introduces new regularization parameters that prevent overfitting and reduce bias.

Introduces new hyperparameters such as:
1. Dynamically determining the adequate depth and complexity of the decision tree.
2. Allows row sampling + column sampling.
3. Randomization parameters
4. Newton’s tree boosting for optimized learning of tree structures.

Stacking

Allows training of multiple models together to get predictions.
Results of the first models are aggregated and used as training data for the meta-learner.

The outputs of each baseline model are used as input for the meta classifier and trained with the actual output.
Using different baseline models will result in better performance of the stacking classifier.
Popular in a competitive ML setting (ex: Kaggle)

Cascading

This ensemble technique is extensively used in critical applications such as fraud detection in credit card transactions and medical diagnosis (ex: cancer detection).

The above diagram is one representation of how ML models can be cascaded.

Note that the data in each model can be different.
Model complexity is increased as the number of models increases.
Each model can be trained to target/classify a particular feature (see example below)

Consider another possibility for cascading by creating a network of models to target features and perform classifications: