Chapter Details

# Ensemble Approaches: Bagging

Bagging (Bootstrap Aggregating): Combine strong classifiers to reduce variance.

In the Bias vs. Variance chapter, we could see that by balancing bias and variance, we could reduce error rates of classifier. In this chapter we look at one approach for strong classifiers.

The figure above shown in the previous chapter illustrates that strong classifiers tend to overfit the data resulting in high variance: large difference between classifiers.

One approach for reducing variance is to build multiple classifiers and combine their results by taking the majority voting for each input data. An analogy of majority voting is obtaining second opinions for medial diagnosis and taking the majority voting to reduce misdiagnosis.

Figure above illustrates this analogy. Suppose we have asked six doctors whether a patient has cancer. Four doctors said yes, and two doctors said no. By taking the majority voting, we would then conclude that the patient has cancer.

Why this would reduce variance? Since we are now combining multiple classifiers (i.e., taking unweighted average of predictions), we remove differences between classifiers. In theory, this would means that variance will be reduced to 0 and bias would remain unchanged. However, against real datasets, bias tends to be slightly increased.

Â

Figure above shows the bagging procedure, where we build multiple classifiers by sampling the data with replacement sampling (bootstrapping) and aggregating the predictions by taking the majority vote. Here, D is the training dataset and the black circle in the middle is a test case. D1, D2, and Dn are training data sets created from D using bootstrapping.

Â

Â