Ensemble Learning

Ensemble Learning uses multiple Machine Learning Models to obtain better predictive performance than could be obtained from any single Model.

The process of Ensemble Learning in mathematical terms looks like this:

$\\E\left ( S,D,M,R,A \right ) = S\left ( \sum_{1}^{n}I\left ( D,M_{n},R,A \right ) \right ) \\\\\textup{where:} \\\\E\,\,\,\,\,\textup{ensemble learning process} \\S\,\,\,\,\,\,\textup{strategy for combining individual modeling processes} \\I\,\,\,\,\,\,\,\textup{individual modeling process} \\D\,\,\,\,\,\textup{data} \\M\,\,\,\,\textup{models} \\R\,\,\,\,\,\,\textup{results} \\A \,\,\,\,\,\,\textup{analysis} \\n \,\,\,\,\,\,\,\textup{number of models}$

Ensemble Learning Process

The ensemble learning process is an iterative process of using individual models to generate predictive results that contribute to the overall Ensemble result.

Strategy

Strategies for combining individual modeling processes include:

Bagging (Bootstrap Aggregating) - models are trained with aggregates of data from a parent set of data
Boosting - incrementally building an ensemble by training each new model instance to emphasize the training instances that previous models mis-classified
Bayesian Model Averaging - seeks to approximate the Bayes optimal classifier by sampling hypotheses from the hypothesis space, and combining them using the Bayes' theorem
Bucketing - a selection algorithm is used to choose the best model
Stacking - training a model to combine the predictions of several other models

Individual Modeling Process

The individual Modeling Processes use data to train a models which produces results for analysis.

Data

Includes data for:

model training
model testing
model performance tracking

Models

Any Machine Learning model can be used for Ensemble Learning. Model types typically used include:

Results

Results include:

model training accuracies
prediction confidence levels and accuracies

Analysis

Includes analysis of:

data
individual model performance
ensemble results