Random Forest Models

Random Forest Models have these key characteristics:

they are an ensemble learning method
they can be used for classification and regression
they operate by constructing multiple decision trees at training time
they correct for overfitting to their training set.

In mathematical terms, it looks like this:

$\\F=S\left ( \left [ T_{1}\left ( d_{1} \right ),T_{2}\left ( d_{2} \right ),..., T_{n}\left ( d_{n} \right )\right ] \right ) \\\\\mathrm{\textup{where:}} \\\\F\,\,\,\,\,\,\mathrm{\textup{final class}} \\S\,\,\,\,\,\,\,\mathrm{\textup{selection function for determining the final class}} \\T_{n}\,\,\,\,\mathrm{\textup{decision tree processing function}} \\d_{n}\,\,\,\,\mathrm{\textup{data for decision tree training}} \\n\,\,\,\,\,\,\,\mathrm{\textup{number of decision tree}}$

Model Training Process

The model training process is roughly as follows:

Randomly select features for n decision trees.
For each decision tree:
- For each decision tree node:
  - Split the node using best split criteria such as Gini impurity, residual sum of squares.

Python Example

To download the code below click here.

"""
random_forest.py
instantiates, trains, and tests a random forest model
"""

# Import needed libraryies.
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Define parameters.
number_of_samples = 1000
number_of_features = 4
number_of_informative_features = 2
number_of_redundant_features = 0
number_of_classes = 2
number_of_clusters_per_class = 2
random_state = 0
shuffle = False
maximum_tree_depth = 3
prediction_test_inputs = [[1, 3, 7, 2]]

# Generate model training inputs.
features, classes = make_classification(n_samples=number_of_samples,
                                        n_features=number_of_features,
                                        n_informative=number_of_informative_features,
                                        n_redundant=number_of_redundant_features,
                                        n_classes=number_of_classes,
                                        n_clusters_per_class=number_of_clusters_per_class,
                                        random_state=random_state,
                                        shuffle=shuffle)

# Print the features and classes generated.
print("features:")
print(features)
print("classes:")
print(classes)

# Instantiate a random forest model.
model = RandomForestClassifier(max_depth=maximum_tree_depth,
                               random_state=random_state)

# Train the model.
model.fit(features, classes)

# Test a prediction.
prediction = model.predict(prediction_test_inputs)

# Print the result.
print("prediction:")
print(prediction)

The results are below:

features:
[[-1.66853167 -1.29901346  0.2746472  -0.60362044]
 [-2.9728827  -1.08878294  0.70885958  0.42281857]
 [-0.59614125 -1.37007001 -3.11685659  0.64445203]
 ...
 [ 0.91711204  1.10596645  0.86766522 -2.25625012]
 [ 0.10027664  1.45875846 -0.44360274 -0.67002328]
 [ 1.0415229  -0.01987143  0.15216419 -1.9405334 ]]
classes:
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1]
prediction:
[1]

Random Forest Models

Model Training Process

Python Example

References