Naive Bayes

Naïve Bayes classifiers are based on applying Bayes' theorem which describes the probability of an event based on prior knowledge of conditions related to the event.

The classifier is called ‘naïve’ because it assumes that no factors other than the known prior conditions effect the classifier predictions, which is not a realistic assumption.

These functions are named after Thomas Bayes (1701-1761).

Naive Bayes algorithms can be used for Cluster Analysis to perform Classification:

Naive Bayes algorithms can also be used in conjunction with Decision Trees for improved accuracy:

Mathematical Model

Key aspects include::

processing is non-iterative, so the algorithm is fast
the algorithm isn’t a ‘black box’; we can know how it operates internally
the algorithm can be used to determine the impact of individual features on results
the algorithm depends on using known prior probabilities
the algorithm depends on assuming the independence of predictor features

The mathematical notation is as follows:

$\\ X = \left ( x_{1},x_{2}...,x_{n} \right ) \\ \\ X_{-i}=\left (x_{1},...,x_{i-1},x_{i+1},...,x_{n} \right )=X\textup{\,without the i-th value} \\ \\ P\left ( y \right )=prior \,measured\,class\,probability \\ \\ P\left ( y\,|X \right ) = \frac{P\left ( y \right )P\left ( X|y \right )}{P\left ( X \right )} \\ \\ P\left ( x_{i}\,|\,y \right )=P\left ( x_{i}|\,y,X_{-i} \right ) = prior\,measured\,P\left ( x_{i} \right )\,given\,y \\ \\ P\left ( y\,|X \right )=\frac{P\left ( y \right )\prod _{i=1}^{n}P\left ( x_{i}|\,y\right )}{P\left ( X \right )} \\ \\ because \,\,P\left ( X \right )\,is\,constant\,given\,the\,input: \\ P\left ( y\,|X \right )\propto P\left ( y \right )\prod _{i=1}^{n}P\left ( x_{i}|\,y\right ) \\ \\ thus: \\ \widehat{y}=\underset{y}{arg\,max}\,P\left ( y \right )\prod _{i=1}^{n}P\left ( x_{i}|\,y\right )$

$\\ \\ \\ \textup{where:} \\ \\ X\,\,\,\textup{vector of dependent features with conditional feature pair independence} \\ y \,\,\,\,\,\,\textup{class prediction} \\ P \,\,\,\,\textup{probability function } \\ n \,\,\,\,\,\textup{number of features} \\ \widehat{y}\,\,\,\,\,\textup{estimated class}$

Python Example

To download the code below, click here.

"""
naive_bayes_multinomial_with_scikit_learn.py
performs classification with discrete features
"""

# Import needed packages.
import numpy as np
from sklearn.naive_bayes import MultinomialNB

# Set parameters.
random_number_seed = 5
maximum_feature_value = 6
number_of_training_feature_records = 6
number_of_prediction_feature_records = 1
number_of_features = 100

# Create a random number generator.
generator = np.random.RandomState(random_number_seed)

# Create model training feature data of 6 groups of 100 numeric features each.
X = generator.randint(maximum_feature_value,
                      size=(number_of_training_feature_records,
                            number_of_features))

# Print the training data.
print("X Feature Training Data:")
print(X)

# Define the classes for training.
y = np.arange(1, number_of_training_feature_records + 1)

# Print the training classes.
print("y Training Classes:")
print(y)

# Instantiate a Naive Bayes model.
model = MultinomialNB()

# Train the model.
model.fit(X, y)

# Create a prediction input.
prediction_input = generator.randint(maximum_feature_value,
                                     size=(number_of_prediction_feature_records,
                                           number_of_features))

# Print the prediction input.
print("Prediction Input:")
print(prediction_input)

# Make a prediction
prediction = model.predict(prediction_input)

# Print the prediction
print("Predicted Class:")
print(prediction)


Outputs are below:

X Feature Training Data:
[[3 5 0 1 0 4 3 0 0 4 1 5 0 3 4 5 3 1 4 5 2 1 1 2 1 1 1 2 0 5 2 0 0 4 4 1
  3 3 2 4 1 3 3 2 1 5 4 4 5 3 3 3 4 1 3 3 3 5 1 1 5 0 2 1 0 5 2 5 3 0 5 3
  0 0 4 4 5 2 0 3 0 0 0 2 4 5 3 5 1 4 5 2 4 3 5 0 0 1 4 3]
 [4 1 0 0 2 5 4 3 2 4 1 2 3 4 3 4 3 1 4 2 3 4 1 4 0 2 4 1 2 2 1 3 0 0 0 3
  1 4 4 3 0 2 4 0 0 5 3 3 3 4 0 2 2 1 3 1 5 1 2 3 0 0 5 1 1 1 0 0 1 4 1 3
  4 2 1 5 4 4 2 2 5 1 2 3 5 1 2 4 1 0 1 2 3 0 2 5 2 5 4 3]
 [2 1 5 1 1 5 1 1 0 4 0 5 0 5 5 2 1 3 4 3 3 0 3 3 3 2 5 2 0 3 4 5 1 3 5 3
  3 5 1 1 2 4 2 5 2 4 0 0 1 4 5 3 1 0 3 2 1 0 3 5 4 4 2 1 1 1 3 0 2 4 4 5
  1 3 1 3 5 4 3 3 5 1 0 0 2 5 2 2 0 1 1 0 3 2 2 2 1 5 1 0]
 [2 2 4 4 1 5 4 1 1 5 0 4 1 5 1 3 0 0 5 0 3 2 3 2 1 0 3 0 2 3 4 4 2 5 0 5
  4 5 3 2 1 5 1 0 3 4 4 4 0 0 3 0 2 5 5 2 4 5 5 5 2 2 5 4 5 1 2 5 0 2 3 5
  5 5 2 4 4 3 1 2 0 0 2 1 0 1 4 4 5 5 3 1 1 5 5 4 1 1 2 0]
 [3 5 5 2 4 3 3 3 3 1 1 5 0 1 5 5 2 4 2 2 2 4 4 5 3 1 1 5 0 1 2 1 2 0 1 1
  5 0 1 4 0 1 3 4 5 4 1 0 4 5 1 2 5 5 2 2 3 5 2 2 3 5 3 2 1 5 5 5 0 3 3 1
  5 5 2 0 5 1 5 0 0 3 5 4 4 0 3 1 1 1 1 2 0 4 5 1 3 5 5 2]
 [2 1 4 2 2 2 2 2 5 3 0 4 2 5 3 0 4 4 0 1 0 0 2 4 1 4 1 4 2 5 3 3 2 3 2 0
  3 4 2 0 0 0 1 3 3 2 5 0 3 4 2 3 1 1 0 4 1 5 5 1 4 3 1 2 4 0 3 5 3 1 0 2
  4 1 1 2 2 1 3 4 1 1 1 3 0 4 2 5 1 2 3 2 1 2 5 0 1 0 3 1]]
y Training Classes:
[1 2 3 4 5 6]
Prediction Input:
[[0 2 1 5 4 1 0 5 5 1 4 5 1 3 2 4 0 2 4 0 3 2 4 5 5 1 1 5 4 5 5 0 4 4 1 5
  3 3 2 5 0 4 1 1 1 5 2 1 2 4 0 4 0 2 1 2 3 2 4 0 4 3 4 2 5 1 2 0 0 2 3 5
  0 3 1 4 1 3 0 4 5 0 1 1 4 5 2 0 3 5 3 2 4 3 4 4 5 4 1 4]]
Predicted Class:
[3]

Naive Bayes

Mathematical Model

Python Example

References