# Full “Laplacianised” posterior naive Bayesian algorithm

- Hamse Y Mussa
^{1}Email author, - John BO Mitchell
^{2}and - Robert C Glen
^{1}

**5**:37

**DOI: **10.1186/1758-2946-5-37

© Mussa et al.; licensee Chemistry Central Ltd. 2013

**Received: **24 May 2013

**Accepted: **12 August 2013

**Published: **23 August 2013

## Abstract

### Background

In the last decade the standard Naive Bayes (SNB) algorithm has been widely employed in multi–class classification problems in cheminformatics. This popularity is mainly due to the fact that the algorithm is simple to implement and in many cases yields respectable classification results. Using clever heuristic arguments “anchored” by insightful cheminformatics knowledge, Xia *et al.* have simplified the SNB algorithm further and termed it the Laplacian Corrected Modified Naive Bayes (LCMNB) approach, which has been widely used in cheminformatics since its publication.

In this note we mathematically illustrate the conditions under which Xia *et al.*’s simplification holds. It is our hope that this clarification could help Naive Bayes practitioners in deciding when it is appropriate to employ the LCMNB algorithm to classify large chemical datasets.

### Results

A general formulation that subsumes the simplified Naive Bayes version is presented. Unlike the widely used NB method, the Standard Naive Bayes description presented in this work is discriminative (not generative) in nature, which may lead to possible further applications of the SNB method.

### Conclusions

Starting from a standard Naive Bayes (SNB) algorithm, we have derived mathematically the relationship between Xia *et al.*’s ingenious, but heuristic algorithm, and the SNB approach. We have also demonstrated the conditions under which Xia *et al.*’s crucial assumptions hold. We therefore hope that the new insight and recommendations provided can be found useful by the cheminformatics community.

### Keywords

Naive Bayes Laplacian Corrected Modified Naive Bayes Classifications Cheminformatics## Background

Broadly speaking there are two conceptually different ways to solve statistical problems: the frequentist and the Bayesian approaches. On the pros and cons of each method there are numerous excellent review articles and text books, such as the recent book by Murphy [1]. Unlike the frequentist approach, in the Bayesian approach any a priori knowledge about the probability distribution function that one assumes might have generated the given data (in the first place) can be taken into account when estimating this distribution function from the data at hand. If the data are noise–free and “complete”, the role of the a priori information in estimating the distribution function diminishes drastically. However, the a priori information can be crucial when the data are noisy and sparse. The latter scenario is typical in realistic large chemical datasets, which, arguably, makes Bayesian based statistics a powerful data analysis tool.

Unfortunately, Bayesian statistics in its fullest form is not computationally feasible in realistic cheminformatics data analyses. However, in recent years, a simplified version of the Bayesian approach, which is commonly known as the “Naive” Bayesian algorithm, has been found to be a useful classification tool in multi–class classification problems in cheminformactics. To this end a Naive Bayesian classifier is built on binary descriptor space. The descriptors/features *x*_{
j
}, representing the compounds to be classified, assume binary values 0 or 1, where (*j* = 1,2,...,*L*) and *L* can typically be more than 1,000. Thus for some cheminformatics practitioners even the Naive Bayesian algorithm in its standard form is computationally prohibitive when the dataset is large. In this regard, Xia *et al*. [2] proposed a simpler version of the standard Naive Bayesian algorithm, albeit for binary classification problems; slight variants of this algorithm for multi–class classification can also be found in [3, 4]. According to Rogers *et al.*[5], Rogers being a co–author of the work presented in [2], “the standard Naive Bayes was modified by considering only the effect of the presence of a feature and not its absence”. There are also a few more noticeable aspects of this proposed simplification: (a) the authors cleverly estimate directly – albeit heuristically – the a posteriori class probability for the present feature; (b) these authors (rather ingeniously) incorporate a Laplacian–correction into the estimated posterior class probability; and (c) the authors deem absent features not discriminating enough and therefore discard their contributions to the estimation of the posterior class. More than anything else it is this omission of the absent features from the Standard Naive Bayes (SNB) algorithm that makes Xia *et al.*’s proposed Naive Bayes Algorithm, termed Laplacian Corrected Modified Naive Bayes (LCMNB), (and its variants by different groups) computationally fast.

It is these three points, (a), (b) and (c), that we expound on in a mathematical setting to demonstrate under which conditions they hold – not only in an abstract sense, but also in the practical sense for a NB practitioner to make an informed decision as to when it is appropriate to employ SNB or LCMNB, in the cheminformatics context.

## Methods

### Naive Bayes

where **x** = (*x*_{1},*x*_{2},...,*x*_{
L
}) and *ω*_{
i
} denote the feature vectors and class labels, respectively; *x*_{
j
} and *L* being as described before, whereas *i* is just an index for the class labels. The terms *p*(*ω*_{
i
}|**x**), *p*(**x**|*ω*_{
i
}), *p*(*ω*_{
i
}), and *p*(**x**) refer to the posterior probability for *ω*_{
i
} given **x**, the descriptor vector distribution conditioned on class *ω*_{
i
}, the a priori probability of class *ω*_{
i
} occurring, and the descriptor vector density function, respectively – for more details, see ref. [3, 4, 6].

*p*(

*ω*

_{ i }|

*x*

_{1},

*x*

_{2},...,

*x*

_{ L }) can be rewritten as

*p*(

*x*

_{1},

*x*

_{2},...,

*x*

_{ L }|

*ω*

_{ i }) as

In practice, it is extremely difficult to estimate *p*(*ω*_{
i
}|**x**) or *p*(**x**|*ω*_{
i
}). This reality inevitably forces one to make concessions over the degree of accuracy the estimated *p*(*ω*_{
i
}|**x**) or *p*(**x**|*ω*_{
i
}) can deliver. One widely employed scheme to obtain these probability distributions with compromised accuracy is to assume that individual descriptors *x*_{
j
}, *j* = 1,2,...,*L*, are independent conditional on *ω*_{
i
}. It is this naive assumption of independence among features to which the term “Naive” in “Naive Bayesian” refers.

*p*(

*x*

_{2}|

*ω*

_{ i }) =

*p*(

*x*

_{2}|

*ω*

_{ i },

*x*

_{1}),

*p*(

*x*

_{3}|

*ω*

_{ i }) =

*p*(

*x*

_{3}|

*ω*

_{ i },

*x*

_{1},

*x*

_{2}),...,

*p*(

*x*

_{ L }|

*ω*

_{ i }) =

*p*(

*x*

_{ L }|

*ω*

_{ i },

*x*

_{1},

*x*

_{2}...,

*x*

_{L−1}). Thus, Eq. 6 modifies to

Clearly $\frac{{\mathrm{\Pi}}_{j=1}^{L}p({x}_{j})}{p({x}_{1},{x}_{2},\mathrm{....},{x}_{L})}$ is common to all classes and therefore plays no role in classification. Thus, in practice (in the Naive Bayes context with which this work is concerned) one is required to estimate *p*(*ω*_{
i
}|*x*_{
j
}) and *p*(*ω*_{
i
}).

Since generative approaches can be informative and “simpler” than their discriminative counterparts [9], we make use of Bayes’ theorem again, i.e., $p({\omega}_{i}|{x}_{j})=\frac{p({\omega}_{i})p({x}_{j}|{\omega}_{i})}{p({x}_{j})}$ and then estimate *p*(*ω*_{
i
}|*x*_{
j
}) through $\frac{p({\omega}_{i})p({x}_{j}|{\omega}_{i})}{p({x}_{j})}$, where $p({x}_{j})=\sum _{i=1}^{C}p({x}_{j}|{\omega}_{i})p({\omega}_{i})$ with *C* referring to the number of classes. *p*(*ω*_{
i
}) denotes the a priori class probability, which is relatively easy to estimate. Thus, in our Bayesian context, the estimation of *p*(*ω*_{
j
}|*x*_{
j
}) boils down in practice to estimating *p*(*x*_{
j
}|*ω*_{
i
}).

### Estimation of *p*(*x*_{
j
}|*ω*_{
i
}), with *x*_{
j
} = 1 *and*0

*p*(

*x*

_{ j }|

*ω*

_{ i }) can be estimated using the given data and assuming a

*Beta*distribution as an a priori distribution for

*p*(

*x*

_{ j }|

*ω*

_{ i }) [10]. (There are other possible prior distributions from which one can choose, but we select the

*Beta*distribution for reasons that will transpire later). As described in Appedix

**A**, a

*Beta*a priori distribution

*Beta*(

*α*

_{ i },

*β*

_{ i }) for

*p*(

*x*

_{ j }|

*ω*

_{ i }) results in a

*p*(

*x*

_{ j }|

*ω*

_{ i }) estimator in the form [11]:

where ${N}_{{\omega}_{i}}$ and *N*_{
ij
}, respectively, denote the number compounds in class *ω*_{
i
}, and number of compounds in this class with descriptor *x*_{
j
} assuming value 1. *β*_{
i
} and *α*_{
i
} are *Beta* distribution hyper–parameters per class and the valid range of values that these hyper–parameter can assume are as defined in Appendix **A**. When *α*_{
i
} and *β*_{
i
} equal 1, *α*_{
i
} and *β*_{
i
} + *α*_{
i
} in Eqs. 12–13 can be viewed as a “Laplacian correction”.

## Results and discussion

### Estimation of *p*(*ω*_{
i
}|*x*_{
j
} = 1) and *p*(*ω*_{
i
}|*x*_{
j
} = 0)

#### Estimation of *p*(*ω*_{
i
}|*x*_{
j
} = 1): In Our Approach

##### Remark 1

Assume that we have *N* chemical compounds (and their activity labels) available for training, where ${N}_{{\omega}_{i}}$ of these compounds belong to class *ω*_{
i
}.

##### Remark 2

Assume that the class a priori distribution is taken as $p({\omega}_{i})=\frac{{N}_{{\omega}_{i}}}{N}$, where ${N}_{{\omega}_{i}}>>{\alpha}_{i}+{\beta}_{i}$ (which is a valid assumption as found in any realistic large chemical dataset).

*p*(

*ω*

_{ i }|

*x*

_{ j }= 1) becomes

(recall that $p({x}_{j})=\sum _{i=1}^{C}p({x}_{j}|{\omega}_{i})p({\omega}_{i})$).

where ${N}_{j}^{+}$ is the number of times *x*_{
j
} assumes the value 1.

#### Estimation of *p*(*ω*_{
i
}|*x*_{
j
} = 1): In Xia *et al.*’s Formulation

*et al.*,

*p*(

*ω*

_{ i }|

*x*

_{ j }= 1) is estimated as

*K*is as defined in Xia

*et al.*and in their paper

*A*

_{ i }is given as

Eq. 16 constitutes what Xia *et al.* term “the Laplacian–Corrected Modified Naive Bayes (LCMNB)” estimator for *p*(*ω*_{
i
}|*x*_{
j
} = 1).

If *α*_{
i
} in Eq. 15 is set to *A*_{
i
}, Eq. 15 is exactly equivalent to Xia *et al.*’s estimator for *p*(*ω*_{
i
}|*x*_{
j
} = 1) as can be seen in Eq. 16.

We note in passing that in Xia *et al.*’s case, *C* = 2 and $p({\omega}_{2})=\frac{1}{K}$, which in their nomenclature denoted by *p*(*Active*) – that is, *A*_{2} = 1 while *A*_{1} = *K* − 1.

Initially we employed the *Beta* a priori distribution for the class conditional distribution to ascertain the equivalence of Eqs. 15 and 16. Fortunately, however, we have ended up with the general equations (Eqs. 14 – 15) that not only encapsulate the LCMNB scheme of Xia *et al.*, but also subsume the other various variants of LCMNB, such as those discussed in Nidhi *et al.* and Nigsch *et al.*’s papers [3, 4].

At any rate, let us proceed to the nub of this work: Identifying the conditions under which the LCMNB algorithm holds with respect to the SNB algorithm. But first we need to describe the estimation of *p*(*ω*_{
i
}|*x*_{
j
} = 0).

#### Estimation of *p*(*ω*_{
i
}|*x*_{
j
} = 0): In Our Approach

*x*

_{ j }= 0, we make use of Remark 1, Remark 2 and Eq. 13, which yield an estimator for

*p*(

*ω*

_{ i }|

*x*

_{ j }= 0) as

### Naive Bayes: scoring function from

For notational convenience let us denote $\frac{{N}_{\mathit{\text{ij}}}+{\alpha}_{i}}{{N}_{j}^{+}+\sum _{i=1}^{C}{\alpha}_{i}}$ in Eq. 15 and $\frac{{N}_{{\omega}_{i}}-({N}_{\mathit{\text{ij}}}+{\alpha}_{i})}{N-({N}_{j}^{+}+\sum _{i=1}^{C}{\alpha}_{i})}$ in Eq. 17 by *ξ*_{
ij
} and *ν*_{
ij
}, respectively.

*p*(

*ω*

_{ i }|

*x*

_{ j }= 1) and

*p*(

*ω*

_{ i }|

*x*

_{ j }= 0) may be written more succinctly as $p({\omega}_{i}|{x}_{j})={\xi}_{\mathit{\text{ij}}}^{{x}_{j}}{\nu}_{\mathit{\text{ij}}}^{(1-{x}_{j})}$, which allows us to express Eq. 11 more compactly as

Now we come to the core of this work, under which conditions does the LCMNB algorithm hold with respect to the SNB algorithm? Before we answer this question, we deem it instructive and more insightful to map Eq. 18 monotonically to a discriminant function, a “scoring function” (so to speak).

where ${S}_{{\omega}_{i}}(\mathbf{\text{x}})={D}_{{\omega}_{i}}(\mathbf{\text{x}})+\left[\sum _{j}\phantom{\rule{.3em}{0ex}}\text{ln}\phantom{\rule{.3em}{0ex}}p({x}_{j})-\phantom{\rule{.3em}{0ex}}\text{ln}\phantom{\rule{.3em}{0ex}}p({x}_{1},{x}_{2},\mathrm{...},{x}_{L})\right]$

### Conditions

*et al.*’s approach, the LCMNB algorithm, is none other than ${\sum}_{j}{x}_{j}\phantom{\rule{.3em}{0ex}}\text{ln}\phantom{\rule{.3em}{0ex}}{\xi}_{\mathit{\text{ij}}}-\phantom{\rule{.3em}{0ex}}\text{ln}\phantom{\rule{.3em}{0ex}}p({\omega}_{i})$ in Eq. 21. This means that in Xia

*et al.*’s scheme the contributions from the terms depending on

*x*

_{ j }= 0 for a given class, i.e.,

*et al.*nor in any other paper on the LCMNB approach has it been demonstrated that (i) the contribution of Eq. 22 is zero, i.e.,

Thus unless one (or more) of the above – (i), (ii) and (iii) – is (are) met, the assumption on which the Modified Naive Bayesian algorithm is based is questionable and therefore its practitioners should pay attention to this discrepancy; clearly it is not justifiable to discard from the onset the contribution of $\sum _{j}(1-{x}_{j})\phantom{\rule{.3em}{0ex}}\text{ln}\phantom{\rule{.3em}{0ex}}{\nu}_{\mathit{\text{ij}}}-(L-1)\times \phantom{\rule{.3em}{0ex}}\text{ln}\phantom{\rule{.3em}{0ex}}p({\omega}_{i})$ simply because features x _{
j
} are absent, i.e. *x*_{
j
} = 0.

*p*(

*ω*

_{1}) =

*p*(

*ω*

_{2}) = ... =

*p*(

*ω*

_{ C }). We hasten to add that this option was not included in the LCMNB scheme. At any rate, by simply repeating the arguments in the preceding sections, it is straightforward to show that one ends up with Eq. 21. In this scenario, though,

*L*× ln

*p*(

*ω*

_{ i }) is common to all classes and therefore does not play a role in classifying a new compound, i.e., ${D}_{{\omega}_{i}}(\mathbf{\text{x}})$ reduces to

## Conclusions

Starting from a standard Naive Bayes (SNB) algorithm, we have derived mathematically the relationship between Xia *et al.*’s ingenious, but heuristic algorithm, and the standard Naive Bayes approach. We also describe the conditions on which Xia *et al.*’s crucial assumption – contributions from absent feature can be discarded – holds. It is our hope that, with this new insight, cheminformaticians may now be able to efficiently use the Modified version of the standard Naive Bayes algorithm, as proposed by Xia *et al.*, and subsequently by Nidhi *et al.* and Nigsch *et al*.

## Appendix

### Appendix A: Estimator of *p*(*x*_{
j
}|*ω*_{
i
})

Here we give for completeness the proof that a priori *Beta* distribution leads to Eqs. 12 and 13 in the text.

For bookkeeping:

*ω*_{
i
}: class label indexed by *i*, *i* = 1,2,...,*C*.

*C*: Number of classes.

${N}_{{\omega}_{i}}$: Number of samples in class *ω*_{
i
}.

*N*_{
ij
}: Number of samples in class *ω*_{
i
} with feature *x*_{
j
} = 1, *j* = 1,2,...,*L*.

*L*: Number of features.

We state from the onset, in the following derivation we follow closely the descriptions given in ref. [10]. We also note, for clarity’s sake, in the following analyses we abuse notation and use *x*_{
jk
} for both the random variable and its realization.

**x**∈ {0,1}

^{ L }, i.e.

*x*

_{ j }∈ {0,1} and suppose that

*x*

_{ j }are independent Bernoulli random variables (and this is in fact the assumption made in the Naive Bayesian approach). Thus, in the Naive Bayesian setting

*p*(

**x**|

*ω*

_{ i }) can be given as

where *μ*_{
ij
} is an estimate for the conditional probability that feature *j* occurs in class *ω*_{
i
}, and is what we are trying to estimate given a set of compounds assumed to belong to class *ω*_{
i
}. (In our context, *μ*_{
ij
} is an estimator for *p*(*x*_{
j
}|*ω*_{
i
}), where *p*(*x*_{
j
}|*ω*_{
i
})is as defined in the text.)

To estimate *μ*_{
ij
} in a Bayesian framework, we first view *μ*_{
ij
} as a random variable, then choose an “appropriate” prior and likelihood for the random variable *μ*_{
ij
}.

*μ*

_{ ij }indicates that

*μ*

_{ ij }is described by a

*Beta*distribution, i.e.,

where *B*(*α*_{
i
},*β*_{
i
}) ensures that the *Beta* distribution is normalised

*μ*

_{ ij }on the training data can be given by

*x*

_{j1},

*x*

_{j2},... and ${x}_{{\mathit{\text{jN}}}_{{\omega}_{i}}}$ denote the

*j*

^{ th }feature of the ${N}_{{\omega}_{i}}$ samples/compounds from class

*ω*

_{ i }. As the samples are assumed independent, then $f({x}_{j1},{x}_{j2},\mathrm{...},{x}_{{\mathit{\text{jN}}}_{{\omega}_{i}}}|{\mu}_{\mathit{\text{ij}}})$ becomes ${\mathrm{\Pi}}_{k=1}^{{N}_{{\omega}_{i}}}f({x}_{\mathit{\text{jk}}}|{\mu}_{\mathit{\text{ij}}})={\mathrm{\Pi}}_{k=1}^{{N}_{{\omega}_{i}}}{\mu}_{\mathit{\text{ij}}}^{{x}_{\mathit{\text{jk}}}}{(1-{\mu}_{\mathit{\text{ij}}})}^{1-{x}_{\mathit{\text{jk}}}}$,

*i.e.*

*π*(

*μ*

_{ ij }|

*x*

_{j1},

*x*

_{j2},...,

*x*

_{ jM }) in Eq. 29 modifies to

*μ*

_{ ij }given the samples ${x}_{j1},{x}_{j2},\mathrm{...},{x}_{{\mathit{\text{jN}}}_{{\omega}_{i}}}$ has the same form as the prior for

*μ*

_{ ij }[11], i.e.,

*Beta*distribution. This means that the Bayes estimator of

*μ*

_{ ij }, which is the estimate we are interested in, is the mean of the posterior distribution obtained [11]:

*QED*.

An accessible description of the derivation of Eq. 37 can be found in ref. [10].

## Declarations

### Acknowledgements

We are in debt to Dr Dave Rogers for his many useful comments on the original LCMNB approach, in particular for helping us understand more about the two–class LCMNB version.

Mussa and Glen would like to thank the Unilever Centre for Molecular Sciences Informatics for its support, whereas Mitchell would like to thank the Scottish Universities Life Sciences Alliance (SULSA).

## Authors’ Affiliations

## References

- Murphy KP: Machine Learning: A Probabilistic Perspective. 2012, Cambridge, MA: MIT PressGoogle Scholar
- Xia X, Maliski EG, Gallant P, Rogers D: Classification of kinase inhibitors using a Bayesian model. J Med Chem. 2004, 47: 4463-4470. 10.1021/jm0303195.View ArticleGoogle Scholar
- Glick M, Davies JW, Jenkins JL, Nidhi: Prediction of biological targets for compounds using multiple-category Bayesian models trained on chemogenomics databases. J Chem Inf Model. 2006, 46: 1124-1133. 10.1021/ci060003g.View ArticleGoogle Scholar
- Nigsch F, Bender A, Jenkins JL, Mitchell JBO: Ligand-target prediction using winnow and naive Bayesian algorithms and the implications of overall performance statistics. J Chem Inf Model. 2008, 48: 2313-2325. 10.1021/ci800079x.View ArticleGoogle Scholar
- Rogers D, Brown RD, Hahn M: Using extended–connectivity fingerprints with Laplacian-modified Bayesian analysis in high–throughput screening follow–up. J Biomol Screen. 2005, 10: 682-686. 10.1177/1087057105281365.View ArticleGoogle Scholar
- Townsend JA, Glen RC, Mussa HY: Note on naive Bayes based on binary descriptors in Cheminformatics. J Chem Inf Model. 2012, 52: 2494-2500. 10.1021/ci200303m.View ArticleGoogle Scholar
- Duda RO, Hart PE: Pattern Classification and Scene Analysis. 1973, New York, NY: John Wiley & Sons, LtdGoogle Scholar
- Koch RK: Introduction to Bayesian Statistics. 2007, Berlin: SpringerGoogle Scholar
- Bishop CM: Pattern Recognition and Machine Learning. 2006, New York: SpringerGoogle Scholar
- Ross SM: Introduction to Probability and Statistics for Engineers and Scientist. 1987, New York: John Wiley & SonsGoogle Scholar
- Davidson AC: Statistical Models (Cambridge Series in Statistical and Probabilistic Mathematics). 2008, Cambridge: Cambridge University PressGoogle Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.