A new method for the comparison of 1H NMR predictors based on tree-similarity of spectra
© Castillo et al.; licensee Chemistry Central Ltd. 2014
Received: 19 February 2014
Accepted: 11 March 2014
Published: 25 March 2014
A methodology based on spectral similarity is presented that allows to compare NMR predictors without the recourse to assigned experimental spectra, thereby making the task of benchmarking NMR predictors less tedious, faster, and less prone to human error. This approach was used to compare four popular NMR predictors using a dataset of 1000 molecules and their corresponding experimental spectra. The results found were consistent with those obtained by directly comparing deviations between predicted and experimental shifts.
Cheminformatics plays an increasingly important role in structure validation by NMR spectroscopy, providing methods and algorithms for computer-assisted NMR spectra assignment and structure elucidation [1–8], as well as prediction and simulation [9–19] of spectra. Those methods heavily rely on the accuracy of predicted NMR parameters and thus along with the introduction of such novel methods comes the need to compare and evaluate available predictors. The established approach for this task consists in comparing the predicted NMR parameters, i.e., chemical shifts and coupling constants, with experimentally determined ones. Such approach comports the need to manually assign experimental data. As an alternative, benchmarking of NMR predictors could be performed using techniques of cheminformatics itself, avoiding errors due to manual assignment.
In a recent article the authors presented a tree-based method for measuring similarity between NMR spectra . It was shown to produce results comparable to those of the binning method , with significant improvement in efficiency by focusing on the regions of the spectrum containing most of the information. Furthermore, this new approach directly operates on raw spectra, i.e., doesn’t relies on peak-picking. These features turn it into an attractive tool for the comparison and evaluation of NMR predictors, as it allows to measure the similarity between predictions and experiments without having recourse to assigned spectra. This article presents such a methodology and validates it against the established approach, for four common predictors.
The success of an NMR prediction algorithm is determined by its ability to reproduce the experimental chemical shifts. Determining the adequacy of a prediction thus implies having assigned the experimental spectra, and having their chemical shifts compared with predicted ones. Peak-picking and assignment are troublesome and time-demanding tasks, however. As an alternative, we propose to evaluate the success of a prediction algorithm by its ability to produce, by means of a proper simulation algorithm, a spectrum that is sufficiently similar to the one given by the experiment. The meaning of sufficiently similar will be discussed later in the text in the Methods section.
Results and discussion
The resulting histograms are also shown in Figure 1b. Again, the performances of predictor A were found superior, producing around 10% more predictions on the two lower error intervals, while the other systems performed similarly, in agreement with the results obtained using our method.
A set of 1000 molecules of up to 33 heavy atoms was randomly selected from the Maybridge catalogue  (see Additional file 1) and the corresponding 1H NMR spectra kindly provided by Maybridge (see Additional file 2). The spectra were acquired with a 250 MHz Bruker spectrometer using a standard bruker pulse sequence (zg30), a relaxation delay of 1 s, a 30° excitation pulse at 27 kHz and an observation window of 20.693 ppm centered at 6.175 ppm. Each spectrum was binned and stored as a 1024 real points vector. For each molecule, the proton chemical shifts were predicted using four different prediction tools, referred to as A, B, C and D. The original spectra (1024 point, jcamp format), the raw predictions and a matrix of simulated spectra of 1024 points are provided in Additional files 3 and 4. We decided to keep predictors anonymous to maintain the focus of this work on the method to rank predictors, rather than the ranking itself. Each prediction was used to simulate a 1024 point spectrum at a frequency of 250 MHz with an algorithm that we described elsewhere . Similarity matrices between simulated and experimental spectra, MRR, and average absolute and relative prediction accuracy were computed for each data set using the methodology described in the previous Section. A subset of 298 randomly chosen molecules were manually assigned in order to perform the evaluation by direct comparison of predicted and observed chemical shifts and compare the results obtained with those produced by our method. A subset was used for this part due to time constraints.
The direct comparison of simulated and experimental spectra using an adequate similarity measure allows for an efficient and fully automatic methodology to evaluate NMR prediction algorithms. Results obtained using this new method are consistent with those obtained by the traditional chemical shift comparison method, but without the need for peak-picking and assignment. We therefore provide a method that can help improving NMR predictors in the future by allowing the comparison of predictors using datasets that are too large to be assigned manually.
where n is the number of queries and rank i is the rank of the correct match in the i-th query.
All queries are located on the upper triangle (gray area), as the similarity measure ranges from 0 to 1.
Points located on the diagonal (dotted line) correspond to those cases where the best match is the correct match.
The accuracy of the prediction in absolute terms (i.e. in terms of the similarity between the correct match and the experimental spectrum) increases as we move up to the extreme at (1,1), where the correct match and the experimental spectrum are identical. We then refer to the magnitude of the component of the query on this direction as the absolute prediction accuracy (see Figure 4).
The accuracy of the prediction relative to the data set (i.e. the ratio between the experimental spectrum’s similarity to the correct match and its similarity to the best match in the data set) decreases as we move away from the identity line. We then refer to the magnitude of the component of the query vector orthogonal to the absolute prediction accuracy as the relative prediction accuracy (see Figure 4).
Low relative prediction accuracy means that the correct match is as similar or almost as similar to the queried spectrum as the best match in the whole database. Good predictions can then be associated with low values of relative accuracy. Note that this approach looks into the actual similarity values between the experimental and simulated spectra regardless of the rank of the correct match, which is exactly the opposite of what we achieved with the MRR. Combining the two approaches we can distinguish between low-ranking queries due to poor prediction and low-ranking queries due to an inadequate similarity measure: as long as the same trends result from evaluating performance in terms of the relative accuracy index or in terms of the MRR, we can be certain that the evaluation is not biased by a poorly discriminating similarity measure.
It follows from the previous discussion that the choice of an appropriate similarity measure is key to the success of the methodology proposed. Here we used the tree-based methodology that has been described in detail elsewhere . In brief, it consists in building a tree representation of each spectrum that summarizes key information on its signal-rich regions, followed by the computation of a similarity measure between these trees. This similarity measure is defined recursively, so that the similarity between two trees at depth k depends on the similarity between nodes located on that level, and on the similarity between the trees at depth k + 1. This technique is similar to the traditional binning technique , but presents the advantage of focusing on regions with high signal intensity, using fewer data points by avoiding large blank or merely noisy zones.
The authors acknowledge Dr. Reiner Dieden for his motivating discussions and Colciencias-Renata (RC-561-2009) for funding.
- Elyashberg ME, Williams AJ, Martin GE: Computer-assitsed structure verification and elucidation tools in NMR-based structure elucidation. Prog Nucl Magn Reson Spectrosc. 2008, 53: 1-104. 10.1016/j.pnmrs.2007.04.003.View ArticleGoogle Scholar
- Nuzillard J-M, Massiot G: Computer-aided spectral assignment in nuclear magnetic resonance spectroscopy. Anal Chim Acta. 1991, 242: 37-41.View ArticleGoogle Scholar
- Vitek O, Bailey-Kellogg C, Craig B, Kuliniewicz P, Vitek J: Reconsidering complete algorithms for protein backbone NMR assignment. Bioinformatics. 2005, 21: 230-236.View ArticleGoogle Scholar
- Christie B, Munk M: The role of two-dimensional nuclear magnetic resonance spectroscopy in computer-enhanced structure elucidation. J Am Chem Soc. 1991, 113: 3750-3757. 10.1021/ja00010a018.View ArticleGoogle Scholar
- Funatsu K, del Carpio C, ichi Sasaki S: Automated structure elucidation system -CHEMICS. Fresenius’ Z Anal Chem. 1986, 324: 750-759. 10.1007/BF00468386.View ArticleGoogle Scholar
- Lindel T, Junker J, Köck M: COCON: from NMR correlation data to molecular constitutions. J Mol Model. 1997, 3: 364-368. 10.1007/s008940050052.View ArticleGoogle Scholar
- Nuzillard J-M, Massiot G: Logic for structure determination. Tetrahedron. 1991, 47: 3655-3664. 10.1016/S0040-4020(01)80878-4.View ArticleGoogle Scholar
- Masui H, Hong H: Spec2D: a structure elucidation system based on 1H NMR and H-H COSY spectra in organic chemistry. J Chem Inf Model. 2006, 46: 775-787. 10.1021/ci0502810.View ArticleGoogle Scholar
- Schaller R, Munk M, Pretsch E: Spectra estimation for computer-aided structure determination. J Chem Inf Comput Sci. 1996, 36: 239-243. 10.1021/ci950141y.View ArticleGoogle Scholar
- Liu X, Balasubramanian K, Munk M: Computer-assisted graph-theoretical construction of 13C NMR signal and intensity patterns. J Magn Reson. 1990, 87: 457-474.Google Scholar
- Golotvin SS, Vodopianov E, Lefebvre BA, Williams AJ, Spitzer TD: Automated structure verification based on 1H NMR prediction. Magn Reson Chem. 2006, 44: 524-538. 10.1002/mrc.1781.View ArticleGoogle Scholar
- Golotvin SS, Vodopianov E, Pol R, Lefebvre BA, Williams AJ, Rutkowse RD, Spitzer TD: Automated structure verification based on a combination of 1D 1H NMR and 2D 1H-13C HSQC spectra. Magn Reson Chem. 2007, 45: 803-813. 10.1002/mrc.2034.View ArticleGoogle Scholar
- ACD/HNMR Predictor, version 9.0. Toronto, Ontario, Canada: Advanced Chemistry Development, Inc, [http://www.acdlabs.com], accessed on February 2014
- Aires-de-Sousa J, Hemmer M, Gasteiger J: Prediction of 1H NMR chemical shifts using neural networks. Anal Chem. 2002, 74 (1): 80-90. 10.1021/ac010737m.View ArticleGoogle Scholar
- Binev Y, Aires-de-Sousa J: Structure-based predictions of 1H NMR chemical shifts using feed-forward neural networks. J Chem Inf Comp Sci. 2004, 44 (3): 940-945. 10.1021/ci034228s.View ArticleGoogle Scholar
- Binev Y, Marques MM, Aires-de-Sousa J: Prediction of 1H NMR coupling constants with associative neural networks trained for chemical shifts. J Chem Inf Model. 2007, 47 (6): 2089-2097. 10.1021/ci700172n.View ArticleGoogle Scholar
- Abraham RJ, Mobli M: A practical approach to 1H NMR calculation and prediction. Modelling 1H NMR Spectra of Organic Compounds. 2008, John Wiley & Sons, Ltd, 349-368.View ArticleGoogle Scholar
- Abraham RJ, Mobli M: The prediction of 1H NMR chemical shifts in organic compounds. Spectrosc Eur. 2004, 16: 16-22.Google Scholar
- Castillo AM, Patiny L, Wist J: Fast and accurate algorithm for the simulation of NMR spectra of large spin systems. J Magn Reson. 2011, 209: 123-130. 10.1016/j.jmr.2010.12.008.View ArticleGoogle Scholar
- Castillo AM, Uribe L, Patiny L, Wist J: Fast and shift-insensitive similarity comparisons of NMR using a tree-representation of spectra. Chemometr Intell Lab. 2013, doi: 10.1016/j.chemolab.2013.05.009Google Scholar
- Bodis L, Ross A, Pretsch E: A novel spectra similarity measure. Chemometr Intell Lab. 2007, 85: 1-8. 10.1016/j.chemolab.2005.10.002.View ArticleGoogle Scholar
- Maybridge.com: 2011, [http://www.maybridge.com], accessed on February 2014
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.