Machine Learning Model Analysis and Data Visualization with Small Molecules Tested in a Mouse Model of Mycobacterium tuberculosis Infection (2014–2015)

Abstract

👁 graphic file with name ci-2016-00004f_0005.jpg

The renewed urgency to develop new treatments for Mycobacterium tuberculosis (Mtb) infection has resulted in large-scale phenotypic screening and thousands of new active compounds in vitro. The next challenge is to identify candidates to pursue in a mouse in vivo efficacy model as a step to predicting clinical efficacy. We previously analyzed over 70 years of this mouse in vivo efficacy data, which we used to generate and validate machine learning models. Curation of 60 additional small molecules with in vivo data published in 2014 and 2015 was undertaken to further test these models. This represents a much larger test set than for the previous models. Several computational approaches have now been applied to analyze these molecules and compare their molecular properties beyond those attempted previously. Our previous machine learning models have been updated, and a novel aspect has been added in the form of mouse liver microsomal half-life (MLM t_1/2) and in vitro-based Mtb models incorporating cytotoxicity data that were used to predict in vivo activity for comparison. Our best Mtbin vivo models possess fivefold ROC values > 0.7, sensitivity > 80%, and concordance > 60%, while the best specificity value is >40%. Use of an MLM t_1/2 Bayesian model affords comparable results for scoring the 60 compounds tested. Combining MLM stability and in vitroMtb models in a novel consensus workflow in the best cases has a positive predicted value (hit rate) > 77%. Our results indicate that Bayesian models constructed with literature in vivoMtb data generated by different laboratories in various mouse models can have predictive value and may be used alongside MLM t_1/2 and in vitro-based Mtb models to assist in selecting antitubercular compounds with desirable in vivo efficacy. We demonstrate for the first time that consensus models of any kind can be used to predict in vivo activity for Mtb. In addition, we describe a new clustering method for data visualization and apply this to the in vivo training and test data, ultimately making the method accessible in a mobile app.

Introduction

Tuberculosis (TB) is a major infectious disease that unfortunately knows no geographic boundaries and accounts for approximately 9 million new cases and 1.5 million deaths each year.¹ TB and its etiological agent, Mycobacterium tuberculosis (Mtb), continue to be the focus of intense international efforts to develop new tools for the control and ultimate elimination² of this devastating disease that is increasingly associated with resistance to first- and second-line drugs.³ The discovery of new TB drug candidates with novel mechanisms of action is of fundamental importance in this regard. The majority of funding for TB research still comes from the NIH NIAID and the Bill and Melinda Gates Foundation. In the past, the European Commission has also funded TB research in the FP7 Program (although nowhere near the levels of the aforementioned organizations). However, no funding for TB small-molecule drug discovery is foreseen in the EC’s Horizon 2020 Program over the next few years. These cuts in funding highlight the need to increase the efficiency of tuberculosis small-molecule drug discovery. Analysis of the recent pipeline at TB Alliance⁴ and elsewhere⁵ reveals that while there are ∼27 projects in preclinical stages and 13 in clinical trials in phases 1–3, only one project is in phase 4 (Figure 1). This indicates a suboptimal pipeline. Of the latter clinical-stage compounds, several do not seem to have progressed since earlier analyses. It is incredibly concerning that we do not have more new molecules in clinical stages, especially with the prescribing limitations surrounding bedaquiline and delamanid because of their cardiovascular side effects from hERG inhibition.^6,7

A major hurdle to progressing molecules into the clinic is identifying compounds that have activity in the mouse models of Mtb infection.⁸ Mice have been used since the 1940s to test drug efficacy.⁹ Of course, the mouse cannot completely model the complex pathology observed in humans, and its drug metabolism and pharmacokinetics also may differ. A recent questionnaire polled different TB laboratories and found that most use BALB/C, C57BL/6, or Swiss mice.⁹ It was concluded that the mouse model may be most useful for rank ordering of compounds to select a drug regimen. However, some laboratories also supplement the in vivo mouse data with in vivo rabbit or marmoset studies of Mtb infection. We previously published a comprehensive assessment of over 70 years of literature resulting in modeling of 773 compounds reported to modulate Mtb infection in mice.⁸ Our detailed analyses of the physiochemical and structural properties of both active and inactive molecules as well as their chemical property space coverage revealed new insights. Furthermore, we used machine learning models to correctly predict in vivo efficacy in the Mtb-infected mouse model for eight of 11 compounds. We identified gaps corresponding to the discovery and approval of new compounds,¹⁰ as highlighted by the 40 years between the approvals of rifampicin and bedaquiline, suggesting that we can learn from earlier drug discovery. Furthermore, there were clear peaks in in vivo testing in the 1940s and 1950s, and it now appears that recent testing in mouse in vivo seems to have peaked,¹⁰ also leading to concerns about the future health of the TB drug pipeline.

Since that report, we have collated an additional 60 molecules from more recent data published in 2014 and 2015 and further evaluated and validated the earlier models with this much larger test set that was not previously available. In addition, we have used recently published models based on in vitro data to create consensus models to predict in vivo activity. Further we describe a new clustering method for data visualization and apply it to all of the in vivo data gathered to date. Our goal is to continue to develop and validate various computational approaches for predicting and visualizing in vivo activity data in the Mtb mouse model, enabling the prediction of new compounds that are the most promising for advancement.

Experimental Section

Results

Discussion

An abundance of data from large in vitro phenotypic screens against Mtb exists in the public domain⁴²⁻⁴⁶ that can readily be used to assist future drug discovery. Since machine learning methods can learn from past data, we have extensively applied Bayesian and other machine learning algorithms to model Mtb inhibition and Vero cell cytotoxicity. We have pioneered the use of dual-event data sets, which use both dose–response data for whole-cell antitubercular activity^47,48 and Vero cell cytotoxicity.^{35,36,42,49−51} We have also used the same approach to model mouse Mtbin vivo data⁸ and most recently MLM stability.³⁴ In addition, we have recently developed a freely available mobile app called TB Mobile¹⁵ that displays over 800 Mtb-active molecule structures and their targets, with links to associated data. This tool was recently enhanced by adding target-specific Bayesian models to rank probable targets.¹⁵ Such Bayesian modeling approaches with FCFP_6 fingerprints have also been integrated into the CDD Vault software,²⁹ and the algorithms were made open source and applied to large numbers of data sets.³¹ Our combined efforts in this area indicate that such machine learning models are a valuable resource and can be used prospectively to suggest molecules to test not only against Mtb but also for other diseases.^52,53

Since our earlier study compiling compounds tested in the mouse Mtb model,⁸ we have continued to collect and curate data with the aim of testing the machine learning models developed and improving upon them. We have now added to our database 60 unique molecules published in 2014–2015 (Table S1 and Supplemental Data 1), of which 41 were classified as actives. The large percentage of active compounds (∼60%) in this new set may represent a publication bias, as noted in our prior analysis for data up to 2014.⁸ Simple molecular property analysis suggested that only the number of hydrogen-bond donors was significantly higher for the 41 in vivo active compounds (Table 1). These 60 molecules broadly cover a similar physiochemical property space as the training set of 784 molecules (Figure 2A) and the over 800 molecules collated in TB Mobile¹⁵ (Figure 2B). These results suggest that we are likely in the applicability domain of the data set. The closest similarity values calculated with the N = 784 Bayesian model (mean closest distance = 0.40 with a standard deviation of 0.15, where a value of zero connotes identity and larger values indicate greater difference; Figure S2) are quite low, suggesting that most are within the applicability domain of the model.

In this study, we have updated the machine learning models and also evaluated whether models for other properties, such as the combined in vitro bioactivity and Vero cell cytotoxicity, MLM t_1/2, or a consensus of these models, could also predict compounds likely to have in vivo activity. When comparing different machine learning approaches such as Bayesian, SVM, and recursive partitioning, we observed little difference based on internal fivefold ROC (Table 2), although predictions for the external test set produced some slight differences in sensitivity, specificity, and concordance (Table 3). These show that the sensitivity values are generally much higher than the specificity values, which is a reversal of what we observed for the fivefold ROC for the Bayesian model training sets (Supplemental Data 1 and 2). This likely suggests that we would have some difficulty classifying inactives with these models while being able to select actives. The updated training set is fairly well balanced, as it was before.⁸ Obviously the use of a computational approach to select compounds presents advantages in likely reducing follow-up costs and lowering numbers of mice used. The t_1/2 MLM stability models follow the same trend with the sensitivity being much higher than the specificity (Table 4), while the in vitroMtb bioactivity TAACF-CB2 dual-event, combined TB dual-event, and MLSMR dual-event models have higher specificity than sensitivity for the test set molecules (Table 4).

Surprisingly, we found that Bayesian models built with in vitro MLM t_1/2 data sets could produce external statistics similar to those for the in vivoMtb models (Table 4), as dual-event models that included in vitroMtb bioactivity and cytotoxicity had positive predicted values of >70% and the best overall hit rates were obtained with a consensus of an Mtb in vitro model and the MLM t_1/2 model (Table 6). In vivo models and in vitro models in the best cases individually had enrichment factors of 1.22 for the top-scoring 10% of compounds, and the combined TB dual-event in vitro Bayesian achieved the best enrichment factor of 1.32 for the top-scoring 10 compounds. Because of the large percentage of active compounds in this new in vivo test set (60%), the maximum enrichment factor that a “perfect” model could produce was 1.46. On the basis of these results (Table 6), it seems that the better PPV hit rates and enrichment factors for two of the consensus approaches are due to (a) superior specificity (filtering out compounds likely to be MLM-unstable, Mtb-inactive, and/or cytotoxic gave ∼2 times the best specificity of an in vivo-trained model) while (b) not having high false positive rates for the consensus approaches. The use of additional external statistics suggested that no single model performed best across all (Table 7). Therefore, these may not be as useful as considering the enrichment factors for test set evaluation (Table 6).

Perhaps part of the accuracy of the MLM Bayesian for predicting in vivo activity is based on the fact that at least 190 of the 894 compounds in the full t_1/2 MLM Bayesian training set were from Mtb and malaria projects (i.e., 20 stable compounds out of 42 were from Mtb projects and 49 stable compounds out of 148 were from malaria projects). Antimalarial compounds sometimes display activity against Mtb,^36,54 and researchers are unlikely to devote the time and money to MLM stability assays unless a compound displays promising therapeutic activity. Thus, perhaps some of the chemical features and properties that are deemed favorable by the MLM stability Bayesian implicitly incorporate both stability and efficacy. Greatly simplified, it is reasonable to correlate in vivo efficacy in the mouse with compounds that at minimum are metabolically stable with regard to phase I metabolism in the mouse and inhibit the growth of Mtb under in vitro conditions mimicking aspects of their actual pathologic environs. Support for this notion is provided by the fact that when evaluating an external test set of known antitubercular molecules, the MLM stability Bayesian recognized most TB drugs as metabolically stable.³⁴

It is interesting that although the TAACF-CB2 and combined TB dual-event Bayesian models had poor overall external sensitivity and concordance values, they displayed enrichment factors for the top-scoring 10% of compounds that were equivalent or superior to the in vivo models. When these two in vitroMtb dual-event models were combined with the t_1/2 MLM stability model to make a novel consensus workflow, the best overall hit rates (and thus the best overall enrichment factors) were achieved (Table 6). This highlights the importance of specificity and of studying more than just the overall external statistics, especially since most laboratories will only be able to perform in vivoMtb studies in mice for a very small number of candidate compounds. In summary, this suggests that together the experimental in vitroMtb and t_1/2 MLM stability models are a good predictor of in vivoMtb efficacy in the mouse model and that a consensus machine learning model may be a useful alternative, at least on the basis of this particular test set as described.

While humans can visualize quite complex data, there have been many approaches to data visualization to produce maps that can explain the terrain of biological data, including Kohonen networks, Sammon maps, and many other approaches.⁵⁵ In this study we have described a new data visualization approach called honeycomb visualization that was used to cluster the in vivo training set and the new external test set (Figures 3 and S1). This shows, for example, how macrocyclic compounds cluster together (Figure 3B) and how the 60 new compounds are dispersed around the map with local clusters of similar analogues. Such an approach, while also relying on the same descriptors, illustrates an alternative way to assess predictions or structure–activity relationships and has been implemented recently in a new iOS free mobile app called PolyPharma⁵⁶ to demonstrate how machine learning and this visualization can be used outside of the desktop. In addition, the use of tools such as CDD Models can enable the sharing of Bayesian models such that they can be run in freely available mobile apps, making the models more accessible.³⁰

In summary, this work adds further weight to the use of machine learning approaches to predict in vivoMtb activity in the mouse efficacy model. We have not seen this kind of approach taken with other disease models and data sets, so this still ranks as a difficult but interesting problem to address. Our previous reported Mtbin vivo model⁸ was tested with only a small test set of 11 molecules, while we now report one that has 60 molecules. For the first time we have shown that MLM stability predictions using a Bayesian model could be a useful adjunct to applying a specific Mtbin vivo mouse machine learning model to predict efficacy. This is potentially of importance because it is likely that MLM stability models are based on more structurally diverse molecules than just antitubercular efficacy models. Having more modeling options for predicting in vivo activity is of value because of the relatively small data set of in vivoMtb values publicly available at this time. The in vitro and cytotoxicity models’ training sets are larger, and these with the MLM t_1/2 models can be additionally used for predicting in vivo activity based on our test set of 60 recently published molecules. There is no shortage of in vitro screening hits (there are likely many thousands across the various public–private partnership, NIH-funded, and commercial screens), and we would propose that computational models such as those described in this study be made available, shared,²⁹ and utilized alongside other selection criteria (medicinal chemistry heuristics or gut feeling) prior to selecting compounds for testing in the animal model in order to expedite TB research and save both time and money. Along these lines, the 177 GSK open source Mtb leads were scored with the top two consensus dual-event and MLM Bayesian workflows, and the intersection of their top predictions was analyzed in order to suggest compounds that should be prioritized for in vivo assays (Tables S5–S7). In due course we will use our models to make prospective predictions prior to in vivo testing in the mouse Mtb model in our own laboratories, and these results will be reported in the future.

In conclusion, we have built on our previous Mtbin vivo and in vitro modeling studies^{8,35,36,47,48,50,51,57,58} to suggest a combination of published data and machine learning models that can be used to harness limited research resources and increasingly contracting funding for TB research. With continual pressure to identify novel in vivo active antituberculars to supplement the currently depleted clinical pipeline, these machine learning models could be considered.

Supporting Information Available

The Supporting Information is available free of charge on the ACS Publications website at DOI: 10.1021/acs.jcim.6b00004.

• Supplemental Data 2–4, Tables S1–S7, and Figures S1 and S2 (PDF)

• Enlarged Figure S1 (PDF)

• Supplemental Data 1 (XLSX)

Author Contributions

^∇ S.E. and A.L.P contributed equally.

The authors declare the following competing financial interest(s): S.E., A.M.C., and R.C.R. are consultants for Collaborative Drug Discovery, Inc.

Notes

All of the computational models are available from the authors upon request. The structures of the molecules used in this study are available in CDD (as described previously) or in previous papers.

Supplementary Material

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.