Research Article Evaluation of 11 Scoring Functions...

Research ArticleEvaluation of 11 Scoring Functions Performance onMatrix Metalloproteinases

Jamal Shamsara

Pharmaceutical Research Center Mashhad University of Medical Sciences Mashhad 91775-1365 Iran

Correspondence should be addressed to Jamal Shamsara shamsarajmumsacir

Received 19 September 2014 Revised 1 December 2014 Accepted 1 December 2014 Published 25 December 2014

Academic Editor Armando Rossello

Copyright copy 2014 Jamal Shamsara This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Matrix metalloproteinases (MMPs) have distinctive roles in various physiological and pathological processes such as inflammatorydiseases and cancerThis study explored the performance of eleven scoring functions (D-ScoreG-Score ChemScore F-Score PMF-Score PoseScore RankScore DSX and X-Score and scoring functions of AutoDock41 and AutoDockVina) Their performancewas judged by calculation of their correlations to experimental binding affinities of 3D ligand-enzyme complexes of MMP familyFurthermore they were evaluated for their ability in reranking virtual screening study results performed on a member of MMPfamily (MMP-12) Enrichment factor at different levels and receiver operating characteristics (ROC) curves were used to assesstheir performance Finally we have developed a PCA model from the best functions Of the scoring functions evaluated F-ScoreDSX and ChemScore were the best overall performers in prediction of MMPs-inhibitors binding affinities while ChemScoreAutodock and DSX had the best discriminative power in virtual screening against the MMP-12 target Consensus scorings didnot show statistically significant superiority over the other scorings methods in correlation study while PCAmodel which consistsof ChemScore Autodock and DSX improved overall enrichment Outcome of this study could be useful for the setting up of asuitable scoring protocol resulting in enrichment of MMPs inhibitors

1 Introduction

Matrix metalloproteinases (MMPs) are zinc-dependent en-dopeptidases that play a central role in various physiologi-cal processes and pathological conditions including cancerand inflammatory diseases One of the main problems fordeveloping a new class of drugs as MMP inhibitors is theissue of selectivity This family shares a very similar activesite that makes traditional chemical approach for developingof selective inhibitors time-consuming In this case thecomputational approaches including molecular docking canhelp the medicinal chemistry [1 2]

As reliability of different scoring functions is very target-dependent [3] in this study we aimed to evaluate someavailable scoring functions in scoring of MMPs-ligandsinteractions Reliability of molecular docking depends onhow the geometry of ligands will be predicted and howthe different pose of a ligand and interaction of differentligands with receptor will be ranked [4] The former hasbeen investigated on a set of 40MMPs complexes [5] In our

paper we focused on successfully ranking the interaction ofdifferent ligands with MMPs Scoring functions are used toestimate the binding affinity of a compound for a receptor ina reasonable time These scoring functions can fall into threecategories [6 7] (1) empirical scoring functions including X-Score [8] F-Score [9 10] andChemScore [11] (2) knowledge-based potentials includingDSX [12] andPMF-Score [13] and(3) force-field based approaches including D-Score [14] andG-Score [15] Knowledge-based scoring functions observeinteratomic contact frequencies andor distances in a largedatabase of protein-ligand complexes 3D structures Theobserved frequency distributions of favorable and unfavor-able molecular interactions are converted to potentials ofmean force or knowledge-based potentials The two othermentioned categories contain scoring methods based onphysical interaction termsThese methods try to estimate thechange in free energy upon ligand binding via decompositionof free energy into a sum of individual contributionsThe firstclass of scoring functionswithin this group (force-field based)directly derives the interaction terms from physicochemical

Hindawi Publishing CorporationInternational Journal of Medicinal ChemistryVolume 2014 Article ID 162150 9 pageshttpdxdoiorg1011552014162150

2 International Journal of Medicinal Chemistry

theory and does not fit them to experimental data Theother class (empirical based) tries to find linear statisticalrelationship between the binding affinity and a number ofligand binding terms in a training set of ligand-proteincomplexes 3D structures with associated binding affinity data[4]

Some proposed consensus docking [16 17] and consensusrescoring [18] protocols are available The two consensusscoring methods so-called rank-by-number and rank-by-rank that had shown promising results [18] were also testedin this study In addition we suggested another methodprincipal component analysis (PCA) for performing a com-bination of multiple scoring functions to rescore and rerankthe compounds after virtual screening on MMP-12 target

Thework reported here seeks to address two questions (1)How can different scoring functions predict the experimentalbinding affinities for MMPs-inhibitor complexes (2) Dothe well-performed scoring functions have also reasonableperformance in an enrichment study on a member of MMPsfamily (MMP-12)

2 Methods

21 Preparation of Protein Test Set for Rescoring Study Thetest set consisted of 100MMPs-ligand complex structuresformed of 10 humanMMPs typesWe excluded the structureswith conflictive reported binding affinitiesThe 3D structureswere taken from PDB (Protein Data Bank) and then under-went some refinements Firstly water and other cocrystal-ized molecules were removed from the retrieved PDB filesThen the protein and corresponding ligand (inhibitor) wereextracted to separate PDB files The file formats changedto mol2 as it was a necessary step for some subsequentanalysis The hydrogens were added to both protein andligand molecules All of the selected PDB structures hadexperimentally determined Ki Kd or IC50 The logarithmof Ki Kd or IC50 was employed as experimental bindingaffinity in our study The detailed structural informationfor each is presented in Supplementary Material availableonline at httpdxdoiorg1011552014162150 For analysisthe pAffinity was employed as dependent variable instead ofbinding affinity which was defined as below

pAffinity = minusLog (Ki Kd or IC50) (120583M)

Metal ion (catalytic zinc ion) was saved as a part of themacromolecule The Gasteiger partial charge was assignedfor ligands All of the above procedures were done usingthe PyMOL (httpwwwpymolorg The PyMOLMolecularGraphics System version 12r3pre Schrodinger LLC) andOpen Babel Package (version 231 httpopenbabelorg)[19]

22 Scoring Functions Various scoring functions have beenevaluated in this study 11 scoring functions including thefive SYBYL built-in scoring functions (D-Score [14] G-Score[15 20] ChemScore [11] F-Score [9 10] and PMF Score[13]) two web based scoring functions (PoseScore [21] andRankScore [21]) two standalone scoring functions (DSX [12]

and X-Score [8]) and scoring functions of AutoDock41 [2223] and AutoDock Vina [24] were employed in this studyFurthermore two consensus scorings were applied on the setAll of the 11 scoring functions were used to compute bindingscores for ligand-protein interactions Some of the scoringfunctions were not able to compute reasonable binding scorefor all of the complexes It has been discussed earlier that suchan incompatibilitymay be raised by the fact that in some casesthere are clashes between protein and ligand molecules [25]However we did not penalize those scoring functions in ourstudy The pairwise deletion strategy was used to deal withmissing data

We employed previously defined consensus scoring(rank-by-number and rank-by-rank methods [18]) to sum-marize the results of multiple scoring functions In rank-by-rank method predicted individual rank was calculated as anaverage of ranks predicted by all the scoring functions Rank-by-number consensus score is an average of the Z-scaledscores calculated by each of the individual scoring functionsIndividual Z-scaled scoring function values (ZScore) arecomputed by

Zscore =(119891

119894minus 120583)

119878

(1)

where 119891119894is the scoring value of an individual scoring

function 120583 is the mean value and 119878 is the standard deviationof this scoring function for entire set

Finally the principal component analysis (PCA) wasapplied on various set of scores of enrichment study toevaluate the discrimination power of PCA on our evaluatedset of compounds PCA is a powerful tool for differentaspects of data evaluation including classification and patternrecognition It can simplify and reduce the dimensionality ofmultivariate data set while preserving as much of the relevantinformation The principal components (PCs) are linearcombinations of the original variables The first principalcomponent (PC1) has the largest possible variance Thesecond principal component (PC2) is uncorrelated to the firstone and it accounts for most of the remaining variance PCAmodel has been employed in our study for discrimination ofactives among decoys in virtual screening results based onobtained scores from various scoring functions In case of ourstudy the PCA was applied to generate linear combinationof different scores and extracted the main variation in thedata as PC1 and subsequent rescoring and reranking of virtualscreening results based on formulated PC1 The contributionof an individual score to the calculated PC can be describedby its loading value

23 Preparation of Docking Set for the Retrospective VirtualScreening on MMP-12 The inhibitors molecules of dockingset were prepared basically from the MMP-12 inhibitorsspreadsheet taken from ChEMBL database [26] Firstlyinactive and low active molecules (IC50 gt 100000 nM)were removed from the spreadsheet Cases with incompleteinformation (unitless activity or inexact IC50 values) andthose which did not fully satisfy Lipinskirsquosrule of five werealso excluded from the spreadsheet The edited spreadsheet

International Journal of Medicinal Chemistry 3

pAffinity 0

2

4

ChemScore

DSX

F-Score

PoseScore

RankScore

minus80

minus80

minus80

minus40

minus80 minus40

minus50

minus50

minus20

minus20

minus20 minus10minus30

minus20

minus10

minus30

minus15 0

minus4

0 2 4minus4

minus30

minus15

0

minus30

minus100

minus250

minus100minus250

Figure 1 Scatter plot for the best performed scoring functions Correlation of each scoring function relative to other scoring functions aswell as experimental binding affinity (pAffinity) is shown

containing SMILES and bioactivities was imported in Canvas16 Some of the selected inhibitors from the previous stepshared similar scaffolds and it could cause biased resultsTo overcome this potential problem finger prints for everyinhibitor were defined by binary fingerprint module Thendiversity selection tool was applied to select the most 30diversified molecules from inhibitor set The 30 inhibitorswere visually inspected to have different scaffolds To generatedecoys which physically resemble active set we used theonline-tool of DUD-E [27] It generated 50 decoys for eachactive molecule In summary this tool tries to make decoyswith similar physical properties including molecular weightcalculated logP number of rotatable bonds and hydrogenbond donors and acceptors for each ligandwhile it minimizesthe 2D topological similarities between generated decoys and

corresponding ligand to make them suitable for true negativecontrol role 3D conformationswere generated for actives anddecoys and subsequent energy minimization partial chargeassignment and ionization were performedThese steps weredone using LigPrep module in Schrodinger It uses force fieldOPL2005 for energy minimization after 2D to 3D conversionof ligands

24 Docking and Preparation of the Protein for the Retrospec-tive Virtual Screening on MMP-12 The Glide (Glide version57 Schrodinger LLC New York NY 2011) was used fordocking studies As mentioned above a set of inhibitors anddecoys was docked in MMP-12 (PDB code 3F17) active siteFor receptor preparation water molecules were removed


False positive rate

True

pos

itive

rate

00 02 04 06 08 10

00

02

04

06

08

10

AUC = 065

(a)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 063

False positive rate00 02 04 06 08 10

(b)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(c)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(d)

Figure 2 ROC curve of (a) Glide-Score (b) DSX (c) Autodock and (d) ChemScore for Glide (HTS) virtual screening results

hydrogens were added and protein structure was minimizedusing protein preparation wizard [28] For Glide two dock-ing runs were conducted a docking procedure with high-throughput virtual screening (HTVS) setting and another onewith standard precision (SP) mode We used default settingsGrid box was centered at cocrystalized ligand and was sizedto 14 angstromThe output files were saved as mol2 format

25 Statistical Analysis The scoring functions were evalu-ated via calculation of the linear correlation between pre-dicted binding affinity scores and experimentally determinedbinding affinities Pearsonrsquos correlation coefficient (119877

119901) and

Spearmanrsquos correlation coefficient (119877119904) were used for quanti-

tative assessment of scoring functions predictivity Pearsonrsquoscorrelation coefficient shows the predictivity of scores while

Spearmanrsquos correlation coefficient indicates the predictiveability of scoring functions to properly rank the ligand-receptor affinities

To evaluate the performance of the scoring functions indiscriminating actives among decoys the scoring functionsperformance was tested on docked active and decoy com-pounds The receiver operating characteristic (ROC) curveand enrichment factor (EF) were applied to determine theperformance of each scoring function The increase in areaunder the curve (AUC) of ROC curve can be used as anindicator of improvement in discrimination between trueligands from decoys AUC can have a value between 0 and1 in which AUC = 05 means that the method of interestperformed like a random selection in average while AUC =1 means the complete discrimination between true and false


00 02 04 06 08 10

True

pos

itive

rate

00

02

04

06

08

10

AUC = 073

False positive rate

(a)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 068

(b)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 069

(c)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 076

(d)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 079

(e)

Figure 3 ROC curve of (a) Glide-Score (b) DSX (c) Autodock (d) ChemScore and (e) PC1 for Glide (SP) virtual screening results


cases (active and decoys) EF is defined as the fractionof active compounds found divided by the fraction of thescreened library

EF = (activessampled

activestotal) times (

119873total119873sampled

) (2)

EF1 and EF2 are shown the ability of a particularscoring method to retrieve true ligands with a high rankamong virtual screening results They could be even moreinformative than AUC of ROC curve index as scoringfunctions with AUC of ROC curve around 05 could still havean acceptable performance at early stage of the curve that canbe detected using EF1 or EF2

All of the statistical test and plotting were done using R(R a language and environment for statistical computing RFoundation for Statistical Computing Vienna Austria URLhttpwwwR-projectorg) including packages enrichvsmissMDA and ROCR [29]

3 Results

31 Correlation of Predicted Scores with Experimental BindingAffinities The minusLog experimental binding affinities (pAffin-ity) for the selected test set of MMPs-ligand complexes rangefrom minus39 to 4 spanning about 8 orders of magnitude with amean value of 140 and STDof 152 (SupplementaryMaterial)The correlation table of scoring functions (scores from allthe 11 scoring functions as well as two consensus scorings)are shown in Supplementary Material Table 1 shows the cor-relation coefficients between different scoring functions andpAffinity Table 2 summarized the main results of the scor-ing functions comparison The consensus scorings did notimprove the prediction more than the best scoring functionsFor scoring functions which had a good correlation withexperimental results (F-Score PoseScore RankScore DSXand ChemScore) correlation plots are shown in Figure 1

32 ROC Curve Analysis and Enrichment Factor Calculationfor the Retrospective Virtual Screening on MMP-12 Based onthe fact that PoseScore and RankScore have online basedinterfaces they were excluded from rescoring assessment

ROC curve plots specificity against sensitivity at differentcutoff values (in this case different scores) The enrichmentability of scoring functions was assessed on a set of dockedcompounds including known inhibitors and decoys Table 3demonstrated the obtained EFs at different level for variousscoring function on docked poses with either Glide standardprecision or Glide HTVS protocols In addition to Glide-native scoring function ChemScore Autodock and DSXshowed better performance than other tested functions inboth rescoring jobs Figures 2 and 3 show representative ROCplots for scoring functionswith the best performances amongscoring programs evaluated in the enrichment study Thecalculated areas under the receiver-operating characteristiccurves values for each scoring program are given in Table 3PC1 obtained from performed PCA on Autodock DSX andChemScore scores led to the best EF1 and AUC for SPdocking runs Principle component 2 (PC2) was plotted

Table 1 Correlation coefficients (Pearsonrsquos and Spearmanrsquos cor-relation coefficients) for 11 individual and two consensus scoringfunctions with pAffinity

Pearsonrsquos correlationcoefficient with

pAffinity

Spearmanrsquoscorrelation coefficient

with pAffinityConsensus(rank-by-rank) 0298 0227

Consensus(rank-by-number) minus0303 minus0211

AutoDock41 minus0049 0019ChemScore minus0253 minus0216D-Score minus0090 minus0048DSX minus0368 minus0255F-Score minus0390 minus0391G-Score minus0178 minus0148PoseScore minus0321 minus0227RankScore minus0311 minus0285PMF-Score minus0148 minus0147Vina minus0078 minus0036X-Score minus0209 minus0109

4

2

2

0

0

minus2

minus2minus4

PC2

PC1

Figure 4 Plot of PC2 against PC1 for Glide virtual screening results(SP) 998771 actives I decoys

against PC1 in Figure 4 As it was shown in Figure 4 PC1 hasan ability to discriminate true binders from decoys as at theleft side the density of true ligands is much higher PC2 is notvery informative in this regard

4 Discussion

It was clear that MMPs are still interesting targets forpharmaceutical studies On the other hand scoring functionshave different performance on different targets [3]We used 11


Table 2 The scoring functions are ranked from the best (1) to the worst (5) according to the correlation with experimental data

Basedon Rp F-Score1 DSX2 PoseScore2 RankScore2 ChemScore3 X-Score4 G-Score4 PMF-Score4 D-Score5 Vina5 AutoDock415

Basedon Rs F-Score1 RankScore2 DSX2 PoseScore2 ChemScore3 G-Score4 PMF-Score4 X-Score4 D-Score5 Vina5 AutoDock415

Table 3 The performance characteristics of scoring functions in discrimination of true binders after docking

Scoring method AUC of ROC curve EF20 EF10 EF2 EF1

Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0

PMF-Score 0501041 0535714 0714286 0 0G-Score 0551684 1428571 1428571 5357143 1071429D-Score 0515996 1428571 1785714 7142857 7142857

ChemScore 0648363 2678571 3571429 7142857 1428571X-Score 056054 125 1785714 1785714 3571429DSX 0632341 2142857 2857143 1785714 0

Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276

scoring functions to predict the binding affinities for MMPs-inhibitor complexes After that the results were tested on amember ofMMPs family (MMP-12)The F-Score PoseScoreRankScore DSX and ChemScore showed the best perfor-mances among the 11 assessed scoring functions in scoringand ranking ligand-receptor binding taken from availableMMPs crystal structures In the next step we evaluated thescoring functions ability to find MMP-12 inhibitors (activecompounds) among set of decoys Our enrichment andROC curve study further validated the results of predictivitystudy for DSX and ChemScore via analysis of MMP-12virtual screening results The PC1 component of PCA modelconsisting of three scoring functions (Autodock DSX andChemScore) had the best performance in enrichment study

The overall performance of scoring functions in predic-tion of experimental binding affinities ofMMPs 3D structuresin presence of inhibitors was not satisfying in comparisonwith those reported in some previous studies on other targets[6 25 30] This could be due to the lack of restrictiveselection criteria in our study We did not apply restrictiveselection criteria on our test set 3D structures since it woulddramatically decrease the statistical power of the analysisUsually test sets for evaluation of dockingscoring functions

include only X-ray crystallography structures with highresolutions Moreover the binary complexes are preferredBut our test set included complexes that are not fully suitablefor dockingscoring studies In addition the binding affinitydata were taken from different sources that could be thesource of noises in analysis

However the scoring functions with top correlation coef-ficients (DSX and ChemScore) associated with the best ROCcurve and EFs in rescoring virtual screening results of MMP-12 This was validated by the applicability of the predictivitypower study results for MMPs As the PCA potential forimproving virtual screening results was demonstrated inprevious reports [31] this approach was also evaluated in ourstudy PCAmodels obtained from different scoring functionswere tested and the one which included Autodock DSX andChemScore scoring functions had improved the ROC as wellas EF of SP Glide virtual screening results

The ultimate goal of this study was to determine whichof the scoring functions or combinations of them wouldyield the best results in terms of enrichment when usedagainst MMPs in a virtual screening study Our study wasretrospective and virtual screening was only performed incase of MMP-12 However due to high similarity between


active site structure and sequence among MMPs family thesimilar results were expected for other members

Conflict of Interests

The author declares that there is no conflict of interests in hiswork

Acknowledgment

This work is financially supported by Mashhad University ofMedical Sciences

References

[1] J Hu P E van den Steen Q-X A Sang and G OpdenakkerldquoMatrix metalloproteinase inhibitors as therapy for inflamma-tory and vascular diseasesrdquoNature Reviews Drug Discovery vol6 no 6 pp 480ndash498 2007

[2] V Vargova M Pytliak and V Mechırova ldquoMatrix metallopro-teinasesrdquo EXS vol 103 pp 1ndash33 2012

[3] M H J Seifert ldquoTargeted scoring functions for virtual screen-ingrdquoDrug Discovery Today vol 14 no 11-12 pp 562ndash569 2009

[4] P F W Stouten and R T Kroemer Docking and ScoringComprehensive Medicinal Chemistry II Elsevier 2007

[5] X Hu S Balaz and W H Shelver ldquoA practical approachto docking of zinc metalloproteinase inhibitorsrdquo Journal ofMolecular Graphics and Modelling vol 22 no 4 pp 293ndash3072004

[6] S-Y Huang S Z Grinter and X Zou ldquoScoring functions andtheir evaluation methods for protein-ligand docking recentadvances and future directionsrdquo Physical Chemistry ChemicalPhysics vol 12 no 40 pp 12899ndash12908 2010

[7] N Brooijmans ldquoDocking methods ligand design and vali-dating data sets in the structural genomic erardquo in StructuralBioinformatics pp 635ndash663 JohnWiley amp Sons New York NYUSA 2009

[8] R Wang L Lai and S Wang ldquoFurther development andvalidation of empirical scoring functions for structure-basedbinding affinity predictionrdquo Journal of Computer-Aided Molec-ular Design vol 16 no 1 pp 11ndash26 2002

[9] M Rarey B Kramer T Lengauer and G Klebe ldquoA fast flexibledocking method using an incremental construction algorithmrdquoJournal of Molecular Biology vol 261 no 3 pp 470ndash489 1996

[10] M Rarey B Kramer and T Lengauer ldquoDocking of hydrophobicligands with interaction-based matching algorithmsrdquo Bioinfor-matics vol 15 no 3 pp 243ndash250 1999

[11] M D Eldridge C W Murray T R Auton G V Paolini and RP Mee ldquoEmpirical scoring functions I The development of afast empirical scoring function to estimate the binding affinityof ligands in receptor complexesrdquo Journal of Computer-AidedMolecular Design vol 11 no 5 pp 425ndash445 1997

[12] G Neudert and G Klebe ldquoDSX a knowledge-based scoringfunction for the assessment of protein-ligand complexesrdquo Jour-nal of Chemical Information and Modeling vol 51 no 10 pp2731ndash2745 2011

[13] IMuegge and Y CMartin ldquoA general and fast scoring functionfor protein-ligand interactions a simplified potential approachrdquoJournal of Medicinal Chemistry vol 42 no 5 pp 791ndash804 1999

[14] E CMeng B K Shoichet and I D Kuntz ldquoAutomated dockingwith grid-based energy evaluationrdquo Journal of ComputationalChemistry vol 13 no 4 pp 505ndash524 1992

[15] G Jones P Willett and R C Glen ldquoMolecular recognition ofreceptor sites using a genetic algorithm with a description ofdesolvationrdquo Journal ofMolecular Biology vol 245 no 1 pp 43ndash53 1995

[16] D R Houston and M D Walkinshaw ldquoConsensus dockingImproving the reliability of docking in a virtual screeningcontextrdquo Journal of Chemical Information andModeling vol 53no 2 pp 384ndash390 2013

[17] T Tuccinardi G Poli V Romboli A Giordano and A Mar-tinelli ldquoExtensive consensus docking evaluation for ligand poseprediction and virtual screening studiesrdquo Journal of ChemicalInformation and Modeling vol 54 no 10 pp 2980ndash2986 2014

[18] R Wang and S Wang ldquoHow does consensus scoring work forvirtual library screening An idealized computer experimentrdquoJournal of Chemical Information and Computer Sciences vol 41no 3ndash6 pp 1422ndash1426 2001

[19] N M OrsquoBoyle M Banck C A James C Morley T Vander-meersch and G R Hutchison ldquoOpen Babel an open chemicaltoolboxrdquo Journal of Cheminformatics vol 3 article 33 2011

[20] G Jones P Willett R C Glen A R Leach and R TaylorldquoDevelopment and validation of a genetic algorithm for flexibledockingrdquo Journal of Molecular Biology vol 267 no 3 pp 727ndash748 1997

[21] H Fan D Schneidman-Duhovny J J Irwin G Dong B KShoichet and A Sali ldquoStatistical potential for modeling andranking of protein-ligand interactionsrdquo Journal of ChemicalInformation and Modeling vol 51 no 12 pp 3078ndash3092 2011

[22] G M Morris H Ruth W Lindstrom et al ldquoAutoDock4 andAutoDockTools4 automated docking with selective receptorflexibilityrdquo Journal of Computational Chemistry vol 30 no 16pp 2785ndash2791 2009

[23] RHuey GMMorris A J Olson andD S Goodsell ldquoSoftwarenews and update a semiempirical free energy force field withcharge-based desolvationrdquo Journal of Computational Chemistryvol 28 no 6 pp 1145ndash1152 2007

[24] O Trott and A J Olson ldquoAuto Dock Vina improving the speedand accuracy of docking with a new scoring function efficientoptimization and multithreadingrdquo Journal of ComputationalChemistry vol 31 no 2 pp 455ndash461 2010

[25] R Wang Y Lu X Fang and S Wang ldquoAn extensive test of14 scoring functions using the PDBbind refined set of 800protein-ligand complexesrdquo Journal of Chemical Information andComputer Sciences vol 44 no 6 pp 2114ndash2125 2004

[26] A Gaulton L J Bellis A P Bento et al ldquoChEMBL a large-scalebioactivity database for drug discoveryrdquoNucleic Acids Researchvol 40 no 1 pp D1100ndashD1107 2012

[27] M M Mysinger M Carchia J J Irwin and B K ShoichetldquoDirectory of useful decoys enhanced (DUD-E) better ligandsand decoys for better benchmarkingrdquo Journal of MedicinalChemistry vol 55 no 14 pp 6582ndash6594 2012

[28] G Madhavi Sastry M Adzhigirey T Day R Annabhimojuand W Sherman ldquoProtein and ligand preparation parametersprotocols and influence on virtual screening enrichmentsrdquoJournal of Computer-Aided Molecular Design vol 27 no 3 pp221ndash234 2013


[29] T Sing O Sander N Beerenwinkel and T Lengauer ldquoROCRvisualizing classifier performance in Rrdquo Bioinformatics vol 21no 20 pp 3940ndash3941 2005

[30] T Cheng X Li Y Li Z Liu and R Wang ldquoComparativeassessment of scoring functions on a diverse test setrdquo Journalof Chemical Information and Modeling vol 49 no 4 pp 1079ndash1093 2009

[31] S Liu R Fu L-H Zhou and S-P Chen ldquoApplication ofconsensus scoring and principal component analysis for virtualscreening against 120573-secretase (BACE-1)rdquo PLoS ONE vol 7 no6 Article ID e38086 2012

Submit your manuscripts athttpwwwhindawicom

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Inorganic ChemistryInternational Journal of

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

International Journal ofPhotoenergy


Carbohydrate Chemistry

International Journal of


Journal of

Chemistry


Advances in

Physical Chemistry

Hindawi Publishing Corporationhttpwwwhindawicom

Analytical Methods in Chemistry

Journal of

Volume 2014

Bioinorganic Chemistry and ApplicationsHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

SpectroscopyInternational Journal of


The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Medicinal ChemistryInternational Journal of


Chromatography Research International


Applied ChemistryJournal of



Theoretical ChemistryJournal of


Journal of

Spectroscopy

Analytical ChemistryInternational Journal of


Journal of


Quantum Chemistry


Organic Chemistry International

ElectrochemistryInternational Journal of



CatalystsJournal of


theory and does not fit them to experimental data Theother class (empirical based) tries to find linear statisticalrelationship between the binding affinity and a number ofligand binding terms in a training set of ligand-proteincomplexes 3D structures with associated binding affinity data[4]

Some proposed consensus docking [16 17] and consensusrescoring [18] protocols are available The two consensusscoring methods so-called rank-by-number and rank-by-rank that had shown promising results [18] were also testedin this study In addition we suggested another methodprincipal component analysis (PCA) for performing a com-bination of multiple scoring functions to rescore and rerankthe compounds after virtual screening on MMP-12 target

Thework reported here seeks to address two questions (1)How can different scoring functions predict the experimentalbinding affinities for MMPs-inhibitor complexes (2) Dothe well-performed scoring functions have also reasonableperformance in an enrichment study on a member of MMPsfamily (MMP-12)

2 Methods

21 Preparation of Protein Test Set for Rescoring Study Thetest set consisted of 100MMPs-ligand complex structuresformed of 10 humanMMPs typesWe excluded the structureswith conflictive reported binding affinitiesThe 3D structureswere taken from PDB (Protein Data Bank) and then under-went some refinements Firstly water and other cocrystal-ized molecules were removed from the retrieved PDB filesThen the protein and corresponding ligand (inhibitor) wereextracted to separate PDB files The file formats changedto mol2 as it was a necessary step for some subsequentanalysis The hydrogens were added to both protein andligand molecules All of the selected PDB structures hadexperimentally determined Ki Kd or IC50 The logarithmof Ki Kd or IC50 was employed as experimental bindingaffinity in our study The detailed structural informationfor each is presented in Supplementary Material availableonline at httpdxdoiorg1011552014162150 For analysisthe pAffinity was employed as dependent variable instead ofbinding affinity which was defined as below

pAffinity = minusLog (Ki Kd or IC50) (120583M)

Metal ion (catalytic zinc ion) was saved as a part of themacromolecule The Gasteiger partial charge was assignedfor ligands All of the above procedures were done usingthe PyMOL (httpwwwpymolorg The PyMOLMolecularGraphics System version 12r3pre Schrodinger LLC) andOpen Babel Package (version 231 httpopenbabelorg)[19]

22 Scoring Functions Various scoring functions have beenevaluated in this study 11 scoring functions including thefive SYBYL built-in scoring functions (D-Score [14] G-Score[15 20] ChemScore [11] F-Score [9 10] and PMF Score[13]) two web based scoring functions (PoseScore [21] andRankScore [21]) two standalone scoring functions (DSX [12]

and X-Score [8]) and scoring functions of AutoDock41 [2223] and AutoDock Vina [24] were employed in this studyFurthermore two consensus scorings were applied on the setAll of the 11 scoring functions were used to compute bindingscores for ligand-protein interactions Some of the scoringfunctions were not able to compute reasonable binding scorefor all of the complexes It has been discussed earlier that suchan incompatibilitymay be raised by the fact that in some casesthere are clashes between protein and ligand molecules [25]However we did not penalize those scoring functions in ourstudy The pairwise deletion strategy was used to deal withmissing data

We employed previously defined consensus scoring(rank-by-number and rank-by-rank methods [18]) to sum-marize the results of multiple scoring functions In rank-by-rank method predicted individual rank was calculated as anaverage of ranks predicted by all the scoring functions Rank-by-number consensus score is an average of the Z-scaledscores calculated by each of the individual scoring functionsIndividual Z-scaled scoring function values (ZScore) arecomputed by

Zscore =(119891

119894minus 120583)

119878

(1)

where 119891119894is the scoring value of an individual scoring

function 120583 is the mean value and 119878 is the standard deviationof this scoring function for entire set

Finally the principal component analysis (PCA) wasapplied on various set of scores of enrichment study toevaluate the discrimination power of PCA on our evaluatedset of compounds PCA is a powerful tool for differentaspects of data evaluation including classification and patternrecognition It can simplify and reduce the dimensionality ofmultivariate data set while preserving as much of the relevantinformation The principal components (PCs) are linearcombinations of the original variables The first principalcomponent (PC1) has the largest possible variance Thesecond principal component (PC2) is uncorrelated to the firstone and it accounts for most of the remaining variance PCAmodel has been employed in our study for discrimination ofactives among decoys in virtual screening results based onobtained scores from various scoring functions In case of ourstudy the PCA was applied to generate linear combinationof different scores and extracted the main variation in thedata as PC1 and subsequent rescoring and reranking of virtualscreening results based on formulated PC1 The contributionof an individual score to the calculated PC can be describedby its loading value

23 Preparation of Docking Set for the Retrospective VirtualScreening on MMP-12 The inhibitors molecules of dockingset were prepared basically from the MMP-12 inhibitorsspreadsheet taken from ChEMBL database [26] Firstlyinactive and low active molecules (IC50 gt 100000 nM)were removed from the spreadsheet Cases with incompleteinformation (unitless activity or inexact IC50 values) andthose which did not fully satisfy Lipinskirsquosrule of five werealso excluded from the spreadsheet The edited spreadsheet


pAffinity 0

2

4

ChemScore

DSX

F-Score

PoseScore

RankScore

minus80

minus80

minus80

minus40

minus80 minus40

minus50

minus50

minus20

minus20


minus20

minus10

minus30

minus15 0

minus4

0 2 4minus4

minus30

minus15

0

minus30

minus100

minus250

minus100minus250






False positive rate

True

pos

itive

rate

00 02 04 06 08 10

00

02

04

06

08

10

AUC = 065

(a)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 063


(b)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(c)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(d)




119901) and






00 02 04 06 08 10

True

pos

itive

rate

00

02

04

06

08

10

AUC = 073

False positive rate

(a)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 068

(b)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 069

(c)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 076

(d)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 079

(e)







) (2)



3 Results






pAffinity





4

2

2

0

0

minus2

minus2minus4

PC2

PC1



4 Discussion








Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0



Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276










Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of


pAffinity 0

2

4

ChemScore

DSX

F-Score

PoseScore

RankScore

minus80

minus80

minus80

minus40

minus80 minus40

minus50

minus50

minus20

minus20


minus20

minus10

minus30

minus15 0

minus4

0 2 4minus4

minus30

minus15

0

minus30

minus100

minus250

minus100minus250






False positive rate

True

pos

itive

rate

00 02 04 06 08 10

00

02

04

06

08

10

AUC = 065

(a)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 063


(b)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(c)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(d)




119901) and






00 02 04 06 08 10

True

pos

itive

rate

00

02

04

06

08

10

AUC = 073

False positive rate

(a)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 068

(b)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 069

(c)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 076

(d)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 079

(e)







) (2)



3 Results






pAffinity





4

2

2

0

0

minus2

minus2minus4

PC2

PC1



4 Discussion








Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0



Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276










Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of


False positive rate

True

pos

itive

rate

00 02 04 06 08 10

00

02

04

06

08

10

AUC = 065

(a)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 063


(b)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(c)

True

pos

itive

rate

00

02

04

06

08

10

AUC = 065


(d)




119901) and






00 02 04 06 08 10

True

pos

itive

rate

00

02

04

06

08

10

AUC = 073

False positive rate

(a)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 068

(b)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 069

(c)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 076

(d)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 079

(e)







) (2)



3 Results






pAffinity





4

2

2

0

0

minus2

minus2minus4

PC2

PC1



4 Discussion








Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0



Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276










Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of


00 02 04 06 08 10

True

pos

itive

rate

00

02

04

06

08

10

AUC = 073

False positive rate

(a)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 068

(b)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 069

(c)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 076

(d)


True

pos

itive

rate

00

02

04

06

08

10

AUC = 079

(e)







) (2)



3 Results






pAffinity





4

2

2

0

0

minus2

minus2minus4

PC2

PC1



4 Discussion








Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0



Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276










Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of






) (2)



3 Results






pAffinity





4

2

2

0

0

minus2

minus2minus4

PC2

PC1



4 Discussion








Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0



Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276










Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of







Glide (HTS)

Glide 0653348 2142857 25 1785714 3571429F-Score 0242776 0 0 0 0



Autodock 0646775 1964286 2142857 7142857 1071429Vina 0560097 125 1071429 0 0

Glide (SP)

Glide 0730062 2758621 3448276 862069 137931F-Score 0409975 0172414 0344828 0 0



Autodock 0690234 1896552 3793103 1206897 137931Vina 0592455 1551724 137931 1724138 3448276PC1 079963 3448276 5862069 1896552 3448276










Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of





Acknowledgment


References










































Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of














Journal of

Chemistry


Advances in

Physical Chemistry



Journal of

Volume 2014














Journal of

Spectroscopy



Journal of


Quantum Chemistry






CatalystsJournal of

Research Article Evaluation of 11 Scoring Functions...

Documents

Transcript of Research Article Evaluation of 11 Scoring Functions...