Gui

IEEETRANSACTIONSONEMERGINGTOPICSINCOMPUTATIONALINTELLIGENCE,VOL.1,NO.3,JUNE2017145WebAPIRec:RecommendingWebAPIstoSoftwareProjectsviaPersonalizedRankingFerdianThung,RichardJ.Oentaryo,DavidLo,andYuanTianAbstract—Applicationprogramminginterfaces(APIs)offeraplethoraoffunctionalitiesfordeveloperstoreusewithoutrein-ventingthewheel.IdentifyingtheappropriateAPIsgivenaprojectrequirementiscriticalforthesuccessofaproject,asmanyfunc-tionalitiescanbereusedtoachievefasterdevelopment.However,themassivenumberofAPIswouldoftenhinderthedevelopers’abilitytoquicklyfindtherightAPIs.Inthislight,weproposeanew,automatedapproachcalledWebAPIRecthattakesasinputaprojectprofileandoutputsarankedlistofwebAPIsthatcanbeusedtoimplementtheproject.Atitsheart,WebAPIRecemploysapersonalizedrankingmodelthatrankswebAPIsspecific(person-alized)toaproject.BasedonthehistoricaldataofwebAPIusages,WebAPIReclearnsamodelthatminimizestheincorrectorderingofwebAPIs,i.e.,whenausedwebAPIisrankedlowerthananun-used(oranot-yet-used)webAPI.Wehaveevaluatedourapproachonadatasetcomprising9883webAPIsand4315webapplica-tionprojectsfromProgrammableWebwithpromisingresults.For84.0%oftheprojects,WebAPIRecisabletosuccessfullyreturncorrectAPIsthatareusedtoimplementtheprojectsinthetop-fivepositions.ThisissubstantiallybetterthantherecommendationsprovidedbyProgrammableWeb’snativesearchfunctionality.We-bAPIRecalsooutperformsMcMillanetal.’sapplicationsearchengineandpopularity-basedrecommendation.IndexTerms—Personalizedranking,recommendationsystem,WebAPI.I.INTRODUCTIONDEVELOPINGasoftwareprojectisnotaneasytask,ascustomersusuallydemandmanyfeaturestobeimple-mented.Toaidtheirjobs,developersoftenusethirdpartyli-brariesthatproviderelevantfunctionalitiesthroughapplicationprogramminginterfaces(APIs)[20].APIsprovidefunction-alitiesforcertaintasksthatcanbe(re)usedbydeveloperstoexpediteprojectdevelopments.UsingAPIspreventsdevelop-ersfromreinventingthewheel,thusallowingthemtofocusonmoreimportanttasksathand.Hence,itisusuallyagoodideatofindsuitableAPIsandusetheminaproject.Moreover,bybuildinguponexistingAPIs,featurescanbecompletedfasterasmanyAPIsarewelldesignedandtheirfunctionalitieshavebeentestedbymanyclientapplications.ManuscriptreceivedNovember6,2016;revisedMarch28,2017;acceptedApril9,2017.DateofcurrentversionMay23,2017.ThisworkwassupportedbytheSingaporeMinistryofEducation(MOE)AcademicResearchFund(AcRF)Tier1grant.(Correspondingauthor:FerdianThung.)TheauthorsarewiththeSchoolofInformationSystems,SingaporeMan-agementUniversity,Singapore178902(e-mail:ferdiant.2013@smu.edu.sg;roentaryo@smu.edu.sg;davidlo@smu.edu.sg;yuan.tian.2012@smu.edu.sg).DigitalObjectIdentifier10.1109/TETCI.2017.2699222FindingtherightAPIs,however,isnotasstraightforwardasitmayseem.ThousandsofAPIshavebeendevelopedtocaterforvariouspurposes,anddevelopersareoftenunawareoftheexistenceofAPIssuitableforaparticularfeatureoftheprojectthattheyaredeveloping.Ofcourse,someAPIsarewellknown,butthemajorityofAPIsdonotenjoysuchluxury[41].More-over,althoughsomeAPIchoicesareobvious(e.g.,ifwewanttoaddFacebooksupport,wedonothavemuchchoiceexceptusingFacebookAPI),thenumberofsuchobviousAPIchoicesisnotmany.Ingeneral,findingAPIsforvariousneeds,e.g.,mu-sicmanagement,typicallyinvolvesmanypossiblealternativesandthechoicewilllargelydependontheprojectrequirement.SomeexamplesofmusicmanagementAPIsareMusicBrainz,Soundiiz,andToma.hk.MusicBrainzcanbeusedtoextractmusicmetadata,Soundiizcanbeusedtocreatemusicplaylist,andToma.hkcanbeusedtoplaymusicfromdifferentsources.ThechoiceofwhichAPItousewoulddependontheneedandrequirementofatargetapplication.ThesefactsnecessitatethedevelopmentofanautomatedrecommendationsystemthatcanhelpdevelopersfindAPIsthattheyneedfortheirprojects.Inthispaper,weproposeanewapproachdubbedWe-bAPIRectorecommendwebAPIsbasedonprojectprofiles.InWebAPIRec,wedefineaprojectprofileasthetextualde-scriptionandkeywordsoftheproject.ItisworthnotingthatourapproachdoesnotrequirethewebAPIsourcecodetobeavailable.ThisrequirementisimportantasmanyproprietaryyetusefulwebAPIsdonotcomewithsourcecode.ExamplesofwebAPIsincludeGoogleMaps,BingMaps,YouTube,andLast.fm,whichareoftenusedaskeycomponentsinmanyprojects.ThesewebAPIsofferessentialfunctionalitiesandusuallycomewithdatathatcanbeusedtocompletevariousfeaturesinamoreefficientway.Givenanewprojectprofile,ourapproachrecommendswebAPIsbyanalyzingpastprojectsandthewebAPIsthattheyuse.WebAPIRecconsistsoftwophases:traininganddeploy-mentphase.Inthetrainingphase,WebAPIRecanalyzespastprojectsandtheirusedwebAPIstobuildapersonalizedrank-ingmodelthataimstominimizerankingerrorsinthetrainingdata.PersonalizedrankingmeansthattherankingofwebAPIsisspecifictoeachproject,andthusdifferentprojectshavedif-ferentwebAPIrankings.ArankingerroroccursifawebAPIusedbysomeprojectisrankedlowerthananunusedwebAPI.Inthedeploymentphase,WebAPIRecanalyzestheprofileofanewprojectusingthetrainedmodel.ItthenassignsarelevancyscoretoeachwebAPI.AhigherrelevancyscoreimpliesthattheAPIisdeemedmorerelevant.Finally,WebAPIRecranks2471-285X©2017IEEE.Personaluseispermitted,butrepublication/redistributionrequiresIEEEpermission.Seehttp://www.ieee.org/publicationsstandards/publications/rights/index.htmlformoreinformation.146IEEETRANSACTIONSONEMERGINGTOPICSINCOMPUTATIONALINTELLIGENCE,VOL.1,NO.3,JUNE2017thewebAPIsinadescendingorderoftheirrelevancyandre-turnsalistofrecommendedwebAPIs.ThislistisintendedtohelpdeveloperstopickwebAPIsmoreefficiently.ItdoesnotexplicitlyreturnacompositionofwebAPIsfortheproject.Toillustratetheusefulnessofourapproach,considerthefollowingscenario.AdeveloperhasnoideawhatwebAPItousefordevelopinghisapplication.Normally,hewillsurfthewebtofindasuitablewebAPI.However,notallwebpagesarerelatedtowebAPIsand,eveniftheyare,hestillneedstoreadthewebAPIdescriptionsanddecidewhethereachofthemisusableornot.IfhethinksawebAPIisusable,hewilltrythewebAPI.Still,aftertryingit,thewebAPImaynotmeethisexpectations.TheremaybenumeroustrialsanderrorsbeforehefindsthewebAPIthatbestmatcheshisneeds.WethusdevelopWebAPIRectoprovideanautomatedrecommendersystemthatcanhelpreducetheeffortneededbyadevelopertofindtherightwebAPI.TovalidateourWebAPIRecapproach,weusethewebapplicationprojectsandwebAPIsextractedfromthePro-grammableWebwebsite.1Thisdatasethasatotalof9,883webAPIsand4,315projects.WeevaluatetheeffectivenessofourapproachintermsofHit@N,MAP@N,MAP,andMRR,whicharepopularmetricsforevaluatingrecommendersystems[22],[25],[26],[29],[33],[36],[42].OurexperimentshowsthatourapproachachievesHit@5,Hit@10,MAP@5,MAP@10,MAP,andMRRscoresof0.840,0.880,0.697,0.687,0.626,and0.750,respectively.TheHit@5scoreimpliesthatfor84.0%oftheprojects,WebAPIReccansuccessfullyreturncorrectwebAPIs,whichareusedtoimplementtheprojectsatthetop-5positions.WehavecomparedtheeffectivenessofourapproachagainstthenativesearchfunctionalityofProgrammableWeb.Weinputtheprofileofaproject(infullorinpart)andevaluatethelistoflibrariesthatthesearchfunctionalityreturns.However,wefindthatthesearchfunctionalityislimitedanditachievesonlyHit@5,Hit@10,MAP@5,MAP@10,MAP,andMRRscoresofatmost0.046,0.047,0.041,0.042,0.042,and0.038re-spectively.WehavealsocomparedourapproachagainstseveralotherbaselinesbasedonMcMillanetal.’sapplicationsearchen-gine[17]andpopularity-basedrecommendation.Wefindthatourapproachoutperformsallofthem.ThebestperformingbaselineachievessignificantlylowerHit@5,Hit@10,MAP@5,MAP@10,MAP,andMRRscoresof0.591,0.675,0.414,0.417,0.363,and0.476respectively.ComparingtheHit@5scoresofWebAPIRecwiththoseofthebaselines,WebAPIRecoutper-formsthebestperformingbaselinebyasubstantialmarginof42.1%.Wesummarizeourmaincontributionsasfollows:1)WeproposeanewapproachnamedWebAPIRecthatrecommendswebAPIsbyanalyzingpastsimilarprojectsandwebAPIsthattheyuse,andmodeltherecommenda-tiontaskasarankingproblem.Toourbestknowledge,WebAPIRecisthefirstapproachthatemploysaperson-alizedrankingmodeltolearnthecorrectorderingofwebAPIsforaspecificproject.Ourapproachrecommends1http://www.programmableweb.com/TABLEIASAMPLEAPIPROFILELast.fmAPIShortDescriptionOnlineaudioserviceLongDescriptionTheLast.fmAPIgivesuserstheabilitytobuildprogramsusingLast.fmdata,whetherontheweb,thedesktopormobiledevices.TheRESTfulAPIallowsforreadandwriteaccesstothefullslateoflast.fmmusicdataresources-albums,artists,playlists,events,users,andmore.ItallowsuserstocallmethodsthatrespondineitherXMLorJSON.KeywordsmusicTABLEIIASAMPLEWEBAPPLICATIONPROJECTPROFILEIvyFM-DiscovernewmusiceverydayLongDescriptionDiscovernewmusiceverydaywithIvyFM.Itplaysgreatsongscontinuouslyineachgenrefromthebestartistsintheworld.Selectyourchannel,listengreatmusic,shareitandenjoy.Keywordsmusic,streamingAPIsLast.fm,Youtubetop-kwebAPIsthatcanmostlikelybeusedtoimplementtheproject.2)WehavecomprehensivelyevaluatedourapproachonadatasetextractedfromProgrammableWeb.Ourexperi-mentshowsthatWebAPIRecisabletoachievesatisfac-toryHit@N,MAP,MAP@NandMRRscores.TheseresultsaresubstantiallybetterthantheresultsforthePro-grammableWeb’snativesearchfunctionality,McMillanetal.’sapplicationsearch,andpopularity-basedrecom-mendation.II.PRELIMINARIESA.ProgrammableWebDatasetProgrammableWebisawebsitethatcollectsinformationaboutAPIsreleasedaswebservicesandwebapplicationprojectsthatusethem.ItcontainsacollectionofthousandsofAPIsimplementingvariousfunctionalities.TableIshowstheprofileofanAPIinourdataset.TheprofileofanAPIcontainsseveralpiecesofinformationsuchasitsname,shortdescription(i.e.,summary),longdescription,andkeywords(i.e.,tags).Inthispaper,werefertoamergedtextthatcontainsthename,shortdescription,andlongdescriptionofanAPIasthetextualdescriptionoftheAPI.WerepresenteachAPIbyitstextualdescriptionsandkeywords.ProgrammableWebcontainsthousandsofwebapplicationprojects.TableIIshowstheprofileofaprojectinourdataset.Theprofilecontainsseveralpiecesofinformationincluding:alongdescriptionoftheprojectandtherelevantkeywords(i.e.,tags).AwebapplicationprojectdoesnothaveashortdescriptioninProgrammableWeb.Werefertothelongdescriptionofawebapplicationprojectasitstextualdescription.SimilartoanAPI,werepresenteachwebapplicationprojectbyitstextualdescriptionsandkeywords.THUNGetal.:WEBAPIREC:RECOMMENDINGWEBAPISTOSOFTWAREPROJECTSVIAPERSONALIZEDRANKING147B.IR&NLPTechniquesWebAPIRecmakeuseofinformationretrieval(IR)andnaturallanguageprocessing(NLP)techniques.Theyincludeparts-of-speech(POS)taggingtechniquefromNLPandtextpreprocessing,vectorspacemodel(VSM),andcosinesimilar-itytechniquesfromIR.Wedescribeeachofthembelow.1)Parts-of-SpeechTagging:POStaggingisanaturallan-guageprocessingtechniquethatassignsapartofspeechlabeltoeverywordinatextualdocument(inourcase:atextualdescriptionofanAPIoraproject).Commonpartsofspeechinclude:noun,verb,adjective,adverb,etc.VariousalgorithmshavebeenproposedtoperformPOStagging.OneofthemostadvancedfamilyofPOStaggingalgorithmsisstochasticPOStaggers,whichconsiderthecontextofawordtodecideitsPOStag[5],[27],[35].Inthiswork,weusethepopularStanford(stochastic)POStagger[35],whichhasalsobeenusedinmanysoftwareengineeringstudies,e.g.,[4].2)TextPreprocessing:Inthisphase,webreakatextdataintoamoresuitablerepresentationthatcanlaterbeconvertedintoanIRmodel.Also,sincetextdataareoftennoisy(i.e.,itcontainsmanyunimportantwords,closelyrelatedwordsthatareindifferenttenses,etc.),additionalpreprocessingstepsareneeded.Inthiswork,thepreprocessingstepsare:1)Tokenization:Itisaprocessofbreakingatextdocumentintoitsconstituentwordtokens.Delimiters,suchaspunc-tuationmarksandwhitespaces,areusedasboundariesbetweenonewordtokenandanother.Attheendofthisprocess,eachtextdocumentisrepresentedbyabag(ormulti-set)ofwordtokens.2)StopWordRemoval:Thisinvolvesremovingwordsthatappearveryfrequentlyandthushelpverylittleindiscrimi-natingonedocumentfromanother.Examplesofthesestopwordsinclude:“I”,“you”,“are”,etc.Inthiswork,weusethelistofEnglishstopwordsfromhttp://jmlr.org/papers/volume5/lewis04a/a11-smart-stop-list/english.stop.3)Stemming:Itisaprocessofconvertingawordtoitsbaseform,typicallybyremovingasuffixfromtheword.Forexample,usingstemming,words“reads”and“reading”wouldallbeconvertedto“read”.Withoutstemming,thesewordswillbeconsideredasdifferentwordsaltogether.WeusethePorterstemmingmethod[19]toreduceeachwordtoitsstemmedform.3)VectorSpaceModel:Textpreprocessingwillconvertatextualdocument—i.e.,aprojectorAPIdescription—intoabagofwords.Inthebagofwordsrepresentation,importantwordsarenotdistinguishedfromunimportantones.Toconsidertherelativeimportanceofwords,IRresearchersproposedthevec-torspacemodel(VSM),whichrepresentsatextualdocumentasavectorofweights[16].Eachweightcorrespondstoawordandindicatestherelativeimportanceofthatword.VSMiscon-structedbyanalyzingmanybagsofwordsrepresentingasetofdocumentsinacorpus(i.e.,acollectionofprojectorAPIdescriptions).Manyweightingschemescanbeusedtoinfertheimportanceofaword.Inthiswork,weusethepopulartermfrequency-inversedocumentfrequency(tf-idf)scheme[21].Thisschemeisbasedontwointuitions.Firstly,words(terms)thatappearfrequentlyinadocumentaremoreimportantthanwordsthatappearrarelyinit.Forexample,adocumentthathasmanyoc-currencesoftheword“Texas”islikelytoberelatedto“Texas”.Secondly,wordsthatappearinmanydocumentsarelessabletodistinguishonedocumentfromanother,andshouldbegivenasmallerweight.Forexample,ifalldocumentsinacorpuscon-tainstheword“Software”,thenthiswordisunimportant,asitcannotdistinguishonedocumentfromanother.GivenadocumentDinacorpusC,wecancomputetheweightofeverywordthatappearsinD.Tocomputethetermfrequency(tf)ofaworddinadocumentD,wesimplycounthowmanytimesthewordappearinD.Tocomputetheinversedocumentfrequency(idf)ofaworddincorpusC,wefirstcomputethedocumentfrequency(df)ofd,whichisthenumberofdocumentsinCthatcontainsd.WethennormalizethisnumberbydividingitbythenumberofdocumentsinC.Theidfissimplythelogarithmofthereciprocalofthisnormalizednumber.Inturn,thetf-idfweightofaworddistheproductofitstermfrequencyandinversedocumentfrequency.Formally,thetf-idfweightofaworddinadocumentDofacorpusC(denotedasw(d,D,C))is:w(d,D,C)=TF(d,D)×IDF(d,C)=TF(d,D)×log(NCDF(d,C))(1)whereTF(d,D)referstothetermfrequencyofwordd,NCreferstothenumberofdocumentsincorpusC,andDF(d,C)referstothedocumentfrequencyofwordd.WedenotetheVSMrepresentationofadocumentDconsid-eringacorpusCasVSMC(D).Inourimplementation,weuseasparsematrixrepresentationfortheAPIandprojectdocuments(i.e.,weonlystorethenon-zeroentries).4)CosineSimilarity:Tocomputethesimilarityoftwodoc-uments,wecantaketheirVSMrepresentationsandcomparethetwovectorsofweightsbycomputingtheircosinesimilar-ity[16].ConsidertwovectorsaandbofsizeN;theircosinesimilarityis:Sim(a,b)=∑Ni=1wi,a×wi,b√∑Ni=1w2i,a√∑Ni=1w2i,b(2)wherewi,areferstotheithweightinvectora.III.APIRECOMMENDATIONSYSTEMThearchitectureofWebAPIRecisoutlinedinFig.1.Ittakesasinput:anewprojectprofile,asetofAPIprofiles,andasetofpastprojects.FromthenewprojectprofileandeachAPIprofile,WebAPIRectakesitstextualdescriptionsandkeywords.Fromeachpastproject,WebAPIRectakesitstextualdescriptions,keywords,andAPIsthatwasused.WebAPIRecanalyzestheseinputsandfinallyproducesarankedlistofAPIstoberecom-mendedtothetargetproject.WebAPIRechastwooperatingphases:trainingphaseanddeploymentphase.Intheformerphase,WebAPIRectakesasinputasetofAPIprofilesandasetofpastprojectsalongwiththeAPIsthattheyuse.Itthenlearnsapersonalizedrankingmodel148IEEETRANSACTIONSONEMERGINGTOPICSINCOMPUTATIONALINTELLIGENCE,VOL.1,NO.3,JUNE2017Fig.1.ArchitectureofWebAPIRec.(seeSectionIV).Inthedeploymentphase,ittakesasinputthenewprojectprofile,asetofAPIprofiles,andthetrainedpersonalizedAPIrankingmodel.ItthenappliesthemodeltothenewprojectprofileandoutputsarankedlistofrecommendedAPIs.Totrainthepersonalizedrankingmodelinthetrainingphase,WebAPIRecneedstorepresenttheprofileofeachpastproject(i.e.,trainingdata)asafeaturevector.WebAPIRecfirstidenti-fiesnounsfromthetextualdescriptionsusingtheStanfordPOStagger.Thesenounscarrymoremeaningthanotherkindsofwords,asadvocatedin[6],[28].WebAPIRecthencombinestheextractednounswiththekeywords,removestopwords,stemeachoftheremainingwords,andconstructaVSMfeaturevec-tor.ThesameprocesscanbedonetoconvertanAPIprofileintoafeaturevector.TheseprojectandAPIfeaturevectorsarethenusedtoconstructasetoftrainingtriples(p,a,a′),whichservesasinputtothepersonalizedrankingmodel.Inatriple(p,a,a′),pisthefeaturevectorofaprojectinthetrainingdata,aisthefeaturevectorofanAPIthatisusedbyprojectp,anda′isthefeaturevectorofanAPInotusedbyprojectp.Attheendofthetrainingphase,therankingmodelwillhavelearnedhowtorankalistofAPIsbasedontheirfeaturevectorsandthefeaturevectorofthetargetproject.Inthedeploymentphase,similartothetrainingphase,We-bAPIRecfirstconstructsfeaturevectorsfromanewprojectprofileandAPIprofiles.UsingthelearnedpersonalizedAPIrankingmodel,WebAPIReccomputestherelevancyofeachAPIandsorttheAPIs(indescendingorder)basedonthesescores.ThesortedAPIsareoutputasalistofrecommendedAPIs.IV.PERSONALIZEDRANKINGWebAPIReccaststheAPIrecommendationproblemasapersonalizedrankingtask.Underthisformulation,ourgoalistoprovidearankedlistofAPIsthatarespecific(i.e.,personal-ized)toeachproject.Specifically,weconsiderthesettingwhereWebAPIRectakesasinputasetoftrainingtriples(p,a,a′)wherepisafeaturevectorofaproject,aisafeaturevectorofanAPIlibraryusedinp,anda′isafeaturevectorofanAPInotusedinp.Basedonthesetrainingtriples,apersonalizedrankingmodellearnshowtorankAPIsforatargetprojectbyjointlyutilizingtheirfeaturevectors.A.NotationandDesiderataWefirstdefineournotationshere.LetPbethesetofallsoftwareprojectsandAthesetofallwebAPIs.Accordingly,therecommendationtaskistoprovideaspecificprojectp∈Pwithatotalordering>pofallAPIsa∈A.Essentially,asoundranking>prequiresseveralcriteriatobefulfilled:∀a,a′∈A:a=a′⇒a>pa′∨a′>pa(3)∀a,a′∈A:a>pa′∧a′>pa⇒a=a′(4)∀a,a′,a′′∈A:a>pa′∧a′>pa′′⇒a>pa′′(5)Theformulae(3)–(5)correspondtotheso-calledtotality(i.e.,aanda′shouldbecomparable),anti-symmetry(i.e.,unlessa=a′,aanda′shouldhavedifferentranks),andtransitivityproperties(i.e.,ifarankshigherthan(orequalto)a′anda′rankshigherthan(orequalto)a′′,thenarankshigherthan(orequalto)a′′),respectively[8].ThepersonalizedrankingmodelwillinturnlearntorankAPIsbasedonasetoftrainingtriplesD:D={(p,a,a′)|a∈Ap∧a′∈A\Ap}(6)whereApreferstothesetofAPIsusedbyaprojectp,andeachelement/triple(p,a,a′)∈DimpliesthatprojectpprefersAPIaoverAPIa′.B.RankingModelOurpersonalizedrankingmodelcomputesacompatibilityscorebetweenaprojectpandanAPIa.Specifically,forany(p,a)pair,ourmodeldefinesthecompatibilityscoref(p,a)asaweightedsumofJinteractionfeatures:f(p,a)=J∑j=1θjxj(p,a)(7)whereeachfeaturexj(p,a)quantifiesaspecifictypeofinter-actionbetweentheprojectpandAPIa,andθjistheweightparametertobeidentifiedbythetrainingprocedure.Furtherde-tailsonwhichfeaturesxj(p,a)weuseintherecommendationtaskwillbegivenlaterinSectionV.Aftertrainingiscompleted,wecancomputeforanewprojectp′thescoref(p′,a)usingtheidentifiedweightparametersθjandfeatureθj(p′,a).Wemaythensortthescoresf(p′,a)com-putedforallAPIsa∈A,andinturnproducetherankedlistofAPIstoberecommendedforp′.C.LossFunctionFormulationTosolvetheAPIrecommendationtask,weneedtoformu-latethelossfunctionthatguidesthetrainingprocessofourrankingmodel.WedefinealossfunctionL(.)toevaluatethegoodnessofthecompatibilityscoref(p,a),andthenfindtheoptimalweightparametersthatminimizeL(.).Asmentioned,THUNGetal.:WEBAPIREC:RECOMMENDINGWEBAPISTOSOFTWAREPROJECTSVIAPERSONALIZEDRANKING149featurevectorsx(p,a)=[x1(p,a),...,xj(p,a),...,xJ(p,a)]arerankedaccordingtof(p,a).Thus,ifthefeaturevectorswithhigherscoresf(p,a)areactuallyrelevant(i.e.,APIaisactuallyusedbyprojectp),thelossshouldbesmall;otherwise,thelossshouldbelarge.Inthiswork,wefocusonarankinglossfunctionoftheformL(y(a>pa′),f(a>pa′)),wheref(a>pa&