rules_of_ml

RulesofMachineLearning:BestPracticesforMLEngineeringMartinZinkevichThisdocumentisintendedtohelpthosewithabasicknowledgeofmachinelearninggetthebenefitofbestpracticesinmachinelearningfromaroundGoogle.Itpresentsastyleformachinelearning,similartotheGoogleC++StyleGuideandotherpopularguidestopracticalprogramming.Ifyouhavetakenaclassinmachinelearning,orbuiltorworkedonamachine­learnedmodel,thenyouhavethenecessarybackgroundtoreadthisdocument.TerminologyOverviewBeforeMachineLearningRule1:Don’tbeafraidtolaunchaproductwithoutmachinelearning.Rule2:Makemetricsdesignandimplementationapriority.Rule3:Choosemachinelearningoveracomplexheuristic.MLPhaseI:YourFirstPipelineRule4:Keepthefirstmodelsimpleandgettheinfrastructureright.Rule5:Testtheinfrastructureindependentlyfromthemachinelearning.Rule6:Becarefulaboutdroppeddatawhencopyingpipelines.Rule7:Turnheuristicsintofeatures,orhandlethemexternally.MonitoringRule8:Knowthefreshnessrequirementsofyoursystem.Rule9:Detectproblemsbeforeexportingmodels.Rule10:Watchforsilentfailures.Rule11:Givefeaturesetsownersanddocumentation.YourFirstObjectiveRule12:Don’toverthinkwhichobjectiveyouchoosetodirectlyoptimize.Rule13:Chooseasimple,observableandattributablemetricforyourfirstobjective.Rule14:Startingwithaninterpretablemodelmakesdebuggingeasier.Rule15:SeparateSpamFilteringandQualityRankinginaPolicyLayer.MLPhaseII:FeatureEngineeringRule16:Plantolaunchanditerate.Rule17:Startwithdirectlyobservedandreportedfeaturesasopposedtolearnedfeatures.Rule18:Explorewithfeaturesofcontentthatgeneralizeacrosscontexts.Rule19:Useveryspecificfeatureswhenyoucan.Rule20:Combineandmodifyexistingfeaturestocreatenewfeaturesinhuman­understandableways.Rule21:Thenumberoffeatureweightsyoucanlearninalinearmodelisroughlyproportionaltotheamountofdatayouhave.Rule22:Cleanupfeaturesyouarenolongerusing.HumanAnalysisoftheSystemRule23:Youarenotatypicalenduser.Rule24:Measurethedeltabetweenmodels.Rule25:Whenchoosingmodels,utilitarianperformancetrumpspredictivepower.Rule26:Lookforpatternsinthemeasurederrors,andcreatenewfeatures.Rule27:Trytoquantifyobservedundesirablebehavior.Rule28:Beawarethatidenticalshort­termbehaviordoesnotimplyidenticallong­termbehavior.Training­ServingSkewRule29:Thebestwaytomakesurethatyoutrainlikeyouserveistosavethesetoffeaturesusedatservingtime,andthenpipethosefeaturestoalogtousethemattrainingtime.Rule30:Importanceweightsampleddata,don’tarbitrarilydropit!Rule31:Bewarethatifyoujoindatafromatableattrainingandservingtime,thedatainthetablemaychange.Rule32:Re­usecodebetweenyourtrainingpipelineandyourservingpipelinewheneverpossible.Rule33:IfyouproduceamodelbasedonthedatauntilJanuary5th,testthemodelonthedatafromJanuary6thandafter.Rule34:Inbinaryclassificationforfiltering(suchasspamdetectionordetermininginterestinge­mails),makesmallshort­termsacrificesinperformanceforverycleandata.Rule35:Bewareoftheinherentskewinrankingproblems.Rule36:Avoidfeedbackloopswithpositionalfeatures.Rule37:MeasureTrainingServingSkew.MLPhaseIII:SlowedGrowth,OptimizationRefinement,andComplexModelsRule38:Don’twastetimeonnewfeaturesifunalignedobjectiveshavebecometheissue.Rule39:Launchdecisionswilldependuponmorethanonemetric.Rule40:Keepensemblessimple.Rule41:Whenperformanceplateaus,lookforqualitativelynewsourcesofinformationtoaddratherthanrefiningexistingsignals.Rule42:Don’texpectdiversity,personalization,orrelevancetobeascorrelatedwithpopularityasyouthinktheyare.Rule43:Yourfriendstendtobethesameacrossdifferentproducts.Yourintereststendnottobe.RelatedWorkAcknowledgementsAppendixYouTubeOverviewGooglePlayOverviewGooglePlusOverviewTerminologyThefollowingtermswillcomeuprepeatedlyinourdiscussionofeffectivemachinelearning:Instance:Thethingaboutwhichyouwanttomakeaprediction.Forexample,theinstancemightbeawebpagethatyouwanttoclassifyaseitheraboutcatsornotaboutcats.Label:Ananswerforapredictiontask­­eithertheanswerproducedbyamachinelearningsystem,ortherightanswersuppliedintrainingdata.Forexample,thelabelforawebpagemightbeaboutcats.Feature:Apropertyofaninstanceusedinapredictiontask.Forexample,awebpagemighthaveafeaturecontainstheword'cat'.FeatureColumn:Asetofrelatedfeatures,suchasthesetofallpossiblecountriesinwhich1usersmightlive.Anexamplemayhaveoneormorefeaturespresentinafeaturecolumn.Afeaturecolumnisreferredtoasa“namespace”intheVWsystem(atYahooMicrosoft),orafield.Example:Aninstance(withitsfeatures)andalabel.Model:Astatisticalrepresentationofapredictiontask.Youtrainamodelonexamplesthenusethemodeltomakepredictions.Metric:Anumberthatyoucareabout.Mayormaynotbedirectlyoptimized.Objective:Ametricthatyouralgorithmistryingtooptimize.Pipeline:Theinfrastructuresurroundingamachinelearningalgorithm.Includesgatheringthedatafromthefrontend,puttingitintotrainingdatafiles,trainingoneormoremodels,andexportingthemodelstoproduction.OverviewTomakegreatproducts:domachinelearninglikethegreatengineeryouare,notlikethegreatmachinelearningexpertyouaren’t.1Google­specificterminology.Mostoftheproblemsyouwillfaceare,infact,engineeringproblems.Evenwithalltheresourcesofagreatmachinelearningexpert,mostofthegainscomefromgreatfeatures,notgreatmachinelearningalgorithms.So,thebasicapproachis:1.makesureyourpipelineissolidendtoend2.startwithareasonableobjective3.addcommon­sensefeaturesinasimpleway4.makesurethatyourpipelinestayssolid.Thisapproachwillmakelotsofmoneyandormakelotsofpeoplehappyforalongperiodoftime.Divergefromthisapproachonlywhentherearenomoresimpletrickstogetyouanyfarther.Addingcomplexityslowsfuturereleases.Onceyou'veexhaustedthesimpletricks,cutting­edgemachinelearningmightindeedbeinyourfuture.SeethesectiononPhaseIIImachinelearningprojects.Thisdocumentisarrangedinfourparts:1.Thefirstpartshouldhelpyouunderstandwhetherthetimeisrightforbuildingamachinelearningsystem.2.Thesecondpartisaboutdeployingyourfirstpipeline.3.Thethirdpartisaboutlaunchinganditeratingwhileaddingnewfeaturestoyourpipeline,howtoevaluatemodelsandtraining­servingskew.4.Thefinalpartisaboutwhattodowhenyoureachaplateau.5.Afterwards,thereisalistofrelatedworkandanappendixwithsomebackgroundonthesystemscommonlyusedasexamplesinthisdocument.BeforeMachineLearningRule1:Don’tbeafraidtolaunchaproductwithoutmachinelearning.Machinelearningiscool,butitrequiresdata.Theoretically,youcantakedatafromadifferentproblemandthentweakthemodelforanewproduct,butthiswilllikelyunderperformbasicheuristics.Ifyouthinkthatmachinelearningwillgiveyoua100%boost,thenaheuristicwillgetyou50%ofthewaythere.Forinstance,ifyouarerankingappsinanappmarketplace,youcouldusetheinstallrateornumberofinstalls.Ifyouaredetectingspam,filteroutpublishersthathavesentspambefore.Don’tbeafraidtousehumaneditingeither.Ifyouneedtorankcontacts,rankthemostrecentlyusedhighest(orevenrankalphabetically).Ifmachinelearningisnotabsolutelyrequiredforyourproduct,don'tuseituntilyouhavedata.Rule2:First,designandimplementmetrics.Beforeformalizingwhatyourmachinelearningsystemwilldo,trackasmuchaspossibleinyourcurrentsystem.Dothisforthefollowingreasons:1.Itiseasiertogainpermissionfromthesystem’susersearlieron.2.Ifyouthinkthatsomethingmightbeaconcerninthefuture,itisbettertogethistoricaldatanow.3.Ifyoudesignyoursystemwithmetricinstrumentationinmind,thingswillgobetterforyouinthefuture.Specifically,youdon’twanttofindyourselfgreppingforstringsinlogstoinstrumentyourmetrics!4.Youwillnoticewhatthingschangeandwhatstaysthesame.Forinstance,supposeyouwanttodirectlyoptimizeone­dayactiveusers.However,duringyourearlymanipulationsofthesystem,youmaynoticethatdramaticalterationsoftheuserexperiencedon’tnoticeablychangethismetric.GooglePlusteammeasuresexpandsperread,resharesperread,plus­onesperread,commentsread,commentsperuser,resharesperuser,etc.whichtheyuseincomputingthegoodnessofapostatservingtime.Also,notethatanexperimentframework,whereyoucangroupusersintobucketsandaggregatestatisticsbyexperiment,isimportant.SeeRule12.Bybeingmoreliberalaboutgatheringmetrics,youcangainabroaderpictureofyoursystem.Noticeaproblem?Addametrictotrackit!Excitedaboutsomequantitativechangeonthelastrelease?Addametrictotrackit!Rule3:Choosemachinelearningoveracomplexheuristic.Asimpleheuristiccangetyourproductoutthedoor.Acomplexheuristicisunmaintainable.Onceyouhavedataandabasicideaofwhatyouaretryingtoaccomplish,moveontomachinelearning.Asinmostsoftwareengineeringtasks,youwillwanttobeconstantlyupdatingyourapproach,whetheritisaheuristicoramachine­learnedmodel,andyouwillfindthatthemachine­learnedmodeliseasiertoupdateandmaintain(seeRule16).MLPhaseI:YourFirstPipelineFocusonyoursysteminfrastructureforyourfirstpipeline.Whileitisfuntothinkaboutalltheimaginativemachinelearningyouaregoingtodo,itwillbehardtofigureoutwhatishappeningifyoudon’tfirsttrustyourpipeline.Rule4:Keepthefirstmodelsimpleandgettheinfrastructureright.Thefirstmodelprovidesthebiggestboosttoyourproduct,soitdoesn'tneedtobefancy.Butyouwillrunintomanymoreinfrastructureissuesthanyouexpect.Beforeanyonecanuseyourfancynewmachinelearningsystem,youhavetodetermine:1.Howtogetexamplestoyourlearningalgorithm.2.Afirstcutastowhat“good”and“bad”meantoyoursystem.3.Howtointegrateyourmodelintoyourapplication.Youcaneitherapplythemodellive,orpre­computethemodelonexamplesofflineandstoretheresultsinatable.Forexample,youmightwanttopre­classifywebpagesandstoretheresultsinatable,butyoumightwanttoclassifychatmessageslive.Choosingsimplefeaturesmakesiteasiertoensurethat:1.Thefeaturesreachyourlearningalgorithmcorrectly.2.Themodellearnsreasonableweights.3.Thefeaturesreachyourmodelintheservercorrectly.Onceyouhaveasystemthatdoesthesethreethingsreliably,youhavedonemostofthework.Yoursimplemodelprovidesyouwithbaselinemetricsandabaselinebehaviorthatyoucanusetotestmorecomplexmodels.Someteamsaimfora“neutral”firstlaunch:afirstlaunchthatexplicitlyde­prioritizesmachinelearninggains,toavoidgettingdistracted.Rule5:Testtheinfrastructureindependentlyfromthemachinelearning.Makesurethattheinfrastructureistestable,andthatthelearningpartsofthesystemareencapsulatedsothatyoucantesteverythingaroundit.Specifically:1.Testgettingdataintothealgorithm.Checkthatfeaturecolumnsthatshouldbepopulatedarepopulated.Whereprivacypermits,manuallyinspecttheinputtoyourtrainingalgorithm.Ifpossible,checkstatisticsinyourpipelineincomparisontoelsewhere,suchasRASTA.2.Testgettingmodelsoutofthetrainingalgorithm.Makesurethatthemodelinyourtrainingenvironmentgivesthesamescoreasthemodelinyourservingenvironment(seeRule37).Machinelearninghasanelementofunpredictability,somakesurethatyouhavetestsforthecodeforcreatingexamplesintrainingandserving,andthatyoucanloadanduseafixedmodelduringserving.Also,itisimportanttounderstandyourdata:seePracticalAdviceforAnalysisofLarge,ComplexDataSets.Rule6:Becarefulaboutdroppeddatawhencopyingpipelines.Oftenwecreateapipelinebycopyinganexistingpipeline(i.e.cargocultprogramming),andtheoldpipelinedropsdatathatweneedforthenewpipeline.Forexample,thepipelineforGooglePlusWhat’sHotdropsolderposts(becauseitistryingtorankfreshposts).ThispipelinewascopiedtouseforGooglePlusStream,whereolderpostsarestillmeaningful,butthepipelinewasstilldroppingoldposts.Anothercommonpatternistoonlylogdatathatwasseenbytheuser.Thus,thisdataisuselessifwewanttomodelwhyaparticularpostwasnotseenbytheuser,becauseallthenegativeexampleshavebeendropped.AsimilarissueoccurredinPlay.WhileworkingonPlayAppsHome,anewpipelinewascreatedthatalsocontainedexamplesfromtwootherlandingpages(PlayGamesHomeandPlayHomeHome)withoutanyfeaturetodisambiguatewhereeachexamplecamefrom.Rule7:Turnheuristicsintofeatures,orhandlethemexternally.Usuallytheproblemsthatmachinelearningistryingtosolvearenotcompletelynew.Thereisanexistingsystemforranking,orclassifying,orwhateverproblemyouaretryingtosolve.Thismeansthatthereareabunchofrulesandheuristics.Thesesameheuristicscangiveyoualiftwhentweakedwithmachinelearning.Yourheuristicsshouldbeminedforwhateverinformationtheyhave,fortworeasons.First,thetransitiontoamachinelearnedsystemwillbesmoother.Second,usuallythoserulescontainalotoftheintuitionaboutthesystemyoudon’twanttothrowaway.Therearefourwaysyoucanuseanexistingheuristic:1.Preprocessusingtheheuristic.Ifthefeatureisincrediblyawesome,thenthisisanoption.Forexample,if,inaspamfilter,thesenderhasalreadybeenblacklisted,don’ttrytorelearnwhat“blacklisted”means.Blockthemessage.Thisapproachmakesthemostsenseinbinaryclassificationtasks.2.Createafeature.Directlycreatingafeaturefromtheheuristicisgreat.Forexample,ifyouuseaheuristictocomputearelevancescoreforaqueryresult,youcanincludethescoreasthevalueofafeature.Lateronyoumaywanttousemachinelearningtechniquestomassagethevalue(forexample,convertingthevalueintooneofafinitesetofdiscretevalues,orcombiningitwithotherfeatures)butstartbyusingtherawvalueproducedbytheheuristic.3.Minetherawinputsoftheheuristic.Ifthereisaheuristicforappsthatcombinesthenumberofinstalls,thenumberofcharactersinthetext,andthedayoftheweek,thenconsiderpullingthesepiecesapart,andfeedingtheseinputsintothelearningseparately.Sometechniquesthatapplytoensemblesapplyhere(seeRule40).4.Modifythelabel.Thisisanoptionwhenyoufeelthattheheuristiccapturesinformationnotcurrentlycontainedinthelabel.Forexample,ifyouaretryingtomaximizethenumberofdownloads,butyoualsowantqualitycontent,thenmaybethesolutionistomultiplythelabelbytheaveragenumberofstarstheappreceived.Thereisalotofspacehereforleeway.Seethesectionon“YourFirstObjective”.DobemindfuloftheaddedcomplexitywhenusingheuristicsinanMLsystem.Usingoldheuristicsinyournewmachinelearningalgorithmcanhelptocreateasmoothtransition,butthinkaboutwhetherthereisasimplerwaytoaccomplishthesameeffect.MonitoringIngeneral,practicegoodalertinghygiene,suchasmakingalertsactionableandhavingadashboardpage.Rule8:Knowthefreshnessrequirementsofyoursystem.Howmuchdoesperformancedegradeifyouhaveamodelthatisadayold?Aweekold?Aquarterold?Thisinformationcanhelpyoutounderstandtheprioritiesofyourmonitoring.Ifyoulose10%ofyourrevenueifthemodelisnotupdatedforaday,itmakessensetohaveanengineerwatchingitcontinuously.Mostadservingsystemshavenewadvertisementstohandleeveryday,andmustupdatedaily.Forinstance,iftheMLmodelforGooglePlaySearchisnotupdated,itcanhaveanimpactonrevenueinunderamonth.SomemodelsforWhat’sHotinGooglePlushavenopostidentifierintheirmodelsotheycanexportthesemodelsinfrequently.Othermodelsthathavepostidentifiersareupdatedmuchmorefrequently.Alsonoticethatfreshnesscanchangeovertime,especiallywhenfeaturecolumnsareaddedorremovedfromyourmodel.Rule9:Detectproblemsbeforeexportingmodels.Manymachinelearningsystemshaveastagewhereyouexportthemodeltoserving.Ifthereisanissuewithanexportedmodel,itisauser­facingissue.Ifthereisanissuebefore,thenitisatrainingissue,anduserswillnotnotice.Dosanitychecksrightbeforeyouexportthemodel.Specifically,makesurethatthemodel’sperformanceisreasonableonheldoutdata.Or,ifyouhavelingeringconcernswiththedata,don’texportamodel.ManyteamscontinuouslydeployingmodelschecktheareaundertheROCcurve(orAUC)beforeexporting.Issuesaboutmodelsthathaven’tbeenexportedrequireane­mailalert,butissuesonauser­facingmodelmayrequireapage.Sobettertowaitandbesurebeforeimpactingusers.Rule10:Watchforsilentfailures.Thisisaproblemthatoccursmoreformachinelearningsystemsthanforotherkindsofsystems.Supposethataparticulartablethatisbeingjoinedisnolongerbeingupdated.Themachinelearningsystemwilladjust,andbehaviorwillcontinuetobereasonablygood,decayinggradually.Sometimestablesarefoundthatweremonthsoutofdate,andasimplerefreshimprovedperformancemorethananyotherlaunchthatquarter!Forexample,thecoverageofafeaturemaychangeduetoimplementationchanges:forexampleafeaturecolumncouldbepopulatedin90%oftheexamples,andsuddenlydropto60%oftheexamples.Playoncehadatablethatwasstalefor6months,andrefreshingthetablealonegaveaboostof2%ininstallrate.Ifyoutrackstatisticsofthedata,aswellasmanuallyinspectthedataonoccassion,youcanreducethesekindsoffailures.Rule11:Givefeaturecolumnownersanddocumentation.Ifthesystemislarge,andtherearemanyfeaturecolumns,knowwhocreatedorismaintainingeachfeaturecolumn.Ifyoufindthatthepersonwhounderstandsafeaturecolumnisleaving,makesurethatsomeonehastheinformation.Althoughmanyfeaturecolumnshavedescriptivenames,it'sgoodtohaveamoredetaileddescriptionofwhatthefeatureis,whereitcamefrom,andhowitisexpectedtohelp.YourFirstObjectiveYouhavemanymetrics,ormeasurementsaboutthesystemthatyoucareabout,butyourmachinelearningalgorithmwilloftenrequireasingleobjective,anumberthatyouralgorithmis“trying”tooptimize.Idistinguishherebetweenobjectivesandmetrics:ametricisanynumberthatyoursystemreports,whichmayormaynotbeimportant.SeealsoRule2.Rule12:Don’toverthinkwhichobjectiveyouchoosetodirectlyoptimize.Youwanttomakemoney,makeyourusershappy,andmaketheworldabetterplace.Therearetonsofmetricsthatyoucareabout,andyoushouldmeasurethemall(seeRule2).However,earlyinthemachinelearningprocess,youwillnoticethemallgoingup,eventhosethatyoudonotdirectlyoptimize.Forinstance,supposeyoucareaboutnumberofclicks,timespentonthesite,anddailyactiveusers.Ifyouoptimizefornumberofclicks,youarelikelytoseethetimespentincrease.So,keepitsimpleanddon’tthinktoohardaboutbalancingdifferentmetricswhenyoucanstilleasilyincreaseallthemetrics.Don’ttakethisruletoofarthough:donotconfuseyourobjectivewiththeultimatehealthofthesystem(seeRule39).And,ifyoufindyourselfincreasingthedirectlyoptimizedmetric,butdecidingnottolaunch,someobjectiverevisionmayberequired.Rule13:Chooseasimple,observableandattributablemetricforyourfirstobjective.Oftenyoudon'tknowwhatthetrueobjectiveis.Youthinkyoudobutthenyouasyoustareatthedataandside­by­sideanalysisofyouroldsystemandnewMLsystem,yourealizeyouwanttotweakit.Further,differentteammembersoftencan'tagreeonthetrueobjective.TheMLobjectiveshouldbesomethingthatiseasytomeasureandisaproxyforthe“true”objective.SotrainonthesimpleMLobjective,andconsiderhavingapolicylayerontopthat2allowsyoutoaddadditionallogic(hopefullyverysimplelogic)todothefinalranking.Theeasiestthingtomodelisauserbehaviorthatisdirectlyobservedandattributabletoanactionofthesystem:1.Wasthisrankedlinkclicked?2.Wasthisrankedobjectdownloaded?3.Wasthisrankedobjectforwardedrepliedtoe­mailed?4.Wasthisrankedobjectrated?5.Wasthisshownobjectmarkedasspampornographyoffensive?Avoidmodelingindirecteffectsatfirst:1.Didtheuservisitthenextday?2.Howlongdidtheuservisitthesite?3.Whatwerethedailyactiveusers?Indirecteffectsmakegreatmetrics,andcanbeusedduringABtestingandduringlaunchdecisions.Finally,don’ttrytogetthemachinelearningtofigureout:1.Istheuserhappyusingtheproduct?2.Istheusersatisfiedwiththeexperience?3.Istheproductimprovingtheuser’soverallwell­being?2Thereisoftenno“true”objective.SeeRule39.4.Howwillthisaffectthecompany’soverallhealth?Theseareallimportant,butalsoincrediblyhard.Instead,useproxies:iftheuserishappy,theywillstayonthesitelonger.Iftheuserissatisfied,theywillvisitagaintomorrow.Insofaraswell­beingandcompanyhealthisconcerned,humanjudgementisrequiredtoconnectanymachinelearnedobjectivetothenatureoftheproductyouaresellingandyourbusinessplan,sowedon’tenduphere.Rule14:Startingwithaninterpretablemodelmakesdebuggingeasier.Linearregression,logisticregression,andPoissonregressionaredirectlymotivatedbyaprobabilisticmodel.Eachpredictionisinterpretableasaprobabilityoranexpectedvalue.Thismakesthemeasiertodebugthanmodelsthatuseobjectives(zero­oneloss,varioushingelosses,etcetera)thattrytodirectlyoptimizeclassificationaccuracyorrankingperformance.Forexample,ifprobabilitiesintrainingdeviatefromprobabilitiespredictedinside­by­sidesorbyinspectingtheproductionsystem,thisdeviationcouldrevealaproblem.Forexample,inlinear,logistic,orPoissonregression,therearesubsetsofthedatawheretheaveragepredictedexpectationequalstheaveragelabel(1­momentcalibrated,orjustcalibrated).Ifyouhaveafeaturewhichiseither1or0foreachexample,thenthesetof3exampleswherethatfeatureis1iscalibrated.Also,ifyouhaveafeaturethatis1foreveryexample,thenthesetofallexamplesiscalibr