下载

1下载券

加入VIP
  • 专属下载特权
  • 现金文档折扣购买
  • VIP免费专区
  • 千万文档免费下载

上传资料

关闭

关闭

关闭

封号提示

内容

首页 ROC_Curves

ROC_Curves.ppt

ROC_Curves

doors
2011-04-20 0人阅读 举报 0 0 暂无简介

简介:本文档为《ROC_Curvesppt》,可适用于自然科学领域

ROCStatisticsfortheLazyMachineLearnerinAllofUsROCStatisticsfortheLazyMachineLearnerinAllofUsBradleyMalinLectureforCOSLabSchoolofComputerScienceCarnegieMellonUniversityWhyShouldICareWhyShouldICareImagineyouhavedifferentprobabilisticclassificationmodelseglogisticregressionvsneuralnetworkHowdoyouknowwhichoneisbetterHowdoyoucommunicateyourbeliefCanyouprovidequantitativeevidencebeyondagutfeelingandsubjectiveinterpretationRecallBasics:ContingenciesRecallBasics:ContingenciesSomeTermsSomeTermsSomeMoreTermsSomeMoreTermsAccuracyAccuracyWhatdoesthismeanWhatisthedifferencebetween“accuracy”andan“accurateprediction”ContingencyTableInterpretation(TruePositives)(TrueNegatives)(TruePositives)(TrueNegatives)(FalsePositives)(FalseNegatives)Isthisagoodmeasure(WhyorWhyNot)NoteonDiscreteClassesNoteonDiscreteClassesTRADITION…ShowcontingencytablewhenreportingpredictionsofmodelBUT…probabilisticmodelsdonotprovidediscretecalculationsofthematrixcells!!!INOTHERWORDS…Regressiondoesnotreportthenumberofindividualspredictedpositive(eghasaheartattack)…well,notreallyINSTEAD…reportprobabilitytheoutputwillbecertainvariable(egor)VisualPerspectiveVisualPerspectiveROCCurvesROCCurvesOriginatedfromsignaldetectiontheoryBinarysignalcorruptedbyGuassiannoiseWhatistheoptimalthreshold(ieoperatingpoint)DependenceonfactorsSignalStrengthNoiseVariancePersonaltoleranceinHitFalseAlarmRateROCCurvesROCCurvesReceiveroperatorcharacteristicSummarizepresentperformanceofanybinaryclassificationmodelModelsabilitytodistinguishbetweenfalsetruepositivesUseMultipleContingencyTablesUseMultipleContingencyTablesSamplecontingencytablesfromrangeofthresholdprobabilityTRUEPOSITIVERATE(alsocalledSENSITIVITY)TruePositives(TruePositives)(FalseNegatives)FALSEPOSITIVERATE(alsocalledSPECIFICITY)FalsePositives(FalsePositives)(TrueNegatives)PlotSensitivityvs(–Specificity)forsamplingandyouaredoneDataCentricExampleDataCentricExampleROCRatesROCRatesROCPlotROCPlotLOGISTICNEURALSidebar:UseMoreSamplesSidebar:UseMoreSamples(Theseareplotsfromamuchlargerdataset–SeeMalin)ROCQuantificationROCQuantificationAreaUnderROCCurveUsequadraturetocalculatetheareaegtrapz(trapezoidalrule)functioninMatlabwillworkExample–Appears“NeuralNetwork”modelisbetterTheory:ModelOptimalityTheory:ModelOptimalityClassifiersonconvexhullarealways“optimal”egNetTreeClassifiersbelowconvexhullarealways“suboptimal”egNaïveBayesNaïveBayesDecisionTreeNeuralNetBuildingBetterClassifiersBuildingBetterClassifiersClassifiersonconvexhullcanbecombinedtoformastrictlydominanthybridclassifierorderedsequenceofclassifierscanbeconvertedinto“ranker”DecisionTreeNeuralNetSomeStatisticalInsightSomeStatisticalInsightCurveArea:TakerandomhealthypatientscoreofXTakerandomheartattackpatientscoreofYAreaestimateofPY>XSlopeofcurveisequaltolikelihood:P(score|Signal)P(score|Noise)ROCgraphcapturesallinformationincontingtableFalsenegativetruenegativeratesarecomplementsoftruepositivefalsepositiverates,respCanAlwaysQuantifyBestOperatingPointCanAlwaysQuantifyBestOperatingPointWhenmisclassificationcostsareequal,bestoperatingpointis…tangenttocurveclosestto(,)coordVerifythismathematically(economicinterpretation)WhyQuickQuestionQuickQuestionAreROCcurvesalwaysappropriateSubjectiveoperatingpointsMustweightthetradeoffsbetweenfalsepositivesandfalsenegativesROCcurveplotisindependentoftheclassdistributionorerrorcostsThisleadsintoutilitytheory(nottouchingthistoday)MuchMuchMoreonROCMuchMuchMoreonROCOh,ifonlyIhadmoretimeYoushouldalsolookupandlearnabout:IsoaccuracylinesSkewdistributionsandwhythelineisn’talways“best”ConvexityvsnonconvexityvsconcavityMannWhitneyWilcoxonsumofranksGinicoefficientCalibratedthresholdsAveragingROCcurvesPrecisionRecall(THISISVERYIMPORTANT)CostCurvesSomeReferencesSomeReferencesGoodBibliography:http:splwebbwhharvardedu:pagespplzourochtmlDrummondCandHolteRWhatROCcurvescanandcan’tdo(andcostcurvescan)InProceedingsoftheWorkshoponROCAnalysisinAIinconjunctionwiththeEuropeanConferenceonAIValencia,SpainMalinBProbabilisticpredictionofmyocardialinfarction:logisticregressionversussimpleneuralnetworksDataPrivacyLabWorkingPaperWP,SchoolofComputerScience,CarnegieMellonUniversitySeptMcNeilBJ,HanleyJAStatisticalapproachestotheanalysisofreceiveroperatingcharacteristic(ROC)curvesMedicalDecisionMaking:ProvostFandFawcettTThecaseagainstaccuracyestimationforcomparinginductionalgorithmsInProceedingsofthethInternationalConferenceonMachineLearningMadison,Wisconsin:SwetsJMeasuringtheaccuracyofdiagnosticsystemsScience():(basedonhisbookInformationRetrievalSystems)

用户评价(0)

关闭

新课改视野下建构高中语文教学实验成果报告(32KB)

抱歉,积分不足下载失败,请稍后再试!

提示

试读已结束,如需要继续阅读或者下载,敬请购买!

文档小程序码

使用微信“扫一扫”扫码寻找文档

1

打开微信

2

扫描小程序码

3

发布寻找信息

4

等待寻找结果

我知道了
评分:

/23

ROC_Curves

VIP

在线
客服

免费
邮箱

爱问共享资料服务号

扫描关注领取更多福利