440_Schaap_CoronaryEvaluation

n m line etz a uer d Standardized evaluation Centerline extraction Tracking Coronaries Computed tomography method is described to create a consensus centerline with multiple observers, (2) well-defined measures are presented for the evaluation of coronary artery centerline extraction algorithms, (3) a database con- taining 32 cardiac CTA datasets with corresponding reference standard is described and made available, and (4) 13 coronary artery centerline extraction algorithms, implemented by different research groups, are quantitatively evaluated and compared. The presented evaluation framework is made available to the medical imaging community for benchmarking existing or newly developed coronary centerline extraction algorithms. � 2009 Elsevier B.V. All rights reserved. * Corresponding author. Address: P.O. Box 2040, 3000 CA Rotterdam, The Netherlands. Tel.: +1 31 10 7044078; fax: +1 31 10 7044722. Medical Image Analysis 13 (2009) 701–714 Contents lists available at ScienceDirect journ E-mail address: michiel.schaap@erasmusmc.nl (M. Schaap). Thomas O’Donnell i, Michel Frenay j, Ola Friman k, Marcela Hernández Hoyos l, Pieter H. Kitslaar j,m, Karl Krissian n, Caroline Kühnel k, Miguel A. Luengo-Oroz p,q, Maciej Orkisz o, Örjan Smedby r, Martin Styner s, Andrzej Szymczak t, Hüseyin Tek u, Chunliang Wang r, Simon K. Warfield v, Sebastian Zambal w, Yong Zhang x, Gabriel P. Krestin c, Wiro J. Niessen a,y aBiomedical Imaging Group Rotterdam, Dept. of Radiology and Med. Informatics, Erasmus MC, Rotterdam, The Netherlands bDept. of Biomedical Engineering, Erasmus MC, Rotterdam, The Netherlands cDept. of Radiology, Erasmus MC, Rotterdam, The Netherlands d Institute for Computer Graphics and Vision, Graz Univ. of Technology, Graz, Austria eCenter for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB), Barcelona, Spain fUniversitat Pompeu Fabra and CIBER-BBN, Barcelona, Spain gCent. for Med. Imaging Validation, Siemens Corporate Research, Princeton, NJ, USA hDept. of Radiology, Univ. of Florida College of Medicine, Jacksonville, FL, USA i Siemens Corporate Research, Princeton, NJ, USA jDivision of Image Processing, Dept. of Radiology, Leiden Univ. Med. Cent., Leiden, The Netherlands kMeVis Research, Bremen, Germany lGrupo Imagine, Grupo de Ingeniería Biomédica, Universidad de los Andes, Bogota, Colombia mMedis Medical Imaging Systems b.v., Leiden, The Netherlands nCentro de Tecnología Médica, Univ. of Las Palmas of Gran Canaria, Dept. of Signal and Com., Las Palmas of G.C., Spain oUniversité de Lyon, Université Lyon 1, INSA-Lyon, CNRS UMR 5220, CREATIS, Inserm U630, Villeurbanne, France pBiomedical Image Technologies Lab., ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain qBiomedical Research Cent. in Bioengineering, Biomaterials and Nanomedicine (CIBER-BBN), Zaragoza, Spain rDept. of Radiology and Cent. for Med. Image Science and Visualization, Linköping Univ., Linköping, Sweden sDept. of Computer Science and Psychiatry, Univ. of North Carolina, Chapel Hill, NC, USA tDept. of Mathematical and Computer Sciences, Colorado School of Mines, Golden, CO, USA u Imaging and Visualization Dept., Siemens Corporate Research, Princeton, NJ, USA vDept. of Radiology, Children’s Hospital Boston, Boston, MA, USA wVRVis Research Cent. for Virtual Reality and Visualization, Vienna, Austria x The Methodist Hospital Research Institute, Houston, TX, USA y Imaging Science and Technology, Faculty of Applied Sciences, Delft Univ. of Technology, Delft, The Netherlands a r t i c l e i n f o Article history: Received 1 November 2008 Received in revised form 15 April 2009 Accepted 11 June 2009 Available online 30 June 2009 Keywords: a b s t r a c t Efficiently obtaining a reliable coronary artery centerline from computed tomography angiography data is relevant in clinical practice. Whereas numerous methods have been presented for this purpose, up to now no standardized evaluation methodology has been published to reliably evaluate and compare the performance of the existing or newly developed coronary artery centerline extraction algorithms. This paper describes a standardized evaluation methodology and reference database for the quantitative eval- uation of coronary artery centerline extraction algorithms. The contribution of this work is fourfold: (1) a Standardized evaluatio coronary artery center Michiel Schaap a,*, Coert T. M Nico R. Mollet c, Christian Ba 1361-8415/$ - see front matter � 2009 Elsevier B.V. A doi:10.1016/j.media.2009.06.003 ethodology and reference database for evaluating extraction algorithms , Theo van Walsum a, Alina G. van der Giessen b, Annick C. Weustink c, , Hrvoje Bogunovic´ e,f, Carlos Castro p,q, Xiang Deng g, Engin Dikici h, Medical Image Analysis al homepage: www.elsevier .com/locate /media ll rights reserved. Coronary artery disease (CAD) is currently the primary cause age 2. Motivation et al., 2004; Renard and Yang, 2008; Schaap et al., 2007; Szymczak of death among American males and females (Rosamond et al., 2008) and one of the main causes of death in the world (WHO, 2008). The gold standard for the assessment of CAD is conven- tional coronary angiography (CCA) (Cademartiri et al., 2007). However, because of its invasive nature, CCA has a low, but non-negligible, risk of procedure related complications (Zanzonic- o et al., 2006). Moreover, it only provides information on the cor- onary lumen. Computed Tomography Angiography (CTA) is a potential alter- native for CCA (Mowatt et al., 2008). CTA is a non-invasive tech- nique that allows, next to the assessment of the coronary lumen, the evaluation of the presence, extent, and type (non-calcified or calcified) of coronary plaque (Leber et al., 2006). Such non-inva- sive, comprehensive plaque assessment may be relevant for improving risk stratification when combined with current risk measures: the severity of stenosis and the amount of calcium (Cademartiri et al., 2007). A disadvantage of CTA is that the current imaging protocols are associated with a higher radiation dose exposure than CCA (Einstein et al., 2007). Several techniques to visualize CTA data are used in clinical practice for the diagnosis of CAD. Besides evaluating the axial slices, other visualization techniques such as maximum intensity projections (MIP), volume rendering techniques, multi-planar reformatting (MPR), and curved planar reformatting (CPR) are used to review CTA data (Cademartiri et al., 2007). CPR and MPR images of coronary arteries are based on the CTA image and a central lu- men line (for convenience referred to as centerline) through the vessel of interest (Kanitsar et al., 2002). These reformatted images can also be used during procedure planning for, among other things, planning the type of intervention and size of stents (Hecht, 2008). Efficiently obtaining a reliable centerline is therefore rele- vant in clinical practice. Furthermore, centerlines can serve as a starting point for lumen segmentation, stenosis grading, and pla- que quantification (Marquering et al., 2005; Wesarg et al., 2006; Khan et al., 2006). This paper introduces a framework for the evaluation of coro- nary artery centerline extraction methods. The framework encom- passes a publicly available database of coronary CTA data with corresponding reference standard centerlines derived from manu- ally annotated centerlines, a set of well-defined evaluation mea- sures, and an online tool for the comparison of coronary CTA centerline extraction techniques. We demonstrate the potential of the proposed framework by comparing 13 coronary artery cen- terline extraction methods, implemented by different authors as part of a segmentation challenge workshop at the Medical Image Computing and Computer-Assisted Intervention (MICCAI) confer- ence (Metz et al., 2008). In the next two sections we will respectively describe our moti- vation of the study presented in this paper and discuss previous work on the evaluation of coronary segmentation and centerline extraction techniques. The evaluation framework will then be out- lined by discussing the data, reference standard, evaluation mea- sures, evaluation categories, and web-based framework. The paper will be concluded by presenting the comparative results of the 13 centerline extraction techniques, a discussion of these re- sults, and a conclusion about the work presented. 1. Introduction 702 M. Schaap et al. /Medical Im The value of a standardized evaluation methodology and a pub- licly available image repository has been shown in a number of medical image analysis and general computer vision applications, for example in the Retrospective Image Registration Evaluation Project (West et al., 1997), the Digital Retinal Images for Vessel Extraction database (Staal et al., 2004), the Lung Image Database project (Armato et al., 2004), the Middlebury Stereo Vision evalua- tion (Scharstein and Szeliski, 2002), the Range Image Segmentation Comparison (Hoover et al., 1996), the Berkeley Segmentation Data- set and Benchmark (Martin et al., 2001), and a workshop and on- line evaluation framework for liver and caudate segmentation (van Ginneken et al., 2007). Similarly, standardized evaluation and comparison of coronary artery centerline extraction algorithms has scientific and practical benefits. A benchmark of state-of-the-art techniques is a prerequi- site for continued progress in this field: it shows which of the pop- ular methods are successful and researchers can quickly apprehend where methods can be improved. It is also advantageous for the comparison of newmethods with the state-of-the-art. Without a publicly available evaluation frame- work, such comparisons are difficult to perform: the software or source code of existing techniques is often not available, articles may not give enough information for re-implementation, and if en- ough information is provided, re-implementation of multiple algo- rithms is a laborious task. The understanding of algorithm performance that results from the standardized evaluation also has practical benefits. It may, for example, steer the clinical implementation and utilization, as a system architect can use objective measures to choose the best algorithm for a specific task. Furthermore, the evaluation could show under which condi- tions a particular technique is likely to succeed or fail, it may there- fore be used to improve the acquisition methodology to better match the post-processing techniques. It is therefore our goal to design and implement a standardized methodology for the evaluation and comparison of coronary artery centerline extraction algorithms and publish a cardiac CTA image repository with associated reference standard. To this end, we will discuss the following tasks below: � Collection of a representative set of cardiac CTA datasets, with a manually annotated reference standard, available for the entire medical imaging community. � Development of an appropriate set of evaluation measures for the evaluation of coronary artery centerline extraction methods. � Development of an accessible framework for easy comparison of different algorithms. � Application of this framework to compare several coronary CTA centerline extraction techniques. � Public dissemination of the results of the evaluation. 3. Previous work Approximately 30 papers have appeared that present and/or evaluate (semi-)automatic techniques for the segmentation or cen- terline extraction of human coronary arteries in cardiac CTA data- sets. The proposed algorithms have been evaluated by a wide variety of evaluation methodologies. A large number of methods have been evaluated qualitatively (Bartz and Lakare, 2005; Bouraoui et al., 2008; Carrillo et al., 2007; Florin et al., 2004, 2006; Hennemuth et al., 2005; Lavi et al., 2004; Lorenz et al., 2003; Luengo-Oroz et al., 2007; Nain et al., 2006; Wang et al., 2007; Wesarg and Firle, 2004; Yang et al., Analysis 13 (2009) 701–714 2005, 2006). In these articles detection, extraction, or segmenta- tion correctness have been visually determined. An overview of these methods is given in Table 1. Table 1 An overview of CTA coronary artery segmentation and centerline extraction algorithms that were qualitatively evaluated. The column ‘Time’ indicates if information is provided about the computational time of the algorithm. Article Patients/ observers Vessels Evaluation details Time Bartz and Lakare (2005) 1/1 Complete tree Extraction was judged to be satisfactory Yes Bouraoui et al. (2008) 40/1 Complete tree Extraction was scored satisfactory or not No Carrillo et al. (2007) 12/1 Complete tree Extraction was scored with the number of extracted small branches Yes Florin et al. (2004) 1/1 Complete tree Extraction was judged to be satisfactory Yes Florin et al. (2006) 34/1 6 vessels Scored with the number of correct extractions No Hennemuth et al. (2005) 61/1 RCA, LAD Scored with the number of extracted vessels and categorized on the dataset difficulty Yes Lavi et al. (2004) 34/1 3 Vessels Scored qualitatively with scores from 1 to 5 and categorized on the image quality Yes Lorenz et al. (2003) 3/1 Complete tree Results were visually analyzed and criticized Yes Luengo-Oroz et al. (2007) 9/1 LAD & LCX Scored with the number of correct vessel extractions. The results are categorized on the image quality and amount of disease Yes Nain et al. (2004) 2/1 Left tree Results were visually analyzed and criticized No Renard and Yang (2008) 2/1 Left tree Extraction was judged to be satisfactory No Schaap et al. (2007) 2/1 RCA Extraction was judged to be satisfactory No Szymczak et al. (2006) 5/1 Complete tree Results were visually analyzed and criticized Yes Wang et al. (2007) 33/1 Complete tree Scored with the number of correct extractions Yes red ract red ient M. Schaap et al. /Medical Image Analysis 13 (2009) 701–714 703 Other articles include a quantitative evaluation of the performance of the proposed methods (Bülow et al., 2004; Busch et al., 2007; Dewey et al., 2004; Larralde et al., 2003; Lesage et al., 2008; Li and Yezzi, 2007; Khan et al., 2006; Marquering et al., 2005; Metz et al., 2007; Olabarriaga et al., 2003; Wesarg et al., 2006; Yang et al., 2007). See Table 2 for an overview of these methods. None of the abovementioned algorithms has been compared to another and only three methods were quantitatively evaluated on both the extraction ability (i.e. how much of the real centerline can be extracted by the method?) and the accuracy (i.e. how accurately can the method locate the centerline or wall of the vessel?). More- Wesarg and Firle (2004) 12/1 Complete tree Sco Yang et al. (2005) 2/1 Left tree Ext Yang et al. (2006) 2/1 4 Vessels Sco pat over, only one method was evaluated using annotations from more than one observer (Metz et al., 2007). Four methods were assessed on their ability to quantify clinically relevant measures, such as the degree of stenosis and the number of calcium spots in a vessel (Yang et al., 2005; Dewey et al., 2004; Khan et al., 2006; Wesarg et al., 2006). These Table 2 An overview of the quantitatively evaluated CTA coronary artery segmentation and centerl (semi-)automatically extracted centerline and the manually annotated centerline. The col algorithm. ‘Method eval.’ indicates that the article evaluates an existing technique and th Article Patients/ observers Vessels Used evaluation measur Bülow et al. (2004) 9/1 3–5 Vessels Overlap: Percentage refe Busch et al. (2007) 23/2 Complete tree Stenoses grading: Comp Dewey et al. (2004) 35/1 3 Vessels Length difference: Diffe Stenoses grading: Comp Khan et al. (2006) 50/1 3 Vessels Stenoses grading: Comp Larralde et al. (2003) 6/1 Complete tree Stenoses grading and c Lesage et al. (2008) 19/1 3 Vessels Same as Metz et al. (200 Li and Yezzi (2007) 5/1 Complete tree Segmentation: Voxel-w Marquering et al. (2005) 1/1 LAD Accuracy: Distance from Metz et al. (2007) 6/3 3 Vessels Overlap: Segments on th positives, false positives similarity indices Accuracy: Average dista Olabarriaga et al. (2003) 5/1 3 Vessels Accuracy: Mean distanc Wesarg et al. (2006) 10/1 3 Vessels Calcium detection: Perf Yang et al. (2007) 2/1 3 Vessels Overlap: Percentage of t Segmentation: Average clinically oriented evaluation approaches are very appropriate for assessing the performance of a method for a possible clinical application, but the performance of these methods for other applications, such as describing the geometry of coronary arteries (Lorenz and von Berg, 2006; Zhu et al., 2008), cannot easily be judged. Two of the articles (Dewey et al., 2004; Busch et al., 2007) evaluate a commercially available system (respectively Vitrea 2, Version 3.3, Vital Images and Syngo Circulation, Siemens). Several other commercial centerline extraction and stenosis grading pack- ages have been introduced in the past years, but we are not aware of any scientific publication containing a clinical evaluation of with the number of correct extractions Yes ion was judged to be satisfactory Yes satisfactory or not. Evaluated in 10 ECG gated reconstructions per Yes these packages. 4. Evaluation framework In this section we will describe our framework for the evalua- tion of coronary CTA centerline extraction techniques. ine extraction algorithms. With ‘centerline’ and ‘reference’ we respectively denote the umn ‘Time’ indicates if information is provided about the computational time of the at no new technique has been proposed. es and details Time Method eval. rence points having a centerline point within 2 mm No ared to human performance with CCA as ground truth No � rence between reference length and centerline length Yes � ared to human performance with CCA as ground truth ared to human performance with CCA as ground truth No � alcium detection: Compared to human performance Yes 7) Yes ise similarity indices No centerline to reference standard Yes e reference standard and centerline are marked as true or false negatives. This scoring was used to construct No nce to the reference standard for true positive sections e from the centerline to the reference No ormance compared to human performance No � he reference standard detected No distance to contours 4.1. Cardiac CTA data The CTA data was acquired in the Erasmus MC, University Med- ical Center Rotterdam, The Netherlands. Thirty-two datasets were randomly selected from a series of patients who underwent a car- diac CTA examination between June 2005 and June 2006. Twenty datasets were acquired with a 64-slice CT scanner and 12 datasets with a dual-source CT scanner (Sensation 64 and Somatom Defini- tion, Siemens Medical Solutions, Forchheim, Germany). A tube voltage of 120 kV was used for both scanners. All data- sets were acquired with ECG-pulsing (Weustink et al., 2008). The maximum current (625 mA for the dual-source scanner and 900 mA for the 64-slice scanner) was used in the window from 704 M. Schaap et al. /Medical Image 25% to 70% of the R–R interval and outside this window the tube current was reduced to 20% of the maximum current. Both scanners operated with a detector width of 0.6 mm. The image data was acquired with a table feed of 3.8 mm per rotation (64-slice datasets) or 3.8 mm to 10 mm, individually adapted to the patient’s heart rate (dual-source datasets). Diastolic reconstructions were used, with reconstruction inter- vals varying from 250 ms to 400 ms before the R-peak. Three data- sets were reconstructed using a sharp (B46f) kernel, all others were reconstructed using a medium-to-smooth (B30f) kernel. The mean voxel size of the datasets is 0:32� 0:32� 0:4 mm3. 4.1.1. Training and test datasets To ensure representative training and test sets, the image qual- ity of and presence of calcium in each dataset was visually assessed by a radiologist with three years experience in cardiac CT. Image quality was scored as poor (defi