Visualization of 3D Ultrasound Data-

Visualization of 3D Ultrasound Data Thomas R. Nelson University of California, San Diego T. Todd Elvins San Diego Supercomputer Center Ultrasound imaging plays a vital role in diagnosis. Future systems will acquire 30 data directly into the imager, where physicians can immediately examine, visualize, and interpret the patient's 30 anatomy ltrasound data acquisition will play an increasing role in U the future of medical imaging. Unlike magnetic resonance imaging (MRI) and computerized tomography (a), ultrasound offers interactive visualization of underlying anatomy in real time. Additionally, ultrasound equipment costs far less and does not use ionizing radiation or require specialized facilities. Most ultrasound imaging procedures are noninvasive com- pared to radiologic or laparoscopic procedures. As outpatient procedures become more common in health care delivery, spe- cialized catheters will further expand the role of ultrasound ex- amination. Sonographic image quality has benefitted from increasingly sophisticated computer technology, with systems integration ensuring better data acquisition, analysis, and display in future systems. The inherent flexibility and cost advantages of ultra- sound imaging will ultimately play a large role in increasing ef- ficiency and reducing health care costs. Multidimensional medical imaging Three-dimensional display of data, while available for some time in CT, single photon emission computed tomography (SPECT), positron emission tomography (PET), and MRI, has not achieved widespread clinical use for two reasons: the time required to obtain high-resolution image data and the often slow postprocessing, which requires significant operator in- volvement. Unlike other tomographic imaging techniques, ul- trasound offers interactive visualization of underlying anatomy while providing flexibility in viewing images from different ori- entations in real time. Currently, much pathology is readily di- agnosed with conventional 2D ultrasound equipment. However, complex cases often make it difficult even for specialists to vi- sualize 3D anatomy. While ultrasound imaging facilitates following the path of a tortuous vessel or complex underlying anatomy with consider- able freedom, patient orientation might limit the image projec- tion angle, making critical views unavailable. Also, although ultrasound's flexibility offers significant imaging advantages, it requires the physician to completely understand the underlying anatomy and to integrate multiple images to obtain a 3D im- pression of the anatomy. While such practice is routine, the physician often needs significant time to completely understand the patient's anatomy. Occasionally, the limitations of 2D ultrasound imaging (such as suboptimal projection angle or large patient size) make it 50 0272 17-16/Y3/1100-0050$03 000 1993 IEEE IEEE Computer Graphics & Applications -~ Authorized licensed use limited to: Xi'an University of Technology. Downloaded on June 4, 2009 at 06:20 from IEEE Xplore. Restrictions apply. Visualization of 30 Ultrasound Data L Video Signal I VIDEO DIGITIZER I difficult to differentiate between a suspicious finding and an artifact. Future ultrasound imaging systems will acquire 3D data directly into the ultrasound imager workstation, where physicians and sonographers can immediately examine, visual- ize, and interpret the patient’s 3D anatomy. Numerous efforts have focused on the development of 3D imaging techniques using ultrasound’s positioning flexibility and data acquisition speed.’ Much of this effort has targeted integrating transducer position information with the greyscale ~onogram.*.~ Ultrasound image signal-to-noise characteristics challenge researchers’ abilities to define organ interfaces and vascular anatomy with sufficient accuracy to permit successful use of standard 3D data analysis and display methods. Ongoing developments in multidimensional transducer technology,4 cou- pled with improvements in workstation performance and re- ductions in cost, should directly benefit 3D ultrasound data acquisition, analysis, and display. Scientific visualization Scientific visualization uses computer graphics techniques to assist scientists and physicians in understanding their data. It helps them extract meaningful information from numerical de- scriptions of complex phenomena using interactive imaging sys- Physicians need these systems not only for their own insight, but also to share results with their colleagues, the insti- tutions that support their research, and the general public. Few technical articles are published without captioned image data of some sort. A key area of scientific visualization is volume visualization,6 which projects a multidimensional data set onto a 2D image plane. The goal is to gain an understanding of the structure con- tained within the volumetric data. Most often researchers define the data set on a 3D lattice with one or more scalar or vector val- ues at each grid point. Methods for visualizing higher dimen- sional grids and irregular grids are active research areas. To be useful in medical imaging, volume visualization tech- niques must offer understandable data representations, quick data manipulation, and fast rendering. Physicians should be November 1993 Figure 1. A block diagram of the 3D ultrasound system showing the essential components used to acquire patient data. A Polhemus six-degree-of- magnetic-field sensor attached to the imaging transducer acquires video ultrasound image data with transducer position information encoded. We input color video and position data to a Sun Sparc Workstation equipped with Parallax’s real- time %bit XVideo video digitizer. After filtering images, we analyzed, rendered, and displayed them on a variety of graphics workstations. freedom (x, Y , z, €4, e,, 0,) able to change parameters and see the resultant image in real time. Presently, few systems can provide this type of perfor- mance. Therefore, volume visualization algorithm development and refinement are important areas of study, with an under- standing of the fundamental algorithms being essential to opti- mize data analysis methods. 3D ultrasound data acquisition Currently, 3D ultrasound image data are acquired as planar images using commercially available grey-scale and color-flow Doppler imaging equipment. Adequate 3D image reconstruc- tions have been obtained despite the need to correct for posi- tional registration differences, which arise from variations in the velocity of sound, and for regionally different spatial reso- lution, due to dynamic focusing. The work described here acquired ultrasound video image data with transducer position information into a graphics work- station, as previously described We encoded trans- ducer position information with a six-degree-of-freedom ( x , y , z , e,, e,, e,) magnetic-field sensor attached to the imaging trans- ducer. While we would prefer direct transfer of ultrasound im- age data from the scanner to the display workstation, the use of video signals simplifies data transfer to the workstation (see Figure 1). Unlike the case with other tomographic methods, ar- eas of acoustic shadowing or attenuation-common in ultra- sound imag ing40 not compromise other, well-visualized areas of the volume. Physicians can further improve study quality by minimizing patient motion with contoured pillows. Respiration effects are generally difficult to eliminate, and correction for heart motion requires synchronization of data acquisition to the same phase of the cardiac cycle. As part of the calibration procedure, we acquire images of a specially constructed test object along with transducer position and angle orientation data to calculate the sensor offsets from the actual image plane. Fiducial points in the test object permit us to determine the exact position and orientation of image pix- els, referenced to a fixed coordinate system and used to calcu- late image scaling constants for a given field of view and offset. 51 Authorized licensed use limited to: Xi'an University of Technology. Downloaded on June 4, 2009 at 06:20 from IEEE Xplore. Restrictions apply. Graphics in Medicine Figure 2. Diagram of volume reprojection techniques used to reproject 2D ultrasound image data into a volume data set. Prior to reprojection, we obtained scaling and offset calibration data yielding an absolute x, y, L coordinate for each pixel in the original image. Subsequently, individual pixels in the 2D ultrasound image are reprojected into the volume data matrix (see Figure 2) . Since all ultrasound image data are acquired via video data, changes in ultrasound image acquisition parameters (such as offset, mag- nification, and so forth) require generation of new calibration parameters. Doppler-shifted data resulting from regions of moving blood also can be acquired and displayed as color-coded images merged with grey-scale image data. Color Doppler data can be repro- jected with 2D image data or extracted and processed separately prior to reprojection into the 3D volume. Optimum image qual- ity depends on obtaining a strong Doppler signal intensity, which requires using an angle of less than 40 degrees with respect to the vessel axis. Ideally, all velocity components in a vessel should have the same direction with respect to the transducer. This helps to distinguish normality from pathology, which often produces ve- locity components having multiple directions. The signal quality of ultrasound volume data benefits from image compounding, which occurs when pixels in multiple planes are reprojected through the same voxel. However. pre- cise image registration is necessary for accurate volume repro- jection. Image compounding also reduces speckle intensity compared to 2D ultrasound. Since speckle intensity of ultra- sound images often exceeds specular echo intensity, traditional boundary extraction and segmentation algorithms have diffi- culty. Signal-to-noise ratios are further improved by filtering volume data using 3D median and Gaussian filters prior to ap- plication of visualization algorithms. Volume visualization methods Volume data are usually treated as an array of elements (vox- els). Since the rendering process often needs to resample the volume between grid points, the voxel approach assumes that the area around a grid point has the same value as the grid point. This has the advantage of making no assumptions about the behavior of data between grid points. That is, only known data values are used for generating an image. Although having a regular grid with identical elements has advantages, most med- ical imaging techniques use a rectilinear grid, where the voxels are axis-aligned rectangular prisms rather than cubes due to nonisotropic image resolution. The 3D ultrasound volume data in the work presented here use a cubic lattice. Many volume visualization algorithms share common steps.','0 The initial step is data acquisition and volume creation, as al- ready described. Next, an algorithm normalizes volume data so that they cover a good distribution of values, are high in contrast, and are free of noise and out-of-range values. Generally, the same processing is applied to the entire volume uniformly. Ideally, the data set is scaled so that the ratio of the dimen- sions is proportional to the ratio of the dimensions of the orig- inal object. It might be necessary to interpolate between values in adjacent slices to construct new slices. replicate existing slices, estimate missing values, or convert an irregular grid or nonorthogonal grid onto a Cartesian grid. Finally, 3D filtering or data enhancement is applied to ultrasound data because of the relatively poor signal-to-noise ratios. Depending on the visualization objective, data classification or thresholding might then be performed. After data classifi- cation, a mapping operation maps the elements into geometric or display primitives. This stage varies the most among volume visualization algorithms. Data classification results in primi- tives that can be stored, manipulated, intermixed with exter- nally defined primitives, shaded, transformed to screen space, and then displayed. Shading and transforming steps can be re- ordered and done in several ways. Volume visualidon algorithms The fundamental algorithms for volume visualization fall into three categories: multiplanar slice projection, surface fitting, and volume rendering. Volume rendering algorithms include approaches such as ray casting, integration methods, and splat- ting. The latter method, splatting, is sometimes called a projec- tion method. Muhiplanar slicing Extraction of a planar image of arbitrary orientation at a par- ticular location in a 3D data set uses well-described coordinate transformations and rotations. This is the most computationally straightforward approach to review data throughout the vol- ume. Processing requirements are minimal, although optimal results require isotropic data or interpolation schemes to pre- serve aspect ratios. Multiplanar slice methods are one of several interactive tech- niques available for physician review of patient data. Interactive display of planar slices offers the physician retrospective eval- uation of anatomy, particularly viewing of arbitrary planes per- pendicular to the primary exam axis and other orientations not possible during data acquisition. Figure 3 shows planar ultra- sound images extracted from a carotid artery and jugular vein study in the region of the carotid bifurcation. 52 IEEE Computer Graphics & Applications Authorized licensed use limited to: Xi'an University of Technology. Downloaded on June 4, 2009 at 06:20 from IEEE Xplore. Restrictions apply. Visualization of 30 Ultrasound Data Surface fitting Surface-fit algorithms (sometimes called feature extraction or isosurfacing) typically fit planar surface primitives, such as poly- gons or patches, to constant-value contour surfaces in volu- metric data sets. The user begins by choosing a threshold value. Then geometric primitives are automatically fit to the high-con- trast contours in the volume that match the threshold. The sur- face-fit approach includes contour connecting, marching cubes,1° marching tetrahedra, dividing cubes, and others. Figure 4 shows a surface-fitted image of the artery and vein derived from blood velocity data that uses the marching cubes approach. Surface-fit methods are typically faster than volume render- ing methods because they only traverse the volume once to ex- tract surfaces. After extracting the surfaces, rendering hardware and well-known rendering methods quickly render the surface primitives each time the user changes a viewing or lighting pa- rameter. Changing the surface-fit threshold value is time con- suming because it requires revisiting all of the cells to extract a new set of surface primitives. Surface-fit methods suffer from several problems, such as oc- casional false positive and negative surface pieces, and incorrect handling of small features and branches in the data. Artifacts can be a serious concern in medicine, since physicians could in- terpret them incorrectly as features in the data. Surface fitting provides a good means of visualizing spatial relationships for the entire volume in a readily comprehended manner, although small features may be poorly visualized. Volume rendering Volume rendering methods are characterized by mapping voxels directly into screen space without using geometric prim- itives as an intermediate representation. One disadvantage of Figure 4. Rendered 3D ultrasound image of the artery and vein surfaces are based on the vessel wall and blood interface determined from color Doppler images derived from the study shown in F i i 3. November 1993 Figure 3. Planar images from a 3D color Doppler ultrasound carotid study showing (left) g r e y d e images and (below) velocity and grey-scale oblique projections of the carotid artery at the bifurcation. Interactive display allows the diagnostician Oexibility in selecting the plane and its orientation, whether parallel to the initial image projection or at an oblique projection for the anatomy of interest, as shown here. 53 Authorized licensed use limited to: Xi'an University of Technology. Downloaded on June 4, 2009 at 06:20 from IEEE Xplore. Restrictions apply. Graphics in Medicine using volume rendering methods is that the entire data set must be traversed each time an image is rendered. A low-resolution pass or random sampling of the data is sometimes used to quickly create low-quality images for parameter checking. In volume rendering, each element in a volume contributes color to the final image. The amount of color contributed by an ele- ment is determined by the color and opacity of the current el- ement and the colors and opacities of all elements between the current element and the image plane. The most often used volume visualization algorithm for pro- ducing high-quality images is ray casting.11-14 Ray casting con- ducts an image-order traversal of the image plane pixels, finding a color and opacity for each. A ray is fired from each pixel through the data volume. The opacities and shaded colors en- countered along the ray are blended to find the opacity and color of the pixel. In ray casting, rays continue in a straight line until the opac- ity encountered by the ray equals unity or the ray exits the rear Figure 6. Mrvrimum intensity images from a 3D ultrasound study of a 22-week fetus. The bones of the developing spine vertebral bodies and ribs are clearly seen in both the coronal limited thickness section (a) and volume (b) projection images. Both images were produced from a subvolume containing the fetus to highlight the strncture of the spine and ribs. Figure 5. Ray-cast 3D ultrasound images of an 18-week fetus incorporating transparency from slightly different projections. The separate bows of the skull are clearly visualized, in addition to the mandible and other facial features. The bones of the chest, abdomen, and pelvis are also dearly visible. of the volume. No shadows or reflections are generated, so these useful visual cues must be added to optimize visual pre- sentation. Figure 5 shows a ray-cast image for a fetus. We can also use ray-casting methods to project the maximum intensity along a ray onto the plane (see Figure 6). Ray casting is CPU intensive, but the images show the entire data set, depending on opacity and intensity values, not just a collection of thin surfaces as in surface fitting. Ray casting can be parallelized at the pixel level, since rays from all of the pix- els in the image plane can be cast independently. A recently developed volume-rendering algorithm called ~plat t ing’~ performs a front-to-back object-order traversal of the voxels in the volumetric data set. Each voxel’s contribution to the image is calculated and composited using a series of table lookups. Some splatting optimizations are given in Hanrahan and Laur.I5 Splatting has the advantage that the viewer sees all of the data values. Figure 7 shows splatting images for a fetus. Data classification Data classification is the most difficult function that a vol- ume visualization user has to perform. Data classification means choosing a threshold if you want to use a surface-fitting algo- rithm or choosing color (brightness) and opacity (light attenu- ation) values to go with each possible range of data values if you IEEE Computer Graphics & Appl