IBM刀片InfiniBand解决方案介绍

IBM BladeCenter InfiniBand Solution Introduction Ma Song 2 InfiniBand Technology Leadership  Industry Standard – Hardware, software, cabling, management – Design for clustering and storage interconnect  Price and Performance – 40Gb/s node-to-node – 120Gb/s switch-to-switch – 1us application latency – Most aggressive roadmap in the industry  Reliable with congestion management  Efficient – RDMA and Transport Offload – Kernel bypass – CPU focuses on application processing  Scalable for Petascale computing & beyond  End-to-end quality of service  Virtualization acceleration  I/O consolidation Including storage InfiniBand Delivers the Lowest Latency The InfiniBand Performance Gap is Increasing Fibre Channel Ethernet 60Gb/s 20Gb/s 120Gb/s 40Gb/s 240Gb/s (12X) 80Gb/s (4X)TodayToday 3 Commercial HPC Enterprise Database Target Markets and Applications Petascale Computing Broadcast Centers Low Latency Trading Apps 4 InfiniBand in the TOP500 Top500 Interconnect Placement 0 10 20 30 40 50 60 70 80 1-100 101-200 201-300 301-400 401-500 Top500 Placement N um be r o f C lu st er s InfiniBand All Proprietary High Speed GigE Top500 InfiniBand Trends 30 82 125 142 0 30 60 90 120 150 N um be r o f C lu st er s Nov-05 Nov-06 Nov 07 Nov 08  Mellanox InfiniBand makes the most powerful clusters – 4 of the top 10 (#1,#3, #6, #10) and 54 of the Top100  All InfiniBand clusters use Mellanox solutions  InfiniBand enables the most power efficient clusters – The only growing high speed interconnect solutions  Nov 08 list includes Mellanox 40Gb/s InfiniBand end-to-end Infiniband - Terminology  Switched Fabric – A computer network topology where many devices connect with each other via switches. – Infiniband Switched Fabrics include hosts, cables, switches - Host Channel Adapters (HCA) and Target Channel Adapters (TCA) Infiniband - Terminology  Host Channel Adapter (HCA) – The infiniband network interface card residing on the host.  Target Channel Adapter (TCA) – The infiniband network interface device residing on the fabric. – Switch, PTM, Bridge… HCA TCA *** MEM CPUCPU HCAHCA Processor Node *** MEM CPUCPU HCAHCA Processor Node Router To other IB Subnets, LAN’s, WAN’s, Internet Subnet Manager I/O M odule TCA I/O M odule TCA I/O M odule TCA I/O Chassis I/O M odule TCA I/O M odule TCA I/O M odule TCA Switch I/O Chassis Controller TCA InfiniBand Storage Subsystem SCSIEthernetVideoGraphics Fibre Channel Drives TCASwitch Switch Switch InfiniBand Elements Infiniband - Technology  Serial Links – Also referred to as lanes – Currently supports 2.5Ghz signal transfer per link. – Equivalent to 2.5Gb/s in data rate terms  Multiple serial links – Number of links (n) multiplied by link-bandwith (b) define total bandwith (T), yielding: n x b = T – 1 x 2.5Gb/s =2.5Gb/s, 4 x 2.5Gb/s =10Gb/s, 12 x 2.5Gb/s =30Gb/s 1X (2.5 (5/10) Gb/s) 4X (10 (20/40) Gb/s) 12X (30 (60/120) Gb/s) Infiniband - Technology  Data Rate – Also referred to as signalling rate or bit rate. – The average number of bits, characters, or blocks per unit, which passes between equipment in a data transmission system.  Double Data Rate (DDR) – Signalling rate is increased to twice the frequency of SDR, or 2Hz – Two clock cycles or bits per unit.  Single Data Rate (SDR) – Example: 1Hz Signalling rate – One clock cycle or bit per unit.  Quadruple Data Rate (QDR) – Signalling rate is increased to four times the frequency of SDR, or 4Hz – Four clock cycles or bits per unit. 10 InfiniBand – The Leading High-Performance Technology 3x0.8W0.25WPower/Gbps 5x$110$22Price/Gbps Less chassis to achieve same overall bandwidth100Gbps560GbpsAggregate Bandwidth per Chassis Matching performance to requirementsYesYesQoS Intelligent ReliabilityFutureYesPriority Flow-Control Maximum ReliabilityFutureYesLossless Fabric True Application Scale-outFutureYesL2 Multi-path Routing 20x2 microseconds100 nanosecondsLatency 4x10Gbps40GbpsBandwidth per port Benefit10GbEInfiniBand  InfiniBand has long secured its place as the interconnect of choice for high performance computing in both the public and commercial sectors  InfiniBand emerging as General Purpose Server Interconnect as customers evaluate how to solve i/o bottleneck due to virtualization and increasing processor density per node 11  40Gb/s InfiniBand deliver up to 100%! faster rendering – Ability to render 3D models at the required frame rates for high image quality Rendering Application- Direct Transport Compositor 12  Highest throughput is critical for high performance and productivity  Mellanox InfiniBand 40Gb/s enables highest scalability – Performance accelerates with cluster size Schlumberger ECLIPSE (FOURMILL) 0 1000 2000 3000 4000 5000 6000 4 8 12 16 20 22 24 Number of Nodes E la ps ed T im e (S ec on ds ) GigE 10GigE InfiniBand Schlumberger Oil and Gas Reservoir Software 13 Intel Nehalem and Voltaire QDR – A Perfect Match En ha nce s p erf orm an ce wh ile ma int ain ing sc ala bili ty Cisco Systems InfiniBand Switch Module Cisco Systems InfiniBand Switch Module (32R1900) Internal ports: – 14 internal 1X (2.5 Gbps) ports External ports: – four 4X (10 Gbps) ports (in the form of one 4X connector and one 12x connector, auto negotiate connection speed) Compatible Blade Expansion Card: Cisco Systems InfiniBand 1X Host Channel Adapter(32R1896) The Cisco 4X InfiniBand Switch Module (32R1756) Internal ports: – 14 internal 4X (10 Gbps) InfiniBand ports to the server blades – 2 internal 4X (10 Gbps) InfiniBand ports to the bridge modules External ports: – four 4X (10 Gbps) ports (in the form of two 4X connector and two 12x connector, auto negotiate connection speed) Compatible Blade Expansion Card: Cisco Systems 4X InfiniBand Expansion Card(32R1760) , 4X InfiniBand Pass-thru Module 4X InfiniBand Pass-thru Module(43W4419) Internal ports: – 14 InfiniBand 4X DDR ports External ports: – 14 InfiniBand 4X DDR ports Compatible Blade Expansion Card: Voltaire 2-port 4x InfiniBand Expansion Card (43W4420) 2-port 4x InfiniBand DDR Expansion Card (43W4423) the 4X InfiniBand DDR Pass-thru Module offers full non-blocking 4X DDR InfiniBand support to all 14 blade servers in a BladeCenter H chassis. It supports both 10Gbps single-data rate (SDR) and 20Gbps double-data rate (DDR) InfiniBand connectivity. The InfiniBand Pass-thru Module is a double-height module, and up to two can be installed in an IBM BladeCenter H, utilizing either switch bays 7 and 8, or switch bays 9 and 10 in the rear of the chassis. Voltaire 40 Gb InfiniBand Switch 16 Voltaire 40 Gb InfiniBand Switch Module(46M6005) Internal ports: – 14 internal 4X QDR/DDR/SDR InfiniBand to the server blades – Two internal 4X InfiniBand ports to the bridge modules External ports: – Up to 16 auto-sensing 4X QDR/DDR/SDR InfiniBand QSFP ports (40, 20, or 10 Gbps auto-negotiate) Blade Expansion Card: 2-port 40Gb InfiniBand Expansion Card(46M6001) (based on proven Mellanox ConnectX IB technology) The double-height InfiniBand switch module offers 14 internal ports, one to each server, and 16 ports out of the chassis per switch. This solution offers a no-compromise, congestion-free solution to meet even the most performance-hungry applications. Voltaire's high speed module also accommodates performance-optimized fabric designs utilizing a single BladeCenter chassis or stacking multiple BladeCenter chassis without requiring an external InfiniBand switch. 17 Hyper-Scale Design with IBM BladeCenter and Voltaire 126 Nodes Complete solution using BCH products only – No need for external switches  Scalable from 14 to 126 nodes  Lowest Latency (<200ns)  Simplest cabling  Aligned with PCIe Gen2 Performance  Most cost effective IB cluster  Integrated with Voltaire UFM enables routing optimization to maximize performance 18 Hyper-Scale Solution Summary  Hyper-Scale is a good fit for majority of application environments  Application Environments requiring 100% non-blocking fabric for node counts larger than 28, must add second tier of Voltaire 40Gb/s switching Less than 200ns64%721269 Less than 200ns65%641128 Less than 200ns67%56987 Less than 200ns69%48846 Less than 200ns71%40705 Less than 200ns71%32564 Less than 200ns86%24423 Less than 200ns100%16282 Less than 200ns100%0141 Switch Latency (node to node) Bi-Sectional Bandwidth* Maximum # of Cables Maximum # of Nodes # of BC Chassis *Bi-Sectional Bandwidth = [N / 2(N-1)] x 16/14 IBM BCH – 28 Blade Servers  2箱14节点刀片系统；  46M6005: Voltaire HSSM For IBM BCH –*14 Internal 4X DDR downlinks –*16 external 4X DDR QSFP uplinks  2台机架式管理、I/O节点；  Non-Blocking IBM BCH – 70 Blade Servers 7条IB线缆  5箱14节点刀片系统；  46M6005: Voltaire HSSM For IBM BCH –*14 Internal 4X DDR downlinks –*16 external 4X DDR QSFP uplinks  2台机架式管理、I/O节点；  2台外置具有管理功能36端 口InfiniBand交换机； –Voltaire 2036 36 4X DDR PORTS INTERNALLY MANAGED, DUAL PS-24 FRU  Non-Blocking IBM BCH – 252 Blade Servers  18箱14节点刀片系统；  46M6005: Voltaire HSSM For IBM BCH – *14 Internal 4X DDR downlinks – *16 external 4X DDR QSFP uplinks  7台外置具有管理功能36端 口InfiniBand交换机； – Voltaire 2036 36 4X DDR PORTS INTERNALLY MANAGED, DUAL PS-24 FRU  Non-Blocking 2条IB线缆 IBM BCH – 504 Blade Servers 1条IB线缆 23 Thanks IBM BladeCenter InfiniBand Solution Introduction InfiniBand Technology Leadership Target Markets and Applications InfiniBand in the TOP500 Infiniband - Terminology Infiniband - Terminology InfiniBand Elements Infiniband - Technology Infiniband - Technology InfiniBand – The Leading High-Performance Technology Rendering Application- Direct Transport Compositor Schlumberger Oil and Gas Reservoir Software Intel Nehalem and Voltaire QDR – A Perfect Match Cisco Systems InfiniBand Switch Module 4X InfiniBand Pass-thru Module Voltaire 40 Gb InfiniBand Switch Hyper-Scale Design with IBM BladeCenter and Voltaire Hyper-Scale Solution Summary IBM BCH – 28 Blade Servers IBM BCH – 70 Blade Servers IBM BCH – 252 Blade Servers IBM BCH – 504 Blade Servers Thanks