构建高效的内存NoSQL数据库 - John Davies

���������������������� �� � ��������� � ��� !�������� ��������������� 1Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Integration - High Volume • Front Office • Very high volume ( 100-100,000 / sec), usually simple messages • Latency is critical (usually < 1ms) • FIX, FAST, ASN.1, IIOP are common payloads and protocols • Light-weight XML only (if any) • Credit card processing • ISO-8583, Binary, NVP, Batch • 10,000 / sec or 180m / day • Betting, Telcos etc. 2Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Numbers • Trading volumes • Average 100,000 per second (recent peek @350k/sec) - Per channel! •@ 512 bytes per message, 8 hours a day • That’s up to 180MB /sec and averaging ~1.5 TB/day, ~400TB/year • Now do that across 20 exchanges (28 with a recent client) • Perfect realm for grid (GemFire, Terracotta, GigaSpaces etc.) • Back-Testing • Data stored in EC2/S3 (~ 1TB per day) - compresses well • Can retrieve and day / time / channel / commodity on demand • Played back in real-time (timestamp modified) 3Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Integration - Complex • Middle Office • Volumes are medium to high (1-1000 / sec), very complex messages • Calculations are complex and grid/HPC is usually required • Derivative contracts on ISDA’s FpML • Corporate Actions (SWIFT, FpML & XBRL) • SEPA on ISO-20022, Murex, SwapsWire, CSVs are also common • XML widely used but usually over MQ & JMS • Tax processing •Wealth records, inheritance, PAYE etc. • Health • Medical records, billing 4Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Integration - High Value • Back Office • Low volume (10-1000 / hour), sometimes just a few per day • Very high value messages, strict compliance and validation • Often over $1T per day • Proprietary networks, mostly SWIFT • Back to numbers... • At 1% interest per year that’s 0.0027% per day. • 0.0027% of €1bn is €27,261.55 • That’s €1m interest per day for each €37bn • In the last week USD/EUR went from 1.3285 to 1.2999 that’s 2.2% • Changes of over 1% in a day on currency exchange are very common • The loss (or gain) of 1% in one day is €10m per €1billion! 5Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. CSV & Text files • One of the simplest but also one of the most common Name,Card Number,Expiry Date,Amount,Currency,Transaction Date,Commission Oliver Twist,4325-6486-3757-2678,10/06,100,GBP,26-09-2006,8,14988603,UK Uriah Heep,4724-7345-4725-7835,11/04,258,USD,21-12-2006,5,15688632,UK Mr Scrooge,4924-7264-1264-8536,04/09,1250.6,USD,13-09-2006,15,66846035,US Charles Dickens,4457-436-0087-0104,05/08,350,EUR,13-11-2006,10,93440252,DE RowCount=4 •While any programmer can parse this in minutes it’s not obvious how to... • Validate semantically & syntactically • Route based on values • Transform into another format • Manage multiple similar formats in a consistent way • Document the format / contract 6Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Let’s start with the basics •We’ll generate some messages and store them in memory... •We will start with Fix messages • Fix is a protocol used in the Front-Office • Typically high volume and low latency • There are dozens of version of Fix and most versions allow user-specific data in addition to the standard • There is a dialect called FAST which is compressed deltas only • There is also FixML but it’s not very common 7Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. • This is a FIX 4.4 Post-Trade Conformation • There’s no time for the “<“ and “>”, this is a low-latency domain 8=FIX.4.4 9=1 35=AK 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING 629=20020101-00:00:00.000 630=1 664=STRING 772=STRING 859=STRING 666=0 773=2 797=N 650=Y 665=4 453=1 448=STRING 447=B 452=1 802=1 523=STRING 803=1 60=20061122-00:00:00.000 75=20061122 55=STRING 65=STRING 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 762=STRING 200=200201 541=20020101 224=20020101 225=20020101 239=RP 226=1 227=1.0 228=1.0 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1.0 947=USD 206=0 231=1.0 223=1.0 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 691=STRING 667=200611 875=99 876=STRING 864=1 865=99 866=20061117 867=4.3 868=STRING 873=20061117 874=20061117 80=400 54=2 862=1 528=A 529=12 863=200 79=STRING 6=1.5 381=123.45 118=115.78 93=6 89=STRING 10=000 FIX 4.4 - Post-Trade Confirmation 8Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. FIX isn’t complex • FIX is “very” simple, it’s basically tag/value pairs 8=FIX.4.1 9=154 35=6 49=BRKR 56=INVMGR 34=236 52=19980604-07:58:48 23=115685 28=N 55=SPMI.MI 54=22 7=200000 44=10100.000000 25=H 10=159 • So, simple, the tag represents the field... • 44 refers to Price • 52 is sending Date/Time • 55 refers to the symbol • Basic but it’s still better than XML when latency comes into play 9Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Generating test message • Test data is vital, you need to test scenarios, those scenarios need to be consistent • You will also need regression tests for your finished code • There are a number of ways to do this • You can create each messages from scratch • This is time consuming and often generates messages with missing elements or inconsistent parts • You can take an existing message and change a few values • This is much easier, read a good message and then change a few values 10Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Java-Binding •We can tokenise the Fix message very easily but it starts to create issues • If we put it into a Map we find some of the keys repeating •We can’t use position as that changes too •We need to look at Java-Binding for the messages we’re using • This makes life easier when we’re using the messages • The can be handled through “bean” interfaces get() and set() • Many technologies word here, we use C24 (www.C24.biz) as they support the majority of Financial messages standards 11Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Create your basic message •We can hard-code the message... • Or we can read it in from a file... •We could store all the messages in files but the volumes start to get crazy • It’s best to re-generate every time if we can 12Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Setting up the data... • Once we have a bound Java object we can start to apply the changes to the data for our tests • For example, setting the message sequence number... • This guarantees that the message remains valid and that the fields/elements remain syntactically correct •We now create our test library 13Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Randomness • If you want a specific message to fail (say one in one million) then you have a few options • If it’s random then “if(i%1000000 == 0)” will work but it’s too random, sometimes you’ll get none, sometimes several • I prefer this method for statistical randomness where the numbers are not exact • It’s usually better to decide up front which message should fail • This can be hard-coded or randomly “decided”. 14Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Queues or Files? • So you’re creating messages in the fly how to we feed these to our test rig? • Typically you’re benchmarking so you don’t really want your test creation to effect the rest of the system • You can write them to file but a million small files tends to kill most operating systems • You could write them to a queue but that requires a lot of memory • One of the easiest ways it to create a zipped file and put them in there 15Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. BlockingQueues are good • If you’re writing a multi-threaded application then use a blockingQueue to feed the reading threads • This is reads from a zipped file into a BlockingQueue... 16Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Now we can parse the messages 17Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. So we’re ready to write • This is the GigaSpaces version... • First we set up the “Space” • Once we have the Space we can simply write objects to it... • Here we’re parsing a SWIFT MT-513 message 18Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. What’s happening •When you write to your “cache” or distributed memory, you are usually writing to several JVMs • This is an example from Coherence (now Oracle) • In this case there is a primary copy and a backup of each object 19Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Master / Worker Pattern • Effectively you can write tasks to the Cache and several “generic” workers can read the tasks and execute them, replacing the result back into the Cache • This is a very effective way of scaling dynamically • You just add more machines with worker threads (and usually memory) to the grid • It works beautifully well 20Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Scaling with Master/Worker • As long as we are writing to a distributed architecture pattern we can usually scale nicely • In this case if we need more calculation engines we simply add them to the cache • This can be done while the system is running Parse & Validate ! Invalid Message Channel Bad XML DB Cache Event-Driven Consumer Calculations Good Reporting Calculations Calculations DB Writer Accounting PoC 2 21Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Scaling with Master/Worker • As long as we are writing to a distributed architecture pattern we can usually scale nicely • In this case if we need more calculation engines we simply add them to the cache • This can be done while the system is running Parse & Validate ! Invalid Message Channel Bad XML DB Cache Event-Driven Consumer Calculations Good Reporting Calculations Calculations DB Writer Accounting PoC 2 Calculations Calculations 21Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Master/Worker in code... •We can listen for objects/entries being inserted to the cache and simply trigger our task... • In this case we’re reading unparsed messages from the cache via the eventListener, parsing them and putting them back into the cache 22Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. GemFire • Connect to the distributed system and get the Cache ref // Create / Find a cache (using the map interface) DistributedSystem ds = DistributedSystem.connect(); // Get the Singleton instance of the cache Cache cache = CacheFactory.create(system); • Instantiate your object and simply put into a Map // Create / Find Data Region “storage” in the cache Map store = (Map ) cache.getRegion(“storage”); // Write the message to the cache... store.put( message.getKey(), message ); • As you can see this couldn’t be easier 23Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Reading a Price from GemFire // Get Access to the Data Region “storage” and cast as a java.util.Map Map store = (Map)cache.getRegion(“storage"); // Retrieve the message ID = MSG-12345 Message message = (Message ) map.get(“MSG-12345” ); • All GemFire Regions are indexed on the key used in put() // Get Access to the Data Region “storage” Region store = cache.getRegion(“storage"); // If the retrieval is not based on primary key, you can use OQL // Retrieve the latest spot price for GBP/NOK SelectResults results = store.query(“getKey() = ‘MSG-123*’”); for (Iterator iter = results.iterator(); iter.hasNext(); ){ Message message = (Message) iter.next(); } • All Regions can be indexed on fields and/or methods 24Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. It’s even easier with Spring Integration •While most of this is a few lines of code a lot of the complexity can be hidden with Spring Integration and Spring Data • The input streams can be abstracted with Spring Integration and the caches with Spring Data • This also works for example with MongoDB •We can now read from multiple sources and write to a multitude of NoSQL targets 25Thursday, 26 April 12 • A SWIFT message generator •Writes MT-513s to unique files • Generates occasional errors • Spring Test • Reads the files, parses them using a C24 parser and writes them to GemFire Simple Demo Message Generator SWIFT MT513 Files File poler SWIFT MT513 C24 SWIFT parser SWIFT validation GemFire Writer C24-iO C24-iO 26Thursday, 26 April 12 Time for a demo • Fingers crossed!!! 27Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Grid technologies • GemFire • Originally an OOD, now has a pure Java implementation • Recently acquired by VMWare • Terracotta • Uses Java VM replication, • Recently acquired by Software AG • GigaSpaces • Originally the only viable implementation of Sun’s JavaSpaces • Coherence • Formally “Tangosol”, now owned by Oracle • Neo4J • The wild-card, a graph database 28Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Finally my “wild-card” • Neo4J is interesting but I’ve yet to find a problem where it’s the obvious solution • If I had to re-implement Twitter or Facebook I’d use Neo4J • I’ve come across a few trading systems that would benefit from the graph-database traversal • However the graphs were not deep so the use-case was not obvious • It is however pretty cool and very fast • I encourage you to take a look as it just need someone with a fresh mind to come up with the “killer use-case” other than social groups 29Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Neo4J • Some code snippet for Neo4J... private static enum RelType implements RelationshipType { ReBooked } GraphDatabaseService graphDb = new EmbeddedGraphDatabase( "./" ); registerShutdownHook( graphDb ); Node firstNode; Node secondNode; Relationship relationship; Transaction tx = graphDb.beginTx(); try { firstNode = graphDb.createNode(); firstNode.setProperty( Trade1.getId(), Trade1 ); secondNode = graphDb.createNode(); secondNode.setProperty( Trade2.getId(), Trade2 ); relationship = firstNode.createRelationshipTo( secondNode, RelType.ReBooked ); relationship.setProperty( “Date”, new java.lang.Date() ); tx.success(); } finally { tx.finish(); } firstNode Id: Trade1 secondNode Id: Trade2 relationship Date: Today Type: ReBooked 30Thursday, 26 April 12 Copyright © 2012 Incept5 Ltd. Thank you! "" 31Thursday, 26 April 12