the little redis book

About This Book License The Little Redis Book is licensed under the Attribution-NonCommercial 3.0 Unported license. You should not have paid for this book. You are free to copy, distribute, modify or display the book. However, I ask that you always attribute the book to me, Karl Seguin, and do not use it for commercial purposes. You can see the full text of the license at: http://creativecommons.org/licenses/by-nc/3.0/legalcode About The Author Karl Seguin is a developer with experience across various fields and technologies. He’s an active contributor to Open- Source Software projects, a technical writer and an occasional speaker. He’s written various articles, as well as a few tools, about Redis. Redis powers the ranking and statistics of his free service for casual game developers: mogade.com. Karl wrote The Little MongoDB Book, the free and popular book about MongoDB. His blog can be found at http://openmymind.net and he tweets via @karlseguin With Thanks To A special thanks to Perry Neal for lending me his eyes, mind and passion. You provided me with invaluable help. Thank you. Latest Version The latest source of this book is available at: http://github.com/karlseguin/the-little-redis-book 1 Introduction Over the last couple years, the techniques and tools used for persisting and querying data have grown at an incredible pace. While it’s safe to say that relational databases aren’t going anywhere, we can also say that the ecosystem around data is never going to be the same. Of all the new tools and solutions, for me, Redis has been the most exciting. Why? First because it’s unbelievably easy to learn. Hours is the right unit to use when talking about length of time it takes to get comfortable with Redis. Secondly, it solves a specific set of problems while at the same time being quite generic. What exactly does that mean? Redis doesn’t try to be all things to all data. As you get to know Redis, it’ll become increasingly evident what does and what does not belong in it. And when it does, as a developer, it’s a great experience. While you can build a complete system using Redis only, I think most people will find that it supplements their more generic data solution - whether that be a traditional relational database, a document-oriented system, or something else. It’s the kind of solution you use to implement specific features. In that way, it’s similar to an indexing engine. You wouldn’t build your entire application on Lucene. But when you need good search, it’s a much better experience - for both you and your users. Of course, the similarities between Redis and indexing engines end there. The goal of this book is to build the foundation you’ll need to master Redis. We’ll focus on learning Redis’ five data structures and look at various data modeling approaches. We’ll also touch on some key administrative details and debugging techniques. 2 Getting Started We all learn differently: some like to get their hands dirty, some like to watch videos, and some like to read. Nothing will help you understand Redis more than actually experiencing it. Redis is easy to install and comes with a simple shell that’ll give us everything we need. Let’s take a couple minutes and get it up and running on our machine. On Windows Redis itself doesn’t officially support Windows, but there are options available. You wouldn’t run these in production, but I’ve never experienced any limitations while doing development. A port by Microsoft Open Technologies, Inc. can be found at https://github.com/MSOpenTech/redis. As of this writing the solution is not ready for use in production systems. Another solution, which has been available for some time, can be found at https://github.com/dmajkic/redis/downloads. You can download the most up to date version (which should be at the top of the list). Extract the zip file and, based on your architecture, open either the 64bit or 32bit folder. On *nix and MacOSX For *nix and and Mac users, building it from source is your best option. The instructions, along with the latest version number, are available at http://redis.io/download. At the time of this writing the latest version is 2.6.2; to install this version we would execute: wget http://redis.googlecode.com/files/redis-2.6.2.tar.gz tar xzf redis-2.6.2.tar.gz cd redis-2.6.2 make (Alternatively, Redis is available via various package managers. For example, MacOSX users with Homebrew installed can simply type brew install redis.) If you built it from source, the binary outputs have been placed in the src directory. Navigate to the src directory by executing cd src. Running and Connecting to Redis If everything worked, the Redis binaries should be available at your fingertips. Redis has a handful of executables. We’ll focus on the Redis server and the Redis command line interface (a DOS-like client). Let’s start the server. In Windows, double click redis-server. On *nix/MacOSX run ./redis-server. If you read the start up message you’ll see a warning that the redis.conf file couldn’t be found. Redis will instead use built-in defaults, which is fine for what we’ll be doing. 3 Next start the Redis console by either double clicking redis-cli (Windows) or running ./redis-cli (*nix/MacOSX). This will connect to the locally-running server on the default port (6379). You can test that everything is working by entering info into the command line interface. You’ll hopefully see a bunch of key-value pairs which provide a great deal of insight into the server’s status. If you are having problems with the above setup I suggest you seek help in the official Redis support group. 4 Redis Drivers As you’ll soon learn, Redis’ API is best described as an explicit set of functions. It has a very simple and procedural feel to it. This means that whether you are using the command line tool, or a driver for your favorite language, things are very similar. Therefore, you shouldn’t have any problems following along if you prefer to work from a programming language. If you want, head over to the client page and download the appropriate driver. 5 Chapter 1 - The Basics What makes Redis special? What types of problems does it solve? What should developers watch out for when using it? Before we can answer any of these questions, we need to understand what Redis is. Redis is often described as an in-memory persistent key-value store. I don’t think that’s an accurate description. Redis does hold all the data in memory (more on this in a bit), and it does write that out to disk for persistence, but it’s much more than a simple key-value store. It’s important to step beyond this misconception otherwise your perspective of Redis and the problems it solves will be too narrow. The reality is that Redis exposes five different data structures, only one of which is a typical key-value structure. Understanding these five data structures, how they work, what methods they expose and what you can model with them is the key to understanding Redis. First though, let’s wrap our heads around what it means to expose data structures. If we were to apply this data structure concept to the relational world, we could say that databases expose a single data structure - tables. Tables are both complex and flexible. There isn’t much you can’t model, store or manipulate with tables. However, their generic nature isn’t without drawbacks. Specifically, not everything is as simple, or as fast, as it ought to be. What if, rather than having a one-size-fits-all structure, we used more specialized structures? There might be some things we can’t do (or at least, can’t do very well), but surely we’d gain in simplicity and speed? Using specific data structures for specific problems? Isn’t that how we code? You don’t use a hashtable for every piece of data, nor do you use a scalar variable. To me, that defines Redis’ approach. If you are dealing with scalars, lists, hashes, or sets, why not store them as scalars, lists, hashes and sets? Why should checking for the existence of a value be any more complex than calling exists(key) or slower than O(1) (constant time lookup which won’t slow down regardless of how many items there are)? 6 The Building Blocks Databases Redis has the same basic concept of a database that you are already familiar with. A database contains a set of data. The typical use-case for a database is to group all of an application’s data together and to keep it separate from another application’s. In Redis, databases are simply identified by a number with the default database being number 0. If you want to change to a different database you can do so via the select command. In the command line interface, type select 1. Redis should reply with an OK message and your prompt should change to something like redis 127.0.0.1:6379[1]>. If you want to switch back to the default database, just enter select 0 in the command line interface.. Commands, Keys and Values While Redis is more than just a key-value store, at its core, every one of Redis’ five data structures has at least a key and a value. It’s imperative that we understand keys and values before moving on to other available pieces of information. Keys are how you identify pieces of data. We’ll be dealing with keys a lot, but for now, it’s good enough to know that a key might look like users:leto. One could reasonably expect such a key to contain information about a user named leto. The colon doesn’t have any special meaning, as far as Redis is concerned, but using a separator is a common approach people use to organize their keys. Values represent the actual data associated with the key. They can be anything. Sometimes you’ll store strings, sometimes integers, sometimes you’ll store serialized objects (in JSON, XML or some other format). For the most part, Redis treats values as a byte array and doesn’t care what they are. Note that different drivers handle serialization differently (some leave it up to you) so in this book we’ll only talk about string, integer and JSON. Let’s get our hands a little dirty. Enter the following command: set users:leto ”{name: leto, planet: dune, likes: [spice]}” This is the basic anatomy of a Redis command. First we have the actual command, in this case set. Next we have its parameters. The set command takes two parameters: the key we are setting and the value we are setting it to. Many, but not all, commands take a key (and when they do, it’s often the first parameter). Can you guess how to retrieve this value? Hopefully you said (but don’t worry if you weren’t sure!): get users:leto Go ahead and play with some other combinations. Keys and values are fundamental concepts, and the get and set commands are the simplest way to play with them. Create more users, try different types of keys, try different values. 7 Querying As we move forward, two things will become clear. As far as Redis is concerned, keys are everything and values are nothing. Or, put another way, Redis doesn’t allow you to query an object’s values. Given the above, we can’t find the user(s) which live on planet dune. For many, this is will cause some concern. We’ve lived in a world where data querying is so flexible and powerful that Redis’ approach seems primitive and unpragmatic. Don’t let it unsettle you too much. Remember, Redis isn’t a one-size-fits-all solution. There’ll be things that just don’t belong in there (because of the querying limitations). Also, consider that in some cases you’ll find new ways to model your data. We’ll look at more concrete examples as we move on, but it’s important that we understand this basic reality of Redis. It helps us understand why values can be anything - Redis never needs to read or understand them. Also, it helps us get our minds thinking about modeling in this new world. Memory and Persistence We mentioned before that Redis is an in-memory persistent store. With respect to persistence, by default, Redis snapshots the database to disk based on how many keys have changed. You configure it so that if X number of keys change, then save the database every Y seconds. By default, Redis will save the database every 60 seconds if 1000 or more keys have changed all the way to 15 minutes if 9 or less keys has changed. Alternatively (or in addition to snapshotting), Redis can run in append mode. Any time a key changes, an append-only file is updated on disk. In some cases it’s acceptable to lose 60 seconds worth of data, in exchange for performance, should there be some hardware or software failure. In some cases such a loss is not acceptable. Redis gives you the option. In chapter 6 we’ll see a third option, which is offloading persistence to a slave. With respect to memory, Redis keeps all your data in memory. The obvious implication of this is the cost of running Redis: RAM is still the most expensive part of server hardware. I do feel that some developers have lost touch with how little space data can take. The Complete Works of William Shakespeare takes roughly 5.5MB of storage. As for scaling, other solutions tend to be IO- or CPU-bound. Which limitation (RAM or IO) will require you to scale out to more machines really depends on the type of data and how you are storing and querying it. Unless you’re storing large multimedia files in Redis, the in-memory aspect is probably a non-issue. For apps where it is an issue you’ll likely be trading being IO-bound for being memory bound. Redis did add support for virtual memory. However, this feature has been seen as a failure (by Redis’ own developers) and its use has been deprecated. (On a side note, that 5.5MB file of Shakespeare’s complete works can be compressed down to roughly 2MB. Redis doesn’t do auto-compression but, since it treats values as bytes, there’s no reason you can’t trade processing time for RAM by compressing/decompressing the data yourself.) 8 Putting It Together We’ve touched on a number of high level topics. The last thing I want to do before diving into Redis is bring some of those topics together. Specifically, query limitations, data structures and Redis’ way to store data in memory. When you add those three things together you end up with something wonderful: speed. Some people think “Of course Redis is fast, everything’s in memory.” But that’s only part of it. The real reason Redis shines versus other solutions is its specialized data structures. How fast? It depends on a lot of things - which commands you are using, the type of data, and so on. But Redis’ performance tends to be measured in tens of thousands, or hundreds of thousands of operations per second. You can run redis-benchmark (which is in the same folder as the redis-server and redis-cli) to test it out yourself. I once changed code which used a traditional model to using Redis. A load test I wrote took over 5 minutes to finish using the relational model. It took about 150ms to complete in Redis. You won’t always get that sort of massive gain, but it hopefully gives you an idea of what we are talking about. It’s important to understand this aspect of Redis because it impacts how you interact with it. Developers with an SQL background often work at minimizing the number of round trips they make to the database. That’s good advice for any system, including Redis. However, given that we are dealing with simpler data structures, we’ll sometimes need to hit the Redis server multiple times to achieve our goal. Such data access patterns can feel unnatural at first, but in reality it tends to be an insignificant cost compared to the raw performance we gain. In This Chapter Although we barely got to play with Redis, we did cover a wide range of topics. Don’t worry if something isn’t crystal clear - like querying. In the next chapter we’ll go hands-on and any questions you have will hopefully answer themselves. The important takeaways from this chapter are: • Keys are strings which identify pieces of data (values) • Values are arbitrary byte arrays that Redis doesn’t care about • Redis exposes (and is implemented as) five specialized data structures • Combined, the above make Redis fast and easy to use, but not suitable for every scenario 9 Chapter 2 - The Data Structures It’s time to look at Redis’ five data structures. We’ll explain what each data structure is, what methods are available and what type of feature/data you’d use it for. The only Redis constructs we’ve seen so far are commands, keys and values. So far, nothing about data structures has been concrete. When we used the set command, how did Redis know what data structure to use? It turns out that every command is specific to a data structure. For example when you use set you are storing the value in a string data structure. When you use hset you are storing it in a hash. Given the small size of Redis’ vocabulary, it’s quite manageable. Redis’ website has great reference documentation. There’s no point in repeating the work they’ve already done. We’ll only cover the most important commands needed to understand the purpose of a data structure. There’s nothing more important than having fun and trying things out. You can always erase all the values in your database by entering flushdb, so don’t be shy and try doing crazy things! Strings Strings are the most basic data structures available in Redis. When you think of a key-value pair, you are thinking of strings. Don’t get mixed up by the name, as always, your value can be anything. I prefer to call them “scalars”, but maybe that’s just me. We already saw a common use-case for strings, storing instances of objects by key. This is something that you’ll make heavy use of: set users:leto ”{name: leto, planet: dune, likes: [spice]}” Additionally, Redis lets you do some common operations. For example strlen can be used to get the length of a key’s value; getrange returns the specified range of a value; append appends the value to the existing value (or creates it if it doesn’t exist already). Go ahead and try those out. This is what I get: > strlen users:leto (integer) 42 > getrange users:leto 27 40 ”likes: [spice]” > append users:leto ” OVER 9000!!” (integer) 54 Now, you might be thinking, that’s great, but it doesn’t make sense. You can’t meaningfully pull a range out of JSON or append a value. You are right, the lesson here is that some of the commands, especially with the string data structure, only make sense given specific type of data. Earlier we learnt that Redis doesn’t care about your values. Most of the time that’s true. However, a few string commands are specific to some types or structure of values. As a vague example, I could see the above append and 10 getrange commands being useful in some custom space-efficient serialization. As a more concrete example I give you the incr, incrby, decr and decrby commands. These increment or decrement the value of a string: > incr stats:page:about (integer) 1 > incr stats:page:about (integer) 2 > incrby ratings:video:12333 5 (integer) 5 > incrby ratings:video:12333 3 (integer) 8 As you can imagine, Redis strings are great for analytics. Try incrementing users:leto (a non-integer value) and see what happens (you should get an error). A more advanced example is the setbit and getbit commands. There’s a wonderful post on how Spool uses these two commands to efficiently answer the question “how many unique visitors did we have today”. For 128 million users a laptop generates the answer in less than 50ms and takes only 16MB of memory. It isn’t important that you understand how bitmaps work, or how Spool uses them, but rather to understand that Redis strings are more powerful than they initially seem. Still, the most common cases are the ones we gave above: sto