NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
2
Table of Contents
Summary 3
Interactive software has changed 4
users – 4
applications – 5
infrastructure – 5
Application architecture has changed 6
Database architecture has not kept pace 7
Tactics to extend the useful scope of RDBMS technology 8
Sharding – 8
Denormalizing – 9
Distributed caching – 10
“NoSQL” database technologies 11
Mobile application data synchronization 13
Open source and commercial NoSQL database technologies 14
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
3
Summary
Interactive software (software with which a person iteratively interacts in real time) has
changed in fundamental ways over the last 35 years. The “online” systems of the 1970s have,
through a series of intermediate transformations, evolved into today’s Web and mobile
applications. These systems solve new problems for potentially vastly larger user populations,
and they execute atop a computing infrastructure that has changed even more radically over
the years.
The architecture of these software systems has likewise transformed. A modern Web
application can support millions of concurrent users by spreading load across a collection of
application servers behind a load balancer. Changes in application behavior can be rolled out
incrementally without requiring application downtime by gradually replacing the software
on individual servers. Adjustments to application capacity are easily made by changing the
number of application servers.
But database technology has not kept pace. Relational database technology, invented in
the 1970s and still in widespread use today, was optimized for the applications, users and
infrastructure of that era. In some regards, it is the last domino to fall in the inevitable
march toward a fully-distributed software architecture. While a number of bandaids have
extended the useful life of the technology (horizontal and vertical sharding, distributed
caching and data denormalization), these tactics nullify key benefits of the relational model
while increasing total system cost and complexity.
In response to the lack of commercially available alternatives, organizations such as Google
and Amazon were, out of necessity, forced to invent new approaches to data management.
These “NoSQL” or non-relational database technologies are a better match for the needs
of modern interactive software systems. But not every company can or should develop,
maintain and support its own database technology. Building upon the pioneering research
at these and other leading-edge organizations, commercial suppliers of NoSQL database
technology have emerged to offer database technology purpose-built to enable the cost-
effective management of data behind modern Web and mobile applications.
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
4
Interactive software has changed
As Table 1 below shows, there are fundamental differences in the users, applications and
underlying infrastructure between interactive software systems of the 1970s and those being
built today.
Users
In 1975, an interactive software system with 2,000 users represented the pinnacle of scale.
Few organizations built, deployed and supported such systems. American Airlines Sabre®
System (first installed in a travel agency in 1976) and Bank of America’s branch banking
automation system represent two notable interactive software systems that scaled to these
heights. But these were exceptions.
Today, applications accessed via the public Web have a potential user base of over two billion
users. Whether an online banking system, a social networking or gaming application, or
an e-commerce application selling goods and services to the public, there are innumerable
examples of software systems that routinely support a population of users many orders of
magnitude beyond the largest of the 1970s. A system with only 2,000 users is the exception
now, assuming the application is not an abject failure.
There is also user growth and churn today not seen in systems of the 1970s. Once rolled out,
the number of travel agents or tellers added to, or removed from, these systems was highly
predictable and relatively easy to manage (albeit somewhat manually and at measured
U
se
rs
In
fr
a
st
ru
ct
u
re
Interactive Software Then and Now
2,000 “online” users = End Point 2,000 “online” users = Starting Point
Static user population Dynamic user population
Data networking in its infancy Universal high-speed data networking
Memory scarce and expensive Memory plentiful and cheap
Centralized compting (Mainframes
and minicomputers
Distributed computing (Network servers and
virtual machines)
Circa 1975
“Online Applications”
Circa 2011
“Interactive Web Applications”
Business process automation Business process innovation
Highly structured data records Structured, semi-structured and unstructured data
A
p
p
li
ca
ti
o
n
s
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
5
pace). Users worked during well-defined office hours, providing windows of opportunity for
scheduled system downtime and maintenance.
Today, Web applications can serve a global population of users 24 hours a day, 365 days
per year. A newly launched software system can grow from no users to over a million users
almost literally overnight. Not all users are active on these systems at any given time, and
some users may use an application only a few times, never to return, and without providing
notice of their intent to leave.
Applications
In 1975, interactive software systems were primarily designed to automate what were
previously tedious, paper-based business processes – teller transactions, flight reservations,
stock trades. These “transactions” typically mirrored what clerical employees had been doing
“by hand” for decades – filling in fields on a structured business form, then filing or sending
forms to other employees who would tally them, update impacted ledgers and notate files
to effect “transactions.” Online transaction processing systems accelerated these tasks and
reduced the probability of error, but in most cases they were automating versus innovating.
Versus simply automating long-standing manual business processes, today’s Web
applications are breaking new ground in every direction. They are changing the nature of
communication, shopping, advertising, entertainment and relationship management. But
they are works in progress. There are no old business forms to simply mimic, or processes to
study and automate. It may be trite, but change is truly the only constant in these systems.
And a database has to be flexible enough to change with them.
Infrastructure
Perhaps the most obvious difference between interactive software then and now is the
infrastructure atop which they execute.
Centralization characterized the computing environment in the 1970s – mainframes and
minicomputers with shared CPU, memory and disk subsystems were the norm. Computer
networking was in its infancy. Memory was an expensive, scarce resource. Today, distributed
computing is the norm. Within a datacenter, servers and virtual machines are interconnected
via high-speed data networks. Users of software systems access them from even more widely
distributed desktop, laptop and mobile computing devices.
The IBM System/360 Model 195 was “the most powerful computer in IBM’s product line”
from August 1969 through the mid-1970s. The most powerful configuration of this system
shipped with 4MB of main (core) memory. Today, a single high-end microprocessor can
have more L1 cache memory on the processor die itself, with support for many orders of
magnitude more main memory.
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
6
Application architecture has changed
Directly addressing the aforementioned changes, and in contrast to the scale-up, centralized
approach of circa 1975 interactive software architecture, modern Web applications are built
to scale out – simply add more commodity Web servers behind a load balancer to support
more users. Scaling out is also a core tenet of the increasingly important cloud computing
model, in which virtual machine instances can be easily added or removed to match demand.
Figure 1: web application – Logic Scales out. to support more users for a web application, you simply
add more commodity web servers. as a result, system cost expands linearly with linear increases in
users, and performance remains constant. this model scales out indefinitely for all practical purposes.
The cost and performance curves are obviously attractive, but ultimately, flexibility is the big
win in this approach.
As users come and go, commodity servers (or virtual machines) can be quickly added or
removed from the server pool, matching capital and operating costs to the difficult-to-
predict size and activity level of the user population. And by distributing the load across
many servers, even across geographies, the system is inherently fault-tolerant, supporting
continuous operations.
As application needs change, new software can be gradually rolled out across subsets of the
overall server pool. Facebook, as an example, slowly dials up new functionality by rolling
out new software to a subset of their entire application server tier (and user population) in
a stepwise manner. If issues crop up, servers can be quickly reverted to the previous known
good build. All this can be done without ever taking the application “offline.”
Web Application - Logic Scales Out. To support more users for a web application, you simply add more commodity web servers.
As a result, system cost expands linearly with linear increases in users, and performance remains constant. This model scales out
indefinitely for all practical purposes.
Web Servers
Users
S
y
st
e
m
C
o
st
A
p
p
li
c
a
ti
o
n
R
e
sp
o
n
se
T
im
e
Load
Balancer
www.wellsfargo.com
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
7
Database architecture has not kept pace
In contrast to the sweeping changes in application architecture, relational database (RDBMS)
technology, a “scale-up” technology that has not fundamentally changed in over 40 years,
continues to be the default choice for holding data behind Web applications. Not surprisingly,
RDBMS technology reflects the realities (users, applications, and infrastructure) of the
environment that spawned it.
Because it is a technology designed for the centralized computing model, to handle more
users one must get a bigger server (increasing CPU, memory and I/O capacity) (see Figure 2).
Big servers tend to be highly complex, proprietary, and disproportionately expensive pieces
of engineered machinery, unlike the low-cost, commodity hardware typically deployed in
Web- and cloud-based architectures. And, ultimately, there is a limit to how big a server one
can purchase, even given an unlimited willingness and ability to pay.
Figure 2: web application – rDbmS Scales up. to support more users, you must get a bigger
database server for your rDbmS. as a result, system cost grows exponentially with linear increases in
users, and application response time degrades asymptotically.
While the scaling economics are certainly inferior to the model now employed at the
application logic tier, it is once again flexibility (or lack thereof) that is the “high-order bit”
to consider.
Upgrading a server is an exercise that requires planning, acquisition and application
downtime to complete. Given the relatively unpredictable user growth rate of modern
software systems, inevitably there is either over- or under-provisioning of resources. Too
much and you’ve overspent, too little and users can have a bad application experience or
the application can outright fail. And with all the eggs in a single basket, fault tolerance and
high-availability strategies are critically important to get right.
Figure 2: Web Application - RDBMS Scales Up. To support more users, you must get a bigger database server for your RDBMS.
As a result, system cost grows exponentially with linear increases in users, and application response time degrades asymptotically.
Relational
Database
RDBMS Software
installes on
comples,
expensive,
big iron.
Web Servers
Users
Won’t
scale
beyond
this
point
S
y
st
e
m
C
o
st
A
p
p
li
c
a
ti
o
n
R
e
sp
o
n
se
T
im
e
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
8
Perhaps the least obvious, but arguably the most damaging downside of using RDBMS
technology behind modern interactive software systems is the rigidity of the database
schema. As noted previously, we are no longer simply automating long-standing and well-
understood paper-based processes, where database record formats are pre-defined and
largely static. But RDBMS technology requires the strict definition of a “schema” prior to
storing any data into the database. Changing the schema once data is inserted is A Big Deal.
Want to start capturing new information you didn’t previously consider? Want to make
rapid changes to application behavior requiring changes to data formats and content? With
RDBMS technology, changes like these are extremely disruptive and therefore are frequently
avoided – the opposite behavior desired in a rapidly evolving business and
market environment.
Tactics to extend the useful scope of
RDBMS technology
In an effort to address the shortcomings of RDBMS technology when used behind modern
interactive software systems, developers have adopted a number of “bandaid” tactics.
Sharding
The RDBMS data model and transaction mechanics fundamentally assume a centralized
computing model – shared CPU, memory and disk. If the data for an application will not
fit on a single server or, more likely, if a single server is incapable of maintaining the I/O
throughput required to serve many users simultaneously, then a tactic known as sharding
is frequently employed. In this approach an application will implement some form of data
partitioning to manually spread data across servers. For example, users that live west of the
Mississippi River may have their data stored in one server, while those who live east of the
river will be stored in another.
While this does work to spread the load, there are undesirable consequences to the approach.
1. When you fill a shard, it is highly disruptive to re-shard. When you fill a
shard, you have to change the sharding strategy in the application itself.
For example, if you had partitioned your database by placing all accounts
east of the Mississippi on one server and all accounts west in another
and then reach the limits of their capacity, you must change the sharding
approach which means changing your application. Where previously the
application had to know “this is an east of the Mississippi customer and
thus I need to look in this database server,” now it must know “if it is east
of the Mississippi and below the Mason-Dixon Line, I need to look in that
server now.”
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
9
2. You lose some of the most important benefits of the relational model. You
can’t do “joins” across shards – if you want to find all customers that have
purchased a pair of wool socks but haven’t purchased anything in over
6 months, you must run a query on every server and piece the results
together in application software. In addition, you can’t do cross-node
locking when making updates. So one must ensure all data that could need
to be atomically operated on is resident on a single server, unless using an
external TP monitor system or complex logic in the application itself.
3. You have to create and maintain a schema on every server. If you have new
information you want to collect, you must modify the database schema on
every server, then normalize, retune and rebuild the tables. What was hard
with one server is a nightmare across many. For this reason, the default
behavior is to minimize the collection of new information.
Denormalizing
Before storing data in an RDBMS, a schema must be created defining precisely what data can
be stored in the database and the relationships between data elements. Data is decomposed
into a “normal form” and a record is typically spread across many interlinked tables. In order
to update a record, all these tables must be locked down and updated atomically, lest the
database become corrupted. This approach substantially limits the latency and throughput
of concurrent updates and is, for most practical purposes, impossible to implement across
server boundaries.
To support concurrency and sharding, data is frequently stored in a denormalized form when
an RDBMS is used behind Web applications. This approach potentially duplicates data in the
database, requiring updates to multiple tables when a duplicated data item is changed, but it
reduces the amount of locking required and thus improves concurrency.
At the limit the relational schema is more or less abandoned entirely, with data simply stored
in key-value form, where a primary key is paired with a data “blob” that can hold any data.
This approach allows the type of information being stored in the database to change without
requiring an update to the schema. It makes sharding much easier and allows for rapid
changes in the data model. Of course, just about all relational database functionality is lost
in the process (though if the database is sharded, much of the functionality was already lost).
Notwithstanding all these problems, many organizations are using relational technology in
precisely this manner given the familiarity of specific RDBMS technologies to developers and
operations teams, and, until recently, the lack of good alternatives.
NoSQL DatabaSe techNoLogy
© 2011 couchbaSe aLL rightS reServeD. www.couchbaSe.com
10
Distributed caching
Another tactic used to extend the useful scope of RDBMS technology has been to employ
distributed caching technologies, such as Memcached. Today, Memcached is a key ingredient
in the data architecture behind 18 of the top 20 largest (by user count) Web applications,
including Google, Wikipedia, Twitter, YouTube, Facebook, Craigslist, and tens of thousands
of other corporate and consumer Web applications. Most new Web applications now build
Memcached into their data architecture from day one.
Figure 3: memcached distributed caching technology extends the useful life of rDbmS technology
behind interactive web applications, spreading data across servers and leveraging the availability and
performance of main memory.
Memcached builds on two of the most important infrastructure transitions over the last 40
years: the shift to distributed computing atop high-speed data networks, and advances in
main memory (RAM) price/performance.
Memcached “sits in front” of an RDBMS system, caching recently accessed data in memory
and storing that data across any number of servers or virtual machines. When an application
needs access to data, rather than going directly to the RDBMS, it first checks Memcached to
see if the data is available there; if it is not, then the database is read by the application and
stored in Memcached for quick access next time it is needed.
While useful and effective to a point, Memcached and similar distributed caching
technologies used for this purpose are no panacea and can even create problems of their own:
1. Accelerates only data reads. Memcached was designed to accelerate the
reading of data by storing it in main memory, but it was not designed to
permanentl
本文档为【NoSQL技术说明书】,请使用软件OFFICE或WPS软件打开。作品中的文字与图均可以修改和编辑,
图片更改请在作品中右键图片并更换,文字修改请直接点击文字进行修改,也可以新增和删除文档中的内容。
该文档来自用户分享,如有侵权行为请发邮件ishare@vip.sina.com联系网站客服,我们会及时删除。
[版权声明] 本站所有资料为用户分享产生,若发现您的权利被侵害,请联系客服邮件isharekefu@iask.cn,我们尽快处理。
本作品所展示的图片、画像、字体、音乐的版权可能需版权方额外授权,请谨慎使用。
网站提供的党政主题相关内容(国旗、国徽、党徽..)目的在于配合国家政策宣传,仅限个人学习分享使用,禁止用于任何广告和商用目的。