Pipeline all the things: Redis performance boost at application level
Posted 4 years ago in
While researching for my latest tutorial at DigitalOcean, I had the chance to play around with Redis and I've run all sorts of benchmark tests on a dozen of servers.
Something that was quite surprising to me was how many more requests per second your Redis server can handle if you just pack some commands together, in what is called pipelining.
This happens because even though Redis is quite fast, a request consumes other resources from the system. It's like buying several single bread slices each wrapped in a package, instead of getting a full loaf.
A regular benchmark with the redis-benchmark tool would go like this:
redis-benchmark -t get,set -q
SET: 95785.44 requests per second
GET: 97370.98 requests per second
Now this will simulate a pipeline of 16 commands:
redis-benchmark -t get,set -q -P 16
SET: 877193.00 requests per second
GET: 1351351.38 requests per second
From 97,370.90 to 1,351,351.38 requests per second!!!
This is an optimization that needs to happen at code level, just so we're clear. You can find more about pipelining with Redis in their official documentation. To give you an idea of how the code looks like, here's an actual Ruby code taken from that documentation, which makes a little benchmark test with and without pipelining:
require 'rubygems'
require 'redis'
def bench(descr)
start = Time.now
yield
puts "#{descr} #{Time.now-start} seconds"
end
def without_pipelining
r = Redis.new
10000.times {
r.ping
}
end
def with_pipelining
r = Redis.new
r.pipelined {
10000.times {
r.ping
}
}
end
bench("without pipelining") {
without_pipelining
}
bench("with pipelining") {
with_pipelining
}
And here is the output (also copied from their docs):
without pipelining 1.185238 seconds
with pipelining 0.250783 seconds
You can also make use of this feature in PHP with the PHP Redis Client library.
Check the Redis clients page for clients supporting pipelining in other languages.
Cheers!