A Web Admin Console for Redis, Part One

Redis seems to grow more popular — and more important — every year. I first heard about it five or so years ago, but it's been around quite a bit longer than that. As a cache it works admirably well: its memory usage scales pretty linearly with its contents, and response times are hard to match. It is just a cache, though — it doesn't come with much in the way of an admin interface. If you install it locally, you get the redis-cli command-line tool as part of the bundle which covers the basics of routine maintenance tasks. Still, searching for keys is a big hassle on the command-line. I often find myself needing to scan through a Redis instance for keys that match a specific pattern and modify or remove them on a case-by-case basis. Although I can, and have, scripted this in Python, there are occasions where I'd like to browse through them. It's come up often enough that I finally broke down and put together a simple web-based Redis browser.

About 20 or so years ago, I would have made this a pure desktop app, but web UI's are too useful to ignore anymore. I can't do it completely in Javascript, though, because even running locally Javascript can't connect to a Redis instance. Instead, I put together a basic web server in Python (because Dear God, anything but Node) that handles the connection to Redis and a web UI that interacts with it.

Writing a basic server-side web app in Python is not too difficult. Modern Python installations include a simple web server that you can easily customize by extending SimpleHTTPRequestHandler and overriding do_verb (where verb is one of GET, POST, DELETE, etc.) to customize behavior. What I want here is a server that opens a persistent connection to a Redis instance and accepts a scan request as a GET query. Since it's supposed to be a single-user "server" I don't need to worry about memory management or re-entrancy much and can just keep the current query in global state rather than managing it in a session as I technically should.

I'll start with a simple server that accepts a Redis host as a command-line parameter and just looks up the value of a key in the redis cache when a properly-formed request is supplied.

import sys
import json
import redis
from http.server import HTTPServer
from http.server import SimpleHTTPRequestHandler

redisConn = None

class RedisHTTPRequestHandler(SimpleHTTPRequestHandler):
  def parseQuery(self, path):
    if path.find('?') > -1:
      query = path.split('?')[1]
      return {key: value for [key, value] in map(lambda x: x.split('='), query.split('&'))}
    else:
      return {}

  def ensure_required_params(self, params, requiredKeys):
    missingKeys = set(requiredKeys).difference(set(params.keys()))
    if len(missingKeys) > 0:
      self.send_error(400, 'Missing required parameters: %s' % (','.join(missingKeys)))
      return False
    else:
      return True

  def send_text(self, obj):
    jsonResponse = json.dumps(obj)

    self.send_response(200, 'OK')
    self.send_header('Content-Type', 'application/json')
    self.send_header('Content-Length', len(jsonResponse))
    self.end_headers()

    self.wfile.write(bytes(jsonResponse, 'UTF-8'))

  def do_GET(self):
    if self.path.startswith('/get'):
      params = self.parseQuery(self.path)
      if self.ensure_required_params(params, ['key']):
        val = redisConn.get(params['key'])
        self.send_text(val)
    else:
      super().do_GET()

if __name__ == '__main__':
  if len(sys.argv) < 2:
    print('Usage: %s [redis host name]' % sys.argv[0])
    sys.exit()

  redisConn = redis.Redis(sys.argv[1], decode_responses=True)
  server_address = ('', 8000)
  httpd = HTTPServer(server_address, RedisHTTPRequestHandler)
  httpd.serve_forever()

Listing 1: Simple Server to query Redis instance

It's a bit long but should be easy enough to follow: if a GET request is received with the path get, it's expected to contain at least a key parameter. If so, it looks up the key and returns the value. Otherwise, it lets the parent class default respond to the GET, whose behavior is to return a response from the local file system (with minimal security checks, so don't run this on a remote server!)

Ok, so far so good, now we need an interface. The default page will be pretty spartan and just prompt the user to input a key. When the get button is pressed, the value of the key will be returned.

<html>
<head>
<title>Redis Maintenance Tool</title>
<style>
#error {
  color: red;
}
</style>
<script>
function issueGetRequest(path, onSuccess)  {
  document.getElementById("error").innerHTML = "";

  var req = new XMLHttpRequest();
  req.open("GET", path);
  req.onreadystatechange = function(evt)  {
    var req = evt.target;
    if (req.readyState == 4)  {
      if (req.status == 200)  {
        onSuccess(req.responseText);
      } else  {
        document.getElementById("error").innerHTML = req.statusText;
      }
    }
  }
  req.send();
}

function get()  {
  var key = document.getElementById("key").value;
  issueGetRequest('/get?key=' + key, function(txt)  {
    document.getElementById("target").innerHTML = txt;
  });
}
</script>
</head>
<body>
Key: <input id="key" /><br/>
<button onclick="get()">Scan</button>
<div id="target"></table>
<div id="error"></div>
</body>
</html>

Listing 2: Landing page

Interesting, but not very useful... I can do the exact same thing from the command line without the UI after all. What I want to be able to do here is put in a key pattern and get back all of the matches. The redis scan command is designed for exactly this, but it's a little bit trickier to use than you might think if you're not familiar with it. If you issue the command scan 0 match cust_* count 100, for instance, you would probably expect to get back the first 100 keys in the cache that match the pattern "cust_*". That's not how scan works, though: instead, it queries the "first" 100 keys and returns the ones that match the pattern, if any. So you might mistakenly believe, if you get back an empty response from scan, that nothing in the cache matches the pattern, but if you keep querying, you may eventually find something.

So what I actually want is a query that keeps running scans until it finds a set number of matches and returns them — and to be useful, I want to be able to save my state and allow paging back and forth through the results. This is a bit tricky because it requires some level of state maintenance. I could do this by scanning the entire keyspace and storing the results, but that's a bit wasteful if I'm only paging through a few hundred at a time. Instead what I'll do is to keep track of the cursor value of the result of the previous page and the current page, along with where in the Redis result the page demarcation landed.

Every time you issue a scan to Redis, you get back a (possibly empty) list of matches along with a "next page" cursor which you're supposed to pass on the next scan to query the next n possible keys. It's designed as forward-only, though — if you want to support paging backwards, you have to store all the previous cursors in order to potentially page back over the responses. Note that I'm not designing for multiple active scans although Redis itself does actually allow for this.

So, with each scan, I can encounter one of three cases:

  1. the scan results include exactly as many elements as I want on a page
  2. the scan results include fewer elements than I want on a page
  3. the scan results include more elements than I want on a page
In the first case, I can return the current page, store the cursor that generated the page (so that I can go back to it), and include the next and previous cursors (the stored cursor of the page before this one) in the response. In the second, I need to issue a new request, but if this is the first result for the current page, to store the cursor that I initially issued so that I can go back to this page. In the third, I need to store the cursor as well, but also keep track of how many results I took from this result list: when a new page is requested, I'll need to start from there. In fact, I may need to start from there a few times if the scan size is much greater than the page size.

I can accomplish this with a new route in my Python server named scan that accepts three parameters: pattern, size and page. I'll maintain (global) state about the currently active query and if its pattern or size change, I'll reset the whole thing.

redisConn = None
activeQuery = {
  'pattern': None,
  'size': 0,
  'pages': [
    {'cursor': '0', 'position': 0}
  ]
}

DEFAULT_SCAN_SIZE=100

class RedisHTTPRequestHandler(SimpleHTTPRequestHandler):
...
  def do_GET(self):
...
    elif self.path.startswith('/scan'):
      params = self.parseQuery(self.path)
      if self.ensure_required_params(params, ['pattern', 'size', 'page']):
        pattern = params['pattern']
        size = int(params['size'])
        page = int(params['page'])
        if activeQuery['pattern'] != pattern or activeQuery['size'] != size:
          # New query, reset the whole thing
          activeQuery['pattern'] = pattern
          activeQuery['size'] = size
          activeQuery['pages'] = [{'cursor': '0', 'position': 0}]
        if page < len(activeQuery['pages']):
          cursor = activeQuery['pages'][page]['cursor']
          position = activeQuery['pages'][page]['position']
          # The keys returned to the user are collected here
          pageKeys = []
          # True unless Redis has no more results
          more = True
          while size > 0:
            (nextCursor, keys) = redisConn.scan(cursor, pattern, DEFAULT_SCAN_SIZE)

            remainingKeys = keys[position:]
            if nextCursor == 0:
              # last scan result from Redis
              pageKeys += remainingKeys
              more = False
              break
            else:
              if len(remainingKeys) > size:
                pageKeys += remainingKeys[:size]
                position += size
              else:
                pageKeys += remainingKeys
                cursor = nextCursor
                position = 0
              size -= len(remainingKeys)

          if page == len(activeQuery['pages']) - 1:
            activeQuery['pages'].append({'cursor': cursor, 'position': position})

Listing 3: Retrieve one page worth of results in (possibly multiple) redis scans

All that's left to do now is to retrieve the key's values themselves. To speed things up, I'll use a pipe to batch up requests and responses:

          pipe = redisConn.pipeline()
          for key in pageKeys:
            pipe.get(key)
          values = pipe.execute()
          self.send_text({'page': list(zip(pageKeys, values)), 'more': more})

Listing 4: Retrieve keys, too

Back to the UI side, I have to update the JavaScript to present the results and a navigation control.

function scan(page) {
  var query = document.getElementById("query").value;
  var size = document.getElementById("size").value;

  issueGetRequest('/scan?pattern=' + query + '&size=' + size + '&page=' + page, function(txt) {
    var result = JSON.parse(txt);
    var tbl = document.getElementById("tbl");
    tblHtml = '<tr><th>Key</th><th>value</th></tr>';
    for (var i = 0; i < result.page.length; i++) {
      tblHtml += '<tr><td>' + result.page[i][0] + '</td></tr>'
    }
    tblHtml += '<tr><td>' +
      (page > 0 ? '<a href="#" onclick="scan(' + (page - 1) + ')">prev</a>&nbsp;' : '') +
    '</td><td>' +
      (result.more ? '<a href="#" onclick="scan(' + (page + 1) + ')">next</a>' : '') +
    '</td></tr>';
    tbl.innerHTML = '<table>' + tblHtml + '</table>';
    if (page == 0 && !result.more)  {
      document.getElementById("error").innerHTML = "No matches found";
    }
  });
}

...
Query: <input id="query" /><br/>
Size: <input id="size" /><br/>
<button onclick="scan(0)">Scan</button>
<div id="tbl"></div>

Listing 5: Display query results in a table

This puts a bit more load on the Redis server than necessary — if the page size is much less than the scan size, I'll end up re-requesting the same results from Redis over and over. I do want the scan size to be pretty big, too: if I query a pattern that doesn't match too many keys, I'll end up scanning the whole keyspace to satisfy it: if my scan size is small relative to my key count, the query will end up taking a long time. A good compromise is to cache the most recent scan result from Redis, but otherwise go back to the server for the next (or previous) page.

I don't worry too much about what happens if the keys change while I'm running the scan — there's really not all that much I can do about that, anyway. Most times when I use this, I'm looking for a relatively small handful of keys that match a specific pattern, anyway. The results are spartan, but useful. Still, once I find what I'm looking for, I usually want to delete it or edit it. I'll tackle modifications in my next post, stay tuned!

Add a comment:

Completely off-topic or spam comments will be removed at the discretion of the moderator.

You may preserve formatting (e.g. a code sample) by indenting with four spaces preceding the formatted line(s)

Name: Name is required
Email (will not be displayed publicly):
Comment:
Comment is required
My Book

I'm the author of the book "Implementing SSL/TLS Using Cryptography and PKI". Like the title says, this is a from-the-ground-up examination of the SSL protocol that provides security, integrity and privacy to most application-level internet protocols, most notably HTTP. I include the source code to a complete working SSL implementation, including the most popular cryptographic algorithms (DES, 3DES, RC4, AES, RSA, DSA, Diffie-Hellman, HMAC, MD5, SHA-1, SHA-256, and ECC), and show how they all fit together to provide transport-layer security.

My Picture

Joshua Davies

Past Posts