Iterate Redis keys using Python

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

Initialize the local environment

virtualenv venv
source venv/bin/activate

Please use these command to install redis-py library

pip install redis

First setup data for testing, we create python file with the code below to produce 1 million cache keys with the value of each.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)
key_count = 0
while key_count < 1000000:
	r.set('foo_{}'.format(key_count), key_count)
	key_count += 1

Now, we try to iterate all the keys to do something like finding out the keys that don’t have any expiration set.

This is the naive approach

r = redis.Redis(host='localhost', port=6379, db=0)
keys = r.keys(pattern='foo_*')
no_expire = []
for key in keys:
	if r.ttl(key) == -1L:
		no_expire.append(key)

This approach will not cause any issue if the number of keys is not too much. Let’s think about the situation that your system is having over 1 million keys with the pattern foo_*, the memory is running out!

Let consider about using scan_iter() instead of using keys()

keys = r.scan_iter(match='foo_*', count=1000)
to_delete = []
while True:
    try:
        while len(to_delete) < 1000:
            key = keys.next()
            if r.ttl(key) == -1L:
                to_delete.append(key)
 
        r.delete(*to_delete)
        to_delete = []
    except StopIteration:
        break

 

Leave a Reply

Your email address will not be published. Required fields are marked *