I use Redis and Redis uses Lua as a scripting language. Today was my first day using Lua in anger, and I am still angry.
My problem is that in Redis I have many sets of keys, e.g. s1 = {1,2,3} s2 = {3,4} and keys 1=a, 2=b, 3=c, 4=d. I want to return all values of all keys in the union of given sets, e.g. f(s1 s2) = a b c d
I could SUNION s1 s2 which returns 1 2 3 4, then MGET 1 2 3 4 to then get a b c d. This is kind of wasteful because it is 2 round trips to Redis when I want to do it in 1.
Enter Lua and EVAL.
Redis lets you send along a Lua script that can do a bunch of stuff all at once; I can call Redis with:
EVAL "
local indexes = redis.call('SUNION', unpack(KEYS))
return redis.call('MGET', unpack(indexes))
" 2 s1 s2
Now this will call the SUNION in the input KEYS, which are s1 s2 and then return the MGET result. unpack here is basically just splat (* in Ruby and Python) taking the Lua table (array) and splitting it out into arguments for the call method.
There, we have done it, and everything works.
Ohhhh, nooooo. It just broke in production!?!
So what I found out was that unpack has a max size (about 8,000). So, if either the number of sets OR the number of keys is greater than 8K Lua throws an error.
So now I have to write more Lua. What is interesting to know is that MGET is actually pretty slow. I don’t know why, but many GETs are faster than 1 MGET [cite]. Don’t know why, but that makes this little bit easy at least.
local indexes = redis.call('SUNION', unpack(KEYS))
local values = {}
for i=1,#indexes do
local value = redis.call('get', indexes[i])
table.insert(values, value)
end
return values
Now onto the harder problem, the number of KEYS being above 8,000.
So, since we are calling from the client, we could just split it up there, e.g. if the client has 18,000 keys we just call the above script three times. But that gets us back to where we were initially, calling Redis multiple times.
So lets do something like:
local splitby = 8000
local indexes = {}
if #KEYS <= splitby then
indexes = redis.call('sunion',
unpack(KEYS, 1, #KEYS)
)
elseif #KEYS <= (splitby * 2) then
indexes = redis.call('sunion',
unpack(KEYS, 1, splitby),
unpack(KEYS, splitby + 1, #KEYS)
)
...
Oh wait, this doesn’t work! When you use unpack more than once in function IT ONLY SELECTS THE FIRST ELEMENT OF THE LIST. I will repeat that, it breaks silently, then only sends the first element of its list as an argument.
This is like 2 hours of my life, including filing a bug against Redis because I didn’t understand this weird behaviour. Lua stole 2hours of my life. If you would like a better description of this and other unpack pains go here.
So here we go again, this time with the correct solution:
local all_indexes = {}
local step = 0
for i=1,1000 do
if #KEYS == step then
break
end
local next_step = step + 8000
if next_step > #KEYS then
next_step = #KEYS
end
local indexes = redis.call('sunion',
unpack(KEYS, step+1, next_step)
)
table.insert(all_indexes, indexes)
step = next_step
end
local values = {}
local seen = {}
for i=1,#all_indexes do
local indexes = all_indexes[i]
for j=1,#indexes do
local getkey = indexes[j]
if seen[getkey] ~= true then
seen[getkey] = true
local value = redis.call('get', getkey)
table.insert(values, value)
end
end
end
return values
First we break apart the incoming KEYS into parts and sunion them in batches adding the results to the all_indexes table.
Then we loop over that table of tables, and get each key, making sure not to get the same key twice.
Done.
To summarise: because unpack has a size limit and weird behaviour, my simple TWO LINE SCRIPT is now at least 30 complicated lines.
I don’t like calling Lua a bad language, it clearly has its place in the world. But Lua hurt me today, and I just wanted to share.