I use Redis and Redis uses Lua as a scripting language. Today was my first day using Lua in anger, and I am still angry.My problem is that in Redis I have many sets of keys, e.g. s1 = {1,2,3} s2 = {3,4} and keys 1=a, 2=b, 3=c, 4=d. I want to return all values of all keys in the union of given sets, e.g. f(s1 s2) = a b c d

I could SUNION s1 s2 which returns 1 2 3 4, then MGET 1 2 3 4 to then get a b c d. This is kind of wasteful because it is 2 round trips to Redis when I want to do it in 1.

Enter Lua and EVAL.

Redis lets you send along a Lua script that can do a bunch of stuff all at once; I can call Redis with: EVAL " local indexes = redis.call('SUNION', unpack(KEYS)) return redis.call('MGET', unpack(indexes)) " 2 s1 s2

Now this will call the SUNION in the input KEYS, which are s1 s2 and then return the MGET result. unpack here is basically just splat (* in Ruby and Python) taking the Lua table (array) and splitting it out into arguments for the call method.

There, we have done it, and everything works.**Ohhhh, nooooo. It just broke in production!?!**So what I found out was that unpack has a max size (about 8,000). So, if either the number of sets OR the number of keys is greater than 8K Lua throws an error.

So now I have to write more Lua. What is interesting to know is that MGET is actually pretty slow. I don’t know why, but many GETs are faster than 1 MGET [cite]. Don’t know why, but that makes this little bit easy at least. local indexes = redis.call('SUNION', unpack(KEYS)) local values = {} for i=1,#indexes do local value = redis.call('get', indexes[i]) table.insert(values, value) end return valuesNow onto the harder problem, the number of KEYS being above 8,000.

So, since we are calling from the client, we could just split it up there, e.g. if the client has 18,000 keys we just call the above script three times. But that gets us back to where we were initially, calling Redis multiple times.

So lets do something like: local splitby = 8000 local indexes = {} if #KEYS <= splitby then indexes = redis.call('sunion', unpack(KEYS, 1, #KEYS) ) elseif #KEYS <= (splitby * 2) then indexes = redis.call('sunion', unpack(KEYS, 1, splitby), unpack(KEYS, splitby + 1, #KEYS) ) ...

Oh wait, this doesn’t work! When you use unpack more than once in function IT ONLY SELECTS THE FIRST ELEMENT OF THE LIST. I will repeat that, it breaks silently, then only sends the first element of its list as an argument.

This is like 2 hours of my life, including filing a bug against Redis because I didn’t understand this weird behaviour. Lua stole 2hours of my life. If you would like a better description of this and other unpack pains go here.So here we go again, this time with the correct solution: local all_indexes = {} local step = 0 for i=1,1000 do if #KEYS == step then break end local next_step = step + 8000 if next_step > #KEYS then next_step = #KEYS end local indexes = redis.call('sunion', unpack(KEYS, step+1, next_step) ) table.insert(all_indexes, indexes) step = next_step end``local values = {} local seen = {} for i=1,#all_indexes do local indexes = all_indexes[i] for j=1,#indexes do local getkey = indexes[j] if seen[getkey] ~= true then seen[getkey] = true local value = redis.call('get', getkey) table.insert(values, value) end end end return values

First we break apart the incoming KEYS into parts and sunion them in batches adding the results to the all_indexes table.

Then we loop over that table of tables, and get each key, making sure not to get the same key twice.

Done.To summarise: because unpack has a size limit and weird behaviour, my simple TWO LINE SCRIPT is now at least 30 complicated lines.

I don’t like calling Lua a bad language, it clearly has its place in the world. But Lua hurt me today, and I just wanted to share.