AWS DynamoDB is popular because it is super fast & scalable. However, when using the Python client boto3 to fetch a large number of documents we started to noticed some unexplained slowness. This was super annoying as some of our queries were taking 20s to process BUT the actual dynamo query returned in only 6s. So we wanted to find out where that 14s was going.Using dynamodb-local I put 250,000 basic items: with table.batch_writer() as batch: for i in range(250000): batch.put_item(Item={ 'id': i, 'updated_at': i, 'name': "some data", 'status': "some other stuff" })

I can then fetch these items in 10k batches with: scan_pages = int(sys.argv[1]) scan_kwargs = { 'FilterExpression': "updated_at > :val", 'ExpressionAttributeValues': { ":val": -1}, 'Limit': 10000 } pages = 0 start_key = None while True: if start_key is not None: scan_kwargs['ExclusiveStartKey'] = start_key response = table.scan(**scan_kwargs) pages +=1 start_key = response.get('LastEvaluatedKey', None) if pages >= scan_pages: break

I wrote the same query in Golang to compare the results against python:

image

Profiling the python code with cProfile: `import cProfile
import pstats
with cProfile.Profile() as pr:

… the query here``stats = pstats.Stats(pr)

stats.sort_stats(pstats.SortKey.TIME)
stats.print_stats()`

For 100k documents this took 4 seconds and printed out: cumtime filename 1.823 /.../botocore/parsers.py:309(_parse_shape)

So it spent nearly 1/2 the time in the _parse_shape method in [botocore/parsers.py](https://github.com/boto/botocore/blob/develop/botocore/parsers.py#L799). This method (called by _handle_json_body) takes the already parsed JSON and looks for more suitable types. For Dynamo this is useless because it already has its own type system, and the payloads are huge so it does lots of recursion.

Lets monkey patch this method away and see what happens: # Monkey Patch from botocore.parsers import PROTOCOL_PARSERS parser = PROTOCOL_PARSERS.get("json") def new_fn(self, raw_body, shape): return self._parse_body_as_json(raw_body) setattr(parser, "_handle_json_body", new_fn)

The speed without _parse_shape:

image

What!?! Without _parse_shape boto3 actually outperforms Golang. So much wasted CPU time!

User beware! This monkey patch affects all boto3 clients you create and some (like KMS) do require this better typing. So beware before you ship this hack.This post is not saying boto3 or DynamoDB are bad, just that (especially for large amounts of documents) they are not well suited for one another.

DAX (the DynamoDB Accelerator) uses a different parser and datatype (CBOR) and has its own client to avoid such issues. Maybe DynamoDB should have its own optimised client as well?Thanks to Bin and Jordan for helping with this post