How to send gzipped requests
with boto3

Luciano Mammino (@loige)

AWS UserGroup Dublin
2021-07-01

loige.link/gzip-boto3

Get these slides!

loige

Let me introduce myself first...

I'm Luciano (🇮🇹🍕🍝) 👋

Senior Architect @ fourTheorem (Dublin 🇮🇪) 👨‍💻

nodejsdp.link

Co-Author of Node.js Design Patterns 👉

Connect with me:

loige.co (blog)

@loige (twitter)

loige (twitch)

lmammino (github)

We are business focused technologists that deliver.

Accelerated Serverless | AI as a Service | Platform Modernisation

We are hiring: do you want to work with us?

loige

In the previous episodes...

SLIC WATCH: fth.link/slic-watch

SLIDES: fth.link/o11y-no-pain

loige

⚠️ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland)

Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).

@wake_me_up_bot

loige

This guy was failing... a lot!

loige

for event in payload['logEvents']:
  # ...
  cw_client.put_metric_data(...)

Sending 1 metric per log line... 🙈

loige

[
    {
        'MetricName': 'SomeMetric1',
        'Dimensions': [
            {
                'Name': 'Dimension1Name',
                'Value': 'Dimension1Value'
            },
            # Up to other 9 dimensions here ...
        ],
        'Unit': 'Count',
        'Values': [217, 220, 221], # Up to 150 items here ...
        'Timestamp': 1624290910000
    },
    # Up to other 19 metric items here ...
]

⚠️ Payload size
limit: 40 KB

loige

🤔

loige

Maybe boto3 gzips the data automatically! 😏

loige

import boto3
import gzip

endpoint_url = "http://localhost:8000/"

cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False)

cw_client.put_metric_data(
    MetricData = [
        {
            'MetricName': 'TEST_BOTO',
            'Dimensions': [
                {
                    'Name': 'APP_VERSION',
                    'Value': '1.0'
                },
            ],
            'Unit': 'None',
            'Value': 17
        },
    ],
    Namespace='BotoTest'
)

loige

... and there is no magic flag like GzipPayload=True! 😭

loige

Can we extend boto3 somehow? 🤔

loige

boto3 has an event system! 🤩

youtu.be/eM8uoGJO2AI

@thekyleknapp

loige

import boto3

lambda_client = boto3.client('lambda')

# our event handler
def add_xtrace_header(request, **kwargs):
    request.headers.add_header('x-trace-id', 'trace-trace')

# get the event system for the lambda_client
event_system = lambda_client.meta.events

# attach an event handler to the client for 
event_system.register('before-sign.lambda.Invoke', add_xtrace_header)

# invoke a lambda function
lambda_client.invoke(FunctionName='my-function')

loige

Event naming convention

<event-type>.<service-name>.<operation-name>

provide-client-params
request-created
before-sign
before-send
response-received

s3
cloudwatch
lambda
...

ListBuckets
PutMetricData
Invoke
...

loige

You can be a 🌟 too!

* (every event)

after-call.*.* (all responses)

after-call.lambda.* (all responses for lambda)

loige

OK, now we know enough to write our own event handler for gzip compression! 🤓

loige

import boto3
import gzip

cw_client = boto3.client('cloudwatch')
event_system = cw_client.meta.events

# Gzip handler
def gzip_request_body(request, **kwargs):
    request.headers.add_header('Content-Encoding', 'gzip')
    gzipped_body = gzip.compress(request.body)
    request.data = gzipped_body

# registers the custom handler
event_system.register('before-sign.cloudwatch.PutMetricData', 
                      gzip_request_body)

cw_client.put_metric_data(...)

loige

Other potential use cases

Log data before/after it is sent to AWS
Add additional headers to AWS requests (tracing)
Additional input validation (your custom rules)
Payload enrichment (e.g. make sure certain tags exist)

loige

What if you don't use Python?

loige

What did we learn?

Always have alarms for your lambdas!
(have you checked SLIC Watch already? 😉)
You can use Gzip compression to send big payloads to AWS
Boto3 is extensible through its event system!

loige