Luciano Mammino PRO
Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com
Luciano Mammino (@loige)
AWS UserGroup Dublin
2021-07-01
Get these slides!
I'm Luciano (🇮🇹🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin 🇮🇪) 👨💻
Co-Author of Node.js Design Patterns 👉
Accelerated Serverless | AI as a Service | Platform Modernisation
We are hiring: do you want to work with us?
SLIC WATCH: fth.link/slic-watch
SLIDES: fth.link/o11y-no-pain
⚠️ ALARM "Custom-Metrics-MetricsFunctionErrorsAlarm" in EU (Ireland)
Threshold crossed: 1 out of the last 1 datapoints [84.0 (23/06/21 09:30:00)] was greater than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).
@wake_me_up_bot
This guy was failing... a lot!
for event in payload['logEvents']:
# ...
cw_client.put_metric_data(...)
Sending 1 metric per log line... 🙈
[
{
'MetricName': 'SomeMetric1',
'Dimensions': [
{
'Name': 'Dimension1Name',
'Value': 'Dimension1Value'
},
# Up to other 9 dimensions here ...
],
'Unit': 'Count',
'Values': [217, 220, 221], # Up to 150 items here ...
'Timestamp': 1624290910000
},
# Up to other 19 metric items here ...
]
⚠️ Payload size
limit: 40 KB
🤔
Maybe boto3 gzips the data automatically! 😏
import boto3
import gzip
endpoint_url = "http://localhost:8000/"
cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False)
cw_client.put_metric_data(
MetricData = [
{
'MetricName': 'TEST_BOTO',
'Dimensions': [
{
'Name': 'APP_VERSION',
'Value': '1.0'
},
],
'Unit': 'None',
'Value': 17
},
],
Namespace='BotoTest'
)
... and there is no magic flag like GzipPayload=True! 😭
Can we extend boto3 somehow? 🤔
boto3 has an event system! 🤩
import boto3
lambda_client = boto3.client('lambda')
# our event handler
def add_xtrace_header(request, **kwargs):
request.headers.add_header('x-trace-id', 'trace-trace')
# get the event system for the lambda_client
event_system = lambda_client.meta.events
# attach an event handler to the client for
event_system.register('before-sign.lambda.Invoke', add_xtrace_header)
# invoke a lambda function
lambda_client.invoke(FunctionName='my-function')
<event-type>.<service-name>.<operation-name>
provide-client-params request-created before-sign before-send response-received
s3 cloudwatch lambda ...
ListBuckets PutMetricData Invoke ...
* (every event) after-call.*.* (all responses) after-call.lambda.* (all responses for lambda)
OK, now we know enough to write our own event handler for gzip compression! 🤓
import boto3
import gzip
cw_client = boto3.client('cloudwatch')
event_system = cw_client.meta.events
# Gzip handler
def gzip_request_body(request, **kwargs):
request.headers.add_header('Content-Encoding', 'gzip')
gzipped_body = gzip.compress(request.body)
request.data = gzipped_body
# registers the custom handler
event_system.register('before-sign.cloudwatch.PutMetricData',
gzip_request_body)
cw_client.put_metric_data(...)
👍 UPVOTE this issue: github.com/boto/botocore/issues/2425
Cover picturea by Moritz Mentges on Unsplash
Thanks doggie by Stefanie Shank
By Luciano Mammino
Most AWS APIs will have limits on the amount of data you can send in one request and sometimes you really need to send a lot of data! To try to maximise the amount of data you can send, while still staying within the limits, some APIs support sending gzip-compressed payloads. But how can you send a gzipped request when using the Python SDK for AWS (boto3)? Well, I needed to answer this question recently and it turned out not to be as easy as I anticipated… Let’s jump into this rabbit hole together and let’s find out the answer!
Cloud developer, entrepreneur, fighter, butterfly maker! #nodejs #javascript - Author of https://www.nodejsdesignpatterns.com , Founder of https://fullstackbulletin.com