Skip to content

systems

Right-sizing your AWS Lambdas

I was recently able to reduce the cost of one of our serverless applications by more than half by reducing the memory allocated to the lambdas.

Possible reasons we didn't do this earlier: - initial cost was low but increased later as traffic increased - team didn't have knowledge/confidence to set lower threshold - we weren't monitoring/alerting on memory usage

AWS Lambda Pricing

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. - https://aws.amazon.com/lambda/

Lambda is charged based on number and duration of requests (AWS Pricing). Duration is measured in GB-seconds which is why it's possible to reduce your cost by reducing the maximum memory provided to you lambdas.

You specify an amount between 128 MB and 3,008 MB in 64 MB increments. Lambda allocates CPU power linearly in proportion to the amount of memory configured. At 1,792 MB, a function has the equivalent of 1 full vCPU (one vCPU-second of credits per second).

There are situations where provisioning far more memory than will be used is a good choice. If the function is CPU bound (as opposed to waiting on responses from the network) then increasing CPU will reduce duration, improving performance without negatively impacting on cost.

The risk when setting memory for a Lambda is that execution halts immediately if the function runs out of memory. Changes to the function over time may alter it's memory usage so we're best to monitor and alert on this.

Checking Memory Usage

It's relatively simple to report on the maximum memory being used by a lambda. This can help you select an appropriate amount.

Lambda logs maxMemoryUsed for each function invocation to CloudWatch Logs. CloudWatch Logs Insights includes a sample query that reports on overprovisioned memory.

The example below is for a function that spends most of it's time waiting on responses from web apis. The report shows it had 976 MB memory and used at most 275 MB in the past three days. Note that the sample query returns figures that may be confusing due to them using a different unit (MiB) than is used for configuring Lambda functions (MB). (I've requested this be fixed).

CloudWatch Logs Insights query displaying overprovisioned memory in Lambda CloudWatch Logs Insights query displaying overprovisioned memory in Lambda

Choose good memory limit for your function

We initially decided to set the memory to 384 MB and setup an alarm to alert us if a function uses 80% of that (307 MB). On checking CloudWatch later we saw function duration increased after the memory was decreased. This was due to the CPU decrease that happens when you reduce memory for the lambda. We decided to manually increase and decrease memory until we found a sweet spot of 512 MB. This was still a 50% decrease in cost with minimal impact on duration.

Monitor and alert in case things change

If our lambda memory usage increases over time, we want to be notified. Below are snippets from the CloudFormation template for our application that write memory used to a custom CloudWatch Metric and alert us if it gets to 80% of the maximum we have set.

CloudWatch Logs Metric Filter

A Metrics Filter parses all logs from the function and writes the max_memory_used value to a custom metric. This provides a convenient way to graph and alert on that metric.

AppMetricFilter:
  Type: AWS::Logs::MetricFilter
  Properties:
    LogGroupName:
      Ref: AppDashserverLogGroup
    FilterPattern: '[ report_label="REPORT", ..., label="Used:", max_memory_used_value, unit="MB" ]'
    MetricTransformations:
      - MetricValue: '$max_memory_used_value'
        MetricNamespace: LogMetrics/Lambda
        MetricName: example-app-memoryUsed'

I'd not come across Metrics Filters before but am glad I have. From whatI can gather, a custom metric costs you $0.30/month but there is no additional charge to have your CloudWatch logs filtered through a Metrics Filter to feed it.

CloudWatch Alarm

We created a CloudWatch alarm to notify us if the maximum memory used bya function exceeded 80% of what it was provisioned with.

AppAlarmLambdaMemory:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmActions:
      - Ref: AwsAlertsAlarm
    AlarmDescription: Lambda memory usage above 80% for example-app-memoryUsed
    ComparisonOperator: GreaterThanThreshold
    EvaluationPeriods: 1
    MetricName: 'example-app-memoryUsed'
    Namespace: LogMetrics/Lambda
    Period: 60
    Statistic: Maximum
    Threshold: 307 # 80% of provisioned memory
    Unit: Megabytes

Checking your lambda's cost

I recommend using AWS Cost Explorer to view at your lambda costs. I generally access it via the AWS Console although I was excited to discover you can also query it via AWSCLI).

Some hints to help you breakdown costs by Lambda:

  • Filters -> Include Only -> Lambda
  • Group By -> Tag: aws:cloudformation:stack-name

Reduced waste and early warning against failure

This work will save us around $600/month running this application. It also provides us with more visibility into memory usage and alerts for when it increases.

It's often a tough call to decide whether ROI on cost savings will justify the effort. You don't know till try it. If you've blown your budget that can be a motivation. Hopefully the information here can help others in their efforts to reduce waste.

Why You Should Enable S3 Block Public Access

Amazon S3 enables you to accidentally share confidential information with the world. The potential impact of misconfiguration justifies implementing controls made available by AWS in November 2018.

Numerous data breaches due to misconfigured AWS Buckets have been reported in recent times and free tools have been released that can be used to scan for them. Even AWS staff have made their buckets world readable by mistake.

S3 Block Public Access allows you to prevent configuration of S3 Buckets and the objects within them from being accessible to the whole world.

It still allows you to share objects with specified targets such as:

  • AWS Services
  • other AWS Accounts
  • specified IP address ranges

How we got here

Amazon S3 was the first AWS Service launched, way back in 2006. Users store file objects in Buckets and can control access to them through a variety of mechanisms, including:

  • Bucket ACLs
  • Object ACLs
  • Bucket Policies
  • IAM Polcies

Objects can be made accessable via:

  • Unauthenticated Web requests (via http/https)
  • AWS API calls (via AWS Web Console, AWSCLI, SDKs, etc)
  • BitTorrent

Confusion around the different methods for controlling access can lead to mistakes. Amazon's "only recommended use case for the bucket ACL is to grant write permission to the Amazon S3 Log Delivery group to write access log objects to your bucket, yet Bucket ACLs still make it easy to make the Bucket world readable (and even writable!).

Detecting Publicly Accessible Buckets

AWS Trusted Advisor's S3 Bucket Permissions Check has been free since Feb 2018.

Business and Enterprise support customers can use these checks to enable automated actions via Trusted Advisor's integration with CloudWatch Events.

Block S3 Public Access

In Nov 2018, AWS launched a new feature that allows you to control against Objects in S3 Buckets being made Public. It consists of four settings which can be applied at the Bucket or Account level. Applying at a Bucket level may enable the rules to be overridden.

Objects intended to be shared publicly (e.g. static websites) can have a Bucket Policy with configured to grant read access to a CloudFront Origin Access Identity.

For situations where CloudFront is considered overkill (it can take ~30 minutes to provision), users may consider granting access to a specific IP Range, AWS Account or IAM Role.

What does Public mean

  • ACLs: AllUsers or AuthenticatedUsers

  • Policies

In order to be considered non-public, a bucket policy must grant access only to fixed values (values that don't contain a wildcard) of one or more of the following:

  • A set of Classless Inter-Domain Routings (CIDRs), using aws:SourceIp. For more information about CIDR, see RFC 4632 on the RFC Editor website.
  • An AWS principal, user, role, or service principal
  • aws:SourceArn
  • aws:SourceVpc
  • aws:SourceVpce
  • aws:SourceOwner
  • aws:SourceAccount
  • s3:x-amz-server-side-encryption-aws-kms-key-id
  • aws:userid, outside the pattern "AROLEID:*"

Enabling S3 Block Public Access on an Account

Applying S3 Block Public Access may break things! Administrators applying this feature should familiarize themselves with the AWS Documentation.

In order to perform Block Public Access operations on an account, use the AWS CLI service s3control.

The four settings that can be configured independantly) are:

  • BlockPublicAcls: Block setting of ACLs if they include public access
  • IgnorePublicAcls: Ignore Public ACLs
  • BlockPublicPolicy: Block setting of Policy that includes public access
  • RestrictPublicBuckets: Restrict buckets with public Policy to same account and AWS Principals

The account-level operations that use this service are:

  • PUT PublicAccessBlock (for an account)
  • GET PublicAccessBlock (for an account)
  • DELETE PublicAccessBlock (for an account)

Example CloudFormation for granting access to Origin Access Identity and IP range

  CFOAI:
    Type: AWS::CloudFront::CloudFrontOriginAccessIdentity
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: !Ref AWS::StackName

  Bucket:
    Type: AWS::S3::Bucket

  BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref Bucket
      PolicyDocument:
        Version: 2012-10-17
        Id: PolicyForCloudFrontPrivateContent
        Statement:
          - Sid: Grant a CloudFront Origin Identity access to support private content
            Action: "s3:GetObject"
            Effect: "Allow"
            Principal:
             CanonicalUser: !GetAtt CFOAI.S3CanonicalUserId
            Resource: !Sub "arn:aws:s3:::${Bucket}/*"
          - Sid: Grant access from Trusted Network
            Action: "s3:GetObject"
            Effect: "Allow"
            Principal: "*"
            Resource: !Sub "arn:aws:s3:::${Bucket}/*"
            Condition:
              IpAddress:
                aws:SourceIp: !Ref OfficeIp

Don't Buck the System, Change it

I don't consider myself a Buddhist but attest to their belief that "Life is Suffering". Not all of it all the time, but there's always some waiting. To some this might sound like a pretty negative view but it doesn't have to be. What gives me strength is the knowledge that not all suffering is out of my control.

This post is about making changes to systems that affect other people. It's not intended to cover changes to personal habits.

Find Things Worth Suffering For

Life is change and change often involves suffering.

Bringing a pet into your life generally means accepting the grief the comes with outliving them (unless you choose a White Cockatoo which can live 40-60 years in captivity).

Improving you health, wealth or education might involve going without things you enjoy and doing things you don't.

System change that affects other people, whether in the workplace, government or community is hard work and success is never guaranteed. If you're going to attempt it, make sure you choose something worth suffering for.

Serenity, Courage, Wisdom

Before the Internet came along, memes thrived in the form of kitchen calendars and fridge magnets. You may be familiar with this one:

Serenity Fridge Magnet Serenity Fridge Magnet

God, grant me the serenity to accept the things I cannot change,
Courage to change the things I can,
And wisdom to know the difference.

Now I know Christian Theologians may not be in vogue these days but they were "putting a bird on it" long before the hipsters caught onto it. The value is in the message, not who said it.

"Accept the Things I Cannot Change"

I would change this to "Accept the things I should not change":

  • A sysadmin with root privileges can invade people's privacy
  • A person in executive government can enact laws that cause unnecessary suffering

Just because you can change the world to better suit you, it doesn't follow that you should.

I accept that I cannot change all the fridge magnets. It's reckon I could get Alcoholics Anonymous to change the version they promote but shouldn't because the change:

  • would provide little benefit to me
  • would provide little benefit to others
  • would require a huge amount of effort (including other people's)

maybe do it

Do you have 'skin in the game'?

Just because something should change, that doesn't neccessarily mean you should be the one to do it. You know best how to scratch your own itch. Avoid trying to lead on change that doesn't impact on your personally. You're unlikely to have the passion, connection and understanding of someone with skin in the game. By all means, support these efforts where you believe in the goals but don't try to own them.

A Pig and a Chicken are walking down the road.
The Chicken says: "Hey Pig, I was thinking we should open a restaurant!"
Pig replies: "Hm, maybe, what would we call it?"
The Chicken responds: "How about 'ham-n-eggs'?"
The Pig thinks for a moment and says: "No thanks. I'd be committed, but you'd only be involved."

It's captured more succintly in the "Nothing About Us Without Us" mantra popular with ethnic, disability and other marginalise groups.

Does anyone else know/care?

Who else would benefit from the change? Who would suffer? If there isn't the likelihood of a net benefit to the group, you're unlikely to get the buy-in required to make the change. Unless you intend to mislead or coerce people into doing things your way, you're probably best accepting this as a thing you shouldn't change.

"Courage to Change the Things I Can"

I don't think most people realise how malleable the world is. The video below says it better than I can.

“When you grow up you tend to get told that the world is the way it is and your life is just to live your life inside the world. Try not to bash into the walls too much. Try to have a nice family life, have fun, save a little money. That's a very limited life. Life can be much broader once you discover one simple fact: Everything around you that you call life was made up by people that were no smarter than you. And you can change it, you can influence it… Once you learn that, you'll never be the same again.”

  • Steve Jobs

"And the Wisdom to Know the Difference"

I'm not sure about wisdom but I hope this post offers some food for thought.

Personal Cloudification

The Urge to Own...

Expensive Paperweights Expensive Paperweights

Something about my personality has seen me always running my own servers. In the late 90s I setup a Pentium 100 with a permanent 28.8 kbps modem connection to the Internet in a wardrobe at my parents house. It hosted my mail and web services for a few years (including 18 months spent living in the U.K.). Something about knowing the system down to the metal, and beyond that even, to the environment has always been important to me. I think amongst systems people I'm not alone.

... versus the Allure of The Cloud

Various factors have led me to abandon (at least for now) my desire to host@home:

  • power consumption (waste & cost) & heat output
  • single points of failure (power, ADSL, theft)
  • desire to eat my own dogfood

A few years back I found other services to be cheaper than Amazon's AWS offerings. I'm currently exploring what has changed since then and how individuals can use Amazon AWS to provide performant, highly available Internet services at low cost.

Mobile is Changing Everything

There are some great tools out there for controlling your Amazon services from mobile devices. Cloud Services Manager is a great app for iOS that enables control of AWS services on the go. My SSH client of choice for connecting from iOS is Prompt by Panic, Inc. These two alone let give you a lot of control from your iPhone (or iPad).

Cloud Services Manager for iPhone Cloud Services Manager for iPhone

Create, start, stop EC2 instances Create, start, stop EC2 instances

Cloudwatch Stats Cloudwatch Stats

Prompt SSH client for iOS Prompt SSH client for iOS

Prompt is remarkably usable Prompt is remarkably usable

ASCII steam locomotive (sl) ASCII steam locomotive (sl)


Cost Savings in Dev Land

When EC2 was first launched you lost everything if you shut an instance down. These days Elastic Block Store (EBS) volumes are the default which means you can stop and restart instances as you need them, paying only for the hours they're running. This can make it more comfortable to tinker knowing you're only paying while you're using it. Running 10 x m1.small instances will cost $0.80 an hour which is quite affordable.

Note that you're charged by the month for the EBS volume the instance uses. The charge includes a charge of approx. $0.10 per GB-month of provisioned storage and $0.10 per 1 million I/O requests. I believe that creating a 10 GB EBS volume and destroying it an hour later would be counted as 10 GB-months but I want to confirm (or refute) this.

To Be Continued...

Back To My Mac, Confessions of a Reluctant Apostate

Rethinking Fanboyism

Two years after replacing my Apple Macbook Pro laptop with a Dell running Ubuntu Linux I'm going back to Apple. I moved away from Apple for a mixture of philosophical and practical reasons. One of the main reasons was my belief in the importance of a viable linux desktop operating system. The backflip came about because I feel happier when using Apple products. In my attempts to get away from Apple I think I've learned more about why they have been so successful.

Android was the Catalyst

I didn't get a smartphone until mid 2011 and even then it was mainly because the battery on my Nokia was needing to be charged nightly. I picked up an HTC Sensation and began hating it almost immediately. I'm usually pretty good at pointing out flaws but the HTC was a bit of a mystery to me. While I can list a few flaws (battery not lasting a day is one!) I just didn't like using it. After three months I decided to buy my first iPhone.

I Feel Love

So this is the fucked up thing. Within 30 minutes I was texting people to tell them how much I love my new iPhone. It may make me sound like a total fanboy but even while I was in the store I was bonding with the device. It feels wrong saying I love a machine but  at the same time I'm curious how it could elicit such emotions. I can only put it down to the design of both hardware and software.

The Cathedral and The Bazaar

Steve Jobs was the high priest at Apple. We're told he was obsessive about detail and that it was his way or the highway. Part of the reason I avoided getting an iPhone was the control exerted over the distribution of Apps through the App Store. I was surprised and disappointed by my first experience of the Android Marketplace. A search for "shopping list" returned hundreds of apps with no indication which were most downloaded or highly rated. Maybe you can't design an OS or phone by committee?

Get Out of my Way

If I'm going to be iPhone boy then life would be easier if I used OSX on my desktop.

It's been about 4 years since I last bought a new Macbook Pro. My efforts to avoid vendor lockin are now being replaced by efforts to avoid unproductive work (like debugging problems printing from Ubuntu 11.10). I want to get on with the fun stuff and OSX just seems to make the fun stuff so much easier.

So What Now for Freedom?

I was shocked earlier this year to hear a techie say he wouldn't care if the linux desktop went away because he's happy with OSX. My issue is that Apple could make OSX unpleasant at some point in the future and without a viable alternative to switch to we would just have to accept it.

I don't think me going back to Apple is going to make an ounce of difference to Linux but what would the world be like if everyone followed this path? I guess I'm no longer willing to sacrifice my own happiness trying to support this particular ideal.

Well Done Steve (and Company)

The iPhone and Macbook Pro are masterpieces. (I didn't mention I've had an iPad for the past 12 months and love it to bits). I tried to extract myself from this web of awesome production but have failed because they are so well designed. I would love to create something as compelling.

RVM in Production

[update] Aussie Rubyists are discussing this on rails-oceania@googlegroups.com

I'm a relative newcomer to Ruby Version Manager (RVM) because I haven't needed it till now.  Last week I started work on a project that uses JRuby and RVM is now a part of my tool chain.

"RVM is a command line tool which allows us to easily install, manage and work with multiple ruby environments from interpreters to sets of gems."

https://rvm.beginrescueend.com/

I think it's a pretty amazing project and Wayne E. Seguin deserves respect for gifting us with such a great tool. I think it's great for dev but...

I'm just not sure I want RVM in Production

As I said, I'm new to RVM. I don't know it that well yet so I'm not qualified to speak about it with any degree of authority. The idea of running rvm on production servers seems wrong to me because it seems to dump a load of non-standard complexity onto sysadmins while ignoring the unix idioms and conventions they know.

Do we really need to be so specific about versions?

In an age of virtualisation and cloud computing, a production server probably shouldn't need to have more than one version of Ruby installed. RVM makes it easy for devs to specify that this web app uses ruby-1.9.2-p180 while that one uses ruby-1.9.2-p290. I'm just not sure they should be expecting operations to be providing them with such specificity.

Don't sysadmins have enough to deal with already?

Developers may spend months or years in the same cosy environment working on software they know backwards. Sysadmins deal with chaos everyday. They put out fires, multitask and deal with hundreds of different bits of software. What makes this possible is conventions within the unix world. Logs go here, start scripts there, this is the library load path, etc... System administration can be like working in a busy kitchen. Special off menu orders make their work harder and are not appreciated!

I just wonder whether developers demanding specific Ruby patch levels will come across like the folks ordering coffee from LA Story?

[Update] I was running into a few difficulties using RVM so at least for now, I've settled on a very simple use for RVM on my workstation. Bundler does a great job managing my gems so I've set RVM to use system Ruby and I'll only use a .rvmrc in projects that are not using my default of ruby-1.9.2.