Skip to content

aws

Right-sizing your AWS Lambdas

I was recently able to reduce the cost of one of our serverless applications by more than half by reducing the memory allocated to the lambdas.

Possible reasons we didn't do this earlier: - initial cost was low but increased later as traffic increased - team didn't have knowledge/confidence to set lower threshold - we weren't monitoring/alerting on memory usage

AWS Lambda Pricing

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. - https://aws.amazon.com/lambda/

Lambda is charged based on number and duration of requests (AWS Pricing). Duration is measured in GB-seconds which is why it's possible to reduce your cost by reducing the maximum memory provided to you lambdas.

You specify an amount between 128 MB and 3,008 MB in 64 MB increments. Lambda allocates CPU power linearly in proportion to the amount of memory configured. At 1,792 MB, a function has the equivalent of 1 full vCPU (one vCPU-second of credits per second).

There are situations where provisioning far more memory than will be used is a good choice. If the function is CPU bound (as opposed to waiting on responses from the network) then increasing CPU will reduce duration, improving performance without negatively impacting on cost.

The risk when setting memory for a Lambda is that execution halts immediately if the function runs out of memory. Changes to the function over time may alter it's memory usage so we're best to monitor and alert on this.

Checking Memory Usage

It's relatively simple to report on the maximum memory being used by a lambda. This can help you select an appropriate amount.

Lambda logs maxMemoryUsed for each function invocation to CloudWatch Logs. CloudWatch Logs Insights includes a sample query that reports on overprovisioned memory.

The example below is for a function that spends most of it's time waiting on responses from web apis. The report shows it had 976 MB memory and used at most 275 MB in the past three days. Note that the sample query returns figures that may be confusing due to them using a different unit (MiB) than is used for configuring Lambda functions (MB). (I've requested this be fixed).

CloudWatch Logs Insights query displaying overprovisioned memory in Lambda CloudWatch Logs Insights query displaying overprovisioned memory in Lambda

Choose good memory limit for your function

We initially decided to set the memory to 384 MB and setup an alarm to alert us if a function uses 80% of that (307 MB). On checking CloudWatch later we saw function duration increased after the memory was decreased. This was due to the CPU decrease that happens when you reduce memory for the lambda. We decided to manually increase and decrease memory until we found a sweet spot of 512 MB. This was still a 50% decrease in cost with minimal impact on duration.

Monitor and alert in case things change

If our lambda memory usage increases over time, we want to be notified. Below are snippets from the CloudFormation template for our application that write memory used to a custom CloudWatch Metric and alert us if it gets to 80% of the maximum we have set.

CloudWatch Logs Metric Filter

A Metrics Filter parses all logs from the function and writes the max_memory_used value to a custom metric. This provides a convenient way to graph and alert on that metric.

AppMetricFilter:
  Type: AWS::Logs::MetricFilter
  Properties:
    LogGroupName:
      Ref: AppDashserverLogGroup
    FilterPattern: '[ report_label="REPORT", ..., label="Used:", max_memory_used_value, unit="MB" ]'
    MetricTransformations:
      - MetricValue: '$max_memory_used_value'
        MetricNamespace: LogMetrics/Lambda
        MetricName: example-app-memoryUsed'

I'd not come across Metrics Filters before but am glad I have. From whatI can gather, a custom metric costs you $0.30/month but there is no additional charge to have your CloudWatch logs filtered through a Metrics Filter to feed it.

CloudWatch Alarm

We created a CloudWatch alarm to notify us if the maximum memory used bya function exceeded 80% of what it was provisioned with.

AppAlarmLambdaMemory:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmActions:
      - Ref: AwsAlertsAlarm
    AlarmDescription: Lambda memory usage above 80% for example-app-memoryUsed
    ComparisonOperator: GreaterThanThreshold
    EvaluationPeriods: 1
    MetricName: 'example-app-memoryUsed'
    Namespace: LogMetrics/Lambda
    Period: 60
    Statistic: Maximum
    Threshold: 307 # 80% of provisioned memory
    Unit: Megabytes

Checking your lambda's cost

I recommend using AWS Cost Explorer to view at your lambda costs. I generally access it via the AWS Console although I was excited to discover you can also query it via AWSCLI).

Some hints to help you breakdown costs by Lambda:

  • Filters -> Include Only -> Lambda
  • Group By -> Tag: aws:cloudformation:stack-name

Reduced waste and early warning against failure

This work will save us around $600/month running this application. It also provides us with more visibility into memory usage and alerts for when it increases.

It's often a tough call to decide whether ROI on cost savings will justify the effort. You don't know till try it. If you've blown your budget that can be a motivation. Hopefully the information here can help others in their efforts to reduce waste.

Why You Should Enable S3 Block Public Access

Amazon S3 enables you to accidentally share confidential information with the world. The potential impact of misconfiguration justifies implementing controls made available by AWS in November 2018.

Numerous data breaches due to misconfigured AWS Buckets have been reported in recent times and free tools have been released that can be used to scan for them. Even AWS staff have made their buckets world readable by mistake.

S3 Block Public Access allows you to prevent configuration of S3 Buckets and the objects within them from being accessible to the whole world.

It still allows you to share objects with specified targets such as:

  • AWS Services
  • other AWS Accounts
  • specified IP address ranges

How we got here

Amazon S3 was the first AWS Service launched, way back in 2006. Users store file objects in Buckets and can control access to them through a variety of mechanisms, including:

  • Bucket ACLs
  • Object ACLs
  • Bucket Policies
  • IAM Polcies

Objects can be made accessable via:

  • Unauthenticated Web requests (via http/https)
  • AWS API calls (via AWS Web Console, AWSCLI, SDKs, etc)
  • BitTorrent

Confusion around the different methods for controlling access can lead to mistakes. Amazon's "only recommended use case for the bucket ACL is to grant write permission to the Amazon S3 Log Delivery group to write access log objects to your bucket, yet Bucket ACLs still make it easy to make the Bucket world readable (and even writable!).

Detecting Publicly Accessible Buckets

AWS Trusted Advisor's S3 Bucket Permissions Check has been free since Feb 2018.

Business and Enterprise support customers can use these checks to enable automated actions via Trusted Advisor's integration with CloudWatch Events.

Block S3 Public Access

In Nov 2018, AWS launched a new feature that allows you to control against Objects in S3 Buckets being made Public. It consists of four settings which can be applied at the Bucket or Account level. Applying at a Bucket level may enable the rules to be overridden.

Objects intended to be shared publicly (e.g. static websites) can have a Bucket Policy with configured to grant read access to a CloudFront Origin Access Identity.

For situations where CloudFront is considered overkill (it can take ~30 minutes to provision), users may consider granting access to a specific IP Range, AWS Account or IAM Role.

What does Public mean

  • ACLs: AllUsers or AuthenticatedUsers

  • Policies

In order to be considered non-public, a bucket policy must grant access only to fixed values (values that don't contain a wildcard) of one or more of the following:

  • A set of Classless Inter-Domain Routings (CIDRs), using aws:SourceIp. For more information about CIDR, see RFC 4632 on the RFC Editor website.
  • An AWS principal, user, role, or service principal
  • aws:SourceArn
  • aws:SourceVpc
  • aws:SourceVpce
  • aws:SourceOwner
  • aws:SourceAccount
  • s3:x-amz-server-side-encryption-aws-kms-key-id
  • aws:userid, outside the pattern "AROLEID:*"

Enabling S3 Block Public Access on an Account

Applying S3 Block Public Access may break things! Administrators applying this feature should familiarize themselves with the AWS Documentation.

In order to perform Block Public Access operations on an account, use the AWS CLI service s3control.

The four settings that can be configured independantly) are:

  • BlockPublicAcls: Block setting of ACLs if they include public access
  • IgnorePublicAcls: Ignore Public ACLs
  • BlockPublicPolicy: Block setting of Policy that includes public access
  • RestrictPublicBuckets: Restrict buckets with public Policy to same account and AWS Principals

The account-level operations that use this service are:

  • PUT PublicAccessBlock (for an account)
  • GET PublicAccessBlock (for an account)
  • DELETE PublicAccessBlock (for an account)

Example CloudFormation for granting access to Origin Access Identity and IP range

  CFOAI:
    Type: AWS::CloudFront::CloudFrontOriginAccessIdentity
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: !Ref AWS::StackName

  Bucket:
    Type: AWS::S3::Bucket

  BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref Bucket
      PolicyDocument:
        Version: 2012-10-17
        Id: PolicyForCloudFrontPrivateContent
        Statement:
          - Sid: Grant a CloudFront Origin Identity access to support private content
            Action: "s3:GetObject"
            Effect: "Allow"
            Principal:
             CanonicalUser: !GetAtt CFOAI.S3CanonicalUserId
            Resource: !Sub "arn:aws:s3:::${Bucket}/*"
          - Sid: Grant access from Trusted Network
            Action: "s3:GetObject"
            Effect: "Allow"
            Principal: "*"
            Resource: !Sub "arn:aws:s3:::${Bucket}/*"
            Condition:
              IpAddress:
                aws:SourceIp: !Ref OfficeIp

Semantic CloudFormation Parameter Values

Here's a pure Cloudformation solution to two annoyances I encounter when managing AWS CloudFormation Parameters. It allows you to optionally specify exported CloudFormation Output values in your CloudFormation Parameters.

Most resources I deploy on AWS are managed via CloudFormation using reusable templates and custom Parameters. Configuring the Parameters often requires looking up resource identifiers for VPCs, Subnets, Route Tables and the like.

Here are the Parameters for a stack that creates routes for a VPC Peering Connection:

[
  {
    "ParameterKey": "RemoteSubnet1CIDR",
    "ParameterValue": "10.0.38.0/24"
  },
  {
    "ParameterKey": "RemoteSubnet2CIDR",
    "ParameterValue": "10.0.39.0/24"
  },
  {
    "ParameterKey": "RouteTable1",
    "ParameterValue": "rtb-01234567"
  },
  {
    "ParameterKey": "RouteTable2",
    "ParameterValue": "rtb-12345678"
  },
  {
    "ParameterKey": "VpcPeeringConnection",
    "ParameterValue": "pcx-11111111111111111"
  }
]

The Annoyances

I love CloudFormation but the file above annoys me for two reasons:

  1. It doesn't convey much about these route tables or subnets

These routes are for the bma-prod VPC to get to internal subnets on failmode-prod. In order to work that out you would need to lookup each value. That's toil.

  1. I had to query AWS to find these values

When creating the Parameters file for the non-prod account, I would need to lookup all these values again. That's toil.

Semantic CloudFormation Parameter Values

The VPCs I deploy export Stack Output values that can be imported by other Stacks. These are given unique names by prepending the stack name to the value identifer.

I resolved both annoyances above by updating my Parameters file to refer to these values:

[
  {
    "ParameterKey": "RemoteSubnet1CIDR",
    "ParameterValue": "import:vpc-failmode-prod-SUBNETINTERNAL1CIDR"
  },
  {
    "ParameterKey": "RemoteSubnet2CIDR",
    "ParameterValue": "import:vpc-failmode-prod-SUBNETINTERNAL2CIDR"
  },
  {
    "ParameterKey": "RouteTable1",
    "ParameterValue": "import:vpc-bma-prod-RTBPRIVATE1"
  },
  {
    "ParameterKey": "RouteTable2",
    "ParameterValue": "import:vpc-bma-prod-RTBPRIVATE2"
  },
  {
    "ParameterKey": "VpcPeeringConnection",
    "ParameterValue": "pcx-11111111111111111"
  }
]

Adding Support to the Stack Template

This pure CloudFormation pattern supports both of the Parameter styles shown above. We define some conditions that look for import: at the start of a Parameter value and this determines whether it should be imported or simply used as a string.

AWSTemplateFormatVersion: '2010-09-09'
Description: VPC Peering Routes
Parameters:
  VpcPeeringConnection:
    AllowedPattern: ^pcx-[a-f0-9]+$
    ConstraintDescription: Must be a valid VPC peering ID
    Description: VPC Peering connection ID
    MinLength: '12'
    MaxLength: '21'
    Type: String
  RemoteSubnet1CIDR:
    Description: CIDR range of Remote Internal subnet 1
    Type: String
  RemoteSubnet2CIDR:
    Description: CIDR range of Remote Internal subnet 2
    Type: String
  RouteTable1:
    Description: Local Route Table 1
    Type: String
  RouteTable2:
    Description: Local Route Table 2
    Type: String

Conditions:
  ImportRemoteSubnet1CIDR: !Equals [ "import", !Select [ 0, !Split [ ":", !Ref RemoteSubnet1CIDR ] ] ]
  ImportRemoteSubnet2CIDR: !Equals [ "import", !Select [ 0, !Split [ ":", !Ref RemoteSubnet2CIDR ] ] ]
  ImportRouteTable1:       !Equals [ "import", !Select [ 0, !Split [ ":", !Ref RouteTable1 ] ] ]
  ImportRouteTable2:       !Equals [ "import", !Select [ 0, !Split [ ":", !Ref RouteTable2 ] ] ]

Resources:

  RouteTable1ToRemoteSubnet1:
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: !If
        - ImportRemoteSubnet1CIDR
        - Fn::ImportValue: !Select [ 1, !Split [ ":", !Ref RemoteSubnet1CIDR ] ]        
        - !Ref 'RemoteSubnet1CIDR'
      RouteTableId: !If
        - ImportRouteTable1
        - Fn::ImportValue: !Select [ 1, !Split [ ":", !Ref RouteTable1 ] ]        
        - !Ref 'RouteTable1'
      VpcPeeringConnectionId: !Ref 'VpcPeeringConnection'

Conclusion

I like this pattern because it: - makes it easier to create and read parameter files - doesn't have any external dependancies - also supports specifying resource ids as strings

Feedback welcome in the comments.