By on

code aws aws-lambda aws-s3

As the web continues to evolve, the selection of HTTP response headers a site is expected to support grows. Some of the security-related headers are now considered mandatory by various online publishers, including Google and Mozilla. If like me you run your site using an S3 bucket of static files, it’s not easy to add the recommended headers and improve your site’s security scores in tools like Mozilla’s Observatory.

CloudFront and Lambda@Edge can fix that. Others have detailed the process more fully than I can but beware that some features may have changed since the older posts were written. I suggest following the article linked previously if you want to implement this for your site. I’ve listed some of the gotchas that slowed me down below.

Adding the right permissions to the Lambda IAM role

While creating the Lambda you will be asked to assign it a role. Make sure you add the Basic Edge Lambda template to the role you create to allow the function trigger to be created. I missed this step the first time and it took me a few tries to figure it out.

Beware of replication

When you deploy the Lambda and attach the trigger, CloudFront will create replicas of the Lambda in various regions. If you then update your function, publish a new version and redeploy, it will create more replicas. The replicas of older versions are not automatically deleted and cannot be deleted by the user at this time so they will pollute your account with potentially large numbers of unused Lambda functions. Hopefully Amazon will fix this issue at some point.

Choose your trigger wisely

CloudFront supports 4 triggers: origin-request, origin-response, viewer-request and viewer-response. The restrictions on run-time and memory size are different for different triggers so pay attention to the delays your functions introduce to the trafic flow.

The viewer triggers are run outside of CloudFront’s caching layer so viewer-request is triggered before the request hits the cache and viewer-response is triggered after the response has exited the cache.

origin-response is triggered after CloudFront has found the resource but before it writes its response to its own cache. That means you can add headers and cache the result, reducing Lambda invokations and delays, keeping the cost down.

Headers and their values

I’ve configured CloudFront to redirect HTTP requests to my site to use HTTPS but browsers like a header to make that even clearer. HSTS or Strict-Host-Security does this job. Other than that, there’s a range of headers that help the browser to mitigate the risks of XSS (Cross-site scripting) vulnerabilities. These are well documented by Mozilla so I won’t rehash it all here.

The interesting one is the CSP (Content-Security-Policy) header. There are a few versions of the syntax supported by different browsers and getting it right is a little tricky. The excellent CSP Evaluator by Google is very helpful for testing wordings of the CSP. Tunine the policy to work properly with Google Analytics and allow inline stylesheets while disabling most other avenues of attack took a few attempts to get right but I’m happy with what I ended up with.

I disabled everything by default with default-src: 'none' then added the permissions I needed for my site. I use a very small inline CSS stylesheet so I needed to enable that with style-src 'self' 'unsafe-inline'. I don’t make much use of images at present but if I do I’ll access them over HTTPS so I enabled that with img-src 'self' https:. Opening up just the needed permissions for scripts was a bit more difficult but the CSP Evaluator helped a great deal. It recommends a strict-dynamic policy for browsers that support it. I only use one script on my site for Google Analytics, so I had to extract the contents (including all whitespace) from the <script> tag, hash it with SHA256, then encode the hash with Base64 and add the result directly to the CSP policy. CSP Evaluator also recommends some fall-back options for browsers that do not yet support strict-dynamic so I end up with script-src 'strict-dynamic' 'sha256-my_script_hash' 'unsafe-inline' https:, where my_script_hash is the Base64-encoded SHA256 hash of the contents of my script. The complete example is in the below code.

Lambda code template

My basic Lambda template for adding custom headers to all HTTP responses on my site is shared below.

'use strict';

exports.handler = (event, context, callback) => {
    function add(h, k, v) {
        h[k.toLowerCase()] = [
                key: k,
                value: v

    const response = event.Records[0].cf.response;
    const headers = response.headers;

    // hsts header at 2 years
    // Strict-Transport-Security: max-age=63072000; includeSubDomains;
    add(headers, "Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload");

    // Reduce XSS risks
    add(headers, "X-Content-Type-Options", "nosniff");
    add(headers, "X-XSS-Protection", "1; mode=block");
    add(headers, "X-Frame-Options", "DENY");
    // TODO: fill in value of the sha256 hash
    const csp = "default-src 'none'" +
        "; frame-ancestors 'none'" +
        "; base-uri 'none'" +
        "; style-src 'self' 'unsafe-inline'" +
        "; img-src 'self' https:" +
        "; script-src 'strict-dynamic' 'sha256-my_script_hash' 'unsafe-inline' https:"
    add(headers, "Content-Security-Policy", csp);

    console.log('Response headers added');

    callback(null, response);

By on

cmake cpack rpmbuild

I’ve been trying to analyse a core dump generated by a C++ application when it seg-faulted. I use CMake 3 to build it and create an RPM with CPack. This application is currently built in debug mode using -DCMAKE_BUILD_TYPE=Debug on the command line that invokes CMake.

The generated binaries have all their debug symbols as expected but the binaries in the RPM package do not. After some searching, I learned that rpmbuild strips binaries by default on some distributions. This makes analysing a core dump way harder than it needs to be so I found a way to turn this feature off using CPack. The trick is to set a variable in the CMakeLists.txt file:

# prevent rpmbuild from stripping binaries when in debug mode
  set(CPACK_RPM_SPEC_INSTALL_POST "/bin/true")

Now my debug packages retain their debug info after installation so it’s possible to get a lot more information out of gdb when looking at a core dump.

This is documented in a roundabout way online, but it took me a while to figure it out so I thought I’d write it up.

By on

core-dump linux

Since Systemd took over as the main init system in Red Hat Linux and derrivatives like CentOS, it has become more difficult to get a core dump out of a daemon application. The traditional approach of running ulimit -c unlimited before executing the binary works when running the application from the command line but does nothing for a daemon managed by Systemd’s unit files.

There is a lot of misleading information online about how to solve this so I thought I’d add a correct solution to the mix in the hope that it’s helpful.

The suggestions I found online include editing /etc/security/limits.conf, adding LimitCore=infinity to the Unit file, and messing around with /etc/systemd/coredump.conf. None of these methods work without customising the kernel configuration first.

Systemd is not configured to handle core dumps by default on CentOS (and by extension RHEL) distributions. The default behaviour is to write to the file core in the process’s working directory, which for daemons is often the root directory.

The obvious problem here is that the daemon probably doesn’t have write access to the root directory (if running as a non-root user). If is possible to change the working directory with the Systemd unit directive WorkingDirectory=/var/run/XXX. This is typically used with RuntimeDirectory=XXX, which creates and manages the lifecycle of /run/XXX (/var/run is a symlink to /run). Unfortunately, we can’t write the core file to a RuntimeDirectory because it gets deleted when the application terminates.

The simplest solution I found is to overwrite the kernel core_pattern setting. This can be edited at runtime by echoing a new value into /proc/sys/kernel/core_pattern:

echo /tmp/core-%e-sig%s-user%u-group%g-pid%p-time%t > /proc/sys/kernel/core_pattern

This will force the kernel to write all core files during the current OS uptime to /tmp with the filename pattern specified. The core manpage has more information on the syntax.

This change will be lost when the machine reboots. To effect the change at kernel startup, you need to edit /etc/sysctl.conf or a file in /etc/sysctl.d/.


Our solution at work was to write a script to create a file in /etc/sysctl.d/ at machine image creation time, so that the config is always there when we roll out to different environments (int, test, live etc.)

It should go without saying that there is no particular reason to use /tmp. The output can be redirected to any location the process has permission to write to. A network share may be more appropriate in some cases.

There may be another solution using systemd-coredump, but it is not part of this release of CentOS (7.2) and not in the yum repository at this time.

By on

code azure azure-functions azure-storage

I’ve written previously about how Amazon’s AWS Lambda feels like the future. I had a little time today to play with Azure Functions, Microsoft’s new competing service.

As a preview, Azure Functions isn’t quite polished yet, but the basic building blocks are in place. You create a Function App in the Azure Portal, much like you would create a Web or Mobile App, then add a few Functions in the Portal blade.

Since the service is built on the mature WebJobs product, there is already strong language support. Azure Functions can be created with specialised C#, NodeJS, PHP, Python, Powershell, Windows Batch scripts (I know), or any standard Windows executable. The Portal UI leaves a lot to be desired for now. I did’t find any obvious way of using different languages or packaging up apps and dependencies, but did manage to get it working. The documentation implies that there are many more features than those exposed by the Portal blade.

For the sake of a quick test, I created a small function to connect to the Azure Storage account and create an empty Table. I had a little trouble setting up the connection string to the storage account but found a solution using magic environment variables here. This may seem normal to regular dotNet and/or Azure developers but it was new to me. The documentation implied that I needed to use a CloudConfigurationManager from this NuGet package but I failed to figure out how to pull in the dependency and resolve the namespace. The solution is to copy a connection string from the Azure Storage account and paste it into the Functions App Service Settings blade’s Application Settings under “Connection Strings”. This effectively makes the string available to the Function in an environment variable at runtime. If our connection string is named StorageConnectionString, and the type is “Custom”, then the value will be available in the environment variable CUSTOMCONNSTR_StorageConnectionString.

Here’s the code I ended up with as shown in the editor in the Azure Portal.

#r "Microsoft.WindowsAzure.Storage"

using System;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Table;

public static void Run(object trigger)
    var storage = CloudStorageAccount.Parse(
    var tableClient = storage.CreateCloudTableClient();
    var membersTable = tableClient


This file is a little weird because it is a csx file. There is some documentation in the Azure Functions Developer Reference but it’s essentially C# with embedded assembly references (#r "Microsoft.WindowsAzure.Storage") and a top-level public static void Run method. Any other classes etc. you need can be included inline in the file.

With the code in place, you have to switch to the ‘integration’ tab to enable a trigger so that the function can be run. I did the minimum needed to test the function and just selected a manual trigger, changed the name to “trigger”, then switched back to the code view and hit the “Run” button at the bottom of the Function App blade. It took a few goes to get the code right, but with everything working, I was able to confirm that it worked by opening the storage account blade in the Azure Portal and looking at the list of Tables in the account. Sure enough, we have our shiny new table.

While creating a table programmatically is about the simplest use case I could think of for testing a Function, I must admit that it feels a bit forced to have to write code for this at all. I prefer Amazon’s model of providing Console-based UIs for most of the core functionality, like creating tables and adding records. There are Storage Explorer tools for this on Azure, but it would be nice to be able to manage everything from the Portal.

There’s a lot more ground to cover here: Function triggering, integration with other services, creating API endpoints and deployment automation are all considerations. I look forward to seeing how Azure Functions evolve in the future.

By on , modified

aws aws-lambda aws-sns aws-cloudformation

Amazon Web Services (AWS) provides many building blocks you can use to create just about anything in the world of web-connected services. A lot of the skill in using this toolkit is in figuring out how to make the various services work together. CloudFormation is critical to this effort, as it let’s you write a config file that can be used to automate the creation of all the infrastructure you need to deliver your product. The big win from using CloudFormation is reproducibility. It let’s you define your whole infrastructure requirement, put it in version control, and reproduce the same setup in a production environment without manually recreating everything.

CloudFormation is not without its problems, however. The Syntax is overly verbose and awkward to author. Some of the services it can create cannot be edited, and it is easy to get into circular reference hell with more complex configurations involving the interactions between multiple products. There are reams of documentation but most of it is useless noise without meaningful explanation. The emerging best-practices are to work in another format (like YAML) and convert it to CloudFormation’s preferred format mechanically, or to pay a third party for their cloud automation tools.

I’ve been struggling to get an SNS topic to trigger a Lambda this week. This is really easy using the AWS console because it automates many of the tricky and barely documented steps required to get it working. Wiring it up in CloudFormation is another hell entirely. You need a few different pieces in the right order to make this work so I’ll go through each one in turn. Each of the below can be pasted into the Resources section of a CloudFormation template.

For completeness, I’ve included a basic template skeleton here.

  "Description": "blah blah",
  "Parameters": { ... },
  "Outputs": { ... },
  "Resources": { ... }

Lambda Execution Role

Before you can build a Lambda Function, you need to create some permissions for it to assume at runtime. Here I present a fairly minimal role suitable for a basic Lambda Function with no external integration points. Additional permissions (e.g. reading from an S3 Bucket) can be added to the list of Statements in the PolicyDocument. This part is actually documented reasonably well.

  "ExecutionRole": {
    "Type": "AWS::IAM::Role",
    "Properties": {
      "Path": "/",
      "Policies": [
          "PolicyName": "CloudwatchLogs",
          "PolicyDocument": {
            "Statement": [
                "Action": [
                "Resource": [ "arn:aws:logs:*:*:*" ],
                "Effect": "Allow"
      "AssumeRolePolicyDocument": {
        "Statement": [
            "Action": [ "sts:AssumeRole" ],
            "Effect": "Allow",
            "Principal": {
              "Service": [ "lambda.amazonaws.com" ]

Lambda Function

The Lambda Function itself is quite easy to set up. I’ve hard-coded placeholders for the S3 Bucket and path to the zip file containing the code. It doesn’t matter what the code is for the purpose of this article.

  "Lambda": {
    "Type": "AWS::Lambda::Function",
    "Properties": {
      "Code": {
        "S3Bucket": "my-personal-bucket",
        "S3Key": "lambdas/test/my-lambda.zip" }
      "Description": "Some Lambda Function",
      "MemorySize": 128,
      "Handler": {"Ref": "LambdaHandler"},
      "Role": {
        "Fn::GetAtt": [ "ExecutionRole", "Arn" ]
      "Timeout": 5,
      "Runtime": "python2.7"
    "DependsOn": [

SNS Topic

The SNS Topic is very easy to create but it cannot be modified using CloudFormation after it has been created. You also have to create all the subscriptions at the same time, so if you use CloudFormation for reproducibility, you can never change the subscriptions of a running event pipeline that relies on SNS. This is a major drawback of SNS and CloudFormation and should be considered with care before you rely too heavily on this set of tools.

The unmodifiable nature of SNS Topics created this way won’t be a problem if you’re creating subscriptions via an API at runtime, but it limits how flexible SNS can be for some workflows, like fanning out to SQS Queues or triggering Lambda Functions.

  "Topic": {
    "Type": "AWS::SNS::Topic",
    "Properties": {
      "Subscription": [
          "Endpoint": {
            "Fn::GetAtt": [ "Lambda", "Arn" ]
          "Protocol": "lambda"
    "DependsOn": [ "Lambda" ]

Permission for the Topic to invoke the Lambda

Unfortunately, creating all the pieces isn’t enough. We still need to grant our SNS topic permission to invoke the Lambda Function directly. This is really important and the documentation is almost completely useless so I’m putting a working example here.

It is absolutely critical to have the SourceArn property refer to the Topic we created earlier. Without this, the Lambda Console will give you inexplicable errors, while the SNS Console claims that everything is correct and working. It can be very frustrating trying to get this right.

  "LambdaInvokePermission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
      "Action": "lambda:InvokeFunction",
      "Principal": "sns.amazonaws.com",
      "SourceArn": { "Ref": "Topic" },
      "FunctionName": {
        "Fn::GetAtt": [ "Lambda", "Arn" ]

Update: An observant reader spotted a bug in the above CloudFormation. I originally had a reference to the SourceAccount in the Properties block:

  "SourceAccount": { "Ref": "AWS::AccountId" },

This was incorrect. It works after removing that line.


That should be enough to get it all working. There are some serious holes in the capabilities of CloudFormation for working with SNS, and the permission model is a poorly documented mess. The only thing I haven’t covered is how to get the Lambda itself to run, but that’s a topic for another day.