Serving compressed (gzipped) static files from Amazon S3 or CloudFront

It’s generally a good idea to serve gzipped versions of plain-text static assets (primarily CSS and JavaScript files) to web browsers. This can significantly reduce file size, which increases perceived website speed. All modern web browsers can transparently decompress gzipped files, thus the only real downside is a bit of additional CPU overhead.

Most default installations of Apache take care of this automatically via mod_deflate (Nginx uses the equivalent HttpGzipModule). Files are compressed on-the-fly and a Content-Encoding: gzip HTTP header is added to the response. However, since Amazon S3 is just a place to store files it lacks the ability to gzip files in real-time before delivering them. When using a website speed test application like WebPageTest, this can result in informational warnings that look like this:

Use gzip compression for transferring compressable responses: 90/100
FAILED - (52.1 KB, compressed = 40.6 KB - savings of 11.5 KB) - http://mybucket.s3.amazonaws.com/css/awesomeness.css

To resolve this, files have to be compressed before being uploaded to S3. From Linux or OSX, this can be easily done with gzip -9 awesomeness.css, which creates a new, compressed version of “awesomeness.css.”

This new file is then uploaded to S3 and the following metadata is set on the bucket object:

Setting content-encoding and content-type on S3 bucket objects

The Content-Type header informs the web browser that the actual contents of the file is CSS markup while Content-Encoding specifies that it’s a gzipped file. Both HTTP headers (“metadata” in AWS-speak) are required for the browser to correctly interprete the file.

Gettin’ fancy

Manually gzipping each file and setting the correct metadata can get very tedious, very quickly. I would recommend that this process be automated as a build step on a continuous integration (CI) server (at RBN we use Jenkins): a post-commit hook on the source repository fires, triggering a build. The CI server performs a SVN export or Git clone, executes a shell script that gzips the files, and uploads them to S3 and sets the proper object metadata. If you’re not currently using CI in your development workflow, a more basic version of this process can be hacked together using only commit hooks. Though I would argue that this would be an opportune time to dip your toes into continuous integration.

10 Replies to “Serving compressed (gzipped) static files from Amazon S3 or CloudFront”

  1. Hi Jamie

    Interesting article. I have exactly the same problem but how do manage this on CloudFront as opposed to S3? Should I simply compress the files as shown and then upload to the server and wait for CloudFront to pick them up?

    Thank you

    Ste

  2. Stephen —

    Cloudfront can use S3 or your own web server as the origin (the canonical location of the files to be distributed on the CDN). If you’re using your own server, I would just recommend that you let your HTTP server (Apache, Nginx, IIS, etc) transparently compress the files on-the-fly. S3 can’t process files before they enter the CDN, which is why the extra step I wrote about here is necessary.

  3. Hi Jamie,

    What would you do if you cannot find the metadata section. It seems as if i dont have this in my account so I cannot add the Content-Type and Content-Encoding?

    Thanks

  4. Richard,

    I am certain that you have the ability to alter the metadata, unless you’re logging in via an IAM account and don’t have permissions to do so. If you’re having problems via the management console, it might be worth trying to alter the keys’ properties using the command line interface or directly using the API.

  5. Richard – You need to apply the metadata to each individual file. Right click on the file (the .gz file in this example) and select “properties”

  6. If you’re precompressing content, and not doing so on the fly via your web server when pages are requested, then I would recommend using Zopfli from Google. It’s a gzip compresser that is ~3-8% more efficient than the standard gzip compresser, and it is fully gzip compliant, so any gzip decompresser can decompress the files it generates.

    The only drawback to using it is that it takes about 10 times longer to compress the files, but unless you’re really concerned about the speed of your build process, this shouldn’t be an issue. There is no change in the time required to decompress files compressed with Zopfli.

  7. What can be done if I have to do it for all folder, subfolder and files, I have multiple subfolder and CSS and JS file, Please suggest what can be done for multiple files with subfolders

  8. Is there a risk that the client/visitor doesn’t accept GZIP encoding and fail to receive a result?

    The AWS docs mention to upload both gzip and uncompressed and build some logic in your app to send the user to the correct version… but is this really necessary ?

Leave a Reply

Your email address will not be published. Required fields are marked *