AWSNinja

Today I’m going to try to answer the question: JavaScript – How do you organize this mess?!? The StackOverflow user who posted this question was primarily concerned with organizing JavaScript classes and namespaces – but many of the replies were about managing and serving script files. Both topics are extremely important, and in this post I’ll be focusing on the latter.

I’ve previously written about managing your site’s images on CloudFront, and this article will basically tackle the same topic for JavaScript. Someday, I will post a third post in this series that deals with CSS.

The Headaches

The headaches that your site’s JavaScript cause can be grouped into two categories, Maintainability Considerations and Performance Considerations. Here are the main points:

JavaScript Maintainability Considerations:

• It’s a lot of code doing a lot of complex stuff. In addition to the functionality you write yourself, you’re including more third-party libraries and plug-ins than ever before.
• You need to manage dependencies. When you include a component on the page, you need to make sure you include the other required components.
• The order that you include the code in your page matters. A dependency needs to appear before the component that depends on it.
• The development environment is completely different from production.
  • In development, your primary concern is finding where the error is, so you want your application to be broken down logically into files and you want the code to be readable for a human being like yourself.
  • On production, you want peak performance which means combining the JavaScript into one file and minimizing the code.

JavaScript Performance Considerations:

• Minimize your code.
• Compress your files.
• Combine your files.
• Get the browser to cache your files (but also make sure the browser isn’t using an outdated version).

These performance considerations (and many, many others) come mainly from the work of Steve Souders. If you’re not familiar with his work in front-end optimizations, go read up. We’ll be here when you get back.

The CloudFrontJavaScript package is my strategy for implementing the best practices in all of these areas. The idea is to keep track of all of your JavaScript files in a couple of database tables, then use a class called the CloudFrontJavaScriptService to manage the resources on CloudFront and generate the SCRIPT tags with the proper source URLs to be inserted into your HTML page.

Before I describe the technical details and how to install and use the package, I am going to highlight a couple of key features of CloudFrontJavaScriptService:

Maintenance Mode

To facilitate website development, the CloudFrontJavaScriptService can be run in Maintenance Mode. In Maintenance Mode, several things are different:

• JavaScripts are sent to the browser as the original source files. They are not minimized, compressed or combined. This is so that your browser development tools (e.g. Firebug) can work effectively.
• A random string (called “cacheBust”) is appended to the script paths on each page refresh. This is done just to make sure that your web browser does not cache any of your JavaScript, avoiding that frustrating development scenario that goes like this:
  • Adjust your code
  • Refresh
  • See no change
  • Make Another Change
  • Refresh
  • Still no change
  • Make Still Another Change
  • Refresh
  • “What’s wrong with me!?!”
  • Make a change that is absolutely guaranteed to completely break the program.
  • Refresh
  • &%^#(%$)!!!
• The script files are served from your web server, not CloudFront.

By providing Maintenance Mode, the CloudFrontJavaScriptService accommodates the needs of the development environment without adding to the work required to maintain the code and without making any compromises in performance on the production environment.

Compression

One of the major downsides of Amazon CloudFront compared to some other CDNs is that CloudFront will not automatically compress (i.e. gzip) your content for web browsers that support it. Amazon is almost certainly developing this feature and could announce its availability at any time. In the meantime, the CloudFrontJavaScriptService handles this for you. The service does this by storing both the compressed and uncompressed versions on CloudFront. When a page request comes in, it examines the headers for the HTTP_ACCEPT_ENCODING header to determine if gzip is supported. It then returns the URL to either the compressed or uncompressed version of the file.

Versioning

Both CloudFront edge locations and your users’ web browsers are going to cache your content. This is what you want them to do in order to get the best possible performance from your website. However, this creates an issue when your content changes because you can’t fully control when your users will see the updated content.

The CloudFrontJavaScript service handles this by maintaining a version number. The version number is part of the URL of the JavaScript file. The number is reset whenever a new version of your site is deployed. This causes the URL of the JavaScript files to change. Because a new URL indicates a completely new file, both CloudFront and the user’s browser are guaranteed to request the new version. Consider the URL of “script 25” in CloudFrontJavaScriptService:

http://d5zgyj1bk6lvu.cloudfront.net/javascript/1285535650_25.gz.js

After deploying a new version, the URL of this resource might look like this

http://d5zgyj1bk6lvu.cloudfront.net/javascript/1285362461_25.gz.js

The large number is the version number and is also a Unix timestamp. That value matches the time that the version was deployed, ensuring that caching does not prevent users from getting the most recent version of your JavaScript.

The Database

To help you get a sense of the underlying architecture of the service, here are the two database tables that keep track of the details:

The tbl_javaScriptScript table contains the following fields:

• id – The auto-incrementing primary key.
• lookupId – This is a unique number that you will assign to the script file and will be the number you use to indicate which script you’re requesting from the CloudFrontScriptService. I prefer to increment the numbers by five and to make the lookupId be the same as the sortOrder.
• fileName – This is the file name of the script. Each script will appear in your filesystem in two versions (one normal and one minimized). In this field, the filename is displayed in the form to be used for the minimized version of the file. The minimized form will be like this “yahoo_yahoo.js.” In your development environment (where you will actually work the files) this file would appear in “yahoo” subdirectory and would appear like this “yahoo/yahoo.js” The CloudFrontScriptService has a method in it for translating one form to the other.
• dependencies – This field contains a comma-delimited string of lookupIds which represent the dependencies of the script. A typical value in this field would be “25,35,40” where the scripts with lookup ids of “25”, “35”, and “40” are dependencies of the current script.
• sortOrder – This field specifies the sort order the scripts. When the CloudFrontScriptService determines which scripts (including dependencies) it needs to include in your webpage, it will order them in the order specified here. Typically, I like to start out with the lookupIds and sortOrder as the same value. For example, the file with a lookupId of 10 will start out with a sortOrder of 10. If I find out later that things need to be reordered, I can just adjust the sortOrder without the much bigger hassle of rearranging the lookupIds.

The tbl_javaScriptCDN table contains the following fields:

• id – An auto-incrementing id. Not the primary key and not currently used. (The primary key is a composite of scriptIds and gzip.)
• scriptIds – This is a comma-separated list of the scriptIds that were requested. If you request the script with lookupId 25, then the value in this field will be “25”. If you request scripts 25 and 35 then the value will be “25,35”. The dependencies of scripts 25 and 35 won’t be included here even though they would be sent to the browser.
• version – This indicates the version number of the scripts that are currently being served from Cloudfront. If the version number in this row is out of date, the CloudfrontJavaScript service knows to rebuild the script package, upload it to CloudFront and update the version number in this table.
• gzip – This indicates whether the combined JavaScript file is gzipped.

Installation and Usage

Download the package from GitHub and put it on your webserver. You will also need the awsninja_core package.

YUI Components
To demonstrate how the CloudFrontJavaScriptService works with a large JavaScript project, the javascript directory in the package contains the JavaScript files from the YUI Library. If you want to test out the ability of the service to serve files to a web browser in Maintenance Mode, you will need to move the files to a location within your web root and change the CDN_JAVASCRIPT_SOURCE_PATH in config.php to match the new path.

Steps to Get Set Up

1. Update the settings on config.samp.php and rename the file to config.php.
2. Run the sql in the ninja_cloudfrontimages.sql against your database to create the needed tables.
3. In the examples directory, run the yuiStoke.php script from the command line (e.g. type “php yuiStoke.php”) to populate the tbl_javaScriptScript table with the YUI script file information. The yuiStoke.php script uses a dependency tree created by jassem.shahrani to set up the YUI files.
4. Run the deploymentTest.php script from the command line. This will minimize and copy the YUI JavaScript files to the js-minified directory. It will also delete the version file, which will cause the CloudFrontJavaScriptService to create a new version number at the first opportunity.
5. Run the serviceTest.php script to see a demonstration of how the CloudFrontJavaScriptService generates HTML Script Tags.

At step five, you know that you’ve got it set up correctly if you see output similar to this:

The code in the example requests script 25, which is the YUI datasource.js script. This script is dependent on event.js (script 10) which itself is dependent on yahoo.js (script 5). In all of the examples, we use the getScriptHTML() method with the first argument 25. See the serviceTest.php source for more details.

The following uses are demonstrated:

• Under “Here’s the uncombined script” you see that the datasource.js script and it’s dependencies are individually included. The scripts are ordered according to the sort_order field in the database and a random “cacheBust” string is included at the end of the URLs. This is ideal for the development environment, because your code will be properly broken out into files that are not minimized. Your browser development tool (e.g. Firebug) will have no problem directing you to your errors by file and line number.
• Under “Here’s the combined uncompressed script:” you see that there is a single file and it is being served from a CloudFront domain name. The three JavaScript files are combined into one and minimized. In this example, they are not compressed because this is an example of how the scripts would be served to browsers that do not support compression.
• Under “Here’s the combined compressed script:” you see something very similar to the previous one. The difference is that before the “.js” extension there is an extra “.gz” in the path. This is the path to the combined, minimized and compressed file.
• Under “Here’s how to let the web browser determine whether to compress or not:” the script echos out the most common way to use the service to insert script into HTML. Basically, you just give it the script number you want and let the service figure out the best way to serve the script.
• Finally, under “In the current context, that code produces this:” the script gives an example of what the script in the previous example outputs in the current context. The context of this output is the command line, so it sends the combined and minimized (but not compressed) script. The reason it does not send the compressed script is that in the command line context, there is no browser HTTP_ACCEPT_ENCODING header.

How I Organize and Add New Scripts to my JavaScript Project

There is one more thing that I would like to share about how I like to manage my scripts. My general strategy is to separate my third party scripts (third-party libraries and plug-ins) from the scripts that I write specifically for the project that I’m working on. I do this by including a large gap of lookupIds between the third-party components and my own components. Here’s an example from one of my projects where I’m using jQuery and a bunch of jQuery plug-ins:

Note that the third party code (jquery and jquery plug-ins) start at lookupId 1000 and my code (the “marketplace” files) start at lookupId 1500. The gap between 1000 and 1500 provides plenty of room for me to add more third party components as needed. The third-party components will always appear above my components, which is good because the third-party scripts are the dependencies.

When I need to add a new JavaScript file (either third-party or my own component), I add the file to the filesystem, then add the row directly to the database table.

Conclusion:
I hope that you find this method of managing your JavaScript useful, and I hope that I was successful in clearly explaining how it works. As always, I’d very much like to know what you think.  If you read this far, then for gosh sakes leave a comment and let me know!  It makes me happy.