Skip to content

on the Edge

February 19, 2014

REST is pretty simple architecture and for the majority of the moving parts within, it’s well understood. Its nature nicely aligns with the stateless HTTP protocol and for the limited server-side performance. When it comes to simple web resources REST is the standard and it looks like that everyone agrees with it. Few problems/questions arise when one attempts to apply this simple principle on a complex web document.

Current web documents are assembled from multiple resources and it is not uncommon to find a HTML page that forces the browser to executes hundreds of requests. We tend to say that this is bad architecture or the page needs optimization but the reality is that a useful document or interface is just complex structure of many, many resources.

To improve the client-side performance we combine, minify and compress any resource we can, Google even multiplexes TCP connections in a shiny new protocol but still – separation of resources forces requests fragmentation.

In most of the cases, the fragmentation is considered the small price to be paid for gaining nice cacheability and order of magnitude in server-side performance, Still – there are requirements like SEO, authentications, inter-resource dependency and so on when fragmentation is reaching unmaintainable levels. Usually in such situations we instinctively reach out for the well known server-side implementation of monolithic documents dynamically generated by the ever growing server farm. This might patch the problem at hand but breaks the REST and more painfully – busts the cache.

Interestingly enough the solution is in the REST definition itself. To cite the wikipedia article” … Layered system : A client cannot ordinarily tell whether it is connected directly to the end server, or to an intermediary along the way. …”. We can combine, verify and swap resources on the fly without invoking any server-side logic or at least any application server logic.

Edge Side Includes (ESI) can solve most request fragmentation and provide elegant yet scalable solutions to search engine optimization, inter-resource dependencies and simple authentication.

Let’s examine usecase where we have a simple resource – an html document but our business logic demands that it is available only to clients with proper API key. Historically we would setup some server-side authentication, which would work but as the number of requests increases so will our server load. Not only that, but we would create a bottleneck in our architecture because URLs like that are completely uncacheable, so our total response time will increase and client-side performance will degrade.

If we implement ESI (Akamai flavor here) we can create a document that contains something like :

<esi:eval src="/check-key?api=$(QUERY_STRING{api})" dca="none"/>
<esi:choose>
 <esi:when test="$(checkResult) == 'OK'">
   <esi:include src="/locked/our-not-so-secured-doc.html"/>
 </esi:when>
 <esi:otherwise>
   <esi:include src="/bad-api-key.html" ttl="365d" />
 </esi:otherwise>
</esi:choose>

This snippet is self explanatory but the idea behind is pretty powerful one.

With ESI we have REST with its simplicity, server-side scalability and client-side performance. SEO with  _escaped_fragment_ should be a breeze and simple auth per asset shouldn’t kill the client.

One might ask if the edge server won’t become another application server (aka bottleneck) but the simplicity of the logic available should eliminate “apps on the edge”.

The one real problem with ESIs is that they are proprietary to the CDNs and there is no real specification or standard. I suspect the as REST is becoming the standard architecture for web and mobile the specifications will solidify and open source implementation will appear.

Cache JSON(P) calls client side

June 10, 2013

Prerequisites

The article assumes the reader knows the basics of AngularJS. The article shows how the cache logic can be written in JavaScript but the UI render is done using AngularJS. A non-angular reader can still choose to continue reading through the article and can get the logic bits from the code. I leave it up to you.

Introduction

In real world, where we have CMS (Content Management System) to assemble our page with modules and each module function independently. The modules are developed by independent developers and the page is assembled by, probably, a different user altogether. AJAX calls are always meant to enhance user experience, but given the fact the each module function independently, there seems to a duplication of AJAX calls made on the page, probably by different modules. This post shows a way, how to cache such AJAX calls with the help of jQuery promise and a JavaScript object.

jQuery Promises

As name suggests, jQuery Promise is a literal promise made by jQuery that a call will be made on the object after its completion. The object is just like an JavaScript object and can be passed around like a ball to any method you want and any number of times you want. For more, read here.

Details

Now that you have an idea of what we are going to do, let me take you through each step of the process.

Constructing an deferred object cache

I will try not to include AngularJS code in the sample but in some places it is unavoidable. Assuming that “command” is the part of the URL and “params” are the parameter key-value map, here is a snapshot of constructing a cache map.

var cacheStorage = {};

function getCacheKey(command, params) {
    var paramStr = command + '-';
    if(params) {
        var keys = [];
        for(var key in params) keys.push(key);
        var sortedKeys = keys.sort();
        for(var count=0; count < sortedKeys.length; count++) {
            var sKey = sortedKeys[count];
            paramStr += (sKey + '-' + params[sKey] + (count < sortedKeys.length-1 ? '-' : ''));
        }
    }
    return paramStr;
};

var ret = {
    get : function(command, params) {
        var paramKey = getCacheKey(command, params);
        var cachedObj;
        if(paramKey.length > 0) {
            cachedObj = cacheStorage[paramKey];
        }
        if($rootScope.debug) {
            $log.log(paramKey + " => " + (cachedObj ? 'hit' : 'undefined'));
        }
        return cachedObj;
    },

    put : function(command, params, deferredObj) {
        var paramKey = getCacheKey(command, params);
        if(paramKey.length > 0) {
            cacheStorage[paramKey] = deferredObj;
        }
    }
};

return ret;

Explanation: If you know Angular, you probably knew about $log and $rootScope. If not, just assume that these are variables injected by Angular API. The cache tries to form a key and save the object in the cache map for the key. We sort the params before forming the key because we do not want to duplicate the same object just because user gave params in a different order.

Data Source Client

Now that, our cache is ready, we need to implement a client which uses this cache and can be a interface to all the modules on the page. The requirements of the client is, to provide a generic interface to all the calls to a particular website because we wrote the cache store for a single domain. If multiple domains are involved, it is only a matter of time we edit the cache storage to modify key that includes domain name or the complete url.

var url_defaults = { key: $rootScope.key, cb: "JSON_CALLBACK" };

function doDS2Cmd( command, params ) {
    if(!params || !params.locid) {
        if(!params) { params={};}
        params.locid = $routeParams.locId;
    }
    var cachedObj = dsCacheStore.get(command, params);
    if(cachedObj) {
        return {
            'deferredObj' : cachedObj,
            'fromCache' : true
        };
    }
    var url = $rootScope.wxdata_server + "/" + command + "/" + params.locid + ".js";
    var deferredObj = $http.jsonp(url, { params: angular.extend( {}, url_defaults, params ) } );

    deferredObj.success(function (data, status) {
            if($rootScope.debug) {
                $log.log(command + ": " + JSON.stringify(data));
            }
        })
        .error(function (data, status) {
            $log.error("error $http failed with " + status + " for " + url);
        });

    dsCacheStore.put(command, params, deferredObj);
    return {
        'deferredObj' : deferredObj,
        'fromCache' : false
    };
};

var ret = {
    executeCommand : function(command, params) {
        return doDS2Cmd(command, params);
    }
};

return ret;

Explanation: We are trying to provide a interface with just one public method: executeCommand – which means executing a JSONP call. The client is trying to read from cache and if not found, it creates a promise object by var deferredObj = $http.jsonp(url, { params: angular.extend( {}, url_defaults, params ) } );. Consider this, as a jQuery equivalent of $.ajax(). Now, this promise is stored in the cache. Next time, when we get a hit from the cache, we get the promise object. Since, you always get a promise object, your module can always use .success on the promise every time it executes. If the call is already completed, your .success callback is called immediately else it waits. Here is the trick, since you are not creating a new promise, AJAX call is NOT made. Instead, it works on the existing promise object and gets the response from the promise – how many ever times you want.

Sample Module Usage

Here is an example of how to call from a module.

var commandOutput = ds2Client.executeCommand(callObj.call, callObj.params);
var fromCache = commandOutput.fromCache;
commandOutput.deferredObj.success(function(response) {
     .....
});

NOTE: The JSONP call is used for demo purposes. However, other protocol also works and you just have to change from $http.jsonp to $http.get.

Here is a complete demo (the link works only from TWC network). For out of network guys, see a non-twc demo here: non-twc demo

Conclusion

What did we just do: We learned a bit about jQuery promise, how to implement a cache store that stores jQuery promises for a given url and params, how to implement a client API which makes of cache store and provide a public interface to make AJAX calls and how to write a module that makes use of the client.

WXForecast Prototype

June 4, 2013

Prerequisites

The following post assumes the user have a basic knowledge on AngularJS and worked on the basics. If not, I would suggest you read through what Angular is and its basic tags here: Egghead IO or in Angular site.

Introduction

In this post, we will be working on implementing a complex weather reporting view with multiple functionalities like Google Maps integration, Reverse Geo-code using Google APIs, etc. The scope of this document is to let you know how to integrate all these APIs together in a Angular page and not for fully understanding the internals of the APIs.

Details

What do we need to achieve a data rep as shown: link to demo – We need hourly weather, daily weather, narration, sunrise, sunset, google charts, google maps, google geo API’s, type ahead, etc.

Aggregation data

$resource is a way to get JSON data object with a simple syntax like object.getMethod(). However, $http is even more flexible with its promise methods and $resource is more useful when performing CRUD operations. The advantage of using a resource is the output of the operations inside the $resource first returns a empty object and later returns the AJAX output once done. This is specifically useful when you assign the resource output directly to UI model. However, it is not particularly useful when you have to do some operations on top of AJAX call. I have used $resource in this demo for just showing usage.

wxModule.factory("mobagg", ['$http', '$routeParams', '$resource', function($http, $routeParams, $resource) {
    return $resource("http://wxdata.weather.com/wxdata/mobile/mobagg/:locID.js",
        {
            cb:'JSON_CALLBACK', locID:'@id'
        },
        {
            getAggregatedInfo: {method:'JSONP', params:{"key" : "2227ef4c-dfa4-11e0-80d5-0022198344f4", "hours" : "48"}, isArray: true}
        }
    );
}]);

The same can be written in $http using

function doDS2Cmd( cmd, params ) {
    var url = $rootScope.wxdata_server + cmd + "/" + params.locid + ".js";
    return $http.jsonp(url, { params: angular.extend( {}, url_defaults, params ) } )
      .success(function (data, status) {
        if($rootScope.debug) {
          $log.log(cmd + ": " + JSON.stringify(data));
        }
      })
      .error(function (data, status) {
        $log.error("error $http failed with " + status + " for " + url);
      });
}

// params contain key, locid.
var deferredObj = doDS2Cmd( 'mobagg', params );

The output of the call looks like below:

MobaggOutput

Routes

The routes are going to be either location key based or latlong:

var wxForecastModule = angular.module('wxforecast', ['ngResource', 'ui', 'ui.bootstrap', 'google-maps', 'googlechart.directives']).config(['$routeProvider','$locationProvider', function($routeProvider, $locationProvider) {
    $routeProvider.
        when('/:locId', {templateUrl: 'partials/skeleton.html'}).
        when('/:lat/:lng', {templateUrl: 'partials/skeleton.html'}).
        otherwise({redirectTo: '/30339'});
}]);

Page Content Partial

<div class="container" ng-controller="ForecastController">
    <h3 class="page-header">WX Forecast Prototype</h3>
    <div class="span12 nomargin-left">
        <div ng-controller="AlertController">
            <alert ng-repeat="alert in alerts" type="alert.type" close="closeAlert($index)">{{alert.msg}}</alert>
        </div>
    </div>
    <div class="well span7 nomargin-left">
        <div>Map with Weather Layer.</div>
        <google-map center="center" draggable="true" zoom="zoom" markers="markers" mark-click="true" fit="false" latitude="latitude" longitude="longitude" class="angular-google-map ng-isolate-scope ng-scope" style="position: relative; background-color: rgb(229, 227, 223); overflow: hidden; -webkit-transform: translateZ(0);"></google-map>
        <div ng-show="address">Exact Location Assumed: {{address}}</div>
        <span ng-repeat="place in places">{{place}}<span ng-hide="$last"> &gt; </span></span>
    </div>
    <div class="span4">
        <div ng-controller="TypeaheadController" class="ta">
            <input type="text" ng-model="selected" typeahead="location as location.displayName for location in searchDS2($viewValue)" ng-change="update()" ui-keypress="{enter:'directLoad($event)'}" placeholder="Search location or enter zip..." />
        </div>
        <a><span tooltip-html-unsafe="{{tooltipString}}"><img src="img/locicon.png" width="40px" height="40px" ng-click="getCurrentLocation()" style="padding-bottom: 10px; float: right;"  /></span></a>
    </div>
    <div class="span4">
        <div class="forecastimg" ng-show="nowWxIcon"><img ng-src="http://s.imwx.com/v.20120328.084208/img/wxicon/120/{{nowWxIcon}}.png" height="180" width="180" alt="Rain Shower" class="wx-weather-icon"></div>
        <div class="header">{{hiradObs.temp}}<sup>&deg;<span class="wx-unit">F</span></sup></div>
    </div>
    <div class="span11">
        <table>
            <tbody ng-repeat="dailyForecast in dailyForecasts" class="span10 modal-header wxrow" ng-init="dayText = ['Today', 'Tomorrow']" ng-click="dailyForecast.isCollapsed = !dailyForecast.isCollapsed">
                <tr>
                    <td class="span2"><span ng-show="$index > 1">{{getDate($index) | date:'MMM dd'}}</span><span ng-show="$index <= 1">{{dayText[$index]}}</span></td>
                    <td class="span7">{{dailyForecast.narration.phrase}}</td>
                    <td class="span2"><div ng-show="dailyForecast.maxTemp">{{dailyForecast.maxTemp}}<sup>&deg;F</sup> <span class="icon-arrow-up"></span></div> <div>{{dailyForecast.minTemp}}<sup>&deg;F</sup> <span class="icon-arrow-down"></span></div></td>
                </tr>
                <tr>
                    <td colspan="3">
                        <div collapse="dailyForecast.isCollapsed">
                            <div class="well wxdetails">
                                <div ng-controller="ChartDataController" ng-show="isHourlyDataAvailable">
                                    <div google-chart chart="chart" style="{{chart.cssStyle}}"/>
                                </div>
                                <div>
                                    <div class="span4">Sunrise: {{getDateFromEpoch(dailyForecast.sunData.rise) | date:'hh:mm a'}}</div>
                                    <div class="span4">Sunrise: {{getDateFromEpoch(dailyForecast.sunData.set) | date:'hh:mm a'}}</div>
                                </div>
                            </div>
                        </div>
                    </td>
                </tr>
            </tbody>
        </table>
    </div>
</div>

As you can see from the partial, the page is quite a bit collection of google maps, google chat for each daily row, sunrise-sunset data, narration data, etc. The population javascript is quite simple, we get the mobagg response and map it to appropriate models.

mobagg.getAggregatedInfo({locID : locId}, function(aggdata) {
            if(aggdata && aggdata[0]) {
                $scope.aggInfo = aggdata[0];
                $scope.weatherAlerts = $scope.aggInfo.WeatherAlerts;
                $scope.hiradObs = $scope.aggInfo.HiradObservation;
                ....
            }
});

Google Maps

Google maps is included by angular-google-maps plugin available on the internet. However, I had to make some modifications to enhance the map such as weather layer, click traversal, etc. The angular directive “google-maps” creates a DOM element inside the directive and provides the DOM element to google maps for map rendering.

// Create our model
var _m = new MapModel(angular.extend(opts, {
      container: element[0],
      center: new google.maps.LatLng(scope.center.latitude, scope.center.longitude),
      draggable: attrs.draggable == "true",
      zoom: scope.zoom
}));

The scope is initialized with default zoom and lat/long we provide. However, we update the latlong on getting the values from mobagg call. The $watch listeners then update the map with the latest center latlong values.

The address resolution in google maps is an amazing functionality and can provide you with an almost accurate address.

(new google.maps.Geocoder()).geocode({latLng: latLng}, function(resp) {
        if (resp[0]) {
              var bits = [];
              for (var i = 0, I = resp[0].address_components.length; i < I; ++i) {
                    var component = resp[0].address_components[i];
                    if ($scope.contains(component.types, 'political')) {
                        bits.push(component.long_name);
                    }
                }
                $scope.places = bits;
                $scope.address = resp[0].formatted_address;
                $scope.$digest();
        }
});

Google Charts

Google charts are provided by angular-google-chart plugin available in the internet. Configuring the chart is a slightly complicated process as the data passed to charts API on each ngRepeat directive. Hence, we have manually update few attributes for each chart data. Hence, we have a separate ChartDataController which populates one.

<div ng-controller="ChartDataController" ng-show="isHourlyDataAvailable">
    <div google-chart chart="chart" style="{{chart.cssStyle}}"/>
</div>

Weather Alerts

The alerts are shown on the top bar on the block level and uses bootstrap’s alert functionality.

<div ng-controller="AlertController">
    <alert ng-repeat="alert in alerts" type="alert.type" close="closeAlert($index)">{{alert.msg}}</alert>
</div>
....
angular.module('wxforecast').controller('AlertController', function($scope, $http) {
    $scope.alerts = [];

    $scope.$watch('weatherAlerts', function(newValue, oldValue) {
        $scope.alerts = [];
        angular.forEach($scope.weatherAlerts, function(weatherAlert) {
            $scope.alerts.push({'type' : (weatherAlert.severity == 1 ? 'error' : 'warning'), 'msg' : weatherAlert.description, 'closeable' : false});
        });
    });

    $scope.closeAlert = function(index) {
        $scope.alerts.splice(index, 1);
    };
});

Conclusion

So, we have just seen how to integrate several APIs in an Angular page and still makes it more responsive and end up with a cleaner code. We have also seen few code examples on how to do certain things like creating Angular alerts bar, google chart, etc along the way.

Improving Icons for Site Performance

December 20, 2012

Once upon a time (before 2007 and Steve Souders’ rise to fame), Designers crafted unique icon sets for the Web and published them as individual, stand-alone images. Developers then added the individual images to Web pages with one icon per IMG tag. As the number of unique icons on a page increased, the page got slower. It both took longer to download the assets on the page and longer to render and display the whole page. Part of the reason for the slowness was due to browsers not being able to simultaneously request more than 2-4 items from a single hostname, and part of the reason was slower Internet connection speeds. It didn’t help that most sites were not yet using CDNs to bring the content closer to the end user, and didn’t yet cache the images properly or effectively.

The first step taken by sites to address the issue was to create single images containing all of the icons, rather than one image per icon. By using CSS to display only a slice of the image containing the appropriate icon in the background of an HTML element, it meant that only one image needed to be downloaded no matter how many unique icons appeared on the page. A single image containing many constituent sub-images was termed a “sprite,” after the video game programming technique of placing many views of a character into a single graphic file. Since all commonly-used browsers now supported CSS positionable backgrounds, this technique was viable. However, there were some serious drawbacks:

  • icons could not be moved around in the image (or re-designed if the size changed) without having to modify CSS as well
  • large buffers needed to be placed around images depending on how much of it needed to be displayed
  • transparent icons were initially limited to 8-bit GIF images because some browsers did not support 24-bit PNGs with alpha transparency (remember the ugly IE 6 Web Behaviors hack with memory leaks?)
  • different variant sprites were needed if the icons were a different color or size
  • sprites tended to get really wide as the height was sometimes fixed, so new icons had to be added to the end of the image

Still, there were big performance gains to be had when used properly, and this was the only solution available at the time. We used this technique sporadically beginning in 2007, and everywhere as a matter of policy by 2010.

The next promising technology to be supported by most browsers was inline (or embedded) images. This technique serializes binary resources like images into a base64-encoded string, which can then be referenced directly in place of a URL. The string generated by this technique can be used both in CSS, and directly in an image tag:

<img width="109" height="184" src="data:image/png;base64,iVBORw0KGgoAAA...">

By using this technique, we gained the advantages of sprites with fewer server requests.
On the other hand, inline images are more difficult to manage and maintain, and if not done right, the size of CSS files can grow dramatically. Ideally, image references in CSS are resolved and converted to inline images at build time. We used this technique in 2011 when we built our new mobile web site targeted to smartphones, since opening new connections is so expensive (in terms of network latency) over cell radios and mobile browsers download fewer simultaneous resources in parallel.

One other fascinating approach we’ve studied is inline vector graphics via SVG in IMG tags. By using vector-based graphics, we could scale imagery like icons to almost any size without loss in quality. The drawbacks are that the colors couldn’t be altered by CSS, and that older IE browsers don’t support it. In fact, even polyfills would require us to maintain two separate sets of imagery.

What are the alternatives to using sprites? One technology we’ve found is Icon Fonts. This is a cross-browser solution supported by all of the major browsers on all platforms. It involves creating a font file in several different binary formats that contains the icons you need to use. Each browser only loads the font file in the format that it supports. This gives us the ability to scale the icons since fonts are scalable. It also allows us to style the fonts using standard CSS.

So, how does the file size of Icon Fonts compare to a similar sprite-based solution? Consider the icons in the following image from our recently-launched Video Player page.
sample icon font output
There are nine icons being used. We have two sizes, one for desktop users and a larger one for tablet users. We also have two states for each icon, normal and hover. This requires us to place 36 separate icon images into a sprite. By keeping things a single color and compressing the image, we can get the file size down to around 8kb. The largest icon font file size is the WOFF format at 3Kb.

To use icon fonts instead of a sprite, you still need to edit both the page (to apply the right class), and the CSS file (to create the selector rules). However, you no longer need to calculate positioning since the icon is now mapped to a character instead of x and y coordinates. It is easy to add new icons to the font since you just map the new icon to a new character. You are not likely to ever need to change the character an icon is mapped to.

To implement icon fonts, we followed the directions outlined at IcoMoon. It allows us to upload our own icons or use those already provided. It maps the icons to a special section of Unicode that is not used by any existing alphabet, so there is no chance that the icon would accidentally show up naturally in any translation of our site. For accessibility, the icons also have a corresponding label that explains the purpose of the icon’s hyperlink. One additional word of caution here: don’t forget to modify your Web server configuration to serve the font files with the proper MIME types, and to add the Access-Control-Allow-Origin header to enable cross domain font file requests.

Using icon fonts for a scalable, customizable solution seems to meet all of our needs.
It works on all supported browsers. It can be easily edited without changing existing markup or CSS. The only issues that keep this from being a perfect solution are that it requires four different files to be edited and published, and that it doesn’t allow icons of more than one color. Hopefully SVG will mature enough to be thoroughly supported, or some other solution will come along like true vector-based images.

Backbone and Dust, Part 2

October 31, 2012

In my last post, I described how weather.com could use Backbone and Dust to deliver weather data from Web Services to the page via client-side technologies. In addition to decreasing the processing load on the server, it would also allow developers to achieve a cleaner separation between the weather data (Model), the presentation of that data (View), and the plumbing needed to retrieve and display the data (Controller). Please note that this is entirely separate from using Semantic HTML5 as a Model, CSS3 as a View and JS as a Controller.

This time, I’d like to take the existing code one step further by creating a Model and View for weather forecast data. Since the data is available for up to ten days out, I’ll build the model to accept a parameter limiting the number of days we’ll use. In our example, I’ll use three days. I’ll also be adding several methods to the Model that will allow us to control how we present Date and Time information regarding the data feed and the days being displayed. In the interests of keeping this example simple, I’ll leave out unit conversions for temperature and wind speed. By tackling multiple days, I’ll also be able to show you how Dust templating allows you to:

  • Iterate over an Array
  • Conditionally display different data
  • Show alternate markup if the Array is empty

Starting with example 2 from the previous blog post, create a Dust template for the forecast module.

<script type="text/template" id="weatherForecastModuleTemplate">
	<h2>{num} Day Forecast</h2>  
	<p>Valid Until: {timestamp}</p>
</script>

In our Model, we’ll be adding an Array named dayData containing the forecast for each day, beginning with today. Add a Dust section for that array, and create an else condition in case the Array is empty.

<script type="text/template" id="weatherForecastModuleTemplate">
	<h2>{num} Day Forecast</h2>  
	<p>Valid Until: {timestamp}</p>
{#dayData}
	<div class="wx-daypart">
	</div> 		
{:else}
	<p>No data found</p>
{/dayData}
</script>

In the daily forecast, the icon code, daily high temperature, phrase, chance of rain, and wind speed and direction all expire at 2PM or thereabouts. Since we don’t want the data to disappear after that time, we need to add conditional logic to use the nighttime values if the daytime values are no longer present. This requires using Dust logic mechanisms.

<script type="text/template" id="weatherForecastModuleTemplate">
	<h2>{num} Day Forecast</h2>  
	<p>Valid Until: {timestamp}</p>
{#dayData}
	<div class="wx-daypart">
		<h3>{dayName} {shortDate}</h3>
		<div>
			<img src="http://s.imwx.com/img/wxicon/100/{?day}{day.icon}{:else}{night.icon}{/day}.png" alt="{?day}{day.bluntPhrase}{:else}{night.bluntPhrase}{/day}" width="70" height="70">
			<p>Hi: {maxTemp}<sup>&deg;{tempUnits}</sup></p>
			<p>Lo: {minTemp}<sup>&deg;{tempUnits}</sup></p>
			<p>{?day}{day.phrase}{:else}{night.phrase}{/day}</p>
		</div> 
		<div>
			<dl>
				<dt>Chance of rain:</dt>
				<dd>{?day}{day.pop}{:else}{night.pop}{/day}%</dd>
			</dl>
			<dl>
				<dt>Wind:</dt>
				<dd>{?day}{day.wDirText}{:else}{night.wDirText}{/day} at {?day}{day.wSpeed}{:else}{night.wSpeed}{/day} {speedUnits}</dd>
			</dl>
		</div> 
	</div> 		
{:else}
	<p>No data found</p>
{/dayData}
</script>

Using the existing observation Model and View as the pattern to follow, we need to create a similar Model and View for the forecast data. In this case, we’ll add helper methods to the Model for Date manipulation, and override the parse function.

	/***
	 *** N Day Forecast
	 ***/
	var fcstModel = Backbone.Model.extend({
		initialize: function(){
			// Do nothing
        },
		// Apply the presentation rules to the data before populating the Model.
		parse: function(data,xhr){
			// If we stored the user's unit preference, 
			//     it would be taken into account here through conditional logic
			var _tempUnits = 'F';
			var _speedUnits = 'mph';
			var _defDate = data[0].validDate;
			// Iterate through the data and add shortDate and dayName
			for (var a=0,b=data.length;a<b;a++){
				data[a].shortDate = this.shortDate(data[a].validDate);
				if (a == 0){
					data[a].dayName = "Today";
				} else {
					data[a].dayName = this.dayName(data[a].validDate);
				}
			}
			// enhance the data with the units, location, timestamp and number of days
			return {
				'tempUnits': _tempUnits,
				'speedUnits': _speedUnits,
				'locid': this.get('locid'),
				'timestamp': this.regDate(_defDate),
				'num': this.get('num'),
				'dayData': data.slice(0,this.get('num'))
			};
		},
		// You gotta start somewhere
        defaults: {
        	locid: 00000,
        	num: 10
        },
        // utility method
        prependZero: function(num){
        	return (num < 10) ? '0' + num : num;
        },
        // regular date used for the timestamp
		regDate: function(str){
			var dateNum = parseInt(str,10);
			var dateSecs = dateNum * 1000;
			var dateDate = new Date(dateSecs);
			return this.dayName(str) + ' ' + this.shortDate(str) + ' ' + this.prependZero(dateDate.getHours()) + ':' + this.prependZero(dateDate.getMinutes()) + ' ET' ;
		},
		// short presentation of the date used for each day
		shortDate: function(str){
			var dateNum = parseInt(str,10);
			var dateSecs = dateNum * 1000;
			var dateDate = new Date(dateSecs);
			return this.MOY(dateDate.getMonth()) + ' ' + dateDate.getDate();
		},
		// short day name used for each day
		dayName: function(str){
			var dateNum = parseInt(str,10);
			var dateSecs = dateNum * 1000;
			var dateDate = new Date(dateSecs);
			return this.DOW(dateDate.getDay());
		},
		// month lookup
		MOY: function(monthNum){
			return ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'][monthNum];
		},
		// day of the week lookup
		DOW: function(dayNum){
			return ['Sun','Mon','Tue','Wed','Thu','Fri','Sat'][dayNum];
		},
		// services call URL
		url: function(){
			return 'http://wxdata.weather.com/wxdata/df/' + this.get('locid') + '.js?cb=?&key=e88ca396-a740-102c-bafd-001321203584&day=all';
		}
	});
	var fcstView = Backbone.View.extend({
		// The forecast view is almost exactly the same as the observation view initialization 
		//     except it uses a different compiled template.
		//    If you compiled the template outside the view and passed it in,
		//    you could use the same view for both modules.
		initialize: function(){
			var _self = this;
			this.myCompiledTemplateKey = 'fcstTempl';
			dust.loadSource(dust.compile($(this.options.tmpl).html(),this.myCompiledTemplateKey));
			this.model.on('change',function(){
				if (_self.model.get('locid') !== '00000'){
					_self.render();
				}
			},this.model);
			this.model.fetch();
		},
		render: function(data){
			// convert the Model data to JSON for templating
			var data = this.model.toJSON();
			// pass Dust the cache key for the compiled template, 
			//     combine with JSON, 
			//     and add a callback that gets passed the rendered HTML fragment as 'out'.
			var _self = this;
			dust.render(this.myCompiledTemplateKey, data, function(err, out) {
				// place the HTML into the target DIV
				_self.$el.html(out);
			});
			// return the instance to support chaining
			return this;
		}
	});
	var myFcstModel = new fcstModel({
		locid: 90210,
		num: 3
	});
	var myFcstView = new fcstView({
		el: '#weatherForecastModule',
		tmpl: '#weatherForecastModuleTemplate',
		model: myFcstModel
	});

Last, make sure you’ve added the weatherForecastModule DIV container to the page as the View expects.

<div id="weatherForecastModule"></div>

View the forecast example page.

REST or back to the future

October 16, 2012

Load’s up, servers go down, too much traffic, out of memory, uptime shows 80, 90, 100 … if that sounds familiar you are not alone. It doesn’t matter how you build your application, what language you use or even how many instances you run – if you get slashdotted, or you are weather.com during a hurricane, sooner or later you will run out of resources. By the virtue of being central your resources are finite. Your users might be finite in number too but now each one of them have a PC, a tablet and a smart phone and the click numbers are very close to infinity …

So what can we do? Rewrite the application? Change the language/platform? Add more servers? You did all of these 3-4 times already and … the application ended in the same place after that 153rd use case, the new language turned out to have new and more complicated bugs, and the new servers are not enough … again. Sounds like trying to explain gravity with Newtonian physics – ain’t gona happen (doesn’t mean that there’s a short supply of people trying it).

Well, let’s forget the current state and see what we are actually trying to solve. Sounds like what Einstein did, so it already feels comfortably right. We have finite resources that–no matter how efficient we use them–won’t be able to serve an infinite number of requests. The only number that will match infinity is well – infinity, kind of obvious but bear with me, in our little equation we have finite number of servers vs infinite number of requests. This seems unsolvable unless … we make the infinity do some of the work and thus balance our equation. No, I am not thinking of putting servers on the clients although this was done too. Let’s looks at the current clients, shall we?

Right now the phone in my pocket is 100s of times more powerful than my first computer. Right now I am writing this post in a HTML page that has 100s more features than the first text editor I used. Right now my network is 100s of times faster than my first storage (floppies anyone?). Hmm … so clients are faster (anyone on a single core?), smarter (thanks Chrome and FF), and well connected. Clients are not dumb, read-only displays anymore (sorry, Mosaic), right now we have this almost infinite number of very capable, well connected clients that we still treat as dumb displays and trying to do everything for them on the server side, where we are struggling to squeeze yet another bit – doesn’t feel right, does it?

Get rest, chill out and conquer the Universe!

Pure REST is great but it does have two major drawbacks – not enough separation between presentation and data (V&M from MVC) and it’s not quite index-able by the almighty search bot. If we forget those two for a moment though we do have an answer to our load/service or finite vs infinity dilemma - serve awesomely CND-able data behind clean URLs in a very gz-able text formats and let the infinite army of clients do 75% of all the work, our servers can chill – more clients doesn’t mean more work, just more traffic but that’s why we have varnish and CDNs. ;)

So REST and AJAX are awesome for offloading our finite resources on the back-end but no matter how you do your client-side page assembly it’s not going to be index-able and soon or later (around 153rd UC) your presentation will be so data-dependent that your MVC will be only VC (no pun intended). Hmm … we solved our gravity issue but we got that speed of light issue now – let’s modify few things, break REST canon a bit and unify space-time, ready?

What if we serve fully-formed html pages, CSS and JavaScript controller logic from a simple HTTP server (nginx, Apache, G-WAN. etc. etc.)? What if we load the data (json, xml, txt coming from an real application server) into the HTML pages on the client? Our URLs are still REST-ful, our clients still do most of the work, we still have infinity vs infinity but now we have clean separation between M and V and our fully formatted HTML is indistinguishable from the old, expensive, sevrer-side rendered one.

So here you have it in few sentences:

If we forget about server-side rendering and split the load of between client and server we can scale almost infinitely. If we serve fully rendered html pages (without the dynamic data) and make the client fill in the data we get the best of both worlds – scalability, index-ability and clean MVC.  Now, how to convince everyone to go back to 1995?

Next: CMS, dynamic pages, UC153 or where that json, xml came from …

Backbone, Underscore and Dust

October 12, 2012

Introduction

Lately I’ve been reading about how various Agile/Lean teams have been refactoring their sites. The goal was to implement MVC/MVVM using a framework such as Backbone or Spine. For templating, they use Underscore (bundled with Backbone) or Dust. That got me to wondering what a similar implementation would look like for weather.com. In this post I’ll describe what it takes to implement a simple weather observation module using Backbone and Dust.

To get your page prepared for using Backbone, you will want to load scripts for jQuery, Underscore, and Backbone in that order inside your HEAD tag:

<script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
<script src="http://underscorejs.org/underscore-min.js"></script>
<script src="http://backbonejs.org/backbone-min.js"></script>

Please note that in the example above I pointed to the original locations for those scripts. In a final product, you’d want to download those scripts and host them locally.

Next, create the HTML container that you’ll want to place the data in, and the template to do the placement:

<script type="text/template" id="weatherObservationModuleTemplate">
	<h1>Weather for <%= presName %></h1>
	<div><img src="http://s.imwx.com/img/wxicon/100/<%= Observation.wxIcon %>.png" alt="<%= Observation.text %>" width="100" height="100" id="myIcon"></div>
	<p><%= Observation.temp %>&deg;F and <%= Observation.text %></p>
</script>
<div id="weatherObservationModule"></div>

In the example above, there are several interesting things taking place. I placed the template inside a script tag in order to ensure that the browser does not render the text. I changed the script type to ‘text/template’ to further ensure that the browser does not try to parse the contents as a script. I then gave it an ID attribute so I could retrieve its contents. I then placed this template immediately above the target container so that they’re in close proximity. There is no technical reason they would need to be placed close together, but it is an intentional choice on my part to make it easier for anyone to maintain the module by keeping related functionality in a single location. The template contents merely use Underscore templating syntax to place variable data verbatim. Notice how it is visually similar to JSP syntax.

Now you’re ready to use Backbone to load the weather data and parse the template. Backbone has a Model class that we’ll extend to create a weatherModel, and a View class that we’ll use to create a weatherView. We’ll then create instances of both to kick off the process. First, instatiate the Model, then instantiate the view so you can pass it a reference to the Model instance. In the View instance’s initialization method, create a listener for the change event, which should fire when the Model instance’s data is populated, and then tell the View instance to render upon successful retrieval. Then tell the model to go fetch the data.

<script>
(function($){
	var weatherModel = Backbone.Model.extend({
		// Overriding one of the Model's built-in methods.
		// The initialize method is called automatically when the Model is instantiated.
		// The fetch method tells the Model to use jQuery to make a JSONP call 
		//     using the url method below.
		// Notice that the fetch success callback 
		//     is invoking the weatherView instance directly.
		initialize: function(){
			// Do nothing
        },
        // The default don't have to be set in this case, 
        //     but it helps indicate what the important properties are, and
        //     if the defaults are displayed, you know something went wrong.
		defaults: {
			'locid': '00000'
		},
		// This is a custom presentation logic method that creates the location name.
		presName: function(data){
			var _l = data.Location;
			return _l.name + ', ' + ((_l.state === '*') ? _l.countryCode : _l.state);
		},
		// Apply the presentation rules to the data before populating the Model.
		parse: function(data,xhr){
			data.presName = this.presName(data);
			return data;
		},
		// This is the URL that we'll use to retrieve the data
		url: function(){
			return 'http://wxdata.weather.com/wxdata/agg/' + this.get('locid') + '.js?cb=?&key=e88ca396-a740-102c-bafd-001321203584';
		}
	});
	var weatherView = Backbone.View.extend({
		initialize: function(){
			var _self = this;
			this.model.on('change',function(){
				if (_self.model.get('locid') !== '00000'){
					_self.render();
				}
			},this.model);
			this.model.fetch();
		},
		render: function(){
			// convert the Model data to JSON for templating
			var data = this.model.toJSON();
			// grab the innerHTML of the template, combine with JSON, and turn into HTML.
			var template = _.template($(this.options.tmpl).html(),data);
			// place the HTML into the target DIV
			this.$el.html(template);
			// return the instance to support chaining
			return this;
		}
	});
	var myWeatherModel = new weatherModel({
		locid: '90210'
	});
	var myWeatherView = new weatherView({
		el: '#weatherObservationModule',
		model: myWeatherModel,
		tmpl: '#weatherObservationModuleTemplate'
	});
})(jQuery);
</script>

View the complete working example.

Dust is a templating language that gives the developer more functionality and the ability to combine templates externally and to precompile them into JS functions for major speed gains. To convert our example from Underscore to Dust requires four small changes. One, add a reference to the Dust library immediately after Backbone. Two, modify the template to replace the JSP-style Underscore variable insertion instructions with curly-brace Dust variable insertions. Three, compile the Dust template in the (previously empty) weatherView initialize function. Four, replace the Underscore templating line and DOM placement of the rendered HTML with a call to the compiled Dust template while passing it a callback to render the result to the DOM.

<script src="dust-full-1.0.0.min.js"></script>
...
<script type="text/template" id="weatherObservationModuleTemplate">
	<h1>Weather for {presName}</h1>
	<div><img src="http://s.imwx.com/img/wxicon/100/{Observation.wxIcon}.png" alt="{Observation.text}" width="100" height="100" id="myIcon"></div>
	<p>{Observation.temp}&deg;F and {Observation.text}</p>
</script>
...
	var weatherView = Backbone.View.extend({
		// read the template from the DOM and compile to a JS function when initializing the view
		initialize: function(){
			var _self = this;
			this.myCompiledTemplateKey = 'weatherTmpl';
			dust.loadSource(dust.compile($(this.options.tmpl).html(),this.myCompiledTemplateKey));
			this.model.on('change',function(){
				if (_self.model.get('locid') !== '00000'){
					_self.render();
				}
			},this.model);
			this.model.fetch();
		},
		render: function(){
			// convert the Model data to JSON for templating
			var data = this.model.toJSON();
			// pass Dust the cache key for the compiled template, 
			//     combine with JSON, 
			//     and add a callback that gets passed the rendered HTML fragment as 'out'.
			var _self = this;
			dust.render(this.myCompiledTemplateKey, data, function(err, out) {
				// place the HTML into the target DIV
				_self.$el.html(out);
			});
			// return the instance to support chaining
			return this;
		}
	});

View the second example.

Follow

Get every new post delivered to your Inbox.