Monday, April 15, 2013

Using Force.com REST API's with Google Dart

INTRODUCTION


Salesforce's Force.com platform is an incredibly powerful solution for building enterprise web applications. However, using Force.com's native VisualForce pages is not aways an attractive solution for creating full-featured interfaces since you are required to host those pages within the Force.com platform.

Fortunately, nearly all of Force.com's capabilities are exposed through their extensive API, including custom objects and apex-code that is annotated as a web service. In this post, I will demonstrate how to use the Force.com RESTful APIs to build a simple web application using Google's new Dart language. 

Dart represents the next generation in building enterprise calibre web applications. While similar in syntax to Java or C#, it still contains many of the most powerful features found in Javascript, such as its asynchronous programming model and treatment of functions as first-class objects. In many respects, it blends the most powerful features of Java and Javascript, while sprinkling in sugar such as integrated unit testing support; DOM-based query and manipulation features like jQuery provides; and, interestingly, can be written for both client and server-side applications (not unlike Javascript and node.js). 

For those of you unfamiliar with Dart, it uses a similar model to Google's GWT. That is, the code you write for client-side browser applications is compiled into Javascript for run-time deployment.  Dart's compilation process deals with the complexity of creating the proper native Javascript for each of the major browsers.  In addition, Google provides a very nice Eclipse-based IDE for development, and a special version of Chrome called Dartium that enables Dart code to be run natively without the Javascript compilation process being required (very handy for rapid development). Because Dart is compiled into Javascript for deployment, it optimizes the Javascript code produced, so that only the exact code needed to run the application is actually getting generated (called "tree shaking" by the Dart folks).

NOTE: The code for this article can be found at: https://github.com/dajevu/sfdc-dart

The application we'll be creating is really pretty simple -- it's a single webpage with tabs used for listing Salesforce Accounts and User objects.  Here's an example of what it looks like:


Since it's currently not possible to invoke the Salesforce API's directly within a web page unless hosting the page through the Force.com infrastructure (due to cross-domain Javascript restrictions), a server component (also written in Dart) is used to proxy the client requests to Salesforce. The graphic below is a high-level overview of the solution:


This sample project includes both the Dart server and client components.

PREREQUISITES


In order to run this application, you'll need at least a Force.com developers account (which is free),  and Dart version 0.4.3.5 or greater (it may work on earlier versions as well, but I didn't test it). You can download the Dart SDK, IDE and Dartium at Dart's website.

SETUP


For security reasons, your instance of Salesforce will need to be setup to allow outside RESTful API calls to be placed. The steps required are outlined in detail at http://www.salesforce.com/us/developer/docs/api_rest/. However, if you are too impatient to follow that explanation (and many of the authentication steps described are only applicable for those using OAuth, which we're not using in this example), what you must do is configure your Salesforce instance for remote access. This is done via the Setup->Develop->Remote Access configuration page, shown below:

After entering all of the required values, you will be provided a consumer key and consumer password, as shown above. When using the force.com API, you can authenticate using either OAuth or via the token/session id. We're using the later in this example.

LOGGING INTO SALESFORCE USING REST


Before we jump into the Dart code and samples, let's first examine how to use the Salesforce's API from the command line. Before we begin placing calls to access the the remote objects, a session or token must be first acquired. Below illustrates how this is done using curl from the command line (broken up into multiple lines for purposes of explanation):

curl -d "grant_type=password&
   client_id=3[consumer-key]&
   client_secret=[consumer-secret]&
   username=[sfdc-login-email-address]&
   password=[sfdc-login-password]"   

In your case, you'll be replacing the values found within the [ ] with those appropriate to your environment. The [consumer-key] and [consumer-secret] are those values captured from the previous screenshot/form.  The other two values represent the regular salesforce.com username and password used to login to salesforce, and similarly for the login URL (notice I'm using a sandbox instance of salesforce, hence it begins with test.).

After invoking the command, you should see a JSON response that resembles:

{
  "id" : "https://test.salesforce.com/id/00DT6EAM/005i000AA",
  "issued_at" : "1365989902442",
  "instance_url" : "https://cs15.salesforce.com",
  "signature" : "GVY9jxn/2Y+gAfewy6rq+l2DAqRza5A8U4g=",
  "access_token" : "[not shown]"
}

The two values that are most relavent are the instance_url, which is the URL location where subsequent calls must be placed, and the access_token, which represent the session id that is also used on subsequent calls (the access_token will remain active for only a few hours after which a new one must be acquired). 

Now that we have the required login credentials, we can place some API calls to retrieve back data from the Salesforce instance. For example, let's place a call to retrieve back some data from the Account object (NOTE: use single quotes around authorization header, as shown below following -H flag). 

curl [instance_url]/services/data/v27.0/query?q=SELECT+id,name,type,description,industry+FROM+account+ORDER+BY+name -H 'Authorization: Bearer [access_token]'

In this example, a GET request is being issued that runs a SQL-type select statement against the Account object in Salesforce. In this case, I'm bringing back the type, description and industry properties from the Account table and ordering the results by the name property. Here's an example of the JSON that is returned:

{
  "totalSize" : 2,
  "done" : true,
  "records" : [
    {
      "attributes" : {
        "type" : "Account",
        "url" : "/services/data/v27.0/sobjects/Account/001e00000GAMYAA4"
      },
      "Id" : "001e0000006GAMYAA4",
      "Name" : "Acme Child Corporation",
      "Type" : "Customer",
      "Description" : "Acme Child is a child of Acme Parent.",
      "Industry" : "Consulting"
    },
    {
      "attributes" : {
        "type" : "Account",
        "url" : "/services/data/v27.0/sobjects/Account/001e0005prfhAAA"
      },
      "Id" : "001e0000005prfhAAA",
      "Name" : "Acme Corporation Parent",
      "Type" : "Customer",
      "Description" : "Description",
      "Industry" : "Insurance"
    }
  ]
}

Now that you have a flavor of how the Salesforce API works, let's dig into Dart.

SERVER-SIDE DART


While you maybe getting pretty excited about the prospect of using the Salesforce API directly from within Javascript on the client, this unfortunately is not possible due to the restrictions found in Javascript when calling AJAX calls from a remote server (cross-domain scripting). Of course, you can build and deploy pages directly within the Force.com infrastructure (i.e., VisualForce pages), but this requires hosting your pages in their instance. While there are some work-arounds (CORS), they remain a bit kludgy (although the Dart sample code for this project illustrates supporting CORS in the Dart server that is used). 

To get around this issue for our example, we'll use server-side Dart as a proxy type service to remotely call the Salesforce API as well as serve up our web application. Using Dart on the server-side is very attractive, as it means that a single language can be used to both develop the backend and front-end portions of our application. This no doubt helps explain why node.js is becoming so popular -- it supports the ability to program server-side applications using Javascript.

While I really don't want to personally argue over the merits of using Javascript, I personally find the language less than ideal, especially coming from a Java background. Understanding how prototypes work as a replacement to classes/inheritance remains somewhat murky for me as, and its heavy use of asynchronous callbacks can create some rather unwieldy code. Dart, in my opinion, marries the best features of Javascript with those of Java. In particular, Dart likewise heavily relies on an asynchronous model, but it deals with its complexity by using a concept known as Futures (granted, there are some similar implements with Javascript, but it's not native). 

Let's examine how we can use Dart to proxy service calls to Salesforce. The first thing we must do is setup a Dart program that will act as a web server. This is accomplished using the Dart:io library. Since we also want to use the server to serve-up web pages and their artifacts (css files, images etc), we need to first instruct the server where to locate these files (setting up a Dart server is a pretty low-level exercise, as you'll see here -- there are some external packages that simplify these steps). This identification is done within the main() method of the server code found in the server.dart file (like Java, a Dart program is invoked through a main method):

main() {
  // Compute base path for the request based on the 
  // location of the script and then start the server.

  File script = new File(new Options().script);
  script.directory().then((Directory d) {
    startServer(d.path.substring(0
    d.path.lastIndexOf('bin')) + 'web/');
  });
}

To be honest, I'm not 100% sure how the Options().script method identifies the local path/directory where the server is running from, but I didn't concern myself overly with such details, as this is a common use pattern.  In this code, we witness our first use of a callback, which can be easily identified as in use via the shorthand then method. In this example, the code states that after the script method is invoked, we're provided a Directory object that we can use to acquire the path location where the server is running. In this case, the server code is running in the bin subdirectory, but I've split apart my web application artifacts into another directory at the same level as bin called web, which I'm simply replacing for bin via the substring method.

Now that I have the location where my web artifacts reside (web), I can then run the method that will actually start the server. Appropriately enough, it's in the startServer method, to which I'm passing the directory path we acquired.  Let's take a look at the startServer method:

startServer(String basePath) {

  HttpServer.bind('127.0.0.1', 9090).then((server) {
    
    print('Server started, Port 9090');
    print('Basepath for content is: ' + basePath);
    
    server.listen((HttpRequest request) {
      print("Received " + request.method + " request, 
             url is: " + request.uri.path);
      
      switch (request.method) {
        case "GET"
          if (request.uri.path.contains('api'))
            handleApi(request, basePath); 
          else
            handleGet(request, basePath);
          break;
        case "POST"
          //handlePost(request); not yet implemented
          break;
        case "OPTIONS"
          handleOptions(request);
          break;
      }  
    });
  });
}

As you can likely discern, the HttpServer.bind static method is used to initiate the web server under localhost port 9090. Once binded, an HttpServer object is assigned to the variable server, and in turn, it begins listening for inbound connections. Once an inbound connection is received, it is routed to other methods for processing based upon a) the inbound connection's HTTP method (i.e., GET, POST etc) and b) in the case of a GET, it is further routed based upon whether the request URL contains "api" or not. 

Before we examine the handleApi method, which is where we invoke the various Salesforce API calls, let's first explore how we can call a RESTful API using Dart's IO library (the io library, by the way, can only be used server-side with Dart, since it includes calls to the local filesystem etc).

PLACING RESTFUL CALLS WITH DART's IO LIBRARY


In this project, we're returning back a list of Accounts and Users from Salesforce.com. Eventually we'll display those lists within a web page. For each of these two object types, I created a separate class that contains both the logic required to place the API call as well some unit tests. Let's look at the Dart Account class (found in bin/classes/account.dart file):

class Account {
  
   static Future getAllAccounts(String sessionId, String host) {
     
     Completer accntResponse = new Completer();
     
     Future reply = 
        SFDCUtils.getRequest('${host}${Constants.QUERY}${Constants.QUERY_ACCOUNT_ALL}'
        sessionId);

     reply.then( (JsonObject response) {
       accntResponse.complete(response);
     });
     
     return accntResponse.future;
   }
}

The first thing to note is that the getAllAccounts method returns a Future object of type JsonObject. What this means is that this method represents an asynchronous call, and when the method is completed, it will notify the calling code accordingly. In other words, it represents a promise that a value will eventually be returned, when processing is completed. Since we are calling an external service, this is sensible approach. However, this code doesn't provide any real insight into how the Salesforce API is invoked, as this logic takes place within the SFDCUtils.getRequest static method. Let's take a look at that code:

  static Future getRequest(String url, String sessionId, 
    [String contentType = Constants.CT_JSON]) {
    
    HttpClient client = new HttpClient();
    
    Completer respBody = new Completer();

    var requestUri = new Uri.fromString(url);
    
    var conn = client.getUrl(requestUri);
    
    conn.then ((HttpClientRequest request) {
      request.headers.add(HttpHeaders.CONTENT_TYPE, contentType);
      request.headers.add('Authorization', 'Bearer ${sessionId}');
      request.close(); 
      request.response.then( (response) {
        
        IOUtil.readAsString(response, onError: null).then((body) {         
          
          respBody.complete(new JsonObject.fromJsonString(body));
        
          client.close();
        });        
      });      
    });    
    return respBody.future;
  }

This could be considered a "helper" method that can be used for any of the GET requests to Salesforce (a similar helper is provided for POST requests, but not covered here). The first thing that we do is to get an instance of Dart:IO's HttpClient. We then follow it by creating an HttpClientRequest object invoking the getUrl method on the client object. With that, we can then setup the required HTTP header used for the Salesforce authentication (we'll see in a bit how the sessionId is first acquired) as well as for specifying the content type (unless provided as an optional parameter to the method, it's set to "application/json"). Lastly, we then invoke the HTTP request asynchronously and wait until the response is received. Once received, an external library is used (IOUtil) to convert the response Stream into a String, represented by the body variable. Lastly, we close the HttpClientRequest using its close method.

So, to circle back to the Account class' getAllAccounts method, once the helper has returned the JSON results from the HTTP, it indicates processing is completed by invoking the complete method on the Future response object.

One thing I haven't discussed yet is the use of the JsonObject. When the response JSON is returned from Salesforce, it is converted to this JsonObject (note -- this functionality is provided via a third-party utility). This object represents a map of the returned JSON, and allows for convenient dot-notation access to the JSON values. We can see this at play first-hand in the accompanying unit test for Account, which we'll discuss next.

UNIT TESTING WITH DART


One of the nicest features of Dart is its built-in support for unit testing (for a more detailed understanding, see this article). The account.dart file, in addition to defining the class and methods for the Account, also contains a main() method that can be invoked to run one or more unit tests.  Below is a unit test to exercise the getAllAccounts method:

  test('Account', () {
    Future url = Credential.getCredentials();
    
    url.then((JsonObject obj) {
      
      String tokenId = obj.access_token;
      String host = obj.instance_url;
      
      Future accnts = Account.getAllAccounts(tokenId, host);
      
      accnts.then((JsonObject jsonObj) {
        expect(jsonObj.totalSize > 0, true);
      });
    });
  }); 

As you can see, the first thing we do is to use the Credential class' getCredentials static method to return the credentials used to login to Salesforce (the Credential class is very similar to Account, so I don't bother covering it here). Once the instance_url and access_token are returned by the getCredentials call, we use those as parameters to invoke the getAllAccounts method. As you may recall, the JSON returned from the Salesforce API contains a totalSize property that identifies how many records are returned. We use Dart's expect method in a similar fashion to jUnit's assert to determine whether the test was successful or not. When running in the Dart IDE, you'll see something like this in the console window indicating the test successfully passed.

unittest-suite-wait-for-done
request url is: https://test.salesforce.com//services/oauth2/token
PASS: Account

All 1 tests passed.
unittest-suite-success

Now that we've seen how the Salesforce API is invoked and the corresponding data returned, let's tie that into the server to see how the proxy functionality works. 

IMPLEMENTING PROXY FUNCTIONALITY IN DART SERVER


If you recall, when a GET HTTP request is issued to the Dart server (found in the file server.dart), if the URL of that request contains "api", the handleApi method is invoked. Let's now examine this method to see how the proxy functionality is implemented:

handleApi (HttpRequest request, String basePath) {
  
  HttpResponse res = request.response;
  addCorsHeaders(request, res);
  
  _getSessionId().then((Map credentials) {
    sfdcCredentials = credentials;
    
    switch (request.uri.path) {

      case '/api/accounts':
          Future accnts = 
             Account.getAllAccounts(sfdcCredentials['access_token'], 
             sfdcCredentials['instance_url']);
          
          accnts.then((JsonObject jsonObj) {
            res.write(jsonObj.toString());
            res.close();
          });
          break;

      case '/api/users':          
          Future orders = 
             User.getAllUsers(sfdcCredentials['access_token'], 
             sfdcCredentials['instance_url']);
          
          orders.then((JsonObject jsonObj) {
            res.write(jsonObj.toString());
            res.close();
          });
          break;
      }
  });  
}

As you can see, it's pretty straightforward. The HttpRequest object's uri path (URL) is extracted and then used as the basis for the switch statement that follows. If, for example, the inbound GET URL contains the prefix of "/api/accounts", then a list of of Account objects is returned wrapped within a JsonObject map. The results are then returned to the calling client as a JSON string.  We follow an identical approach with the "/api/users" call.  Most of the real work was offloaded to the individual Dart classes such as Account

NOTE: CORS supported was added to the server (see the method addCorsHeaders), so cross-domain Javascript support is available. To read more on how it's implemented with Dart, see this article

Now that we've got the proxy server completed, we can dive into how the UI is developed within Dart.

CLIENT-SIDE DART


The initial screenshot provided at the start of this article depicted the UI we'll be creating. As a quick refresher, it's just a web page with two tabs -- one to display a list of Salesforce Accounts, and the other tab Users. Let's begin by looking at a snippet of the HTML used to construct the page (sfdcdemo.html):

The design for the page uses Twitter Bootstrap for the CSS layout, and the new Dart UI Widgets library, which mirrors the Javascript functionality provided in Twitter Bootstrap, for the widget support. In this example, we're using the tab widget, which is implemented by way of the custom HTML Dart tag introduced called <x-tabs>(for more information on creating custom Dart tags, see this article).  As we will see in a moment, the Dart client code interacts with the above HTML by way of the id values assigned to some of the HTML elements. For example, when the list of Accounts is populated by Dart, it simply appends the tabular data to the table identified by the element id named accounttable. In this fashion, Dart's query method works similarly to jQuery.

Let's take a look at the fragment of Dart client code that corresponds to this HTML (sfdcdemo.dart) that is used for populating the Accounts tab.
When the page is first opened, the main method in sfdcdemo.dart invokes the populateAccountsTable method shown above.  The first thing that is done is to invoke an Ajax request to the server API proxy method we described previously (obviously, in a production type environment, the host wouldn't be hard-coded as above).  Once the results of the call are returned, a JsonObject is populated with the data, and then we simply iterate through the results to dynamically construct the HTML table body. Once the HTML is constructed, we use  Dart's query mechanism to locate the target element id in the DOM, and then append the new dynamic HTML results. Couldn't be much easier!

The sfdcdemo.dart file also contains a method called populateUsersTable, and it's used to populate the User tab in the HTML. It follows an identical pattern as populateAccountsTable.

The easiest way to run this project is to a) open the project in the Dart IDE and b) right-click on the sfdcdemo.html file and select "Run in Dartium". Of course, you will first need to follow the steps outline in the pre-requisites and setup sections.

In a future posting, I hope to replace the existing HttpClient mechanism on the Dart client for placing the AJAX calls to populate the tabular data with web sockets.

Thursday, September 27, 2012

Spring Security using API Authentication

Background


While there are many blog posts that detail how to use Spring Security, I often still find it challenging to configure when a problem domain lies outside of the standard LDAP or database authentication. In this post, I'll describe some simple customizations to Spring Security that enable it to be used with a REST-based API call. Specifically, the use case is where you have an API service that will return a user object that includes a SHA-256 password hash.

Setup 

The prerequisites for running this sample is Git and Maven, and your choice of IDE (tested with both Eclipse and IntelliJ).

The source code can be found at: https://github.com/dajevu/Spring3SecurityUsingAPI. After pulling down the code, perform the following steps:
  1. In a terminal window, cd to the Shared directory located under the root where the source code resides.
  2. Issue the command mvn clean install. This will build the Shared sub-project and install the jar into your local mvn repository.
  3. Within Eclipse or IntelliJ, import the project as a Maven project. In Eclipse, this will result in 3 projects being created: Shared, SpringWebApp, and RestfulAPI. In IntelliJ, this will be represented as sub-projects. No errors should exist after the compilation process is complete.
  4. Change directory to RestfulAPI. Then, issue the command mvn jetty:run to run the API webapp.  You can then issue the following URL that will bring back a User object represented in JSON: http://localhost:9090/RestfulAPI/api/v1/user/john
  5. Open up a new terminal window, cd to SpringWebApp directory located under the project root. Issue the command mvn jetty:run. This will launch a standard Spring webapp that incorporates Spring Security. You can access the single HTML page at: http://localhost:8080/SpringWebApp/. After clicking the Login link, login with the username of john and a password of doe. You should be redirected to a Hello Admin page. 

In order to demonstrate the solution, three maven modules are used, which are illustrated below:

  • SpringWebApp. This is a typical Spring webapp that serves up a single JSP page. The contents of the page will vary depending upon whether the user is currently logged in or not. When first visiting the page, a Login link will appear, which directs them to the built-in Spring Security login form. When they attempt to login, a RESTEasy client is used to place a call to the API service (described below), which returns a JSON string that is converted into a Java object via the RESTEasy client. The details of how Spring Security is configured is discussed in the following sections.
  • RestfulAPI. An API service that serves JSON requests. It is configured using RESTEasy (a JAX-RS implementation), and is described in more detail in the next section.
  • Shared. This contains a few Java classes that are shared between the other two projects. Specifically, the User object DTO, and the RESTEasy proxy definition (it's shared because it can also be used by the RESTEasy client).

RestfulAPI Dissection


The API webapp is configured using RESTEasy's Spring implementation. The RESTEasy documentation is very thorough, so I won't go into a detailed explanation of its setup. A single API call is defined (in UserProxy in the Shared project) that returns a static JSON string. The API's proxy (or interface) is defined as follows:

For those of you familiar with JAX-RS, you'll easily follow this configuration. It defines an API URI that will respond to requests sent to the URL path of /api/v1/user/{username} where {username} is replaced with an actual username value. The implementation of this service, which simply returns a static response, is shown below:

About the only thing remotely complicated is the use of the SHA-256 hashing of the user's password. We'll see shortly how this get's interpreted by Spring Security. When the URL is accessed, the following JSON string is returned:

The webapp's web.xml contains the setup configuration to service RESTEasy requests, so if you're curious, take a look at that.

SpringWebApp Dissection


Now we can look at the Spring Security configuration. The web.xml file for the project configures it as a Spring application, and specifies the file applicationContext-security.xml as the initial Spring configuration file.  Let's take a closer look at this file, as this is where most of the magic occurs:

Let's go through each of line numbers to describe their functionality.  Lines 3 through 5 instructs Spring to look for Spring-backed classes in the com.acme directory and that Spring annotations will be supported.  Line 7 is used to load the properties specified in the application.properties file (this is used to specify the API host). Lines 9 through 11 enable Spring Security for the application. Normally, as a child element to http, you would specify which pages should be protected using roles, but to keep this example simple, that wasn't configured.

Lines 13-17 are where the customizations to base Spring Security begin. We define a custom authentication-provider called userDetailsSrv through its bean ref. That bean is implemented through the custom class com.acme.security.UserDetailsService (line 19). Let's take a closer look at this class:

As you can see, this class implements the Spring interface org.springframework.security.core.userdetails.UserDetailsService.  This requires overriding the method loadUserByUsername.  This method is responsible for retrieving the user from the authentication provider/source. The returned user (or if no matching user is found, a UsernameNotFoundException is thrown - line 28) must contain a password property in order for Spring Security to compare against what was provided in the form. In this case, as we've seen previously, the password is returned in a SHA-256 hash.

In our API implementation, the user lookup is pulled using the APIHelper class, which we'll cover next. The returned API data is then populated in the custom class called UserDetails.  This implements the Spring interface with the same name.  That interface requires an concrete implementation of the getUsername() and getPassword() methods.  Spring will invoke those in the next processing step of Security to compare those values against what was recorded in the web form.

How does Spring go about comparing the password returned in the SHA-256 against the form password value. If you look back at the XML configuration, it contained this setting:
Notice the passwordEncoder -- this reference points to the Spring class ShaPasswordEncoder. This class will compute an SHA-256 password of the password provided through the web form, and then Spring will compare that computed value against what we returned via the API.

Let's close this out by looking at the APIHelper class:

The first thing you'll on lines 8 and 9 is the injection of the API.host property. As you recall, this was set in the application.properties file. This identifies the host in which to post the API call (since it's running locally, localhost is specified).  Lines 17 through 20 use one of the RESTEasy client mechanisms to post a JSON RESTful call (RESTEasy also has what is called client proxy implementation, which is easier to use/less code, but doesn't provide as much low-level control).  The resulting response from the API is then converted from JSON into the User Java object by way Jackson in line 26. That Java object is then returned to the UserDetails service.


Summary/Wrap-up


As you can see, the actual work involved in customizing Spring Security to authenticate against an API call (or really any external service) is really rather straightforward. Only a few classes have to be implemented, but it can be tricky trying to figure this out for the first time. Hence, the reason I included the complete end-to-end example. 

Thursday, August 2, 2012

GWT 2.5 SuperDevMode and Google App Engine - Now an Amazing Combination

Prior to version 2.5 of GWT, I found it rather tedious to do development with GWT and Google App Engine. Both had separate development modes that didn't always play nice together, and often I found the code-test cycle to be very time-consuming (launching GAE's local development environment is pretty slow).

However, with the advent of GWT's new SuperDevMode (SDM), you can easily modify your Java client code and see it quickly reflected in GAE's development environment without any restart necessary. This is an addition to the other native Java debugging features introduced as a result of SDM. Below is a quick screencast of this capability:



For those of you without access to YouTube, you can download it directly here.

Look forward to your comments/suggestions!

(see my previous blog on setting up SDM in Eclipse).

Thursday, July 26, 2012

Setting-up GWT 2.5's SuperDevMode

Setting up SuperDevMode in GWT 2.5

One of the most exciting features of GWT 2.5 is what is called SuperDevMode (read about it at: https://developers.google.com/web-toolkit/articles/superdevmode). As a replacement for the previous DevMode, it allows you to debug your Java code (and Javascript) directly through Chrome's Browser's Developer Tools. Perhaps because it is still an experimental feature, instructions are a bit vague on how to set it up. Since it took me a while to figure it out, I thought I would spare you the pain by sharing the screencast below (it's a little rough, but should be sufficient to get you started).





Note: For those of you unfortunate enough to work for a company that blocks YouTube (like you can't access it via your smartphone or tablet, duh) -- you can download from this link.

In my next blog, I discuss how SDM can be used in powerful combination with Google's App Engine development environment.

Wednesday, May 30, 2012

Authenticating for Google Services, Part 2

Introduction

In the last blog entry I described how to use OAuth for access/authentication for Google's API services. Unfortunately, as I discovered a bit later, the approach I used was OAuth 1.0, which has apparently now been officially deprecated by Google in favor of version 2.0 of OAuth. Obviously, I was a bit bummed to discovered this, and promised I would create a new blog entry with instructions on how to use 2.0. The good news is that, with the 2.0 support, Google has added some additional helper classes that make things easier, especially if you are using Google App Engine, which is what I'm using for this tutorial.

The Google Developers site now has a pretty good description on how to setup OAuth 2.0. However, it still turned out to be a challenge to configure a real-life example of how it's done, so I figured I'd document what I've learned.

Tutorial Scenario

In the last tutorial, the project I created illustrated how to access a listing of a user's Google Docs files. In this tutorial, I changed things up a bit, and instead use YouTube's API to display a list of a user's favorite videos. Accessing a user's favorites does require authentication with OAuth, so this was a good test.

Getting Started

(Eclipse project for this tutorial can be found here).

The first thing you must do is follow the steps outlined in Google's official docs on using OAuth 2.0. Since I'm creating a web-app, you'll want to follow the section in those docs titled "Web Server Applications". In addition, the steps I talked about previously for setting up a Google App Engine are still relevant, so I'm going to jump right into the code and bypass these setup steps.

(NOTE: The Eclipse project can be found here -- I again elected not to use Maven in order to keep things simple for those who don't have it installed or are knowledgeable in Maven).

The application flow is very simple (assuming a first-time user):
  1. When the user accesses the webapp (assuming you are running it locally at http://localhost:8888 using the GAE developer emulator), they must first login to Google using their gmail or Google domain account.
  2. Once logged in, the user is redirected to a simple JSP page that has a link to their YouTube favorite videos.
  3. When the click on the link, a servlet will initiate the OAuth process to acquire access to their YouTube account. The first part of this process is being redirected to a Google Page that prompts them whether they want to grant the application access.
  4. Assuming the user responds affirmative,  a list of 10 favorites will be displayed with links. 
  5. If they click on the link, the video will load.
Here's the depiction of the first 3 pages flow:


And here's the last two pages (assuming that the user clicks on a given link):


While this example is specific to YouTube, the same general principles apply for accessing any of the Google-based cloud services, such as Google+, Google Drive, Docs etc. They key enabler for creating such integrations is obviously OAuth, so let's look at how that process works.

OAuth 2.0 Process Flow

Using OAuth can be a bit overwhelming for the new developer first learning the technology. The main premise behind it is to allow users to selectively identify which "private" resources they want to make accessible to an external application, such as we are developing for this tutorial. By using OAuth, the user can avoid having to share their login credentials with a 3rd party, but instead can simply grant that 3rd party access to some of their information.

To achieve this capability, the user is navigated to the source where their private data resides, in this case, YouTube. They can then either allow or reject the access request. If they allow it, the source of the private data (YouTube) then returns a single-use authorization code to the 3rd party application. Since it's rather tedious for the user to have to grant access every time access is desired, there is an additional call that can be played that will 'trade-in" their single use authorization for a longer term one. The overall flow for the web application we're developing for this tutorial can be seen below.

OAuth Flow
The first step that takes place is to determine whether the user is already logged into Google using either their gmail or Google Domain account. While not directly tied to the OAuth process, it's very convenient to enable users to login with their Google account as opposed to requiring them to sign up with your web site. That's the first callout that is made to Google. Then, once logged in, the application determines whether the user has a local account setup with OAuth permissions granted. If they are logging in for the first time, they won't.  In that case, the OAuth process is initiated.

The first step of that process is to specify to the OAuth provider, in this case Google YouTube, what "scope" of access is being requested.  Since Google has a lot of services, they have a lot of scopes. You can determine this most easily using their OAuth 2.0 sandbox

When you kickoff the OAuth process, you provide them the scope(s) you want access to, along with the OAuth client credentials that Google has provided you (these steps are actually rather generic to any provider that supports OAuth).  For our purposes, we're seeking access to the user's YouTube account, so the scope provided by Google is: https://gdata.youtube.com/.

If the end-user grants access to the resource identify by the scope, Google will then post back an authorization code to the application. This is captured in a servlet. Since the returned code is only a "single-use" code, it is exchanged for a longer running access token (and related refresh token). That step is represented above by the activity/box titled "Access & Refresh Token Requested". 

Once armed with the access token, the application can then access the users' private data by placing an API call along with the token. If everything checks out, the API will return the results.

It's not a terrible complicated process -- it just involves a few steps. Let's look at some of the specific implementation details, beginning with the servlet filter that determines whether the user has already logged into Google and/or granted OAuth access.

AuthorizationFilter

Let's take a look at the first few lines of the AuthorizationFilter (to see how it's configured as a filter, see the web.xml file).

The first few lines simply cast the generic servlet request and response to their corresponding Http equivalents -- this is necessary since we want access to the HTTP session. The next step is to determine whether a CredentialStore is present in the servlet context. As we'll see, this is used to store the user's credentials, so it's convenient to have it readily available in subsequent servlets. The guts of the matter begin when we check to see whether the user is already present in the session using:
if (session.getAttribute(Constant.AUTH_USER_ID) == null) {
If not,  we get their Google login credentials using Google's UserService class. This is a helper class available to GAE users to fetch the user's Google userid, email and nickname. Once we get this info from UserService, we store some of the user's details in the session.

At this point, we haven't done anything with OAuth, but that will change in the next series of code lines:
try {
    Utils.getActiveCredential(request, credentialStore);
} catch (NoRefreshTokenException e1) {
    // if this catch block is entered, we need to perform the oauth process
    LOGGER.info("No user found - authorization URL is: " +   

                 e1.getAuthorizationUrl());
    response.sendRedirect(e1.getAuthorizationUrl());
}

A helper class called Utils is used for most of the OAuth processing. In this case, we're calling the static method getActiveCredential().  As we will see in a moment, this method will return a NoRefreshTokenException if no OAuth credentials have been previously captured for the user. As a custom exception, it will return URL value that is used for redirecting the user to Google to seek OAuth approval.

Let's take a look at the getActiveCredential() method in more detail, as that's where much of the OAuth handling is managed.


The first thing we do is fetch the Google userId from the session (they can't get this far without it being populated).  Next, we attempt to get the user's OAuth credentials (stored in the Google class with the same name) from the CredentialStore using the Utils static method getStoredCredential(). If no credentials are found for that user, the Utils method called getAuthorizationUrl() is invoked. This method, which is shown below, is used to construct the URL that the browser is redirected to which is used to prompt the user to authorize access to their private data (the URL is served up by Google, since it will ask the user for approval).


As you can see, this method is using the class (from Google) called GoogleAuthorizationCodeRequestUrl. It constructs an HTTP call using the OAuth client credentials that is provided by Google when you sign up for using OAuth (those credentials, coincidentally, are stored in a file called client_secrets.json. Other parameters include the scope of the OAuth request and the URL that the user will be redirected back to if approval is granted by the user.  That URL is the one you specified when signing up for Google's OAuth access:


Now, if the user had already granted OAuth access, the getActiveCredential()method would instead grab the credentials from the CredentialStore.

Turning back to the URL that receives the results of the OAuth credentials, in this case, http://localhost:8888/authSub, you maybe wondering, how can Google post to that internal-only address? Well, it's the user's browser that is actually posting back the results, so localhost, in this case, resolves just fine. Let's look that the servlet called OAuth2Callback that is used to process this callback (see the web.xml for how the servlet mapping for authSub is done).


The most important take-away from this class is the line:
AuthorizationCodeResponseUrl authResponse = 
       new AuthorizationCodeResponseUrl(fullUrlBuf.toString());
The AuthorizationCodeResponseUrl class is provided as a convenience  by Google to parse the results of the OAuth request. If the getError() method of that class isn't null, that means that the user rejected the request. In the event that it is null, indicating the user approved the request, the method call getCode() is used to retrieve the one-time authorization code. This code value is placed into the user's session, and when the Utils.getActiveCredential() is invoked following the redirect to the user's target URL (via the filter), it will exchange that authorization code for a longer-term access and refresh token using the call:
credential = exchangeCode((String) request.getSession().getAttribute("code"));
The Utils.exchangeCode() method is shown next:


This method also uses a Google class called GoogleAuthorizationCodeTokenRequest that is used to call Google to exchange the one-time OAuth authorization code for the longer-duration access token.

Now that we've (finally) got our access token that is needed for the YouTube API, we're ready to display to the user 10 of their video favorites.

 

Calling the YouTube API Services

With the access token in hand, we can now proceed to display the user their list of favorites. In order to do this, a servlet called FavoritesServlet is invoked. It will call the YouTube API, parse the resulting JSON-C format into some local Java classes via Jackson, and then send the results to the JSP page for processing. Here's the servlet:


Since this post is mainly about the OAuth process, I won't go into too much detail how the API call is placed, but the most important line of code is:
feed = YouTube.fetchFavs(credential.getAccessToken());
Where feed is an instance of VideoFeed. As you can see, another helper class called YouTube is used for doing the heavy-lifting. Just to wrap things up, I'll show the fetchFavs() method.


It uses the Google class called HttpRequestFactory to construct an outbound HTTP API call to YouTube. Since we're using GAE, we're limited as to which classes we can use to place such requests.  Notice the line of code:
url.access_token = accessToken;
That's where we are using the access token that was acquired through the OAuth process.

So, while it took a fair amount of code to get the OAuth stuff working correctly, once it's in place, you are ready to rock-and-roll with calling all sorts of Google API services!



Wednesday, May 23, 2012

Authenticating for Google Services in Google App Engine


(NOTE: This tutorial uses the older, and now deprecated Oauth 1.0 approach -- See my new blog entry for an OAuth 2.0 tutorial).

Introduction


This post will illustrate how to build a simple Google App Engine (GAE) Java application that authenticates against Google as well as leverages Google's OAuth for authorizing access to Google's API services such as Google Docs.  In addition, building on some of the examples already provided by Google, it will also illustrate how to persist data using the App Engine Datastore and Objectify.

Project Source Code

The motivation behind this post is that I struggled to previously find any examples that really tied these technologies together. Yet, these technologies really represent the building-blocks for many web applications that want to leverage the vast array of Google API services. 

To keep things simple, the demo will simply allow the user to login via a Google domain; authorize access to the user's Google Docs services; and display a list of the user's Google Docs Word and Spreadsheet documents. Throughout this tutorial I do make several assumptions about the reader's expertise, such as a pretty deep familiarity with Java.

Overview of the Flow


Before we jump right into the tutorial/demo, let's take a brief look at the navigation flow. 
While it may look rather complicated, the main flow can be summarized as:
  1. User requests access to listFiles.jsp (actually any of the JSP pages can be used).
  2. A check is make to see if the user is logged into Google. If not, they are re-directed to a Google login page -- once logged in, they are returned back. 
  3. A check is then made to determine whether the user is stored in the local datastore. If not, the user is added along with the user's Google domain email address.
  4. Next, we check to see if the user has granted OAuth credentials to the Google Docs API service. If not, the OAuth authentication process is initiated. Once the OAuth credentials are granted, they are stored in the local user table (so we don't have to ask each time the user attempts to access the services).
  5. Finally, a list of Google Docs Spreadsheet or Word docs is displayed.
This same approach could be used to access other Google services, such as YouTube (you might display a list of the user's favorite videos, for example).

Environment Setup


For this tutorial, I am using the following:
  • Eclipse Indigo Service Release 2 along with the Google Plugin for Eclipse (see setup instructions).
  • Google GData Java SDK Eclipse plugin version 1.47.1 (see setup instructions). 
  • Google App Engine release 1.6.5. Some problems exist with earlier versions, so I'd recommend making sure you are using it. It should install automatically as part of the Google Plugin for Eclipse.
  • Objectify version 3.1. The required library is installed already in the project's war/WEB-INF/lib directory.
 After you have imported the project into Eclipse, your build path should resemble:


The App Engine settings should resemble:


You will need to setup your own GAE application, along with specifying your own Application ID (see the Google GAE developer docs). 

The best tutorial I've seen that describes how to use OAuth to access Google API services can be found here. One of the more confusing aspects I found was how to acquire the necessary consumer key and consumer secret values that are required when placing the OAuth request. The way I accomplished this was:
  1. Create the GAE application using the GAE Admin Console. You will need to create your own Application ID (just a name for your webapp). Once you have it, you will update your Application ID in the Eclipse App Engine settings panel that is shown above.
  2. Create a new Domain for the application. For example, since my Application ID was specified above as "tennis-coachrx", I configured the target URL path prefix as: http://tennis-coachrx.appspot.com/authSub. You will see how we configure that servlet to receive the credentials shortly. 
  3. To complete the domain registration, Google will provide you an HTML file that you can upload. Include that file the root path under the /src/war directory and upload the application to GAE. This way, when Google runs it's check, the file will be present and it will generate the necessary consumer credentials. Here's a screenshot of what the setup looks like after it is completed:

Once you have the OAuth Consumer Key and OAuth Consumer Secret, you will then replace the following values in the com.zazarie.shared.Constant file:

final static String CONSUMER_KEY = "";
final static String CONSUMER_SECRET = "";

Whew, that seemed like a lot of work! However, it's a one-time deal, and you shouldn't have to fuss with it again.

Code Walkthrough


Now that we got that OAuth configuration/setup out of the way, we can dig into the code. Let's begin by looking at the structure of the war directory, where your web assets reside:

The listFiles.jsp is the default JSP page that is displayed when your first enter the webapp. Let's now look at the web.xml file to see how this is configured, along with the servlet filter which is central to everything.

The servlet filter called AuthorizationFilter is invoked whenever a JSP file located in the html directory is requested. The filter, as we'll look at in a moment, is responsible for ensuring that the user is logged into Google, and if so, then ensures that the OAuth credentials have been granted for that user (i.e., it will kick off the OAuth credentialing process, if required).

The servlet name of Step2 represents the servlet that is invoked by Google when the OAuth credentials have been granted -- think of it as a callback. We will look at this in more detail in a bit.

Let's take a more detailed look at the AuthorizationFilter.

AuthorizationFilter Deep Dive

The doFilter method is where the work takes place in a servlet filter.  Here's the implementation:


Besides the usual housekeeping stuff, the main logic begins with the line:

AppUser appUser = LoginService.login(request, response);

As we will see in a moment, the LoginService is responsible for logging the user into Google, and also will create the user in the local BigTable datastore. By storing the user locally, we can then store the user's OAuth credentials, eliminating the need for the user to have to grant permissions every time they access a restricted/filtered page.

After LoginService has returned the user (AppUser object), we then store that user object into the session (NOTE: to enable sessions, you must set sessions-enabled in the appengine-web.xml file):

session.setAttribute(Constant.AUTH_USER, appUser);

We then check to see whether the OAuth credentials are associated with that user:

if (appUser.getCredentials() == null) {

   session.setAttribute(Constant.TARGET_URI, request.getRequestURI());

   OAuthRequestService.requestOAuth(request, response, session);
   return;
} else 
   session.setAttribute(Constant.DOC_SESSION_ID,LoginService.docServiceFactory(appUser));

If getCredentials() returns a null, the OAuth credentials have not already been assigned for the user. This means that the OAuth process needs to be kicked off. Since this involves a two-step process of posting the request to Google and then retrieving back the results via the callback (Step2 servlet mentioned above), we need to store the destination URL so that we can later redirect the user to it once the authorization process is completed. This is done by storing the URL requested into the session using the setAttribute method.

We then kick off the OAuth process by calling the OAuthRequestService.requestOAuth() method (details discussed below).

In the event that if getCredentials() returns a non-null value, this indicates that we already have the user's OAuth credentials from their local AppUser entry in the datastore, and we simply add it to the session so that we can use it later.

LoginService Deep Dive

The LoginService class has one main method called login, followed by a bunch of JPA helper methods for saving or updating the local user in the datastore. We will focus on login(), since that is where most of the business logic resides.


The first substantive thing we do is use Google UserService class to determine whether the user is logged into Google:

UserService userService = UserServiceFactory.getUserService();

User user = userService.getCurrentUser();

If the User object returned by Google's call is null, the user isn't logged into Google, and they are redirected to a login page using:

res.sendRedirect(userService.createLoginURL(URI));

If the user is logged (i.e., not null), the next thing we do is determine whether that user exists in the local datastore. This is done by looking up the user with their logged-in Google email address with appUser = findUser(userEmail). Since JPA/Objectify isn't the primary discussion point for this tutorial, I won't go into how that method works. However, the Objectify web site has some great tutorials/documentation.

If the user doesn't exist locally, the object is populated with Google's email address and created using appUser = addUser(userEmail). If the user does exist, we simply update the login timestamp for logging purposes.

OAuthRequestService Deep Dive

As you may recall from earlier, once the user is setup locally, the AuthorizationFilter will then check to see whether the OAuth credentials have been granted by the user. If not, the OAuthRequestService.requestOAuth() method is invoked. It is shown below:


To simplify working with OAuth, Google has a set of Java helper classes that we are utilizing. The first thing we need to do is setup the consumer credentials (acquiring those was discussed earlier):

GoogleOAuthParameters oauthParameters = new GoogleOAuthParameters();
oauthParameters.setOAuthConsumerKey(Constant.CONSUMER_KEY);
oauthParameters.setOAuthConsumerSecret(Constant.CONSUMER_SECRET);

Then, we set the scope of the OAuth request using:
oauthParameters.setScope(Constant.GOOGLE_RESOURCE);

Where Constant.GOOGLE_RESOURCE resolves to https://docs.google.com/feeds/. When you make an OAuth request, you specify the scope of what resources you are attempting to gain access. In this case, we are trying to access Google Docs (the GData API's for each service have the scope URL provided).  Next, we establish where we want Google to return the reply.

oauthParameters.setOAuthCallback(Constant.OATH_CALLBACK);

This value changes whether we are running locally in dev mode, or deployed to the Google App Engine. Here's how the values are defined in the the Constant interface:

// Use for running on GAE
//final static String OATH_CALLBACK = "http://tennis-coachrx.appspot.com/authSub";

// Use for local testing
final static String OATH_CALLBACK = "http://127.0.0.1:8888/authSub";

When then sign the request using Google's helper:

GoogleOAuthHelper oauthHelper = new GoogleOAuthHelper(new OAuthHmacSha1Signer());

We then generate the URL that the user will navigate to in order to authorize access to the resource. This is generated dynamically using:

String approvalPageUrl = oauthHelper.createUserAuthorizationUrl(oauthParameters);

The last step is to provide a link to the user so that they can navigate to that URL to approve the request. This is done by constructing some simple HTML that is output using res.getWriter().print().

Once the user has granted access, Google calls back to the servlet identified by the URL parameter /authSub, which corresponds to the servlet class RequestTokenCallbackServlet. We will examine this next.

RequestTokenCallbackServlet Deep Dive

The servlet uses the Google OAuth helper classes to generate the required access token and secret access token's that will be required on subsequent calls to to the Google API docs service.  Here is the doGet method that receives the call back response from Google:


The Google GoogleOAuthHelper is used to perform the housekeeping tasks required to populate the two values we are interested in:

String accessToken = oauthHelper.getAccessToken(oauthParameters);
String accessTokenSecret = oauthParameters.getOAuthTokenSecret();

Once we have these values, we then requery the user object from the datastore, and save those values into the AppUser.OauthCredentials subclass:

appUser = LoginService.getById(appUser.getId());
appUser = LoginService.updateUserCredentials(appUser,
          new OauthCredentials(accessToken, accessTokenSecret));
req.getSession().setAttribute(Constant.DOC_SESSION_ID,
LoginService.docServiceFactory(appUser));

In addition, you'll see they are also stored into the session so we have them readily available when the API request to Google Docs is placed.

Now that we've got everything we need, we simply redirect the user back to the resource they had originally requested:

RequestDispatcher dispatcher = req.getRequestDispatcher((String) req
.getSession().getAttribute(Constant.TARGET_URI));
dispatcher.forward(req, resp);

Now, when they access the JSP page listing their documents, everything should work!

Here's a screencast demo of the final product:



Hope you enjoyed the tutorial and demo -- look forward to your comments!