Adempiere Integration Framework: Part 1

Adempiere, an open-source ERP system who's heritage is from Compiere, offers several convenient mechanisms for integrating with external services. Perhaps the most flexible and powerful are Adempiere's Service Processes (we'll call them Processes for short). They essentially represent callouts that you can invoke from within Adempiere. They can be triggered by user actions, such as clicking on a button, or programmatically through workflows. What's particularly enticing about them is how simple they are to develop -- A topic I will cover at a future time. My focus on this article is how they can be used, in conjunction with a integration bus, or ESB, to effectively integrate with external services.

External services can be any system that Adempiere needs to communicate with. For example, integrating with a shipper or credit card processor comes to mind. Or, synchronizing data with popular CRM systems such as Salesforce.com or SugarCRM. While people have been doing this sort of thing with Compiere/Adempiere for a long time, more often than not, it's a point-to-point integration. The disadvantage to this approach is that it can quickly become unmanageable, and tightly couples the integration code into Adempiere. This limits flexibility, and can be fragile and difficult to maintain.

The increasing popularity of Enterprise Service Buses (ESB) has occurred because of the recognized limitations and issues surrounding point-to-point integrations. In addition, ESB's generally provide a bevy of reusable adapters, such as SOAP, HTTP, Email, FTP etc., that eliminate the need to recode these solutions over-and-over again. Often, they also have sophisticated business process orchestration features such as BPEL, built-in, to help design and manage complex workflows. Popular open-source ESB's include Mule, Apache Servicemix, OpenESB , PetalsESB and Celtix.

In the scenario for today's topic I want to discuss a framework that can be use invoking external services from within Adempiere using a Process. The Process, in turn, publishes the outbound request into a JMS queue or topic. From there, it is lifted for processing by an ESB. The ESB, using it's rich set of pre-built adapters, transforms the request to the destination format, and submits it for processing. This process can be asynchronous or synchronous, depending up the client's requirements.

Why not just invoke the ESB directly from within the Adempiere Process. Again, it's about keeping things loosely coupled. We might want to change ESB's at some future point, and hardwiring specific ESB code (most likely proprietary in nature, using it's APIs) into the Adempiere Process callout limits such flexibility.

Using JMS, however, we can define a standard XML schema for the data transfer. As long as the request/response adheres to the proper format, the Adempiere Process could really care less about the implementation details.

Below is a diagram of what I'm describing:



As you can see, in this diagram, I'm using the Servicemix ESB (which I do have a preference for, given it's JBI-compliance, maturity, breadth, and Spring-based architecture). A Servicemix JMS component (in JBI parlance, that's a "Binding Component") lifts the JMS message sent by the Adempiere Process, then performs some transformation and business rule processing using pre-built components designed for that purpose ("Service Engines", in JBI). Then, when the XML is in the appropriate format/schema, a SOAP adapter is used to call the external web service that's providing the service desired.

This architecture is highly flexible, and allows for a wide variety of implementations. For example, the JMS and Servicemix instances can be running in a distributed fashion on one more services (both ActiveMQ and Servicemix provide clustering capabilities as well).

In my next article, I'll describe how an ESB can be used to receive inbound request generated externally from Adempiere. For example, a sales order that is received from an external channel, such as an ecommerce web site.

As always, I welcome your thoughts/comments.

jeff

Add to Technorati Favorites
Add to Digg

Adempiere (Compiere) Web Services

I've been struggling for some time trying to come up with the right way to expose Adempiere (or Compiere) through SOAP-based web services (nothing against REST, but I'm still not convinced it's the way to go in an enterprise world, where security and vendor tool support are essential). A few months back, I wrote an Axis2 service wrapper that leveraged Compiere's model classes in a generic fashion. However, treating all Compiere/Adempiere documents using a single XML schema, while programatically simple, was less then elegant. Using this approach, I ended up with generic-looking XML that resembled:

<idal:mField idal:name="DatePrinted">
<idal:newvalue/>
</idal:mField>
<idal:mField idal:name="Processed">
<idal:newvalue>true</idal:newvalue>
</idal:mField>
<idal:mField idal:name="DateAcct">
<idal:newvalue>11/01/2003</idal:newvalue>
</idal:mField>
<idal:mField idal:name="IsPrinted">
<idal:newvalue>false</idal:newvalue>
</idal:mField>
<idal:mField idal:name="IsInvoiced">
<idal:newvalue>true</idal:newvalue>
</idal:mField>

This is really not ideal, as it's very verbose, and it circumvents the power of XML, which is it's self-describing vocabulary.

A more preferred approach to the above is something like:

<DatePrinted/>
<Processed>true</Processed>
<DateAcct>11/01/2003</DateAcct>
<IsPrinted>false</isPrinted>
<isInvoiced>true</isInvoice>

Obviously, in addition to being more succinct, it's also more human readable, which is an important consideration.

Recently, after a taking a few months off on the project, I believe it is possible to relatively easily dynamically generate a SOAP WSDL that is fully expressive, and accomodates changes to the Compiere/Adempiere data dictionary. Further, by using XSD extension mechanisms, we can limit the number of operations, or verbs, to a reasonable number. So, under this scenario, we'd have operations (verbs) such as "Create", "Delete", "Update" and "Search", which can be applied to common documents (nouns) such as PO, Invoice etc.

Another key consideration is security. Passing username/password values, in clear text, even over SSL, is not sufficiently secure for most organizations. After all, the SOAP XML is probably littered throughout log files, leaving credentials easily exposed in clear text to any would-be hacker. Instead, the preferred approach is to use WS-Security, which through profiles, can support a variety of encryption solutions. For example, a binary hash token can be used that is highly secure. Further, WS-Security is one of the few WS-* standards that is actually more-or-less universally supported. For instance, the UsernameToken WS-Security standard is nicely supported by both .NET and Java (Axis Rampart), and presumably others as well.

Notwithstanding my earlier contention that SOAP is the best way to go in an enterprise environment, it's foolish to think everyone concurs with me on that. So, it also makes sense to expose the services through other protocols as well, including JMS and REST (or even email). My previous approach to using Axis2 really limited the solution to SOAP. What I'm proposing this time around is to use Servicemix as the integration broker, whereby it could receive the inbound messages in a variety of protocols. Thus, we're no longer just limited to SOAP.

ARCHITECTURE

From an architecture standpoint, what I had in mind was:



In the above scenario, Servicemix "binding components" are used to manage the inbound requests. From there, it's routed through a profile validation services (I'll explain more of this in a future posting), then ultimately to the implementation bean/service that will fulfill the request.

RELEASE PHASES

Obviously, this is a fairly ambitious undertaking. So, I propose the follwing phases:

Phase 1:

• Authorization/Authentication via Profile service
• Dynamic WSDL Generation
• Core API set (Create, Delete, Update, Insert)
• WS-I Basic Compatibility for SOAP.

Phase 2:

• WS-Security Authentication using Mediator Service (Apache Synapse or Spring-WS, which has strong WS-Security support)
• Additional API Calls.
• Performance Improvements

OPEN-SOURCE

I anticipate releasing the code under the same license as Adempiere, or even BSD, if appropriate. A Sourceforge project will be launched as soon as enough code exists to seed the project (nobody is interested in vaporware projects -- real code has to exist to excite the community). I have begun working on the code, an expect anticipate a May delivery timeframe for the first seed.



Add to Technorati Favorites
Add to Digg