WCF in QBO 3

Post date: Sep 28, 2011 12:16:3 PM

Background

Windows Communication Foundation (WCF) is Microsoft's abstraction of calling services across various platforms. QBO 2.0 invokes functionality by:

• • calling a web service (think QDS service orders or QBO import framework calls), or

  • creating an Interface instance via Reflection, and calling an Interface method (think ProcessEvent and ProcessStep)

WCF allows for many more scenarios:

• • calling a method via a web service

  • calling a method via HTTP without the service running under IIS

  • calling a method via TCP (akin to .NET Remoting)

  • calling a method via MSMQ

  • calling a method via a customized extension (such as an Amazon SQS binding)

Many of these methods support duplex communications, essentially meaning that the service can issue a callback to a client.

There are a few problems solved in the QBO world with WCF:

• • trapping boundary errors,

  • splitting application processing among multiple machines via a GUI, and

  • building a callback structure into all QDS calls, so we can recover elegantly from synchronous service order vendor errors

Trapping Boundary Errors

A credit report order behaves along these lines:

• • QBO > QDS > Credit Bureau > QDS > QBO, or more precisely

Each of these "systems" requires a lot of moving parts to be working correctly for the order to process successfully. The following is a short list of issues that might break such a call:

• • QDS web server is offline

  • Credit bureau is offline

  • QDS database is running maintenance

  • VPN to a credit bureau times out

  • QBO is incorrectly configured when attempting to process the results

In QBO 2.0, if any of these things happen, at least 2 parties (and often all 3 parties) must be involved in recovering from an error.

With QBO 3 with WCF, a credit report order can be configured along these lines:

• • QBO > MSMQ > QDS > Amazon SQS > Credit Bureau > Amazon SQS > QDS > MSMQ > QBO

Wait, what?  More moving parts to break?  Well, yes! 

However, configuration of queues (MSMQ, Amazon SQS, or SQL Service Broker) are easier to maintain 24x7 up time on than a standard QBO, QDS, or Vendor instance. A queue is about as easy to maintain as a file system folder. Revisiting the typical issues above:

• • QDS web server is offline: the request sits in MSMQ until QDS is back online

  • Credit bureau is offline: the request sits in Amazon SQS until the bureau is back online

  • QDS database is running maintenance: the request sits in MSMQ until QDS is back online

  • VPN to a credit bureau times out: no longer applicable

  • QBO is incorrectly configured when attempting to process the results: the request sits in MSMQ until QBO is reconfigured; no QDS coordination is required

Not so sure this works in all use cases?  Of course it doesn't.  WCF can be reconfigured (via configuration files) to do this instead:

• • QBO > https > QDS > https > Credit Bureau > https > QDS > https > QBO, or

  • QBO > tcp > QDS > https > Credit Bureau > https > QDS > tcp> QBO

Ah, these look at lot more like what happens in QBO 2.0. In fact, the QBO 2.0 version is better represented as:

• • QBO > https > QDS > https > Credit Bureau > https > QDS > https > QBO

Note 1: obviously not all vendors will offer queue-based ordering. For those that don't, QDS already engineers some level of re-try into it's processing. Introduction of WCF makes no real difference in this regard.

Note 2: QDS may still hit unexpected data conditions that cause errors in QDS.  QBO may still hit configuration issues that cause errors in QBO. WCF merely helps isolate those errors by having the request message stored in a queue, so the message can be reprocessed at the convenience of QDS/QBO, rather than requiring manual support intervention after the error condition has been fixed.

Splitting Application Configuration

QBO 2.0 divides workload across machines via configuration of the qbo.EventQueueServiceMSMQ windows service. This has some limitations, including:

• • only one machine may monitor any given Event Queue, 

  • if the Event Queue Service crashes, all processing for a machine stops,

  • all queuing must be made through the EventQueue table, which can grow extraordinarily large,

  • processing done across machines must be done asynchronously through the qbo.EventQueue

With WCF, one creates a service, and can "launch" the service telling it to monitor an endpoint (such as a port or TCP or HTTP, or a queue such as MSMQ or Amazon SQS). Once a WCF service is running, it behaves on it's own, and is not dependent on an overall service like qbo.EventQueueServiceMSMQ to continue operating. WCF services can be launched through code, so a web GUI can be created to allow administrators to start and stop instances, or create new instances, as workload demands.

Multiple instances can monitor the same queue, so high volume queues can have throughput increased by adding multiple machines to it. With qbo.EventServiceMSMQ, additional threads can be brought to bear on a queue, but not multiple machines. This is ideal for use cases like generation of PDFs from HTML.

Today, synchronous document generation happens on the web server. Generation of documents via Aspose (and other IAttachmentTemplate plugins) can be resource intensive. If the document generation processing is pegging the web server CPU, only QBO 2.0 alternative is to create an EventQueue row and process the document asynchronously. With WCF, QBO 3 can stand up a document generation service endpoint on an application server, and the web server's Attachment.Generate() call can invoke the Attachment.Generate code on another box.  Technically, this is an async call, but done in near-real-time, minimizing resource usage on the web server.

Callback Structures

Still working this one out, but it's built into WCF.

WCF Configuration Observations

WCF is a very powerful and flexible infrastructure, and as a result, is difficult to configure correctly. Sound familiar? 

Key nuances observed include:

• • passing credentials to a QBO class instance within WCF: see the qbo.Security.Service namespace classes for this

  • configuration files must be precise, and are pretty long: QBO classes will make standard assumptions if the configuration files are missing or incomplete

  • both the WCF client and server must use a certificate to encrypt credentials: we use the *.quandis.net / *.quandis.com SSL cert for this purpose

  • more to come

References

• http://www.neovolve.com/post/2008/04/07/wcf-security-getting-the-password-of-the-user.aspx

  • http://blog.functionalfun.net/2008/05/how-to-create-server-certificate-for.html 

• http://stackoverflow.com/questions/2454850/how-do-i-pass-values-to-the-constructor-on-my-wcf-service