Message Queueing

Post date: Sep 12, 2011 1:37:21 PM

QBO 3 abstracts the notion of message queuing to enable use of EventQueue, MSMQ, SQL Service Broker, or Amazon's SQS.

Use Case: Event Queue

QBO 2.0 relies heavily on the use of the Event and EventQueue tables for asynchronous processing.  QBO 3 will continue to support this infrastructure.

Use Case: MSMQ (Direct) for Attachment Generation

When generating Attachments, QBO 3 will enable offloading the processing to multiple application machines. The mechanism will be to queue an attachment generation request directly to a MSMQ, which is in turn monitored by a WCF instance that knows how to generate the requested attachment. This implies one or more distinct queues for each attachment plugin, such as:

• • Aspose Word Generator: 3 queues

  • Google Map Generator: 1 queue

  • XHTML Generator: 1 queue

  • HTML to PDF Generator: 2 queues

The following of communications for generation of a Word document via Aspose might be:

• • A web component calls qbo.Attachment.Invoke('Generate', {parameters}), which

  • Creates a MSMQ in the Aspose Word Generator queue, logging that fact to a queuing logging sink

  • The MSMQ is read by the Aspose plugin WCF, logging that fact to a queuing logging sink

  • The Aspose plugin generates the document, logging that fact to a queuing logging sink

  • The Aspose plugin WCF removes the message from the MSMQ, logging that fact to a queuing logging sink

In this scenario, there is no use made of the EventQueue table. However, the use of a logging sink enables more detail tracking that currently allowed by the EventQueue table. For example, a logging sink might be:

• • The QBO Message table, binding each log entry to a message bound to the attachment

  • Amazon Simple DB, with the QBO UI providing a popup window view on the messages associated with an Object/ID

  • A local file system

  • A customized table / database designed precisely for such logging

Use Case: Amazon Simple Queuing Services (SQS)

When calling QDS for build-in functionality like geo-coding or credit reporting, QBO 3 will enable offloading the processing to external parties, such as Quandis Data Services (QDS). The mechanism will follow MSMQ, but leveraging the cross-network convenience of Amazon SQS.

Use Case: SQL Service Broker

SQL Service Broker offers an alternative to MSMQ, with the distinct advantage of being accessible for TSQL reporting and insertion. However, it may not be available during backups or heavy SQL usage. We shall create an IQueue instance of SQL Service Broker should a use case for it arise, possibly including:

• • Batch processing 

  • Async triggers?

Configuration

QBO 3 will include automatic configuration of queues, as well as a dashboard to quickly display the status of each queue participating in the installation's processing.  The dashboard will include key information such as:

• • Queue: name of the queue

  • Type: plugin used to communicate with the queue (MSMQ, Amazon SQS, etc.)

  • Monitored From: 1 or more machines monitoring the queue

  • Monitored By: plugin being used to monitor the queue

  • Messages in Queue: number of messages currently in the queue

  • Last Processed: age of the last message processed

  • Processing Rate: rate at which messages are being processed

  • Average Age: of items in queue

  • Check endpoints

  • Schema Validation

  • Max message size?

See the attached spreadsheet as an example.

Functionality provided by this dashboard will include:

• • Start, Stop and Restart the machine/plugin (WCF endpoint) being used to monitor a queue

  • Add a machine/plugin (WCF endpoint) to monitoring a queue

  • Prioritize queues that share a common machine/plugin (WCF endpoint)

Payloads

• • Intra-system: Object/ID

  • All: method, dictionary

  • Extra-system: XML Payload

Questions

• • Restart functionality: does it become obsolete?

  • Heartbeat infrastructure?

  • Is batching involved?