Queue Service 2.2

Queue Service has been updated to version 2.2 which includes automatic configuration refresh, increased ability to reset queues and performance improvements with queue processors.

Release Notes

Summary of Features
  • QBO Configuration Monitoring is now directly integrated with the queue service. This allows users to fully manage queue changes the queue service will automatically detect changes without the need to restart the queue service. Additionally and changes to QBO configuration that are facilitated through ConfigurationEntry (Eg. Statements, Filters, etc) for any object will be automatically detected.
  • Global reset of queue service can be facilitated without having to restart the queue service
  • Queue processor performance has been dramatically increased.
  • Callback Queue added to QueueList. This is a system queue which allows UI access to increase visibility to QBO callbacks triggered from the application tier

Integration with QBO Configuration Monitoring
  • Each Queue Manager (defined by QueueService.exe.config QueueSource) now contains a QBO Configuration Monitor. The Configuration Monitor runs on a timer and detects any changes made to ConfigurationEntry. If any changes are detected it instructs each Queue Processor (1:1 with Queues listed in queue config) to automatically refresh their configuration which detects the same changes at the Queue Processor level. Propagation occurs for all servers running queue service for a given application. See Configuration Monitoring section for a detailed explanation of the monitoring algorithm. 
  • Queue Resets and web server resets can now be facilitated globally. Double clicking on the "Design" menu will now initiate a global recycle event. Users can now reset the QBO application on all web servers and app servers with one command. Both web and application servers which have configuration monitoring enabled will detect the reset event. With the queue service, it will stop all processing of queues and recycle each queue processor. All activity is logged to QueueLog under each queue as QueueLogType 'Debug'. Global resets are use typically useful when:
    • Users execute a configuration change which is not facilitated through ConfigurationEntry and they want all servers to detect the change
    • ConfigurationEntry record was removed and the record was NOT an override of an existing item. Examples are:
      • Removing a user added queue
      • Removing a user added statement
Queue Processor Performance Improvements
  • Queue processors have now been optimized to include a callback to "process again" once an item in the queue has been processed. This allows for continual processing of items and bypasses the dependence on the processor timer in a duty cycle scenario. Consider the following:
    • 5000 items are queued which take approximate 100 milliseconds to process each
    • Processor timer is set to run every 500 milliseconds
    • 3 threads assigned to queue
    • Before this enhancement, the processors would process 3 items every 500 milliseconds and have to wait for the timer to elapse before processing 3 items again. Since 100 ms < 500 ms each timer cycle would process each item and there would be no overlap. Therefore 1 second would have to elapse to be eligible to process 6 items. Extrapolation reveals ~833 seconds to process 5000 items.
    • With the enhancement, each item will initiate a callback to process again resulting in no timer latency. Once the processing starts it's continually processing. Therefore each thread can now process ~10 per second (1000 ms / 100 ms). 3 threads * 10 items per second = 30 items per second. 5000 items / 30 = ~166 seconds. These times are approximate and do not factor in all latency incurred with processing queues such as checking counts, clock cycles, etc.

Steps to update to version 2.2:

  1. Ensure Steps to update to version 2.1 and previous versions have been met.
  2. Get latest from trunk:
    1. Qbo.Application/qbo.ApplicationWeb
    2. Qbo.Security/qbo.SecurityWeb
    3. /qbo v3/Installation/qbo.QueueService.2.20.msi
    4. V2 – qbo.Application.Database
      1. /Indexes/ConfigurationEntry.indexes.sql
      2. /Scripts/Post-Deployment/Queuing.2.X.ObjectQueue.QueueLog.Sync.sql
    5. Queuing Plug-ins:
      1. qbo.Queue.MSSQL
      2. qbo.Queue.ObjectQueue
  3. Ensure Queue Service is stopped
  4. Backup
    1. QueueService.exe.config
    2. /Config/Queuing.config
    3. /Config/QueueLog.config
    4. Web.config
  5. Build and deploy all projects
  6. Run Index script on target database
  7. Run Queuing.2.X.ObjectQueue.QueueLog.Sync.sql. This will retrofit any ObjectQueue records (Future, Failed, Current, Callback) and add associated QueueLog records enabling these records to be searchable from Queue.ashx
  8. web.config
    1. Add categorySource ‘Configuration’. Ensure there are no active listeners under the categorySource as the logging is very verbose
    2. Ensure qbo.Application.Properties.Settings.ConfigurationMonitoring is set to true or does not exist (by default it's set to true)
  1. Uninstall Queue Service
  2. Install qbo.QueueService.2.20.msi
  3. Replace QueueService.exe.config with backup version
  4. Run Test Scripts from browser
    1. Application – Queue Testing - All steps must pass to use this build. Note in some testing the DB Server time and Web Server time had to be manually synchronized for script to succeed:
      1. Control Panel->Set the time and date->Internet Time->Change Settings->Update Now
    2. Any additional test scripts you desire
  5. Start Queue Service
  6. Manually queue an item for immediate processing 
Configuration changes to consider:
  1. Adjust threading on all Queues to not exceed 3 threads. Suggested settings to start with:
    1. CurrentQueue - 1 thread
    2. Queues that were set to >= 5 threads - now start with 2 or 3 threads
  2. QueueService.exe.config - qbo.QueueService.ProcessorIntervalMS. Try 2000 ms instead of 500 ms

Comments