Queue Service 3.0

Queue Service, BOT Service, IQueue and IBOT interfaces have been modified to support asynchronous processing. 

Summary Of Updates

• • All queue messaging has been converted to async processing. Items are processed using AbstractObject.InvokeAsync()

  • All projects that implement IQueue have been updated to support IQueue async methods

  • All Abstract classes that support extensibility have been converted to support async processing. In each class all async abstract methods have been implemented as Virtual and invoke the synchronous counterparts. This will allow for backward compatibility and for projects to be converted to async as time/resources permit.

  • Queue Service solution has been updated to Visual Studio 2015 and test project qbo.Queue.Tests has been added

  • Queue Service count metrics have been optimized to no longer query queue counts if queues are not being reported to Cloudwatch. Queue counts will not be reported if count is zero as well. This allows for optimized Cloudwatch alarms when BOTS are used

Installation - READ FIRST

Ensure .NET Framework 4.6.1 is installed on all machines running QBO. You can install here. Note the install will force a machine restart.

Important: IQueueMessage serialization has been updated such that serialization of entire objects occurs in IQueueMessage. The new codebase can process queued items that contain the older format and will convert any new items to the new format. This means you must install the new codebase to ALL machines and cannot have a mix of old and new codebases in any given environment.

Pre-Installation - 

• Run all tests from project qbo.Queue.Tests in the QueueService solution.

QBO Queue Service

• • Build and deploy latest codebase from Trunk

  • Obtain Trunk/qbo.3/Installation/qbo.QueueService.3.0.0.x64.msi or qbo.QueueService.3.0.0.x86.msi (x86 version).

    • Backup qbo.QueueService.exe.config before installing new Queue Service. Ensure that supportedRuntime is updated to reflect .NET 4.6.1:

 <startup>

    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1"/>

  </startup>

• • • Ensure that web.config supportedRuntime is updated to reflect .NET 4.6.1. You may want to allow for various version of Newtosoft.JSON by modifying web.config here.

QBO BOT Service

A new AWS Image with AMI Name QBO-BOT-2-0-0 has been created. Please look for latest version in AMI (Eg. QBO-BOT-2-0-1) as this post will age. The intent is for all QBO applications utilizing BOTS (as part of an Auto-Scale strategy) to use this AMI. This image contains:

• • .NET 4.6.1 and subsequent Windows updates

  • Bot Service 2.0.0 install

  • Permissions adjusted for QBO BOT Service to allow System account to start service

This image can be used as a generic BOT image by AutoScaling. Steps to configure:

• • Create a Launch Configuration Group (Eg. SCRA BOTS). This will be used for any platform (DEV, UAT, PROD)

    • Choose the AMI QBO-BOT-2-0-0 - (please look for latest version in AMI (Eg. QBO-BOT-2-0-1)

  • Select an instance type (Eg. t2.small)

  • Under Configure Details:

    • Provide User Data as text. The Powershell script is responsible for:

      • Copying appropriate BotService.exe.config (depending on environment)

      • Copying any Plug-In dlls the BOTS invoke. They generally have to reside in the BOT Service install folder.

      • Starting the QBO BOT Service

    • IP Address Type: Assign IP Address to every Instance

    • Ensure Security Group is correct. In most cases this is VPCWebSG

  • Configure New Auto Scale Group

    • Choose the Launch Configuration just created

    • Set desired machine size

    • Ensure Subnet match any custom (or segregated) Subnet

    • Assign AutoScale policy IAW the desired Cloudwatch alarm

• Modify Pre-Existing Auto Scale Groups

  •  Select Auto Scale Group

    • Scale Policies > Delete All Scaling Policies

    • Create 2 new polices: 

      • Add Policy > Name: Increase, Add X Instances. Also ensure you choose the newly created Clouldwatch Alarm. Ensure "Create a Simple Policy is checked"

      • Add Policy > Name: Decrease, Remove Y Instances. Also ensure you choose the newly created Clouldwatch Alarm. Ensure "Create a Simple Policy is checked"

    • Navigate back to Clouldwatch alarm

      • Adjust "Decrease" auto scale action to be based off status "OK"

If you elect to use another AMI:

• Ensure .NET Framework 4.6.1

  • Obtain Trunk/qbo.3/Installation/qbo.BotService.2.0.0.x64.msi

  • Backup BotService.exe.config before installing new BOT Service. Ensure that supportedRuntime is updated to 4.6.1

  • Ensure that any dependent plug-in dlls are built to .NET 4.6.1

BOT Status Synchronization

The BOT Service has the ability to perform a status check against the parent application and synchronize BOT status based on application Queue status. To enable, the following settings must be in place:

BotService.exe.config

• • RefreshIntervalMS: Must be set to a number >=5000 ms

        <BotSource Name="QBO" Active="true" ConfigFile="C:\Program Files\Quandis\Qbo Bot Service\BotService.exe.config" Debug="false" RefreshIntervalMS="10000">

        </BotSource>

• • BaseUrl: Must be set to the application host name and accessible from the bot machine

      <setting name="BaseUrl" serializeAs="String">

        <value>http://devscra.quandis.net/</value>

      </setting>

• • Parent application web servers (web.config) - Application/Queue.ashx must be configured to allow anonymous access:

  <location path="Application/Queue.ashx">

    <system.web>

      <authorization>

        <allow users="?"/>

      </authorization>

    </system.web>

  </location>

BOT Service will poll the application Url to obtain status and dynamically set status of BOTs.

Cloudwatch Alarm Modifications

• • Consolidate Auto-Scale Cloudwatch alarms for your specific BOT Herd:

    • Current pattern is to have 2 alarms

      • Decrease Alarm - this will decrease the number of BOTS when number of items is <= X

      • Increase Alarm - this will increase the number of BOTS when the number of items is >= X

Create a new alarm that will consolidate both the increase and decrease:

• Ensure that alarm is triggered when number of items is >= X (same as increase)

• Implement 3 Auto-scale Actions in this alarm

  • State is OK - Decease Group Size

  • State is ALARM - Increase Group Size

  • State is INSUFFICIENT - Decease Group Size

Queue Log Performance Enhancement

Queue activity logging and searchability database performance has been improved. The following patch is applicable to Queuing versions 2.4 - 3.0 and can be applied without deploying  application layer code. Steps to deploy:

• • Stop Queue Service on all app servers

  • qbo.Db.Standard solution

    • Publish table QueueLogCurrent and all indexes

    • From SQL Management Studio, run /Scripts/Script.QueueLog.QueueLogCurrent.Seed.sql. Note you will need to un-comment SQL block

    • Ensure QueueLog indexes, triggers are in sync with Trunk. This should result in adding QueueLog trigger QueueLogQueueCurrentInsert and dropping QueueLog indexes:

    • IX_QueueLogByQueueAndQueueLogType

    • IX_QueueLogByExecuteDate

    • IX_QueueLogByNullSuperceded

    • IX_QueueLogByParentMessageIDIteration

  • Deploy to all web and application servers:

    • qbo.Application.Web - 

      • Templates/Configuration/Queue.Home.xslt

      • Scripts/Specs/Application/Spec.Queue.js

    • qbo.Logging.Web

      • Config/QueueLog.config

  • Restart website on all IIS Servers

  • Run Queue Test Spec: Design > Specifications > Application > Queue (Do not proceed and resolve any errors.)

  • Restart Queue Service on application Servers