Queue Service - Deploying using MS Build

posted Jul 6, 2018, 4:22 PM by Greg Kent   [ updated Jul 27, 2018, 11:44 AM ]
Queue Service and BOT Service deployment is now facilitated using MSBuild and should be deployed whenever a new build is made. 
  • MSI installs have been deprecated
  • MSBuild 2017 is supported
Additional Updates:
  • Queue Statistic logic updated including bug fixes
  • Queue Herd and Logging settings are now controlled using Application Settings

Queue Service Details

  • Queue Service solution (<root>/QueueService/QueueService.sln) has been refactored and simplified. All BOT projects have been removed.
  • qbo3.QueueServiceTargets supports the deployment and registration of Queue Service on the target machine. Targets:
    • QueueService - stops and unregisters (if existing), deploys and registers QueueService. It will NOT automatically start Queue Service. 
    • qsStart - Starts Queue Service
    • qsStop - Stops Queue Service
    • qsDeploy - Deploys content of Queue Service
    • qsCreate - Registers Queue Service
    • qsDelete - UnRegisters Queue Service
  • QueueService now deploys to <WebsiteRoot>/bin vs. MSI install location of C:\Program Files\Quandis\Queue Service. This allows QueueService to utilize the same dlls as the application.
  • QueueService.exe.config
    • Now deploys to <WebsiteRoot>/bin
    • Config File Autodetect - Auto detection of application's web.config file is now supported. If Queue Service is installed in <WebsiteRoot>/bin there is no need to specify a path for the QueueSource.ConfigFile. Instead the value "autodetect" is specified. This is the recommended pattern.
    • This file is usable "as-is". If the pre-existing QueueService.exe.config file has a custom Herd specified, please take note of Herd value and implement using the pattern specified in section Queue Herd Updates below. Note all Queue Service settings can be configured using SystemDefaults. 
      • If customization is required beyond standard settings, the suggested pattern is to create a copy of QueueService.exe.config in the respective theme project at same level as web.config and set "Copy To Output Director" = "Copy if newer", Build Action = "None"
  • To Build and Deploy:
    • Save existing QueueService.exe.config file in MSI install location of C:\Program Files\Quandis\Queue Service. Compare to QueueServiceDeploy/QueueService.exe.config to see if there is any customization.
    • Uninstall Queue Service from add/remove programs
    • From main proj file, CallTarget "QueueService". See qbo3.Sample.2017 for usage.
      • SiteFolder must be specified as a parameter if SiteFolder is other than C:\inetpub\wwwroot (specified in qbo3.Powershell.Targets)

Queue Statistic Logging Update

Queue Statistics are now logged using the application instead of the queue service. This simplifies configuration and allows more control over settings. The following settings control statistics logging behavior:

  • qbo.Application
    • QueueServiceLogIntervalMS - Sets logging interval of statistics to Cloudwatch. Suggested setting is 60000. (60 seconds). If set to 0 logging is disabled. Default value is 0.
    • QueueServiceQueueStatisticsList - List of queues or Queue Statistics Group to log statistics on. Names are separated by comma. Eg "DefaultQueue,QCCBots"
    • QueueServiceLogExecutionTime - Records execution times of processed items as a separate metric. Default value is False which disables this feature.
    • QueueServiceSourceName - Sets namespace of metrics in CloudWatch
  • QueueService.exe.config
    • qbo.QueueService.StatisticIntervalMS - sets the timer intveral in Queue Service to poll for logging statistics. Suggested setting is 10000. (10 seconds)
For statistics logging to function, the following above settings must be defined:
  • QueueServiceLogIntervalMS must be > 0
  • QueueServiceQueueStatisticsList must contain at least one queue name or Queue Statistics Group
  • qbo.QueueService.StatisticIntervalMS must be > 0
Note Statistics logging only functions on queues that are part of active Herds.

The following settings/updates must also occur:
  • Web.config update
    • loggingConfiguration > listeners > AmazonCloudWatchQueueStatistics (must be present)
    • loggingConfiguration > formatters > Queue Metric Formatter (must be present)
    • loggingConfiguration > categorySources > QueueStatistics (must be present)
See qbo.Core > ApplicationWeb > Web.config

  • Application Setting name update. Two settings have been renamed for easier identification:
    • qbo.Application.Properties.Settings.QueueSourceName has been renamed to qbo.Application.Properties.Settings.QueueServiceSourceName
    • qbo.Application.Properties.Settings.QueueStatisticLogIntervalMS has been renamed to qbo.Application.Properties.Settings.QueueServiceLogIntervalMS
If these are seeded as SystemDefaults, a simple update can adjust the names:

Update SystemDefault SET SystemDefault = 'qbo.Application.Properties.Settings.QueueServiceSourceName' , UpdatedDate = GETDATE()
WHERE SystemDefault = 'qbo.Application.Properties.Settings.QueueSourceName'

     Update SystemDefault SET SystemDefault = 'qbo.Application.Properties.Settings.QueueServiceLogIntervalMS', UpdatedDate = GETDATE()
where SystemDefault = 'qbo.Application.Properties.Settings.QueueStatisticLogIntervalMS'

Queue Herd Updates

Queue Service Herds are now managed using Application Settings. To manage queue herds, Application Settings can leverage the standard QBO Configuration Filter Pattern to target specific machines for custom herds. The following Application Setting sets the Herd(s):
  • QueueServiceHerd - sets Queue Herd(s) to process. Default value is Default as all Queues are part of the Default herd unless specified. Multiple Herds can be separated with a comma. Eg. System,Default
To configure a custom herd:
  1. Seed a custom setting: "QueueServiceHerd" with Filter "DocGenerators" with value Docgeneration. Eg.
    <SystemDefaultItem>
      <SystemDefault>qbo.Application.Properties.Settings.QueueServiceHerd</SystemDefault>
      <Value>DocGeneration</Value>
      <Filter>DocGenerators</Filter>
    </SystemDefaultItem>
  1. For each queue to be processed by custom herd, set the Queue.Herd to Docgeneration
  2. In web.config on herd machines, ensure setting exists:
<setting name="ConfigurationFilter" serializeAs="String">
  <value>DocGenerators</value>
</setting>

BOT Service Details

  • BOTService.sln has been created and only contains BOT Service projects and has been decoupled from Queue Service
  • BOT Service deployment follows same pattern as Queue Service:
  • qbo3.BOT ServiceTargets supports the deployment and registration of BOT Service on the target machine. Targets:
    • BotService - stops and unregisters (if existing), deploys and registers BOT Service. It will NOT automatically start BOT Service. 
    • bsStart - Starts BOT Service
    • bsStop - Stops BOT Service
    • bsDeploy - Deploys content of BOT Service
    • bsCreate - Registers BOT Service
    • bsDelete - UnRegisters BOT Service
  • BOTService.exe.config
    • Now deploys to <WebsiteRoot>/bin
    • Updated to support auto detection of  QueueService.exe.config.  Set BOTSource.ConfigFile  = "autodetect". This is the recommended pattern.
    • EC2 Autoscale and UserData dynamic config file copies, change:
      • $targetFolder = <WebsiteRoot>\bin. eg. C:\inetpub\wwwroot\bin
      • Any reference to "QBO BOT Service" to "qbo BOT Service". Note case sensitivity
  • To Build and Deploy:
    • Uninstall BOT Service from add/remove programs
    • From main proj file, CallTarget "BOTService"
      • SiteFolder must be specified as a parameter if SiteFolder is other than C:\inetpub\wwwroot (specified in qbo3.Powershell.Targets)



Comments