Queue Service has been updated to version 2.6 which includes: Queue item retry enhancements, Queue pause / auto restart enhancements, Queue Herding, Logic Monitor Notification strategies and minor optimizations/bug fixes.
- Queue item retry behavior allows for additional configuration and now supports:
- Retry Models - The retry model determines the calculation model of retry interval. Options are:
- Straight - No exponential back-off. Uses same retry interval every time an item is retried.
- Exponential Models - Note with both models, maximum wait and maximum wait interval must be supplied.
- Slow - Item retry interval will slowly increase starting with retry wait (initial retry amount) until the Retry Max Wait reached.
- Fast - Item retry interval will quickly increase starting with retry wait (initial retry amount) until the Retry Max Wait reached.
- Item retry count now allows for infinite retry along with specified tries
- Queues Automatic Retry - Queues can now automatically restart based on settings. Pause/Restart settings allow for:
- Pause a queue after X number of failures
- Restart X times allowing infinite restarts if desired
- Restart after X minutes|hours|days
- Queues can now support the concept of a "Failure Strategy". The purpose of Failure Strategy is to choose a "preset" of commonly used retry strategies. Currently, there are 7 strategies:
| Strategy | Description | | None | No item retry; no queue retry | | Standard | Intended to cover the majority of use cases. Straight backoff with queue pause with quick retry | | Database | Intended for items that perform DB operations. Fast backoff with a low amount of restarts | | Network | Intended for items that connect to network endpoints. Slow backoff with a finite amount of restarts | | Third Party | Intended for third party endpoints with a fast backoff with an infinite queue restart | | Cost Sensitive | Intended for queues that process items that incur cost and require minimum retry with a long queue pause and finite amount of restarts | | Advanced | Users can manually configure all retry settings. Note users can choose a preset strategy and then choose advanced and modify the preset strategy settings |
- Queue Herds - Queues can be specified to process under a given "Herd". The concept of herd implies that a single or group of machines will process specified queues. Application servers running queue service can be configured to process specific queues thus filtering what they process. This is useful for auto-scale strategies with volume spikes.
- Each queue can be set to process by a specific Herd. The default Herd is Default
- If a Queue Herd is set to a custom Herd, there must be an application server configured to process that Herd.
See below for more examples.
- Windows Application Logging - Queue Pause and Restart Events now log to Event Viewer which are monitored by Logic Monitor and can notify via an escalation list. Queue Events have custom EventID mappings in the Windows Application Log:
| Queue Event | Windows Application Log EventId | | Queue Pause | 11 | | Queue Resume | 12 | | Queue Catastrophic Error | 13 |
All Queue Pause and Resume events are logged to Windows Application log. Queue Pause events will log with a severity of Information when a queue is paused and a restart is eligible. If a queue is paused and cannot restart, the severity is Error which allows for further escalation within Logic Monitor. - UI Updates
- Queue Status - The Queue Summary page will now indicate if a queue is paused based on Queue color:
- White - Queue is not paused and is in good health
- Yellow - Indicates Queue is paused but will restart based on calculated restart date
- Red - Queue is paused and will not restart
Queue Status now supports hover and will display queue status, pause count and a restart date.
- Queue Logic - Item and Queue retry settings will now display when hovered over under Queue Logic. A search icon will also display the retry schedule calculated based on queue settings. This allows users to view and adjust settings to achieve a desired retry schedule
Failure Strategy (Includes Item restart and Queue resume) - Example 1 - Power user wants to adjust a given queue such that it's retry will:
- Retry 10 times, retrying immediately and eventually waiting 5 minutes. They also want this queue to pause but restart infinitely waiting 30 minutes between restarts.
- Suggestion: Advanced, applying above settings
- Example 2 - Power user manages 3 queues (Default, MSP, FTP Upload) and wants the queue service to retry with different strategies which fit best. For their queues, we would suggest
| Queue | Strategy | Reason | | Default | Standard | Standard retry covers most item use cases | | MSP | Cost Sensitive | Low retry which saves money and quick pause to restrict multiple errors | | FTP Upload | Network | Multiple retry with slow backoff |
- Example 3 - Power user manages 3 queues (Default, Letter Generation, Credco) and wants the queue service to retry with different strategies which fit best. For their queues, we would suggest
| Queue | Strategy | Reason | | Default | Standard | Standard retry covers most item use cases | | Letter Generation | Advanced | Need to customize retry strategy to fit specific characteristics and pause/retry attempts due to SLA requirements | | CredCo | Third Party | Leverage the fast back-off and retry. Fast back-off can start with an immediate retry and quickly escalate to retry to 30 minutes.
|
Queue Herds
- Use Case 1 - Simple Install. QBO install has 2 application servers for redundancy and no queues that volume spike beyond reasonable levels.
- Recommended Herding: Default
- Settings:
- Application Servers 1 & 2 - QueueService Herd: Default
- Queue settings: All queues set to Default
Reason: Both application servers can handle volume and do not need to be split out.
- Use Case 2 - QBO install has 2 application servers for redundancy and a letter generation queue that periodically encounters volume spikes that must be met by a certain SLA.
- Recommended Herding:
- APP 1 & APP 2: Herd = Default, LetterGeneration
- Auto Scale Servers: Herd = LetterGeneration
- Settings:
- Application Servers 1 & 2 - QueueService Herd: Default, LetterGeneration
- Queue settings: All queues set to Default except for Letter Generation queue which is set to LetterGeneration
Reason: APP 1 & APP 2 can process all queues. When volume spikes occur, auto scale specific Letter Generation app servers. Note that LetterGeneration is a separate Herd so it must be specified in the Herd list if a given application server is going to monitor multiple herds. The auto scale machines will specifically process LetterGeneration and
|
|