Auditing

Background

Auditing is a subset of scoring, typically performed manually by a human. The most common use case is to score the quality of work performed by a user after completing a Custom Form. For example:
  • 10% of Foreclosure Referral forms may be audited for regulatory compliance
  • 100% of BPO forms may be audited for quality control
  • 25% of the work performed by a new employee may be audited by a supervisor

Setup

The steps required to set up an audit include:
  • Create a custom form to serve as a task for users to complete (e.g. Foreclosure Referral)
    • This will automatically create a Smart Worklist Template matching the name of the form (e.g. Foreclosure Referral)
  • From Design > Smart Worklists, search for and click on the worklist to be audited
  • From Smart Worklist Template Summary page:
    • click on the Audit panel
    • leave the Matrix dropdown on the 'Create a Matrix' option, and
    • Click the Save button
    • this will create a new Matrix, with some pre-populated dimensions (see below)
  • From Design > Scores, create a Score Template containing the data points you wish to collect during your audit
  • From Worklist > Manage Worklists, sort by Template, select the templates to be audited, and choose Options > Audit
    • this will scan every completed worklist item, and determine if the items matches a row in the Audit Matrix
    • for each row of the Audit Matrix, a subset of matching items will be added to an auditing worklist
Data Affected

Based on the setup above, we have:
  • An ImportFormTemplate called 'Foreclosure Referral'
  • Two SmartWorklistTemplates: 'Foreclosure Referral', and 'Audit: Foreclosure Referral'
  • A Matrix called 'Audit: Foreclosure Referral'
  • A ScoreTemplate called 'Audit: Foreclosure Referral'

Audit Matrix

The Audit Matrix is very powerful: it can group worklist items by an number of data points (Matrix Dimensions), and audit a subset of that group. The most import output (Measure) of the Audit Matrix is a Template: this is the name of the 'auditing' Smart Worklist Template to use. For example, if you are auditing a Foreclosure Referral worklist with 100 items in it, the system will create a new Smart Worklist Template called 'Audit: Foreclosure Referral Worklist'.

When choosing how many worklist items to audit, you may define a percentage, minimum and maximum.  For example, assume you have 100 items in a worklist:
  • Setting Percent to 20 will cause 20% * 100 = 20 items to be audited
  • Setting Minimum to 5 would make no difference in this case
  • Setting Minimum to 25 would cause 25 items to be audited (since 25 > 20% * 100 = 20)
  • Setting Maximum to 50 would make no difference in this case
  • Setting Maximum to 15 would cause 15 items to be audited (since 15 < 20% * 100 = 20)
  • Setting Minimum to 20 and Maximum to 10 would cause 10 items (but this is a silly setting, so don't confuse yourself)

Audit Worklist

The audit worklist will route the auditor to the same URL that a user used to edit the data being audited, but will include an &Audit={Audit Score} parameter.  The target URL is responsible for implementing the Audit behavior, as follows:

<div class="btn-group" data-behavior="Audit" data-audit-options="{'data': {'Object': '<some object>', 'ObjectID': '<some object id>'} }">...</div>

This behavior will, upon detection of an Audit parameter in the URL, render the Audit UI.

Audit Score

The audit score determines the questions an auditor will see when navigating to a page to be audited. Each Score Item defined in the score template will be rendered as a question to be answered by an auditor.  

A Scoring dashboard displays all scores performed by template, including roll-up statistics on the values entered by auditors for each score.

Troubleshooting

An audit is launched by calling SmartWorklist/Audit?ID={some SmartWorklistIDs}. This calls the SmartWorklist/Audit method, which does several things:
  • executes the SmartWorklist/AuditCheck statement, which returns all Smart Worklist Members that have been completed, but have not already been marked for an audit
  • for each Smart Worklist Member, the SmartWorklistMember/Audit methods is called, which calls Matrix.Lookup(SmartWorklistMember) to determine if the member is eligible to be audited. If the matrix returns a row, the SmartWorklistMember.AuditMeasureID will be used to store which matrix 'row' was returned.  If the Matrix lookup returns no rows, SmartWorklistMember.AuditMeasureID is set to 0, indicating that the member is not eligible for auditing.
At this point, all completed Smart Worklist Members in the Smart Worklists we are auditing will have SmartWorklistMember.AuditMeasureID set to 0 (not eligible) or some MatrixMeasureID (indicating the 'type' of audit to be performed). Keep in mind that audits are usually performed on a subset of work completed; we next must calculate that subset.

The SmartWorklist/Audit method will next call the SmartWorklist/Audit statement, which calculates which of the eligible Smart Worklist Members to actually audit.  The Audit statement is pretty slick, doing the following:
  • for each distinct Matrix Measure, count how many Smart Worklist Members are associated with the 'row'
  • ensure a Smart Worklist Template exists matching the Matrix row's Template column
    • the MemberUrl for this template should be the same as the Smart Worklist Template being audited, with '&Audit={Template}' added to the end
  • ensure a Smart Worklist exists matching the Matrix row's Template column
  • inserts enough Smart Worklist Members into the audit Smart Worklist to ensure we meet the Percentage, Minimum and Maximum values specified by the Audit Matrix
If the auditing is not working as expected, key points to check include:
  1. Does the Smart Worklist Template being audited have an Audit Matrix defined?  If not, create one in accordance with the instructions above
  2. Does the Audit Matrix have a Template output (measure)? This template value needs to match the name of both the Smart Worklist Template that tracks the audits, as well as the Score Template used to enter the auditing data.
    • if there is no Smart Worklist Template matching the Audit Matrix's Template column, that's okay; one will be created on the fly by the SmartWorklist/Audit statement
    • if there is no Score Template matching the Audit Matrix's Template column name, that's not okay; we cannot predict what data points are to be audit; you need to create one!
    • if you want to change the name of the Score Template being used, you can: you must:
      • Ensure a Smart Worklist Template exists matching the Audit Matrix Template name, and
      • Set that Smart Worklist Template's MemberUrl to include &Audit={your custom Score Template name here}

Future Use Case: Generic Load Balancing

Background

Clients may wish to distribute their work to vendors based on target percentages.  For example, a mortgage servicer may elect to split their collections work among 3 collection agencies:
  • 40% to Company A
  • 30% to Company B
  • 30% to Company C
Of course, real-world use cases are never that simple, and may need to use a matrix to determine eligibility to work on a particular loan.

Example

Assume we have:
  • 5000 open processes (already assigned)
  • 125 inbound new processes (need to be assigned)
  • of the 125 inbound, you are limited to assigning them to "eligible" attorneys
    • e.g. in CA, there may be 7 attys
    • e.g. in MI, there may be 2 attys
    • e.g. in NY|MA|RI|CT Freddie mac may required routing to Harmon
    • e.g. very high value loans may all be routed to a central atty, regardless of state
  • we can convince our clients to maintain a "simple" set of rules: split work by % of active portfolio
So, we do this:
  • Create a SWL containing all "active" processes
    • one for all processes, or split between BK and FC, doesn't really matter
    • this is not for "next item" use; it's for calculating load balancing
  • Create a Matrix that will be used to identify "segments" of a list of rows to be load balanced, and output a pool of eligible candidates
    • e.g. State, Investor, UPB Range as inputs
    • Output is an XML doc like this:
<Organizations>
  <Organization id={Trott} target="30%">
  <Organization id={Orlans} target="30%">
  <Organization id={MCR} target="40%">
</Organizations>

We need a slick UI to build this XML -- let's assume such a creature exists.

Then, on some schedule / event:
  • adds the 125 new processes to the SWL; now the "active" count = 1500 + 125 = 5125
  • call SmartWorklist/LoadBalance
    • for each row without AuditMeasureID populated, call the matrix to find out which "segment" the process falls into
    • now we're ready for some math in a slick SQL statement, similar to SWL/Audit
    • for each "segment" (group by AuditMeasureID), calculate:
      • target % (from MatrixMeasure XML) for each atty
      • currently assigned items for each atty
      • % of new total they have (current items / active count)
      • % need to hit target = target % - % of new total
      • Number to assign = % need to hit target * new active count
    • update the SWLM rows accordingly
    • update the Process.AssignedOrganizationID accordingly
  • for each row, call the matrix go get determine which "segment" the process belongs to (e.g. a normal CA FC,a MA Freddie loan,etc.)
    • this means we add a MatrixMeasureID to each row in our data table 
  • feed the data table back to SQL, and calculate assignment


Comments