Post date: Mar 26, 2014 3:25:34 PM
There are several use cases that require some task to be executed for each row output by a data reader, including:
DecisionStep.StartMethodLoop
Import.BatchEngine (processing Excel rows)
AbstractObject.Listen
The typical approach is something like this:
using (IDataReader reader = myQboObject.InvokeDataReader(operation, parameters)) {
while (reader.Read()) {
IDictionary<string, object> data = Functions.ToProperties(reader); // This creates a dictionary from a single row in the data reader
AbstractObject myOtherQboObject = AbstractObject.Create(someString, User);
myOtherQboObject.Invoke(someOperation, data);
}
}
Assume for a moment our data reader is returning a lot of day; perhaps 50K rows. While data readers are very memory-efficient, this approach is single-threaded, which means we are calling myOtherQboObject.Invoke(someOperation, data) in series, 50K times. Slow.
Fortunately, the .NET Task Parallel Library (which is wicked cool) allows use to use an enumerator and leverage parallelism, like this:
MethodSignature signature = new MethodSignature("MyQboObject/operation?{Parameters}");
Parallel.ForEach(signature.EnumerateReader(), data => {
AbstractObject myOtherQboObject = AbstractObject.Create(someString, User);
myOtherQboObject.Invoke(someOperation, data);
});