Category Archives: batch

Database allOrNone and exceptions

In my batch classes, I’m a big user of the allOrNone argument to Database.insert or Database.update. This is mostly because some orgs have dirty data and dealing with those as non-fatal exceptions is better for the business than rolling back all DML from a batch execute() that might be dealing with 199 good records and only one bad one.

So, the normal pattern would be

Database.SaveResult[] srList = Database.update(listOfSobjs,false);
for (Integer i =0; i < listOfSobjs.size(); i++)
  if (!srList[i].isSuccess() {
      // log the error somewhere for later admin action - typically to a persistent sobj
  }

But what if you had this coding fragment where the allOrNone argument was a variable, sometimes true, sometimes false?

Database.SaveResult[] srList = Database.update(listOfSobjs,allOrNoneVbl);
for (Integer i =0; i < listOfSobjs.size(); i++)
  if (!srList[i].isSuccess() {
      // log the error somewhere for later admin action - typically to a persistent sobj
  }

Well, and again the doc isn’t completely clear on this, if allOrNoneVbl is true, no Database.SaveResults are returned and an exception is thrown. Here’s proof:

try {
   Database.SaveResult[] srList = Database.insert(new List<Account>{
                                                    new Account(),
                                                    new Account()},
                                                 true);
   system.assert(false,'allOrNothing=true does not throw exception');
}
catch (Exception e) {
    system.assert(false,'allOrNothing=true does throw exception');
}

Debug log:
DML_END|[2]
EXCEPTION_THROWN|[2]|System.DmlException: Insert failed. First exception on row 0;
first error: REQUIRED_FIELD_MISSING,
Required fields are missing: [Name]: [Name]
EXCEPTION_THROWN|[6]|System.AssertException: Assertion Failed: allOrNothing=true does throw exception
FATAL_ERROR|System.AssertException: Assertion Failed: allOrNothing=true does throw exception

Conclusion: If your batch execute() is intended to log errors and you sometimes use isAllOrNone as true and sometimes as false in the same execute() (because you are doing multiple DML operations), your logging code is more complex as the source of the error message is found in different places (i.e. Database.SaveResult method getErrors() versus caught exception getMessages() ).

Testing a Batchable + Queueable + Schedulable

It isn’t super clear in the documentation (V36) what happens in a test method when a Batchable, Queueable, and Schedulable are involved within the Test.startTest()...Test.stoptest() execution scope.

The system executes all asynchronous processes started in a test method synchronously after the Test.stopTest statement

So, I decided to do a simple experiment:

The class (acts as both a batchable, queueable, and schedulable)

public with sharing class FooBatchableQueueable 
             implements Database.Batchable<Sobject>, Queueable {
    
    
    public Database.QueryLocator start(Database.BatchableContext bc) {
    	System.debug(LoggingLevel.INFO,'Entered Batchable start()...');
    	return  Database.getQueryLocator([select id from Group 
                                              where DeveloperName = 'Foo']);
    }
    
    public void execute(Database.BatchableContext bc, List<Group> scope) {
    	System.debug(LoggingLevel.INFO,'Entered Batchable execute()...');
    	System.enqueueJob(new FooBatchableQueueable());
    	System.debug(LoggingLevel.INFO,'within Batchable execute(), after enqueuing the job...');
    }
    public void finish(Database.BatchableContext bc) {
    	System.schedule('FooSchedulable','0 0 0 1 1 ?', new FooSchedulable());
    	System.debug(LoggingLevel.INFO,'within Batchable finish(), after scheduling');
    }
    
    public void execute(QueueableContext qc) {
    	System.debug(LoggingLevel.INFO,'reached Queueable execute()');
    }
    

    public class FooSchedulable implements Schedulable {
    	public void execute(SchedulableContext sc) {
    		System.debug(LoggingLevel.INFO,'reached Schedulable execute()');
    	}   	
    }
}

And the testmethod

@isTest
private with sharing class FooBatchableQueueableTest {
    
    @isTest private static void testBatchableQueueable() {
    	insert new Group(DeveloperName='Foo', name='Foo', type='Regular');
    	Test.startTest();
    	Database.executeBatch(new FooBatchableQueueable());
    	Test.stoptest();
    	//	Async batchable should execute, then queueable,  
        //      then schedulable. Or do they? See debug log
    }
}

And, what does happen?

  1. The batchable start() and execute() execute fine.
  2. The execute() calls System.enqueueJob(..)
  3. The Queueable job starts, and its execute() method is invoked. See the debug Log
  4. The batchable finish() method executes. It does a System.schedule() on a new object.
  5. The schedulable’s execute does not start.

Debug log

Entered Batchable start()…
Entered Batchable execute()…
within Batchable execute(), after enqueuing the job…
reached Queueable execute()
Entered Batchable finish()…
within Batchable finish(), after scheduling

Conclusion

  • Both the batchable and the queueable, as async transactions are executed “synchronously” once the Test.stopTest() is reached in the testmethod.
  • You definitely cannot assume that the batch finish() will execute before the queueable execute().
  • The constructor for the schedulable class will get invoked, but not its execute() method. You can see no debug log from within the schedulable’s execute().
  • You will need to explicitly test the schedulable by mocking the environment prior to its scheduling and then invoking in a separate testmethod.

Batchables-Limits Exceptions

A bad thing happened the other day. Here was the sequence:

  1. Batch job started. start() method returned 5000 + Opportunity rows.
  2. Database.stateful used to record an internal log of activity for subsequent posting in the finish() method.
  3. Each batch of 200 was passed to execute() method.
  4. Execute method added bits to the stateful log (a string variable).
  5. Batch 21 (out 28) blew up on a Limits Heap Size exception. Blow up continued on batches 22-28.
  6. finish() method started and took value from Database.Stateful variable and persisted to Sobject Log__c (s).
  7. AND HERE IS WHERE BAD THINGS HAPPENEDfinish() method started a “finalize” batch job, passing a list of sobject IDs that had exceptions in any previous batch execute. The finalize batch job (i.e. the chained batch job), made updates to all Opportunities that weren’t selected from the previous batches start() method and weren’t already marked as exceptions. In my case, these Opportunities were marked as closed lost.

So .. because the Opportunities in batches 21-28 were never processed and never marked with an exception (because of the uncatchable Limits exception), the chained (second) batch job blithely assumed that the Opportunities in batches 21-28 had never been fetched in the previous batch job’s start() method. Hence, perfectly good Opportunities got marked as closed lost.

Uh-oh.


So, what should I have done differently?

First, I wrongly assumed that a Limits exception would terminate the entire batch job, not just the currently running execute() batch.

And, since of this misconception, the finish() method unconditionally executes without knowing if all of the batches passed without uncaught exceptions. And, any work the finish() method performs that involves DML-type work, including scheduling a subsequent chained job may lead to incorrect behavior.

  1. The finish() method has access to the BatchableContext and can get, via getJobId(), the AsyncApexJob that represents the batch job.
  2. AsyncApexJob has a field NumberOfErrors that identifies how many batches are in error. If greater than zero, appropriate business logic should be applied.

Of course, the Limits Exception needs to be avoided in the first place by taking a different approach to stateful logging. I’ll investigate this in a subsequent post (but don’t stay up waiting for it!)

Batchable class – test data – callouts: How to test?

The short hand title for this post might also be System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out

Here’s the problem I recently faced:

  1. Batchable class that does callouts in the start() method.
  2. Testmethod needs to setup some test data before the batchable class executes

As you probably know if you are reading this ..

  • Testmethods can’t do callouts, you have to mock out the callout response using Test.setMock(..)
  • Async tasks like Database.executeBatch() or @future methods don’t execute in a test context until the Test.stopTest() method.

Naively, I constructed this basic testmethod:

insert new Account(name = '00test'); // create test data here
Test.startTest();
  Test.setMock(HttpCalloutMock.class, myMultiMockObject); // setup of multiMockObject omitted for clarity
  MyBatchableClass bClass = new MyBatchableClass();
  Database.executeBatch(bClass,2);  // scope is 2
Test.stopTest();  // SFDC executes the async task here
System.assert(...) //verify results

When you execute this test, you get System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out

I then tried to setup the test data in a different context like this:

System.runAs(someUserOtherThanRunningUser) {
   insert new Account(name = '00test'); // create test data here
}
Test.startTest();
  Test.setMock(HttpCalloutMock.class, myMultiMockObject); // setup of multiMockObject omitted for clarity
  MyBatchableClass bClass = new MyBatchableClass();
  Database.executeBatch(bClass,2);  // scope is 2
Test.stopTest();  // SFDC executes the async task here
System.assert(...) //verify results

When you execute this test, you get System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out

Aargh. SFDC doc as of V30 wasn’t much help here. Much Googling ensued. Thanks to tomlogic for the solution essence.

Here is what you have to do –avoid the use of Database.executeBatch() in the testmethod if one of the batchable class’s start(), execute() or finish() methods does a callout. Thus, you have to invoke the start(), execute(), finish() methods explicitly to verify your batchable class logic and get code coverage.

insert new Account(name = '00test'); // create test data here
Test.startTest();
  Test.setMock(HttpCalloutMock.class, myMultiMockObject); // setup of multiMockObject omitted for clarity
  Database.BatchableContext 	bc;
  MyBatchableClass bClass = new MyBatchableClass();
  
  // we execute the start() and prepare results for execute()
  // in my use case, start() does the callout; 
  // thus the testmethod mocks the results of the callout (assumed here to be accounts)
  // setup of custom Iterable and Iterator not shown
  MyIterable  itrbl = (MyIterable)bclass.start();  //start() returns an iterable, in my case, a custom iterable. Note the casting
  MyIterator  itrator = (MyIterator) itrbl.iterator();  // continue simulation of start() by constructing the iterator
  List<Account> aScopeList = new List<Account> ();      // create scope for execute() by iterating against the result of the mocked callout
  while (itrator.hasNext()) {
     aScopeList.add(itrator.next());

  // Now invoke execute() w/ Account list built via mocked callout invoked by start()
  bClass.execute(bc,aScopeList);

  // Finally, invoke finish()
  bClass.finish(bc);
Test.stopTest();  // SFDC executes the async task here
System.assert(...) //verify results

To recap, this approach accomplishes the following:

  1. It will test code coverage as start(), execute(), and finish() are all invoked as if SFDC were invoking them by Database.executeBatch().
  2. It avoids the System.CalloutException: You have uncommitted work pending. Please commit or rollback before calling out
  3. It allows you to test against the returned data from the mocked callout(s) thus isolating your testmethod to predictable responses.
  4. It tests your custom iterator, if you are using one
  5. It only requires a bit more setup to individually invoke start(), execute(), and finish() plus manual construction of what execute() would get if database.executeBatch() were called