1. Java Unit Testing
2. A Simple Unit Test
3. Assert Methods
4. Matchers
5. Testing for Exceptions
6. Stub, Mock and Proxy Testing
7. Subclass Mock Objects
8. Mock Testing isn't Always Enough
9. IO Testing
10. Servlet Unit Testing
11. Unit Testing with Dependency Injection Containers
12. Database Unit Testing - Rollback Transactions
13. Database Unit Testing - CRUD Testing
14. Database Unit Testing - Test Data Creation
15. Running Unit Tests With Ant
16. Running Unit Tests With Maven
17. Running Unit Tests Inside IntelliJ IDEA
18. Code Coverage
19. Design for Testability
20. Test First Development

• When Mocks aren't Enough

There is no doubt that unit testing with stubs and mock objects is useful. In several situations mock testing is easier than testing with the real collaborators of a class. But there are situations where mock object testing isn't enough.

When Mocks aren't Enough

There are cases where a unit depends on more than the interface of its collaborator. Situations where a mock object cannot precisely imitate the real implementation. One such case is classes that use JDBC. DAO classes for instance. When testing a DAO you may use a mock java.sql.Connection and java.sql.ResultSet object with the DAO to check that the DAO makes the correct JDBC calls. It is not visible from the value returned by the DAO if it remembered to call ResultSet.close() and Connection.close() before returning.

The problem with JDBC is that the behaviour of the methods in the JDBC interfaces isn't always 100% clearly defined. This results in slight variations of behaviour between JDBC drivers. Even for interface behaviour that is unambiguously defined the driver implementor may not follow the specification. Some JDBC methods may not even be implemented in the driver.

The above situation was exactly the case for the GenericDao class in Mr. Persister. The method executeUpdate(String sql, Object[] parameters) should throw an exception if the number of parameters (the ? -characters) in the sql string is not the same as the number of parameters in the parameters array. Ideally the JDBC driver would detect that when calling the PreparedStatement.executeUpdate() method. If we were to mock the PreparedStatement used by the GenericDao, this is most likely what the mock would do - throw an SQLException. Unfortunately the HSQLDB driver doesn't do that. Instead it just returns an empty ResultSet. Not very intuitive when debugging, that missing prepared statement parameters results in an empty ResultSet. One would think that there was something wrong with the sql. Therefore we decided to compare the PreparedStatement.getParameterMetaData().getParameterCount() with parameters.length before calling PreparedStatement.executeUpdate(). If the numbers were not equal the Mr. Persister API would throw an exception. This works with the HSQLDB driver, but the MySQL driver haven't implemented the PreparedStatement.getParameterMetaData() method. Instead it throws an exception. So the HSQLDB driver didn't work with the first solution, and the MySQL driver didn't work with the second.

Fortunately the MySQL driver does throw an exception if the ?-signs and the parameters set doesn't match when calling PreparedStatement.executeUpdate(). The solution, or workaround rather, was to check if the driver used is the MySQL driver. If it is, no parameter count check is made. If it is not the MySQL driver, a parameter count check is performed. Not an ideal situation but reality is far from always ideal.

The point of this story is: The issues with the differing implementations of the JDBC driver for the two databases, could not have been caught with ordinary mock object testing. A naive mock object would behave according to the JDBC specification. JDBC is probably not the only case where interface implementations may have unspecified behaviour or side effects that a naive mock object wouldn't have caught. So, sometimes you need to test your units with the real collaborator and not just a mock.

My dynamic mock object testing API, Butterfly Testing Tools is able to make assertions about what methods were called on real collaborators. Instead of creating a mock for a collaborator interface, you can create a proxy for a real collaborator instance. The proxy has an associated mock object that can check what methods were called on the proxy. That way you can use the real collaborator proxy as a mock as well as the real collaborator. You can also stub return values on the proxy, thus making the proxy behave like a stupid stub. You can switch between stubbed return values and real collaborator calls (+ return values) on the fly.