Butterfly Container Performance
Jakob Jenkov |
Performance is often less relevant when wiring components together, since the wiring often only happen once, at startup time, or infrequently, like at the start of the processing of an HTTP request. Even when you use the DI container to wire up complex components repeatedly at later runtime, the instantiation and wiring time of those components is often only a fraction of the time these components take to execute their business logic.
Even if object instantiation is not often a hot spot in applications, it can still be useful to have an idea of the overhead DI adds compared to instantiation with 'new'. Especially when deciding whether or not to use DI inside code you know will be an application hot spot.
We have put together a small performance test to shed some light on the overhead added to instantiation by Butterfly Container. We even included Google Guice (another DI framework) just to have another similar product to compare with. Only you yourself can determine if the overheads of Butterfly (or Guice) are acceptable in your project. Hopefully these numbers can help you. The results are listed below, ordered after time:
NOTE: Over time this test suite will be expanded to include more instantiation scenarios like more complex object graphs, singletons etc. For now it only benchmarks simple no-arg constructor instantiation.
This test was performed using Java 5, Butterfly Container 1.9.11-beta, and Guice 1.0.
The test performs 10.000.000 instantiations of a TestProduct class using a no-arg constructor, for each DI mechanism. All tests were repeated 10 times to allow the JVM to warm up. We have also tried to change the order of the tests when run to see if the sequence changed the result, but it didn't seem to make any noticable differences. Here are the numbers of the first and last of the 10 test runs:
-- NEW BASED INSTANTIATION ---------------------- Java new : 100%, 687 Butterfly Java : 245%, 1688 Butterfly Java 2 : 136%, 937 Guice Provider : 866%, 5953 Guice Provider 2 : 420%, 2891 -- REFLECTION BASED INSTANTIATION ---------------- Java Reflection : 386%, 2656 Butterfly Script : 891%, 6125 Butterfly Script 2 : 707%, 4860 Guice Reflection : 1846%, 12687 Guice Reflection 2 : 1394%, 9578 ================================================= -- NEW BASED INSTANTIATION ---------------------- Java new : 100%, 750 Butterfly Java : 229%, 1719 Butterfly Java 2 : 135%, 1016 Guice Provider : 806%, 6047 Guice Provider 2 : 391%, 2937 -- REFLECTION BASED INSTANTIATION ---------------- Java Reflection : 362%, 2719 Butterfly Script : 845%, 6343 Butterfly Script 2 : 672%, 5047 Guice Reflection : 1706%, 12797 Guice Reflection 2 : 1310%, 9828 =================================================
The numbers pretty much speak for themselves, but here are a few comments anyways:
There seems to be a small relative speedup in all mechanisms except 'new' from the first to the last run, but the speedup is not revolutionary. The observant reader may have noticed that this relative speedup seems to have come from 'new' being slower in the last run. Not the other mechanisms having sped up. In fact they have slowed down too. Just not as much as 'new'. We have seen this relative speedup in many other test runs, where 'new' didn't actually get slower, but where the other mechanisms got a little faster. Still, the speedups where small.
The first five tests use instantiation mechanisms based on Java's new, meaning they do not use reflection. The last three tests use reflection based instantiation mechanisms. Therefore the first five tests are of course faster than the last three.
If you check the test code (below) you will notice that two different instantiation methods were used: The first method asks the Butterfly Container / Guice Injector directly for an instance of some type. This requires a lookup inside the container / injector to find the factory/provider matching the requested type. The second method obtains the factory / provider from the container / injector once, then call that directly for all instantiations during the test. Not surprisingly it is faster to use the factory / provider directly than go through the container. That might be worth remembering if using DI in a hot spot.
It seems Butterfly Container has a factor 2 to 4 performance advantage over Guice. According to Bob Lee (Guice main developer) this is caused by some checks Guice does internally, for instance checking for circular dependencies. Butterfly solves these problems in a different way, so it doesn't have these checks at instantiation time.
The tests were run singlethreadedly. When we get around to designing a multithreaded test, we will publish the results here to.
The full test code is included below. To run this test yourself copy the class below. Add jenkov-butterfly-container-1.9.10-beta.jar and guice-1.0.jar to the classpath. Then call main(...).
import com.google.inject.*;
import com.jenkov.container.TestProduct;
import com.jenkov.container.Container;
import com.jenkov.container.IContainer;
import com.jenkov.container.itf.factory.IGlobalFactory;
import com.jenkov.container.script.ScriptFactoryBuilder;
import com.jenkov.container.java.JavaFactoryBuilder;
import com.jenkov.container.java.JavaFactory;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
/**
*/
public class PerformanceTestGuice {
public static int iterationsPerTest = 10 * 1000 * 1000;
public static int iterationsOfAllTests = 10;
public static void main(String[] args)
throws NoSuchMethodException, InvocationTargetException,
InstantiationException, IllegalAccessException {
/* NEW INSTANCE TEST */
for(int i=0; i<iterationsOfAllTests; i++){
long javaNewTime = javaNewTest();
long javaReflectionTime = javaReflectionTest();
long butterflyScriptTime = butterflyScriptTest();
long butterflyScript2Time= butterflyScriptTest2();
long butterflyJavaTime = butterflyJavaTest();
long butterflyJava2Time = butterflyJava2Test();
long guiceTime = guiceTest();
long guice2Time = guiceTest2();
long guiceProviderTime = guiceProviderTest();
long guiceProvider2Time = guiceProvider2Test();
long javaReflectionTimeRatio =
(javaReflectionTime * 100) / javaNewTime;
long butterflyJavaTimeRatio =
(butterflyJavaTime * 100) / javaNewTime;
long butterflyJava2TimeRatio =
(butterflyJava2Time * 100) / javaNewTime;
long butterflyScript2TimeRatio =
(butterflyScript2Time * 100) / javaNewTime;
long butterflyScriptTimeRatio =
(butterflyScriptTime * 100) / javaNewTime;
long guiceTimeRatio =
(guiceTime * 100) / javaNewTime;
long guiceTime2Ratio =
(guice2Time * 100) / javaNewTime;
long guiceProviderTimeRatio =
(guiceProviderTime * 100) / javaNewTime;
long guiceProvider2TimeRatio =
(guiceProvider2Time * 100) / javaNewTime;
System.out.println("-- NEW BASED INSTANTIATION --------------");
System.out.println("Java new : 100%, "
+ javaNewTime);
System.out.println("Butterfly Java : "
+ butterflyJavaTimeRatio + "%, " + butterflyJavaTime);
System.out.println("Butterfly Java 2 : "
+ butterflyJava2TimeRatio + "%, " + butterflyJava2Time);
System.out.println("Guice Provider : "
+ guiceProviderTimeRatio + "%, " + guiceProviderTime);
System.out.println("Guice Provider 2 : "
+ guiceProvider2TimeRatio + "%, " + guiceProvider2Time);
System.out.println("-- REFLECTION BASED INSTANTIATION ------");
System.out.println("Java Reflection : "
+ javaReflectionTimeRatio + "%, " + javaReflectionTime);
System.out.println("Butterfly Script : "
+ butterflyScriptTimeRatio + "%, " + butterflyScriptTime);
System.out.println("Butterfly Script 2 : "
+ butterflyScript2TimeRatio + "%, " + butterflyScript2Time);
System.out.println("Guice Reflection : "
+ guiceTimeRatio + "%, " + guiceTime);
System.out.println("Guice Reflection 2 : "
+ guiceTime2Ratio + "%, " + guice2Time);
System.out.println("========================================");
}
}
protected static long javaNewTest(){
long newStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
new TestProduct();
}
long newEnd = System.currentTimeMillis();
return newEnd - newStart;
}
protected static long javaReflectionTest()
throws NoSuchMethodException, InvocationTargetException,
IllegalAccessException, InstantiationException {
Constructor constructor = TestProduct.class.getConstructor(null);
long newStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
constructor.newInstance(null);
}
long newEnd = System.currentTimeMillis();
return newEnd - newStart;
}
protected static long butterflyScriptTest(){
IContainer container = new Container();
ScriptFactoryBuilder builder = new ScriptFactoryBuilder(container);
builder.addFactory("script = * com.jenkov.container.TestProduct();");
long butterflyScriptStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
container.instance("script");
}
long butterflyScriptEnd = System.currentTimeMillis();
return butterflyScriptEnd - butterflyScriptStart;
}
protected static long butterflyScriptTest2(){
IContainer container = new Container();
ScriptFactoryBuilder builder = new ScriptFactoryBuilder(container);
builder.addFactory("script = * com.jenkov.container.TestProduct();");
IGlobalFactory<TestProduct> factory =
container.getFactory("script");
long butterflyScriptStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
factory.instance();
}
long butterflyScriptEnd = System.currentTimeMillis();
return butterflyScriptEnd - butterflyScriptStart;
}
protected static long butterflyJavaTest(){
IContainer container = new Container();
JavaFactoryBuilder javaBuilder =
new JavaFactoryBuilder(container);
javaBuilder.addFactory("java", TestProduct.class,
new JavaFactory(){
public Object instance(Object... parameters) {
return new TestProduct();
}
});
long butterflyJavaStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
container.instance("java");
}
long butterflyJavaEnd = System.currentTimeMillis();
return butterflyJavaEnd - butterflyJavaStart;
}
protected static long butterflyJava2Test(){
IContainer container = new Container();
JavaFactoryBuilder javaBuilder =
new JavaFactoryBuilder(container);
javaBuilder.addFactory("java", TestProduct.class,
new JavaFactory(){
public Object instance(Object... parameters) {
return new TestProduct();
}
});
IGlobalFactory<TestProduct> javaFactory =
container.getFactory("java");
long butterflyJavaStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
javaFactory.instance();
}
long butterflyJavaEnd = System.currentTimeMillis();
return butterflyJavaEnd - butterflyJavaStart;
}
protected static long guiceTest(){
/* GUICE TEST */
Injector injector = Guice.createInjector(new Module(){
public void configure(Binder binder) {
binder.bind(TestProduct.class);
}
}) ;
long guiceStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
injector.getInstance(TestProduct.class);
}
long guiceEnd = System.currentTimeMillis();
return guiceEnd - guiceStart;
}
protected static long guiceTest2(){
/* GUICE TEST */
Injector injector = Guice.createInjector(new Module(){
public void configure(Binder binder) {
binder.bind(TestProduct.class);
}
}) ;
Provider<TestProduct> provider =
injector.getProvider(TestProduct.class);
long guiceStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
provider.get();
}
long guiceEnd = System.currentTimeMillis();
return guiceEnd - guiceStart;
}
protected static long guiceProviderTest(){
final Provider<TestProduct> provider =
new Provider<TestProduct>(){
public TestProduct get() {
return new TestProduct();
}
};
/* GUICE TEST */
Injector injector = Guice.createInjector(new Module(){
public void configure(Binder binder) {
binder.bind(TestProduct.class).toProvider(provider);
}
}) ;
long guiceStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
injector.getInstance(TestProduct.class);
}
long guiceEnd = System.currentTimeMillis();
return guiceEnd - guiceStart;
}
protected static long guiceProvider2Test(){
Injector injector = Guice.createInjector(new Module(){
public void configure(Binder binder) {
binder.bind(TestProduct.class).toProvider(new
Provider<TestProduct>() {
public TestProduct get() {
return new TestProduct();
}
});
}
}) ;
Provider<TestProduct> provider =
injector.getProvider(TestProduct.class);
long guiceStart = System.currentTimeMillis();
for(int i=0; i< iterationsPerTest; i++){
provider.get();
}
long guiceEnd = System.currentTimeMillis();
return guiceEnd - guiceStart;
}
| Tweet | |
Jakob Jenkov | |
