java


Why is my application running faster in IntelliJ compared to command-line?


We have an application that imports a large amount of files by splitting the data and sorting it. When running the JUnit test case, the whole process takes about 16 minutes.
Same test, done with mvn clean test -Dtest=MyTest run in 34 minutes.
We are calling in to /bin/sort to sort the files. The sort seems to be taking longer. I don't understand what is different.
Looking at IntelliJ it runs with
/Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/Contents/Home/bin/java -Didea.launcher.port=7532 -Didea.launcher.bin.path=/Applications/IntelliJ IDEA 10.app/bin -Dfile.encoding=UTF-8 -classpath %classhpath% com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 xxx.IntTestImportProcess,testImportProcess
I am on OS X. All the classes are injected using Spring. What are some possible suggestions are theories at what is behind this performance gain in IntelliJ? The tests are identical. I can't share all of the code because there is just so much. But I can add any detail if requested.
Here is my main class and how I am running both.
public static void main(String... args) throws IOException {
if(args.length != 2) {
System.out.println("Usage: \n java -jar client.jar spring.xml data_file");
System.exit(1);
}
ApplicationContext applicationContext = new FileSystemXmlApplicationContext(args[0]);
PeriodFormatter formatter = new PeriodFormatterBuilder()
.appendMinutes()
.appendSuffix("minute", "minutes")
.appendSeparator(" and ")
.appendSeconds()
.appendSuffix("second", "seconds")
.toFormatter();
URI output = (URI) applicationContext.getBean("workingDirectory");
File dir = new File(output);
if(dir.exists()) {
Files.deleteDirectoryContents(dir.getCanonicalFile());
}
else {
dir.mkdirs();
}
ImportProcess importProcess = applicationContext.getBean(ImportProcess.class);
long start = System.currentTimeMillis();
File file = new File(args[1]);
importProcess.beginImport(file);
Period period = new Period(System.currentTimeMillis() - start); // in milliseconds
System.out.println(formatter.print(period.toPeriod()));
}
I have decided to remove JUnit and just use a main() method. The result are exactly the same. IntelliJ is again. Here is the crazy log.
With IntelliJ
DEBUG [ main] 2011-08-18 13:05:16,259 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/usage]
DEBUG [ main] 2011-08-18 13:06:09,546 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/customer]
With java -jar
DEBUG [ main] 2011-08-18 12:10:16,726 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/usage]
DEBUG [ main] 2011-08-18 12:15:55,893 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/customer]
The sort command is
sort -t' ' -f -k32,32f -k18,18f -k1,1n
As you can see above, sorting in Intellij take 1 minutes but in java -jar takes 5 minutes!
Update
I ran everything using /Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/Contents/Home/bin/java and the sorting still takes well over 5+ mins.
Thank you everybody for helping. It turns out IntelliJ starts sort with LANG=C. Mac OS X terminal sorts by default in UTF8 which explains the performance loss. Hopefully this answer will help somebody.
Is mvn clean doing a rebuild of the project? Is the run under IDEA not doing that? Does building the project with Maven take 18 minutes (I wouldn't be surprised if it did, given that Maven is the absolute pits)?
If the answers to all these questions are 'yes', then i think you have a conclusion.
The solution is to take Maven to the woods, shoot it, then bury it in an unmarked grave.
A guess more than substantiated answer:
A lot may depend on I/O buffering. Sort over 500K records is going to output a lot of data, so the right buffer size may matter a lot. I think the tty is typically line buffered, so it is going to do 500K read & write ops, and the IDE may simply read in much larger buffers.
Additionally, it is possible that OSX has process or I/O scheduling which heavily favours GUI apps over console ones (which could be detected through being bound to a tty), so it might be that you have to wait & idle a lot more time from the console than from within the IDE.
This question's not exactly properly defined and far from clear.
Are you saying you read out a list of files in a folder using Java IO and then pass it to external process to sort them?
Sounds as a bit strange solution to me, but anyway, this has to be memory related. Connect to both apps using JConsole and look for clues in charts.
mvn is probably launching a JVM with different options than your IDE.
Real differences between "java -server" and "java -client"? can cause substantial differences in performance for long running applications, but sometimes differences in gc flags can cause performance differences when your app uses memory in different ways.
If your process is memory bound and so thrashes at some memory level, heap flags like -Xmx can drastically affect performance too. Memory profiling can easily rule this in or out.
To diagnose these differences, look at your mvn configuration files to figure out how it is launching a JVM, and look at your IDE java app launcher configuration.
Have you tried using a profiler such as VisualVM to see what is is the bottleneck? Also compare the CPU usage graphs of your computer (Mac should have some system monitor). Maybe the process is blocking at some point and not working effectively, which could be seen as a CPU usage graph of different shape.

Related Links

Is there clean way to pass context data to #Asynchronous ejb call?
numetriclabz / numandroidchart how to space out the bars in a stacked bar chart / stacked percentage column chart
How to get first element of list in jsp [duplicate]
Android Studio 2.0 using external projects with jar and source files
How I can load a file and save this in a list of objects in Android?
Java GUI checkbox control logic needed
Create an inclinometer Android
connect intellij idea to sql server database
JUNG - Large graphs and vertex dragging
Evosuite: failure in executing test cases
Third option on JCheckBox? [duplicate]
How to recursively fill a tree with 4 children below each node in Java
how to use java code connection remote hadoop hdfs?
Maven Dependency & Java Service Provider Interface
How do I remove the background from image in Java using OpenCV with only single Image?
App won't build? DebugResources error?

Categories

HOME
log4j
listview
single-sign-on
google-play
relative-path
relayjs
getelementsbytagname
programming-languages
tizen-web-app
cross-browser
ezpublish
echarts
networkx
azure-storage-tables
podio
directx
virtualization
elasticsearch-hadoop
cloudkit
spring-xd
google-cloud-ml
apache-cayenne
gz
size
event-handling
fallback
highlight.js
http-status-code-504
interop
shared-hosting
visual-composer
excel-2007
devops
wijmo
android-widget
smb
sparse-matrix
wpfdatagrid
applozic
log4js-node
webkitspeechrecognition
r-raster
tinymce-4
uisplitview
usbserial
lxd
swisscomdev
space-complexity
sqlite2
user-controls
galen
reportingservices-2005
youcompleteme
multipeer-connectivity
python-c-api
paxos
typescript1.8
aurelia-validation
snmptrapd
chord-diagram
color-picker
log4c
bonobo
home-directory
connect-by
markojs
flash-cs5
essence
make-install
livequery
tcpserver
xna-4.0
teamcity-8.0
wordml
tableau-online
ibaction
responsive-images
remobjects
jsapi
csquery
jscript.net
codeigniter-url
java.util.concurrent
markers
google-reader
dml
jubula
valuechangelistener
flexmojos
spring-io
lcs
http-unit
jspinclude
ticoredatasync
heartbeat
actionview
infrastructure
user-friendly
ti-dsp
data-driven
yagni
pascal-fc
gacutil
port-number
dbisam
django-notification

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App