java


Why is my application running faster in IntelliJ compared to command-line?


We have an application that imports a large amount of files by splitting the data and sorting it. When running the JUnit test case, the whole process takes about 16 minutes.
Same test, done with mvn clean test -Dtest=MyTest run in 34 minutes.
We are calling in to /bin/sort to sort the files. The sort seems to be taking longer. I don't understand what is different.
Looking at IntelliJ it runs with
/Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/Contents/Home/bin/java -Didea.launcher.port=7532 -Didea.launcher.bin.path=/Applications/IntelliJ IDEA 10.app/bin -Dfile.encoding=UTF-8 -classpath %classhpath% com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 xxx.IntTestImportProcess,testImportProcess
I am on OS X. All the classes are injected using Spring. What are some possible suggestions are theories at what is behind this performance gain in IntelliJ? The tests are identical. I can't share all of the code because there is just so much. But I can add any detail if requested.
Here is my main class and how I am running both.
public static void main(String... args) throws IOException {
if(args.length != 2) {
System.out.println("Usage: \n java -jar client.jar spring.xml data_file");
System.exit(1);
}
ApplicationContext applicationContext = new FileSystemXmlApplicationContext(args[0]);
PeriodFormatter formatter = new PeriodFormatterBuilder()
.appendMinutes()
.appendSuffix("minute", "minutes")
.appendSeparator(" and ")
.appendSeconds()
.appendSuffix("second", "seconds")
.toFormatter();
URI output = (URI) applicationContext.getBean("workingDirectory");
File dir = new File(output);
if(dir.exists()) {
Files.deleteDirectoryContents(dir.getCanonicalFile());
}
else {
dir.mkdirs();
}
ImportProcess importProcess = applicationContext.getBean(ImportProcess.class);
long start = System.currentTimeMillis();
File file = new File(args[1]);
importProcess.beginImport(file);
Period period = new Period(System.currentTimeMillis() - start); // in milliseconds
System.out.println(formatter.print(period.toPeriod()));
}
I have decided to remove JUnit and just use a main() method. The result are exactly the same. IntelliJ is again. Here is the crazy log.
With IntelliJ
DEBUG [ main] 2011-08-18 13:05:16,259 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/usage]
DEBUG [ main] 2011-08-18 13:06:09,546 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/customer]
With java -jar
DEBUG [ main] 2011-08-18 12:10:16,726 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/usage]
DEBUG [ main] 2011-08-18 12:15:55,893 [er.DelimitedTextUnixDataSorter] Sorting file [/Users/amirraminfar/Desktop/import-process/customer]
The sort command is
sort -t' ' -f -k32,32f -k18,18f -k1,1n
As you can see above, sorting in Intellij take 1 minutes but in java -jar takes 5 minutes!
Update
I ran everything using /Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/Contents/Home/bin/java and the sorting still takes well over 5+ mins.
Thank you everybody for helping. It turns out IntelliJ starts sort with LANG=C. Mac OS X terminal sorts by default in UTF8 which explains the performance loss. Hopefully this answer will help somebody.
Is mvn clean doing a rebuild of the project? Is the run under IDEA not doing that? Does building the project with Maven take 18 minutes (I wouldn't be surprised if it did, given that Maven is the absolute pits)?
If the answers to all these questions are 'yes', then i think you have a conclusion.
The solution is to take Maven to the woods, shoot it, then bury it in an unmarked grave.
A guess more than substantiated answer:
A lot may depend on I/O buffering. Sort over 500K records is going to output a lot of data, so the right buffer size may matter a lot. I think the tty is typically line buffered, so it is going to do 500K read & write ops, and the IDE may simply read in much larger buffers.
Additionally, it is possible that OSX has process or I/O scheduling which heavily favours GUI apps over console ones (which could be detected through being bound to a tty), so it might be that you have to wait & idle a lot more time from the console than from within the IDE.
This question's not exactly properly defined and far from clear.
Are you saying you read out a list of files in a folder using Java IO and then pass it to external process to sort them?
Sounds as a bit strange solution to me, but anyway, this has to be memory related. Connect to both apps using JConsole and look for clues in charts.
mvn is probably launching a JVM with different options than your IDE.
Real differences between "java -server" and "java -client"? can cause substantial differences in performance for long running applications, but sometimes differences in gc flags can cause performance differences when your app uses memory in different ways.
If your process is memory bound and so thrashes at some memory level, heap flags like -Xmx can drastically affect performance too. Memory profiling can easily rule this in or out.
To diagnose these differences, look at your mvn configuration files to figure out how it is launching a JVM, and look at your IDE java app launcher configuration.
Have you tried using a profiler such as VisualVM to see what is is the bottleneck? Also compare the CPU usage graphs of your computer (Mac should have some system monitor). Maybe the process is blocking at some point and not working effectively, which could be seen as a CPU usage graph of different shape.

Related Links

How to convert TIS-620 string to UTF-8 string in Java? [closed]
How to do a check before allowing editing of a given row in a JTable
Logging user actions
Meaning of serial port parameters in Java?
Netbeans GUI Designer & Fixed-Size Application Panels
When using Spring Security, what is the proper way to obtain current username (i.e. SecurityContext) information in a bean?
Open Browser window from Java program
comparison of code performance, threaded versus non-threaded
Java GUI LayoutManagers
Write XML file (using XStream) to filesystem in Java
Static initializer in Java
JMS Message Driven Bean worker synchronization
is there a way to force a transactional rollback without encountering an exception?
how to workaround a java image scaling bug
Best implementation for hashCode method
Java User Interface Specification

Categories

HOME
java
fluentd
alfresco
jsp-tags
programming-languages
jxls
retrofit
convolution
webrequest
virtualization
communication
spring-kafka
nstableview
google-apps-marketplace
timeout
foselasticabundle
pugjs
evopdf
plunker
opencover
hammerspoon
dxf
rundeck
p-value
cas
srcset
r-raster
google-cloud-endpoints-v2
pim
stringtemplate
lxd
accelerate-framework
android-kernel
android-nestedscrollview
revolution-slider
restlet
multilingual
sas-visual-analytics
galen
elmah
texmaker
http-live-streaming
apple-news
xml-documentation
sage-one
csound
diagnostics
media-player
linode
django-scheduler
messenger
word-vba-mac
savon
moveit
hover-over
bluegiga
freelancer.com-api
execute
teamcity-8.0
marching-cubes
cannon.js
computer-algebra-systems
kcachegrind
wyam
drawbitmap
sortedlist
p2
wordpress-theme-customize
gulp-less
google-reader
navigationservice
floating-point-precision
ccss
comaddin
orchardcms-1.7
factory-method
javaspaces
chrono
spyware
deobfuscation
coredump
actionview
datareader
wise
microsoft-virtualization
uimenucontroller
anti-piracy
ugc
windows-live-messenger
paul-graham

Resources

Database Users
RDBMS discuss
Database Dev&Adm
javascript
java
csharp
php
android
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App