log4j


yarn logs not showing until I kill my spark streaming application


How can BigInsights on cloud 4.2 Basic and Enterprise be configured for spark-streaming to log when deployed to yarn cluster?
I'm setting up my log4j.properties like this:
log4j.rootLogger=INFO, rolling
log4j.appender.rolling=org.apache.log4j.RollingFileAppender
log4j.appender.rolling.layout=org.apache.log4j.PatternLayout
log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n
log4j.appender.rolling.immediateFlush=true
log4j.appender.rolling.maxFileSize=50MB
log4j.appender.rolling.maxBackupIndex=5
log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log
log4j.appender.rolling.encoding=UTF-8
log4j.logger.org.apache.spark=WARN
log4j.logger.org.eclipse.jetty=WARN
I'm then using spark-submit as follows:
spark-submit --class "MovieRating" \
--master yarn \
--deploy-mode cluster \
--properties-file spark_streaming.conf \
--files ./log4j.properties \
--conf "spark.driver.extraJavaOptions=-Diop.version=4.2.0.0,-Dlog4j.configuration=log4j.properties" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" \
--packages cloudant-labs:spark-cloudant:1.6.4-s_2.10 \
./movie-rating_2.10-1.0.jar
However, if I try to inspect the logs:
$ yarn logs -applicationId application_1480680664469_0040
/app-logs/snowch/logs/application_1480680664469_0040 does not have any log files.
If I then kill the yarn job with yarn application -kill ... the logs will show up with the yarn logs ... command.
Update:
I've tried hardcoding log4j.appender.rolling.file=/var/log/spark/spark.log but this file is not created. I've also tried running a org.apache.log4j.net.SimpleSocketServer on the mastermanager and then using a org.apache.log4j.net.SocketAppender but nothing gets logged to this either. I don't think my log4j.properties is getting applied to my spark job. It seems that the default log4j properties are still getting applied.

Related Links

How to disable log4j default logging?
TIBCO MDM 9 : log4j:WARN No appenders could be found for logger (RulebaseCustomFunction)
Log4J DailyRollingFileAppender file name
How to configure JDBCAppender programmatically
how to use markerfilter with old log4j
How to setup log rotation in AEM web console?
Websphere 8.5.3, SLF4j Log4j, Multiple EARs, multiple log files not generating
How to configure logging level for JAWR using log4j.xml
In which classes we are not using Log4J
log4j - why am I getting this ludicrous amount of DEBUG even if there's no log4j.properties?
log4j 2.5 Rolling file appender does not release lock on files with Delete feature
configure log4j in dockerfile
No logs when tomcat is stopped
log4j doesn't print preparestatement sql log
Running a web application on Tomcat through Windows service
How can I assign a variable to a logger at log4j

Categories

HOME
netbeans
dictionary
json.net
spagobi
malloc
disassembler
windows-server
azure-media-services
uber-api
google-translate
offline
hapi
caml
iolanguage
vaadin7
worldwind
tostring
jqwidget
nodatime
jprofiler
visjs
tar
sylius
google-search-api
php-openssl
qhull
maquette
ejabberd-module
dbclient
sql-server-2012-express
fabric8
react-chartjs
nameservers
google-cloud-endpoints-v2
semantic-versioning
ghost4j
devextreme
dartium
mesos-chronos
fedex
jvm-languages
turbogears
babel-core
taffy
pearson
objective-c-swift-bridge
word-vba-mac
abcpdf9
deadbolt-2
grails-tomcat-plugin
gridpane
sts-springsourcetoolsuite
adxstudio-portals
multi-level
pcf
transmitfile
ubuntu-10.04
skype4py
atk4
django-unittest
spidermonkey
libressl
map-projections
ionic
arcanist
splash
event-bubbling
device-orientation
system.reflection
jsonpickle
ng-animate
coldbox
asp.net-dynamic-data
javafx-webengine
heisenbug
message-driven-bean
gnu-smalltalk
aqtime
plasma
funscript
broken-links
javaspaces
windows-phone-7.1.1
postgresql-performance
asyncfileupload
transactionscope
django-tagging
invite
appendto
oncheckedchanged
gallio
ti-dsp
anti-piracy

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App