log4j


yarn logs not showing until I kill my spark streaming application


How can BigInsights on cloud 4.2 Basic and Enterprise be configured for spark-streaming to log when deployed to yarn cluster?
I'm setting up my log4j.properties like this:
log4j.rootLogger=INFO, rolling
log4j.appender.rolling=org.apache.log4j.RollingFileAppender
log4j.appender.rolling.layout=org.apache.log4j.PatternLayout
log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n
log4j.appender.rolling.immediateFlush=true
log4j.appender.rolling.maxFileSize=50MB
log4j.appender.rolling.maxBackupIndex=5
log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log
log4j.appender.rolling.encoding=UTF-8
log4j.logger.org.apache.spark=WARN
log4j.logger.org.eclipse.jetty=WARN
I'm then using spark-submit as follows:
spark-submit --class "MovieRating" \
--master yarn \
--deploy-mode cluster \
--properties-file spark_streaming.conf \
--files ./log4j.properties \
--conf "spark.driver.extraJavaOptions=-Diop.version=4.2.0.0,-Dlog4j.configuration=log4j.properties" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" \
--packages cloudant-labs:spark-cloudant:1.6.4-s_2.10 \
./movie-rating_2.10-1.0.jar
However, if I try to inspect the logs:
$ yarn logs -applicationId application_1480680664469_0040
/app-logs/snowch/logs/application_1480680664469_0040 does not have any log files.
If I then kill the yarn job with yarn application -kill ... the logs will show up with the yarn logs ... command.
Update:
I've tried hardcoding log4j.appender.rolling.file=/var/log/spark/spark.log but this file is not created. I've also tried running a org.apache.log4j.net.SimpleSocketServer on the mastermanager and then using a org.apache.log4j.net.SocketAppender but nothing gets logged to this either. I don't think my log4j.properties is getting applied to my spark job. It seems that the default log4j properties are still getting applied.

Related Links

Apache Kafka 0.8 log4j2.xml appender getting timeout error
Flink with High-availability with zookeeper: Submitted job is not acknowledged by Job manager
SoapUI does not change log file name/path after changes in soapui-log4j.xml
How to export lo4j from Ambari?
Log4j don't catch javax.el exception
'infinispan-core' not showing logs
Tomcat log each application into separate file
slf4j-log4j bridge with log4j 2.8.1
logging to ELK stack from karaf
Changing Log4J configuration in Tibco BW/Designer does not have the desired effect
Write to SYSLOG with RootLogger OFF
How to write separate log files for each error transactions in Java using log4j
log4j on jboss - subpackage level
converting log4j into log4j2. With RollingFileAppender, size and file name pattern
Log4j NoSQL/Mongo appender in Tibco BusinessWorks 5
jcl104-over-slf4j version 1.4.2 is safe to be replaced by jcl-over-slf4j 1.7.2

Categories

HOME
variables
date
zeromq
angular-material
xmpp
rsync
jira
yarn
sharepoint-designer
umd
ckeditor
indesign
communication
tomcat6
numeral.js
quicklook
postgres-xl
quickfix
ups
reactcsstransitiongroup
http-status-code-504
crystal-reports-2008
lucene.net
nas
conemu
autocad-plugin
kryo
telerik-reporting
angular2-aot
smb
xpages-ssjs
microsoft-sync-framework
spark-jobserver
django-storage
assistant
windowbuilder
google-cloud-nl
media-queries
javascriptcore
ecto
typed.js
rotational-matrices
libvpx
withings
objectlistview
impersonation
galen
auto-update
chain-builder
apache-fop
errordocument
celery-task
blogengine.net
tropo
font-size
windows-mobile-6.5
r-forge
removechild
dstu2-fhir
phishing
fody
measures
pagedlist
dataview
livequery
rhino-servicebus
goose
iad
django-unittest
ami
jscript.net
gui-test-framework
qcodo
navigationservice
elliptic-curve
system.net.webexception
xceed-datagrid
soundtouch
file-locking
batterylevel
isnullorempty
cascalog
code-cleanup
php-parser
stage
krl
actionview
android-sdk-2.1
radcombobox
sortable-tables
dmx512
sproutcore-2
gamma
preference
nsviewanimation
fixed-width
substrings
phonon
ugc

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App