log4j


yarn logs not showing until I kill my spark streaming application


How can BigInsights on cloud 4.2 Basic and Enterprise be configured for spark-streaming to log when deployed to yarn cluster?
I'm setting up my log4j.properties like this:
log4j.rootLogger=INFO, rolling
log4j.appender.rolling=org.apache.log4j.RollingFileAppender
log4j.appender.rolling.layout=org.apache.log4j.PatternLayout
log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n
log4j.appender.rolling.immediateFlush=true
log4j.appender.rolling.maxFileSize=50MB
log4j.appender.rolling.maxBackupIndex=5
log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log
log4j.appender.rolling.encoding=UTF-8
log4j.logger.org.apache.spark=WARN
log4j.logger.org.eclipse.jetty=WARN
I'm then using spark-submit as follows:
spark-submit --class "MovieRating" \
--master yarn \
--deploy-mode cluster \
--properties-file spark_streaming.conf \
--files ./log4j.properties \
--conf "spark.driver.extraJavaOptions=-Diop.version=4.2.0.0,-Dlog4j.configuration=log4j.properties" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" \
--packages cloudant-labs:spark-cloudant:1.6.4-s_2.10 \
./movie-rating_2.10-1.0.jar
However, if I try to inspect the logs:
$ yarn logs -applicationId application_1480680664469_0040
/app-logs/snowch/logs/application_1480680664469_0040 does not have any log files.
If I then kill the yarn job with yarn application -kill ... the logs will show up with the yarn logs ... command.
Update:
I've tried hardcoding log4j.appender.rolling.file=/var/log/spark/spark.log but this file is not created. I've also tried running a org.apache.log4j.net.SimpleSocketServer on the mastermanager and then using a org.apache.log4j.net.SocketAppender but nothing gets logged to this either. I don't think my log4j.properties is getting applied to my spark job. It seems that the default log4j properties are still getting applied.

Related Links

Tomcat log each application into separate file
slf4j-log4j bridge with log4j 2.8.1
logging to ELK stack from karaf
Changing Log4J configuration in Tibco BW/Designer does not have the desired effect
Write to SYSLOG with RootLogger OFF
How to write separate log files for each error transactions in Java using log4j
log4j on jboss - subpackage level
converting log4j into log4j2. With RollingFileAppender, size and file name pattern
Log4j NoSQL/Mongo appender in Tibco BusinessWorks 5
jcl104-over-slf4j version 1.4.2 is safe to be replaced by jcl-over-slf4j 1.7.2
Log4j2 main ERROR AmazonRollingRandomAccessFile contains an invalid element or attribute
Is it possible to set an SMTP header in SMTPAppender?
log4j rotate and compress files
Log4J 2 failing to configure
log4j not writing logs to a file
Log4J : Console Error in log4j.xml

Categories

HOME
maven
testing
osgi
clips
netsuite
omnet++
fluentd
heroku
react-virtualized
appx
mean-stack
tizen-web-app
amazon-ecs
dxl
callback
angular-ui-bootstrap
datastax-java-driver
quickbooks
google-cloud-spanner
alignment
gz
orchardcms
fallback
msp430
try-catch
serverless-framework
uitypeeditor
highlight.js
http-status-code-504
normalizr
swingx
tdd
fish
url-scheme
strncpy
compatibility
primitive
sql-server-2012-express
bcd
exuberant-ctags
pdb
gtrendsr
host
suricata
hockeyapp
temporary-files
trim
awt
broadcastreceiver
ruby-on-rails-3.1
gulp-sourcemaps
grid.mvc
nxlog
isbn
heightmap
http-redirect
flickr-api
hibernate-tools
jquery-validate
setter
akka-cluster
sqlbulkcopy
nbconvert
watchconnectivity
abcpdf9
savon
file-writing
disque
hover-over
yaws
ipconfig
associative-array
kendonumerictextbox
cartesian-product
cyclomatic-complexity
twirl
ami
iiviewdeckcontroller
skos
lustre
phalanger
asp.net-web-api-odata
client-side-templating
websocket4net
elliptic-curve
openlaszlo
geos
xsockets.net
inbox
hungarian-algorithm
spring-validator
drools-planner
rdoc
runas
subgurim-maps
infobox
iphone-web-app
suppress
multi-tier
premature-optimization
gacutil
mdac
lzh

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App