spark streaming在yarn-client或者本地local執(zhí)行的時(shí)候，由于長時(shí)間運(yùn)行日志會(huì)越來越多泊脐，查看起來很不方便，可以利用log4j對(duì)spark streaming的日志進(jìn)行配置。
配置步驟如下：
- 1.新建driver端的log4j配置文件driver-log4j.properties
log4j.rootLogger =INFO,stdout,I,E
### output to console ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm} %5p %t %c{2}:%L - %m%n
### output error to files ####
log4j.appender.E=org.apache.log4j.DailyRollingFileAppender
log4j.appender.E.layout=org.apache.log4j.PatternLayout
log4j.appender.E.layout.conversionPattern=%-d{yyyy-MM-dd HH:mm:ss} %5p %t %c{2}:%L - %m%n
log4j.appender.E.maxFileSize=100MB
log4j.appender.E.maxBackupIndex=5
log4j.appender.E.Append = true
log4j.appender.E.Threshold = ERROR
log4j.appender.E.file=/home/root/log/streaming/stderror.log
log4j.appender.E.encoding=UTF-8
### output info to files ####
log4j.appender.I=org.apache.log4j.DailyRollingFileAppender
log4j.appender.I.layout=org.apache.log4j.PatternLayout
log4j.appender.I.layout.conversionPattern=%-d{yyyy-MM-dd HH:mm:ss} %5p %t %c{2}:%L - %m%n
log4j.appender.I.maxFileSize=100MB
log4j.appender.I.maxBackupIndex=5
log4j.appender.I.Append = true
log4j.appender.I.Threshold = INFO
log4j.appender.I.file=/home/root/log/streaming/stdout.log
log4j.appender.I.encoding=UTF-8
以上分別配置了控制臺(tái)壶熏，info日志輸出和error的輸出
執(zhí)行腳本如下：
BASEDIR=$(dirname $(readlink -f $0))
echo $BASEDIR
nohup spark-submit \
--class com.MainFunc \
--master local[8] \
--driver-memory 2g \
--executor-memory 3g \
--conf spark.eventLog.enabled=false \
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:driver-log4j.properties \
--files $BASEDIR/driver-log4j.properties \
$BASEDIR/../jars/my-streaming-1.0.jar > /dev/null 2>&1 &
注意結(jié)尾，輸出到 /dev/null浦译。
