启动spark历史服务失败问题处理

1.场景
执行启动spark历史服务器 start-history-server.sh报错
[root@manager file]# $SPARK_HOME/sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /home/bigdata/spark/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-manager.out
[root@manager file]# more /home/bigdata/spark/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-manager.out
Spark Command: /usr/java/jdk1.8.0_112/bin/java -cp /home/bigdata/spark/conf/:/home/bigdata/spark/jars/*:/home/bigdata/hadoop/e
tc/hadoop/ -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=zook_node1:2181,zook_node2:2181,zook_node3:2181 
-Dspark.deploy.zookeeper.dir=/home/bigdata/zookeeper -Xmx1g org.apache.spark.deploy.history.HistoryServer
========================================
2023-09-18 16:46:29,971 INFO history.HistoryServer: Started daemon with process name: 18867@manager
2023-09-18 16:46:29,974 INFO util.SignalUtils: Registering signal handler for TERM
2023-09-18 16:46:29,975 INFO util.SignalUtils: Registering signal handler for HUP
2023-09-18 16:46:29,975 INFO util.SignalUtils: Registering signal handler for INT
2023-09-18 16:46:30,192 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-ja
va classes where applicable
2023-09-18 16:46:30,271 INFO spark.SecurityManager: Changing view acls to: root
2023-09-18 16:46:30,272 INFO spark.SecurityManager: Changing modify acls to: root
2023-09-18 16:46:30,273 INFO spark.SecurityManager: Changing view acls groups to: 
2023-09-18 16:46:30,273 INFO spark.SecurityManager: Changing modify acls groups to: 
2023-09-18 16:46:30,274 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with vi
ew permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify 
permissions: Set()
2023-09-18 16:46:30,360 INFO history.FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; group
s with admin permissions: 
2023-09-18 16:46:31,011 INFO util.log: Logging initialized @1850ms to org.sparkproject.jetty.util.log.Slf4jLog
2023-09-18 16:46:31,062 INFO server.Server: jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e
53ef315a249d63ba9332; jvm 1.8.0_112-b15
2023-09-18 16:46:31,084 INFO server.Server: Started @1924ms
2023-09-18 16:46:31,130 INFO server.AbstractConnector: Started ServerConnector@5ae76500{HTTP/1.1, (http/1.1)}{0.0.0.0:18080}
2023-09-18 16:46:31,131 INFO util.Utils: Successfully started service 'HistoryServerUI' on port 18080.
2023-09-18 16:46:31,156 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e094740{/,null,AVAILABLE,@Spark}
2023-09-18 16:46:31,157 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@24a1c17f{/json,null,AVAILABLE,@Spar
k}
2023-09-18 16:46:31,158 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73511076{/api,null,AVAILABLE,@Spark
}
2023-09-18 16:46:31,167 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@514eedd8{/static,null,AVAILABLE,@Sp
ark}
2023-09-18 16:46:31,168 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1a2e2935{/history,null,AVAILABLE,@S
park}
2023-09-18 16:46:31,170 INFO history.HistoryServer: Bound HistoryServer to 0.0.0.0, and started at http://manager:18080
Exception in thread "main" java.io.FileNotFoundException: Log directory specified does not exist: hdfs://mycluster/spark_logs
        at org.apache.spark.deploy.history.FsHistoryProvider.startPolling(FsHistoryProvider.scala:291)
        at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:236)
        at org.apache.spark.deploy.history.FsHistoryProvider.start(FsHistoryProvider.scala:413)
        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:311)
        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://mycluster/spark_logs
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1756)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
        at org.apache.spark.deploy.history.FsHistoryProvider.startPolling(FsHistoryProvider.scala:281)
        ... 4 more
[root@manager file]# 

2.日志分析
关键是这一句：Log directory specified does not exist: hdfs://mycluster/spark_logs
意思是：找不到指定文件

3.解决方案
需要预先创建文件即可

[root@manager conf]#  hadoop dfs -mkdir hdfs://mycluster/spark_logs
[root@manager conf]#  hdfs dfs -chmod 777   /home/bigdata/spark/conf/spark_logs

4.重新启动验证
[root@manager file]# $SPARK_HOME/sbin/stop-history-server.sh 
stopping org.apache.spark.deploy.history.HistoryServer
[root@manager file]# $SPARK_HOME/sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /home/bigdata/spark/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-manager.out
[root@manager file]# tail -f /home/bigdata/spark/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-manager.out
2023-09-18 17:10:07,780 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@56102e1c{/api,null,AVAILABLE,@Spark}
2023-09-18 17:10:07,789 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@51f49060{/static,null,AVAILABLE,@Spark}
2023-09-18 17:10:07,790 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@687a762c{/history,null,AVAILABLE,@Spark}
2023-09-18 17:10:07,791 INFO history.HistoryServer: Bound HistoryServer to 0.0.0.0, and started at http://manager:18080
2023-09-18 17:10:07,952 INFO history.FsHistoryProvider: Parsing hdfs://mycluster/spark_logs/local-1695027842090.zstd for listing data...
2023-09-18 17:10:07,952 INFO history.FsHistoryProvider: Parsing hdfs://mycluster/spark_logs/local-1695027960132.zstd for listing data...
2023-09-18 17:10:08,694 INFO history.FsHistoryProvider: Finished parsing hdfs://mycluster/spark_logs/local-1695027842090.zstd
2023-09-18 17:10:08,694 INFO history.FsHistoryProvider: Finished parsing hdfs://mycluster/spark_logs/local-1695027960132.zstd
2023-09-18 17:10:08,700 INFO history.FsHistoryProvider: Parsing hdfs://mycluster/spark_logs/local-1695027671578.zstd.inprogress for listing data...
2023-09-18 17:10:08,863 INFO history.FsHistoryProvider: Finished parsing hdfs://mycluster/spark_logs/local-1695027671578.zstd.inprogress