spark Structured报错解决

报错，不想看原因的直接去解决方案试试

Exception in thread "main" java.lang.IllegalArgumentException: Pathname /C:/Users/Administrator/AppData/Local/Temp/1/temporary-611514af-8dc5-4b20-9237-e5f2d21fdf88/metadata from hdfs://master:8020/C:/Users/Administrator/AppData/Local/Temp/1/temporary-611514af-8dc5-4b20-9237-e5f2d21fdf88/metadata is not a valid DFS filename.
	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:197)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
	at org.apache.spark.sql.execution.streaming.StreamMetadata$.read(StreamMetadata.scala:51)
	at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:122)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.<init>(MicroBatchExecution.scala:49)
	at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:258)
	at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:299)
	at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:296)
	at com.gugu.book.basespark.chapter08.StructuredNetworkWordCount$.main(StructuredNetworkWordCount.scala:25)
	at com.gugu.book.basespark.chapter08.StructuredNetworkWordCount.main(StructuredNetworkWordCount.scala)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

看报错信息好像是要赋权，路径不存在，但是这个路径有点古怪
1

hdfs://master:8020/C:/Users/Administrator/AppData/Local/Temp/1/temporary-611514af-8dc5-4b20-9237-e5f2d21fdf88/metadata is not a valid DFS filename

按道理不应该出现C:/。。。。这个是我本地的一个地址，但是却拼接到了hdfs文件系统下，应该是什么配置的有问题
顺着报错信息

只需要checkpointRoot修改就行这个地址来源于checkpointLocation
org.apache.spark.sql.streaming.StreamingQueryManager#createQuery

也就是说直接改变“checkpointLocation”的值就行

解决方案

在start之前设置

            .option("checkpointLocation", "file:///D:\\applicationfiles\\data\\kafka")
1

解决，完美
环境问题，在windows下有些默认位置拼接到了hdfs上