一段时间后，stop-dfs.sh关不掉Hadoop3.1.3集群，stop-hbase.sh关不掉HBase集群

一段时间后，Hadoop3.1.3集群关不掉

问题

一段时间后，使用 stop-dfs.sh和 stop-yarn.sh关不掉Hadoop 3.1.3集群
是因为Hadoop默认进程ID存储在 /tmp， /tmp里的 冇被访问的文件 会被定期清理

解决

修改进程ID存储路径

1、编辑配置文件

vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh
1

2、修改 配置文件中的 进程ID储存目录，保存退出

# export HADOOP_PID_DIR=/tmp
export HADOOP_PID_DIR=/opt/module/hadoop/pid/
1
2

3、分发配置文件

rsync.py $HADOOP_HOME/etc/hadoop/hadoop-env.sh
1

4、Hadoop3群起脚本

#!/bin/bash
if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit ;
fi
case $1 in
"start")
        echo " =================== 启动 hadoop集群 ==================="

        echo " --------------- 启动 hdfs ---------------"
        ssh hadoop105 "/opt/module/hadoop/sbin/start-dfs.sh"
        echo " --------------- 启动 yarn ---------------"
        ssh hadoop106 "/opt/module/hadoop/sbin/start-yarn.sh"
        echo " --------------- 启动 historyserver ---------------"
        ssh hadoop105 "/opt/module/hadoop/bin/mapred --daemon start historyserver"
;;
"stop")
        echo " =================== 关闭 hadoop集群 ==================="

        echo " --------------- 关闭 historyserver ---------------"
        ssh hadoop105 "/opt/module/hadoop/bin/mapred --daemon stop historyserver"
        echo " --------------- 关闭 yarn ---------------"
        ssh hadoop106 "/opt/module/hadoop/sbin/stop-yarn.sh"
        echo " --------------- 关闭 hdfs ---------------"
        ssh hadoop105 "/opt/module/hadoop/sbin/stop-dfs.sh"
;;
*)
    echo "Input Args Error..."
;;
esac
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

5、使用kill关闭每台机的Hadoop相关进程（DataNode、NodeManager、DataNode等）

[root@hadoop107 ~]# jps | grep -v Jps
5041 NodeManager
4931 DataNode
[root@hadoop107 ~]# kill -9 5041 4931
1
2
3
4

6、启动

[hjw@hadoop105 hadoop]$ hdp.sh start
 =================== 启动 hadoop集群 ===================
 --------------- 启动 hdfs ---------------
Starting namenodes on [hadoop105]
hadoop105: WARNING: /opt/module/hadoop/pid/ does not exist. Creating.
Starting datanodes
hadoop107: WARNING: /opt/module/hadoop/pid/ does not exist. Creating.
hadoop106: WARNING: /opt/module/hadoop/pid/ does not exist. Creating.
Starting secondary namenodes [hadoop106]
 --------------- 启动 yarn ---------------
Starting resourcemanager
Starting nodemanagers
 --------------- 启动 historyserver ---------------
1
2
3
4
5
6
7
8
9
10
11
12
13

7、查看进程文件

[hjw@hadoop105 hadoop]$ cluster.py ls $HADOOP_HOME/pid/
ssh hadoop105 'ls /opt/module/hadoop/pid/'
hadoop-hjw-datanode.pid
hadoop-hjw-historyserver.pid
hadoop-hjw-namenode.pid
hadoop-hjw-nodemanager.pid
ssh hadoop106 'ls /opt/module/hadoop/pid/'
hadoop-hjw-datanode.pid
hadoop-hjw-nodemanager.pid
hadoop-hjw-resourcemanager.pid
hadoop-hjw-secondarynamenode.pid
ssh hadoop107 'ls /opt/module/hadoop/pid/'
hadoop-hjw-datanode.pid
hadoop-hjw-nodemanager.pid
1
2
3
4
5
6
7
8
9
10
11
12
13
14

8、查看进程

[hjw@hadoop105 hadoop]$ jps.py 
---------------hadoop105----------------
31651 DataNode
32138 JobHistoryServer
31530 NameNode
31967 NodeManager
---------------hadoop106----------------
14394 SecondaryNameNode
14285 DataNode
14541 ResourceManager
14655 NodeManager
---------------hadoop107----------------
6694 NodeManager
6583 DataNode
1
2
3
4
5
6
7
8
9
10
11
12
13
14

HBase2.0.5启动一段时间后也关不掉

stop-hbase.sh关不掉HBase集群，原因也是进程ID文件默认在/tmp

1、编辑配置文件

vim $HBASE_HOME/conf/hbase-env.sh
1

2、修改进程ID存储路径

# The directory where pid files are stored. /tmp by default.
# export HBASE_PID_DIR=/var/hadoop/pids
export HBASE_PID_DIR=/opt/module/hbase/pids
1
2
3

3、分发

rsync.py $HBASE_HOME/conf/hbase-env.sh
1

4、使用kill结束HBase进程

kill -9
1

5、启动HBase

start-hbase.sh
1

6、查看进程ID文件

[hjw@hadoop105 phoenix]$ ll /opt/module/hbase/pids
total 16
-rw-rw-r-- 1 hjw hjw  5 Dec  1 18:53 hbase-hjw-master.pid
-rw-rw-r-- 1 hjw hjw 30 Dec  1 18:53 hbase-hjw-master.znode
-rw-rw-r-- 1 hjw hjw  5 Dec  1 18:53 hbase-hjw-regionserver.pid
-rw-rw-r-- 1 hjw hjw 40 Dec  1 18:53 hbase-hjw-regionserver.znode
1
2
3
4
5
6