基于Centos7.6安裝CDH6集群

前提：所有cdh相关安装文件均放在master节点（cdh节点）的/root/cdh目录下
在这里插入图片描述

配置环境（所有节点）
1.1配置主机名
hostnamectl set-hostname cdh
hostnamectl set-hostname cdh01
hostnamectl set-hostname cdh02
hostnamectl set-hostname cdh03
注意：这四条命令各自在各自机器上执行，除了这四条命令，下面命令需在所有机器上执行。

1.2配置hosts文件
echo -e ‘10.0.24.6 cdh\n10.0.20.2 cdh01\n10.0.20.7 cdh02\n10.0.20.10 cdh03’ >> /etc/hosts

cat /etc/hosts
在这里插入图片描述

1.3关闭selinux
setenforce 0

1.4关闭防火墙
关闭防火墙
systemctl stop firewalld.service
关闭开机自启动
systemctl disable firewalld.service
查看防火墙状态
systemctl status firewalld.service
在这里插入图片描述

1.5配置免密登陆
ssh-keygen -t rsa
然后一路回车
在这里插入图片描述

ssh-copy-id -p6323 cdh
ssh-copy-id -p6323 cdh01
ssh-copy-id -p6323 cdh02
ssh-copy-id -p6323 cdh03
注意：这四条命令要在所有主机上执行，担心有的人不理解，特此提醒。

选择yes，然后会让你输入主机ssh连接的密码

注意：因为我用的是云服务器，之前服务器被黑，所以将ssh端口改为了6323。如果你未改过ssh端口，命令直接打ssh-copy-id cdh即可，若是换为了其它端口，将6323换为其它端口号即可。下面所有涉及-p6323指定端口号的操作均一样，就不一一介绍了。
在这里插入图片描述

设置authorized_keys权限（只是为了安全，可不设置）
chmod 600 /root/.ssh/authorized_keys

1.6配置时间同步
查看是否安装了ntp服务
rpm -qa | grep ntp
在这里插入图片描述

图中说明已经安装了ntp服务，若是没安装，则不会有任何返回

若是没有ntp服务，执行以下命令安装
yum install -y ntp

编辑ntp.conf
vim /etc/ntp.conf
加入以下内容（未测试，因使用腾讯云服务器，该部分被注释，添加的为云服务时间）
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

上面内容不行，可尝试下边内容
disable monitor
server time1.tencentyun.com iburst
server time2.tencentyun.com iburst
server time3.tencentyun.com iburst
server time4.tencentyun.com iburst
server time5.tencentyun.com iburst
interface ignore wildcard
interface listen eth0

启动ntp服务
systemctl start ntpd
设置开机自启动
systemctl enable ntpd
同步服务器时间
ntpdate -u 0.centos.pool.ntp.org
将硬件时钟调整为与目前的系统时钟一致
hwclock --systohc

1.7安装JDK
jdk需要安装在/usr/java目录下
mkdir /usr/java
cd /usr/java

拷贝安装包到当前目录（注意最后那个点是命令的一部分）
scp -P6323 cdh:/root/cdh/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm .

安装
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

安装完之后会出现jdk1.8.0_181-cloudera
在这里插入图片描述

配置环境变量
echo ‘export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera’ >> /etc/profile
echo ‘export JRE_HOME= ${JAVA_HOME}/jre' >> /etc/profile echo 'export CLASSPATH=.:$ {JAVA_HOME}/lib: ${JRE_HOME}/lib:$ CLASSPATH’ >> /etc/profile
echo ‘export JAVA_PATH= ${JAVA_HOME}/bin:$ {JRE_HOME}/bin’ >> /etc/profile
echo ‘export PATH= $P A T H :$ {JAVA_PATH}’ >> /etc/profile
查看环境变量
cat /etc/profile
在这里插入图片描述

使环境变量生效
source /etc/profile

验证
java -version
在这里插入图片描述

1.8禁用透明大页
查看
cat /sys/kernel/mm/transparent_hugepage/defrag
在这里插入图片描述
cat /sys/kernel/mm/transparent_hugepage/enabled

echo never > /sys/kernel/mm/transparent_hugepage/defrag
执行完命令后，再查看

echo never > /sys/kernel/mm/transparent_hugepage/enabled
执行完命令后再查看

永久关闭（将命令加入开机启动）
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/defrag’ >> /etc/rc.d/rc.local
echo ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local

1.9修改linux swappiness参数
cat /proc/sys/vm/swappiness
在这里插入图片描述
sysctl vm.swappiness=10

修改生效
sysctl -p

永久修改（加入开机启动）
echo ‘vm.swappiness=10’>> /etc/sysctl.conf
cat /etc/sysctl.conf
在这里插入图片描述

2.安装mysql（master节点，安装目录/usr/local）
2.1下载mysql安装源
cd /usr/local
wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
在这里插入图片描述
2.2安装mysql安装源
yum -y localinstall mysql57-community-release-el7-11.noarch.rpm

2.3在线安装mysql（这里取消了gpgkey验证）
yum -y install mysql-community-server --nogpgcheck

2.4启动mysql
systemctl start mysqld

2.5设置开机启动
systemctl enable mysqld
systemctl daemon-reload

2.6登陆mysql
寻找mysql密码
cat /var/log/mysqld.log
在这里插入图片描述

登陆
mysql -uroot -p’nl,pAIO4<5uX’

修改密码
ALTER USER ‘root’@‘localhost’ IDENTIFIED BY ‘P@ss123123’;
在这里插入图片描述
设置远程登陆
GRANT ALL PRIVILEGES ON . TO ‘root’@‘%’ IDENTIFIED BY ‘P@ss123123’ WITH GRANT OPTION;

2.7修改mysql配置文件/etc/my.cnf
源文件备份
mv /etc/my.cnf /etc/my.cnf.bk

创建新文件
vim /etc/my.cnf

添加以下内容
[client]
port=3306
socket=/var/lib/mysql/mysql.sock
[mysqld]
port=3306
user=mysql
socket=/var/lib/mysql/mysql.sock
basedir=/var/lib/mysql
datadir=/var/lib/mysql
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
transaction-isolation = READ-COMMITTED
character-set-server=utf8
collation-server=utf8_general_ci
lower_case_table_names=1
sql_mode=“STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO”

Disabling symbolic-links is recommended to prevent assorted security risks;

to do so, uncomment this line:

symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace ‘/var/lib/mysql/mysql_binary_log’ with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

InnoDB settings

innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

重启mysql
systemctl restart mysqld

2.8创建元数据库
重新登陆mysql
mysql -uroot -pP@ss123123

###降低密码安全策略验证等级强度
###set global validate_password_policy=LOW;

###修改密码最短位数
###set global validate_password_length=6;

创建元数据库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO ‘scm’@‘%’ IDENTIFIED BY ‘scm@123’;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO ‘amon’@‘%’ IDENTIFIED BY ‘amon@123’;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO ‘rman’@‘%’ IDENTIFIED BY ‘rman@123’;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO ‘hue’@‘%’ IDENTIFIED BY ‘hue@123’;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO ‘metastore’@‘%’ IDENTIFIED BY ‘metastore@123’;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO ‘sentry’@‘%’ IDENTIFIED BY ‘sentry@123’;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO ‘nav’@‘%’ IDENTIFIED BY ‘nav@123’;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO ‘navms’@‘%’ IDENTIFIED BY ‘navms@123’;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO ‘oozie’@‘%’ IDENTIFIED BY ‘oozie@123’;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hive.* TO ‘hive’@‘%’ IDENTIFIED BY ‘hive@123’;
CREATE DATABASE cm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON cm.* TO ‘cm’@‘%’ IDENTIFIED BY ‘cm@123’;

2.9安装mysql-connector（所有节点）
mkdir /root/cdh1
cd /root/cdh1

分发安装文件到所有节点
scp -P6323 cdh:/root/cdh/mysql-connector-java-8.0.22-1.el7.noarch.rpm .

安装
yum install -y /root/cdh1/mysql-connector-java-8.0.22-1.el7.noarch.rpm
安装完成后，在/usr/share/java下有如下文件
在这里插入图片描述

3.安装cloudera-manager
3.1制作yum源（master节点）
安装httpd
yum install -y httpd
在这里插入图片描述
启用httpd
systemctl start httpd
systemctl enable httpd
启用成功后，浏览器输入ip可以看到（cdh为我在本地hosts配置的映射）

mkdir -p /var/www/html/cloudera-repos/cm6
创建完此文件夹，可以看到如图

在这里插入图片描述

cp /root/cdh/*.rpm /var/www/html/cloudera-repos/cm6/
cp /root/cdh/allkeys.asc /var/www/html/cloudera-repos/cm6/

cd /var/www/html/cloudera-repos/cm6

安装createrepo（制作yum源的包）
yum install -y createrepo

制作yum源
createrepo .

在这里插入图片描述

同时该目录下会多一个文件夹
在这里插入图片描述

创建文件cloudera-manager.repo（需要所有节点都有这个文件）
vim /etc/yum.repos.d/cloudera-manager.repo
加入以下内容
[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://cdh/cloudera-repos/cm6
gpgkey=https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera
gpgcheck=0
enabled=1
autorefresh=0
type=rpm-md

分发到其它节点
scp -P6323 /etc/yum.repos.d/cloudera-manager.repo cdh01:/etc/yum.repos.d/
scp -P6323 /etc/yum.repos.d/cloudera-manager.repo cdh02:/etc/yum.repos.d/
scp -P6323 /etc/yum.repos.d/cloudera-manager.repo cdh03:/etc/yum.repos.d/

清楚yum缓存
yum clean all

将安装包进行缓存
yum makecache
在这里插入图片描述

这三个出现说明成功

3.2安装cloudera-manager
在master节点执行
yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

在其它子节点执行（子节点只安装客户端就可以）
yum install -y cloudera-manager-agent

3.3parcel文件设置（master节点）
移动parcel文件
cp /root/cdh/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /root/cdh/manifest.json /opt/cloudera/parcel-repo/

mkdir -p /var/www/html/cdh6
cp /root/cdh/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /var/www/html/cdh6/
cp /root/cdh/manifest.json /var/www/html/cdh6/

执行校验
cd /opt/cloudera/parcel-repo
sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk ‘{ print $1 }’ > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
在这里插入图片描述