• Hadoop（一）

    目录

    Hadoop运行环境搭建

    模板虚拟机环境准备

    克隆虚拟机

    编写集群分发脚本

    SSH无密登录配置

    Hadoop运行环境搭建

    模板虚拟机环境准备

    1.关闭防火墙，关闭防火墙开机自启

    1. [doudou@localhost ~]$ su root
    2. Password: 
    3. [root@localhost doudou]# systemctl stop firewalld
    4. [root@localhost doudou]# systemctl disable firewalld.service
    5. Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
    6. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

    小贴士：

    在xshell中怎样添加复制粘贴功能

    1.点击工具，选择里面的按键对应

    2.点击新建，输入组合键，类型选择菜单，菜单选择复制或粘贴

     

    2.创建一个用户，并修改用户密码

    1. [root@localhost hadoop100]# useradd doudou
    2. [root@localhost hadoop100]# passwd doudou
    3. 更改用户 doudou 的密码 。
    4. 新的 密码：
    5. 无效的密码： 密码少于 8 个字符
    6. 重新输入新的 密码：
    7. passwd：所有的身份验证令牌已经成功更新。

     3.配置用户具有root权限，方便后期加sudo执行root命令

    [root@localhost doudou]# vim /etc/sudoers
    1. ## Allow root to run any commands anywhere 
    2. root    ALL=(ALL)       ALL
    3. doudou  ALL=(ALL)       NOPASSWD:ALL

    4.在/opt目录下创建文件夹，并修改所属组和用户组

    1. [root@localhost doudou]# mkdir /opt/module
    2. [root@localhost doudou]# mkdir /opt/software
    1. [root@localhost doudou]# chown doudou:doudou /opt/module
    2. [root@localhost doudou]# chown doudou:doudou /opt/software
    1. [root@localhost doudou]# cd /opt/
    2. [root@localhost opt]# ll
    3. total 0
    4. drwxr-xr-x. 2 doudou doudou 6 Aug  1 08:07 module
    5. drwxr-xr-x. 2 root   root   6 Oct 30  2018 rh
    6. drwxr-xr-x. 2 doudou doudou 6 Aug  1 08:08 software

    5.卸载虚拟机自带的JDK

    1. [root@localhost opt]# cd ~
    2. [root@localhost ~]# rpm -qa | grep -i java | xargs -nl rpm -e --nodeps
    3. xargs: invalid number for -n option
    4. Usage: xargs [OPTION]... COMMAND INITIAL-ARGS...
    5. Run COMMAND with arguments INITIAL-ARGS and more arguments read from input.
    6.  
    7. Mandatory arguments to long options are mandatory for short options too.
    8. Non-mandatory arguments are indicated by [square brackets]
    9.   -0, --null                   Items are separated by a null, not whitespace.
    10.                                Disables quote and backslash processing
    11.   -a, --arg-file=FILE          Read arguments from FILE, not standard input
    12.   -d, --delimiter=CHARACTER    Input items are separated by CHARACTER, not by
    13.                                blank space. Disables quote and backslash
    14.                                processing
    15.   -E END                       If END occurs as a line of input, the rest of
    16.                                the input is ignored.
    17.   -e [END], --eof[=END]        Equivalent to -E END if END is specified.
    18.                                Otherwise, there is no end-of-file string
    19.   --help                       Print a summary of the options to xargs.
    20.   -I R                         same as --replace=R (R must be specified)
    21.   -i,--replace=[R]             Replace R in initial arguments with names
    22.                                read from standard input. If R is
    23.                                unspecified, assume {}
    24.   -L,-l, --max-lines=MAX-LINES Use at most MAX-LINES nonblank input lines per
    25.                                command line
    26.   -l                           Use at most one nonblank input line per
    27.                                command line
    28.   -n, --max-args=MAX-ARGS      Use at most MAX-ARGS arguments per command
    29.                                line
    30.   -P, --max-procs=MAX-PROCS    Run up to max-procs processes at a time
    31.   -p, --interactive            Prompt before running commands
    32.   --process-slot-var=VAR       Set environment variable VAR in child
    33.                                processes
    34.   -r, --no-run-if-empty        If there are no arguments, run no command.
    35.                                If this option is not given, COMMAND will be
    36.                                run at least once.
    37.   -s, --max-chars=MAX-CHARS    Limit commands to MAX-CHARS at most
    38.   --show-limits                Show limits on command-line length.
    39.   -t, --verbose                Print commands before executing them
    40.   --version                    Print the version number
    41.   -x, --exit                   Exit if the size (see -s) is exceeded
    42.  
    43. Report bugs to .

    6.重启虚拟机

    [root@localhost ~]# reboot

    克隆虚拟机

    1.克隆模板虚拟机，注意要选择完整克隆

    2.修改克隆机的IP和名称

    1. [doudou@localhost ~]$ su root
    2. 密码：
    3. [root@localhost doudou]# ifconfig
    4. ens33: flags=4163  mtu 1500
    5.         inet 192.168.149.131  netmask 255.255.255.0  broadcast 192.168.149.255
    6.         inet6 fe80::fe9e:c2f3:2656:bcbd  prefixlen 64  scopeid 0x20
    7.         inet6 fe80::b406:2768:a50c:e991  prefixlen 64  scopeid 0x20
    8.         inet6 fe80::afc4:b77f:4ea9:33af  prefixlen 64  scopeid 0x20
    9.         ether 00:0c:29:01:27:af  txqueuelen 1000  (Ethernet)
    10.         RX packets 287  bytes 61246 (59.8 KiB)
    11.         RX errors 0  dropped 0  overruns 0  frame 0
    12.         TX packets 192  bytes 31239 (30.5 KiB)
    13.         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    14.  
    15. lo: flags=73  mtu 65536
    16.         inet 127.0.0.1  netmask 255.0.0.0
    17.         inet6 ::1  prefixlen 128  scopeid 0x10
    18.         loop  txqueuelen 1000  (Local Loopback)
    19.         RX packets 64  bytes 5568 (5.4 KiB)
    20.         RX errors 0  dropped 0  overruns 0  frame 0
    21.         TX packets 64  bytes 5568 (5.4 KiB)
    22.         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    23.  
    24. virbr0: flags=4099  mtu 1500
    25.         inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
    26.         ether 52:54:00:ae:e3:4b  txqueuelen 1000  (Ethernet)
    27.         RX packets 0  bytes 0 (0.0 B)
    28.         RX errors 0  dropped 0  overruns 0  frame 0
    29.         TX packets 0  bytes 0 (0.0 B)
    30.         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    31.  
    32. [root@localhost doudou]# vim /etc/hostname
    33. [root@localhost doudou]# reboot

    编写集群分发脚本

    1. [doudou@hadoop101 ~]$ cd
    2. [doudou@hadoop101 ~]$ cd bin
    3. [doudou@hadoop101 bin]$ vim xsync
    1. #脚本内
    2. #!/bin/bash
    3. if [ $# -lt 1 ]
    4. then
    5.   echo Not Enough Arguement!
    6.   exit
    7. fi
    8.  
    9. for host in hadoop101 hadoop102 hadoop103
    10. do
    11.   echo =================== $host ==================
    12.   for file in $@
    13.   do
    14.     if [ -e $file ]
    15.     then
    16.       pdir=$(cd -P $(dirname $file); pwd)
    17.       fname=$(basename $file)
    18.       ssh $host "mkdir -p $pdir"
    19.       rsync -av $pdir/$fname $host:$pdir
    20.     else
    21.       echo $file does not exists!
    22.     fi
    23.   done    
    24. done     
    25.           
    1. [doudou@hadoop101 bin]$ chmod 777 xsync
    2. [doudou@hadoop101 bin]$ ll
    3. total 4
    4. -rwxrwxrwx. 1 doudou root 431 Aug  2 02:46 xsync

    SSH无密登录配置

    [doudou@hadoop101 ~]$ cd .ssh/

    [doudou@hadoop101 .ssh]$ ssh-keygen -t rsa

    1. [doudou@hadoop101 .ssh]$ ssh-copy-id hadoop101
    2. [doudou@hadoop101 .ssh]$ ssh-copy-id hadoop102
    3. [doudou@hadoop101 .ssh]$ ssh-copy-id hadoop103

  • 相关阅读:
    【C/C++】C编程技巧
    Rust Yew应用开发的事件初探
    Qt 停靠布局QDockWidget使用
    走进Elasticsearch
    C++：string的模拟实现
    如何获得淘宝/天猫app商品详情原数据API数据
    在typora中利用正则表达式，批量处理图片
    Go 语言变量
    day25-Listener监听器
    使用Flink完成流数据统计
  • 原文地址：https://blog.csdn.net/qq_61741696/article/details/126111655