• Apache Knox 2.0.0使用

    目录

    介绍

    使用

    gateway-site.xml

    users.ldif

    my_hdfs.xml

    my_yarn.xml

    其它

    介绍

            The Apache Knox Gateway is a system that provides a single point of authentication and access for Apache Hadoop services in a cluster. The goal is to simplify Hadoop security for both users (i.e. who access the cluster data and execute jobs) and operators (i.e. who control access and manage the cluster). The gateway runs as a server (or cluster of servers) that provide centralized access to one or more Hadoop clusters. In general the goals of the gateway are as follows:

    • Provide perimeter security for Hadoop REST APIs to make Hadoop security easier to setup and use
      • Provide authentication and token verification at the perimeter
      • Enable authentication integration with enterprise and cloud identity management systems
      • Provide service level authorization at the perimeter
    • Expose a single URL hierarchy that aggregates REST APIs of a Hadoop cluster
      • Limit the network endpoints (and therefore firewall holes) required to access a Hadoop cluster
      • Hide the internal Hadoop cluster topology from potential attackers

    使用

     解压后，目录如下：

    进入conf目录:

    文件说明：

    gateway-site.xml

    网关文件，修改如下:

    1.     <property>
    2.         <name>gateway.portname>
    3.         <value>18483value>
    4.         <description>The HTTP port for the Gateway.description>
    5.     property>
    6.     <property>
    7.         <name>gateway.pathname>
    8.         <value>my/mimivalue>
    9.         <description>The default context path for the gateway.description>
    10.     property>
    11.     <property>
    12.         <name>gateway.dispatch.whitelistname>
    13.         <value>.*$value>
    14.     property>

    users.ldif

    用户文件，全部如下

    1. # Licensed to the Apache Software Foundation (ASF) under one
    2. # or more contributor license agreements.  See the NOTICE file
    3. # distributed with this work for additional information
    4. # regarding copyright ownership.  The ASF licenses this file
    5. # to you under the Apache License, Version 2.0 (the
    6. # "License"); you may not use this file except in compliance
    7. # with the License.  You may obtain a copy of the License at
    8. #
    9. #     http://www.apache.org/licenses/LICENSE-2.0
    10. #
    11. # Unless required by applicable law or agreed to in writing, software
    12. # distributed under the License is distributed on an "AS IS" BASIS,
    13. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14. # See the License for the specific language governing permissions and
    15. # limitations under the License.
    16.  
    17. version: 1
    18.  
    19. # Please replace with site specific values
    20. dn: dc=hadoop,dc=apache,dc=org
    21. objectclass: organization
    22. objectclass: dcObject
    23. o: Hadoop
    24. dc: hadoop
    25.  
    26. # Entry for a sample people container
    27. # Please replace with site specific values
    28. dn: ou=people,dc=hadoop,dc=apache,dc=org
    29. objectclass:top
    30. objectclass:organizationalUnit
    31. ou: people
    32.  
    33. dn: ou=myhadoop,dc=hadoop,dc=apache,dc=org
    34. objectclass:top
    35. objectclass:organizationalUnit
    36. ou: myhadoop
    37.  
    38. # Entry for a sample end user
    39. # Please replace with site specific values
    40. dn: uid=guest,ou=people,dc=hadoop,dc=apache,dc=org
    41. objectclass:top
    42. objectclass:person
    43. objectclass:organizationalPerson
    44. objectclass:inetOrgPerson
    45. cn: Guest
    46. sn: User
    47. uid: guest
    48. userPassword:123456
    49.  
    50. dn: uid=myclient,ou=people,dc=hadoop,dc=apache,dc=org
    51. objectclass:top
    52. objectclass:person
    53. objectclass:organizationalPerson
    54. objectclass:inetOrgPerson
    55. cn: Myclient
    56. sn: Client
    57. uid: myclient
    58. userPassword:qwe
    59.  
    60. dn: uid=myhdfs,ou=myhadoop,dc=hadoop,dc=apache,dc=org
    61. objectclass:top
    62. objectclass:person
    63. objectclass:organizationalPerson
    64. objectclass:inetOrgPerson
    65. cn: myhdfs
    66. sn: myhdfs
    67. uid: myhdfs
    68. userPassword:123456
    69.  
    70. dn: uid=myyarn,ou=myhadoop,dc=hadoop,dc=apache,dc=org
    71. objectclass:top
    72. objectclass:person
    73. objectclass:organizationalPerson
    74. objectclass:inetOrgPerson
    75. cn: myyarn
    76. sn: myyarn
    77. uid: myyarn
    78. userPassword:123
    79.  
    80.  
    81. # entry for sample user admin
    82. dn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
    83. objectclass:top
    84. objectclass:person
    85. objectclass:organizationalPerson
    86. objectclass:inetOrgPerson
    87. cn: Admin
    88. sn: Admin
    89. uid: admin
    90. userPassword:123456
    91.  
    92. dn: uid=root,ou=myhadoop,dc=hadoop,dc=apache,dc=org
    93. objectclass:top
    94. objectclass:person
    95. objectclass:organizationalPerson
    96. objectclass:inetOrgPerson
    97. cn: Admin
    98. sn: Admin
    99. uid: root
    100. userPassword:123456
    101.  
    102. # entry for sample user sam
    103. dn: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org
    104. objectclass:top
    105. objectclass:person
    106. objectclass:organizationalPerson
    107. objectclass:inetOrgPerson
    108. cn: sam
    109. sn: sam
    110. uid: sam
    111. userPassword:sam-password
    112.  
    113. # entry for sample user tom
    114. dn: uid=tom,ou=people,dc=hadoop,dc=apache,dc=org
    115. objectclass:top
    116. objectclass:person
    117. objectclass:organizationalPerson
    118. objectclass:inetOrgPerson
    119. cn: tom
    120. sn: tom
    121. uid: tom
    122. userPassword:tom-password
    123.  
    124. # create FIRST Level groups branch
    125. dn: ou=groups,dc=hadoop,dc=apache,dc=org
    126. objectclass:top
    127. objectclass:organizationalUnit
    128. ou: groups
    129. description: generic groups branch
    130.  
    131. # create the analyst group under groups
    132. dn: cn=analyst,ou=groups,dc=hadoop,dc=apache,dc=org
    133. objectclass:top
    134. objectclass: groupofnames
    135. cn: analyst
    136. description:analyst  group
    137. member: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org
    138. member: uid=tom,ou=people,dc=hadoop,dc=apache,dc=org
    139. member: uid=myhdfs,ou=myhadoop,dc=hadoop,dc=apache,dc=org
    140. member: uid=myyarn,ou=myhadoop,dc=hadoop,dc=apache,dc=org
    141.  
    142. # create the scientist group under groups
    143. dn: cn=scientist,ou=groups,dc=hadoop,dc=apache,dc=org
    144. objectclass:top
    145. objectclass: groupofnames
    146. cn: scientist
    147. description: scientist group
    148. member: uid=sam,ou=people,dc=hadoop,dc=apache,dc=org
    149.  
    150. # create the admin group under groups
    151. dn: cn=admin,ou=groups,dc=hadoop,dc=apache,dc=org
    152. objectclass:top
    153. objectclass: groupofnames
    154. cn: admin
    155. description: admin group
    156. member: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
    157. member: uid=root,ou=myhadoop,dc=hadoop,dc=apache,dc=org

    注意，我在这里面添加了两个用户 myhdfs，myyarn

     进入knox-2.0.0/conf/topologies 文件，会发现一个sandbox.xml文件

    它是一个模板文件，使用时改一个名字，我的如下

    my_hdfs.xml

    1. "1.0" encoding="UTF-8"?>
    2. <topology>
    3.    <name>my_hdfsname>
    4.    <gateway>
    5.       <provider>
    6.          <role>authenticationrole>
    7.          <name>ShiroProvidername>
    8.          <enabled>trueenabled>
    9.          <param>
    10.             <name>sessionTimeoutname>
    11.             <value>30value>
    12.          param>
    13.          <param>
    14.             <name>main.ldapRealmname>
    15.             <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealmvalue>
    16.          param>
    17.          <param>
    18.             <name>main.ldapContextFactoryname>
    19.             <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactoryvalue>
    20.          param>
    21.          <param>
    22.             <name>main.ldapRealm.contextFactoryname>
    23.             <value>$ldapContextFactoryvalue>
    24.          param>
    25.  
    26.  
    27. 		<param>
    28. 			<name>main.ldapRealm.contextFactory.systemUsernamename>
    29. 			<value>uid=myclient,ou=people,dc=hadoop,dc=apache,dc=orgvalue>
    30. 		param>
    31. 		<param>
    32. 			<name>main.ldapRealm.contextFactory.systemPasswordname>
    33. 			<value>${ALIAS=ldcSystemPassword}value>
    34. 		param>
    35. 		<param>
    36. 			<name>main.ldapRealm.userSearchBasename>
    37. 			<value>ou=myhadoop,dc=hadoop,dc=apache,dc=orgvalue>
    38. 		param>
    39. 		<param>
    40. 			<name>main.ldapRealm.userSearchAttributeNamename>
    41. 			<value>uidvalue>
    42. 		param>
    43. 		<param>
    44. 			<name>main.ldapRealm.userSearchFiltername>
    45. 			<value>(&(objectclass=person)(uid={0})(uid=myhdfs))value>
    46. 		param>
    47.  
    48.          <param>
    49.             <name>main.ldapRealm.contextFactory.urlname>
    50.             <value>ldap://localhost:33389value>
    51.          param>
    52.          <param>
    53.             <name>main.ldapRealm.contextFactory.authenticationMechanismname>
    54.             <value>simplevalue>
    55.          param>
    56.          <param>
    57.             <name>urls./**name>
    58.             <value>authcBasicvalue>
    59.          param>
    60.       provider>
    61.       <provider>
    62.          <role>identity-assertionrole>
    63.          <name>Defaultname>
    64.          <enabled>trueenabled>
    65.       provider>
    66.       <provider>
    67.          <role>hostmaprole>
    68.          <name>staticname>
    69.          <enabled>trueenabled>
    70.          <param>
    71.             <name>localhostname>
    72.             <value>my_hdfsvalue>
    73.          param>
    74.       provider>
    75.    gateway>
    76.    
    77.    <service>
    78.       <role>HDFSUIrole>
    79.       <url>http://hadoop02:50070url>
    80.       <version>2.7.0version>
    81.    service>
    82.  
    83. topology>

    my_yarn.xml

    1. "1.0" encoding="UTF-8"?>
    2. <topology>
    3.    <name>my_yarnname>
    4.    <gateway>
    5.       <provider>
    6.          <role>authenticationrole>
    7.          <name>ShiroProvidername>
    8.          <enabled>trueenabled>
    9.          <param>
    10.             <name>sessionTimeoutname>
    11.             <value>30value>
    12.          param>
    13.          <param>
    14.             <name>main.ldapRealmname>
    15.             <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealmvalue>
    16.          param>
    17.          <param>
    18.             <name>main.ldapContextFactoryname>
    19.             <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactoryvalue>
    20.          param>
    21.          <param>
    22.             <name>main.ldapRealm.contextFactoryname>
    23.             <value>$ldapContextFactoryvalue>
    24.          param>
    25.          
    26. 		 
    27.         <param>
    28. 			<name>main.ldapRealm.contextFactory.systemUsernamename>
    29. 			<value>uid=myclient,ou=people,dc=hadoop,dc=apache,dc=orgvalue>
    30. 		param>
    31. 		<param>
    32. 			<name>main.ldapRealm.contextFactory.systemPasswordname>
    33. 			<value>${ALIAS=ldcSystemPassword}value>
    34. 		param>
    35. 		<param>
    36. 			<name>main.ldapRealm.userSearchBasename>
    37. 			<value>ou=myhadoop,dc=hadoop,dc=apache,dc=orgvalue>
    38. 		param>
    39. 		<param>
    40. 			<name>main.ldapRealm.userSearchAttributeNamename>
    41. 			<value>uidvalue>
    42. 		param>
    43. 		<param>
    44. 			<name>main.ldapRealm.userSearchFiltername>
    45. 			<value>(&(objectclass=person)(uid={0})(uid=myyarn))value>
    46. 		param>
    47. 		 
    48. 		<param><name>csrf.enabledname><value>truevalue>param>
    49. 		<param><name>csrf.customHeadername><value>X-XSRF-Headervalue>param>
    50. 		<param><name>csrf.methodsToIgnorename><value>GET,OPTIONS,HEADvalue>param>
    51. 		<param><name>cors.enabledname><value>falsevalue>param>
    52. 		<param><name>xframe.options.enabledname><value>truevalue>param>
    53. 		<param><name>xss.protection.enabledname><value>truevalue>param>
    54. 		<param><name>strict.transport.enabledname><value>truevalue>param>
    55. 		<param><name>rate.limiting.enabledname><value>truevalue>param>
    56.  
    57.          <param>
    58.             <name>main.ldapRealm.contextFactory.urlname>
    59.             <value>ldap://localhost:33389value>
    60.          param>
    61.          <param>
    62.             <name>main.ldapRealm.contextFactory.authenticationMechanismname>
    63.             <value>simplevalue>
    64.          param>
    65.          <param>
    66.             <name>urls./**name>
    67.             <value>authcBasicvalue>
    68.          param>
    69.       provider>
    70.       <provider>
    71.          <role>identity-assertionrole>
    72.          <name>Defaultname>
    73.          <enabled>trueenabled>
    74.       provider>
    75.       <provider>
    76.          <role>hostmaprole>
    77.          <name>staticname>
    78.          <enabled>trueenabled>
    79.          <param>
    80.             <name>localhostname>
    81.             <value>my_yarnvalue>
    82.          param>
    83.       provider>
    84.    gateway>
    85.    <service>
    86.       <role>YARNUIrole>
    87.       <url>http://hadoop03:8088url>
    88.    service>
    89.    <service>
    90.     <role>JOBHISTORYUIrole>
    91.     <url>http://hadoop03:19888url>
    92.   service>
    93.  
    94. topology>

    说明，配置中的${ALIAS=ldcSystemPassword}是生成的密文

    参考：

    如果测试不成功改名实际密码测试

    之后进入knox-2.0.0/bin

    执行(注意，不能使用root用户)

    1. ./ldap.sh start
    2. ./knoxcli.sh create-master
    3. ./gateway.sh start

    网站访问：

    https://192.168.200.11:18483/my/mimi/my_yarn/yarn

    账号：myyarn 密码 123

    https://192.168.200.11:18483/my/mimi/my_hdfs/hdfs

    账号：myhdfs 密码 123456

    其它

    通过我的多次测试，成功实现了，不同用户访问不同页面。下面是几个问题

    1 必须有多个单独的xml才能实现不同用户访问不同页面

    2 我测试成功的只有hdfs、yarn、hbase。之后准备测试hue，发现死活不成功

    注意：文中xml单用户访问页面可以直接复制，然后修改。官网描述很多测试失败。

  • 相关阅读:
    Linux设置禁止SSH空密码登录
    Hadoop完全分布式环境搭建
    天软特色因子看板 (2023.09 第04期)
    Swift开发中：非逃逸闭包、逃逸闭包、自动闭包的区别
    100张照片带你了解澳大利亚
    typeScript简单封装axios
    CSS定位
    tiup cluster upgrade
    线上研讨会 | CATIA助力AI提升汽车造型设计
    Sentinel 哨兵数据 更新下载地址 2023年11月
  • 原文地址：https://blog.csdn.net/qq_40209679/article/details/138563912