CDH安装与测试

发布 : 2016-02-12 分类 : 大数据 浏览 :

CDH安装方式

1
2
3
4
Cloudera Manager
Yum
Rpm
Tarball

Cloudera Manager

1
2
3
4
5
6
Cloudera Manager是一个管理CDH的端到端的应用
作用：
管理
监控
诊断
集成

架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Server
管理控制台服务器和应用程序逻辑
负责软件安装. 配置
启动和停止服务
管理服务运行的群集

Agent
安装在每台主机上
负责启动和停止进程，配置，监控主机

Management Service
由一组角色组成的服务，执行各种监视.报警和报告功能

Database
数据库支撑

Cloudera Repository
组件.软件下载的文档

Clients

Admin Console

API

Cloudera Manager安装前系统环境准备

1.网络配置

1
2
3
4
5
6
7
8
vi /etc/sysconfig/network
vi /etc/hosts
192.168.230.100 cdh1
192.168.230.101 cdh2
192.168.230.102 cdh3

vi /etc/sysconfig/network-scripts/ifcfg-eth0
vi /etc/resolv.conf

2.SSH免密钥登录

1
2
3
4
5
6
7
8
9
10
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
   
cdh1发送公钥给cdh2.cdh3
[root@cdh1 ~]# scp ~/.ssh/id_dsa.pub root@cdh2:/opt/
[root@cdh1 ~]# scp ~/.ssh/id_dsa.pub root@cdh3:/opt/

cdh2.cdh3将cdh1发过来的公钥加入到认证文件中去
[root@cdh2 ~]$ cat /opt/id_dsa.pub >> ~/.ssh/authorized_keys
[root@cdh3 ~]$ cat /opt/id_dsa.pub >> ~/.ssh/authorized_keys

3.防火墙关闭

1
2
3
4
5
6
7
8
9
查看防火墙状态
    service iptables status

关闭防火墙
service iptables stop
chkconfig iptables off

查看防火墙状态
chkconfig iptables --list

4.SELINUX关闭

1
2
setenforce 0
vi /etc/selinux/config (SELINUX=disabled)

5.安装JDK并配置环境变量

5.1.使用rpm命令安装jdk

1
[root@cdh1 software]# rpm -ivh jdk-7u67-linux-x64.rpm


1
2
[root@cdh1 ~]# rpm -qa | grep jdk
[root@cdh1 ~]# rpm -ql jdk


5.2.配置jdk环境变量

1
2
3
4
5
6
[root@cdh1 ~]# vi .bash_profile

export JAVA_HOME=/usr/java/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

[root@cdh1 ~]# source .bash_profile

6.安装NTP

1
2
3
4
5
6
7
设置开机启动

[root@cdh1 ~]# chkconfig ntpd on

设置时间同步

[root@cdh1 ~]# ntpdate -u ntp.sjtu.edu.cn

7.安装并配置mysql

1
三台机器都进行mysql安装配置

7.1.安装mysql客户端

1
2
3
[root@cdh1 ~]# yum -y install mysql
[root@cdh2 ~]# yum -y install mysql
[root@cdh3 ~]# yum -y install mysql

1
2
查看mysql是否安装成功
[root@cdh1 ~]# rpm -ql mysql

7.2.安装mysql服务端

1
2
3
4
5
6
7
[root@cdh1 ~]# yum -y install mysql-server
[root@cdh2 ~]# yum -y install mysql-server
[root@cdh3 ~]# yum -y install mysql-server
 
[root@cdh1 ~]# yum -y install mysql-devel
[root@cdh2 ~]# yum -y install mysql-devel
[root@cdh3 ~]# yum -y install mysql-devel

7.3.配置mysql编码格式

1
2
3
[root@cdh1 ~]# vi /etc/my.cnf
[root@cdh2 ~]# vi /etc/my.cnf
[root@cdh3 ~]# vi /etc/my.cnf

1
2
添加
default-character-set=utf8

7.4.添加开机启动项

1
2
[root@cdh1 ~]# chkconfig --add mysqld
[root@cdh1 ~]# chkconfig mysqld on

1
2
3
4
5
[root@cdh2 ~]# chkconfig --add mysqld
[root@cdh2 ~]# chkconfig mysqld on

[root@cdh3 ~]# chkconfig --add mysqld
[root@cdh3 ~]# chkconfig mysqld on

7.5.启动mysql

1
2
3
[root@cdh1 ~]# service mysqld start
[root@cdh2 ~]# service mysqld start
[root@cdh3 ~]# service mysqld start

7.6.配置root用户并设置密码

1
cdh1主机.cdh2主机.cdh3主机，三台主机的mysql均进行如下配置
1
2
3
4
5
6
7
8
[root@master ~]# mysql -uroot -p
mysql> use mysql;
mysql> select user,host from user;
mysql> delete  from user where user = "";
mysql> select user,host from user;
mysql> update user set password = PASSWORD('123456') where user = 'root';
mysql> update user set host='localhost' where host='%';
mysql> select user,host,password from user;

7.7.赋予权限

1
2
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
mysql> flush privileges;


8.配置本地yum源

9.下载第三方依赖包

1
2
[root@cdh1 ~]# yum -y install chkconfig  python  bind-utils  psmisc  libxslt  zlib  sqlite
[root@cdh1 ~]# yum -y install cyrus-sasl-gssapi  fuse  fuse-libs  redhat-lsb

Cloudera Manager安装

1.安装Cloudera Manager Server. Agent

1
2
[root@cdh1 ~]# mkdir /opt/cloudera-manager
[root@cdh1 ~]# tar -zvxf cloudera-manager*.tar.gz -C /opt/cloudera-manager

1
2
[root@cdh2 ~]# mkdir /opt/cloudera-manager
[root@cdh2 ~]# tar -zvxf cloudera-manager*.tar.gz -C /opt/cloudera-manager

1
2
[root@cdh3 ~]# mkdir /opt/cloudera-manager
[root@cdh3 ~]# tar -zvxf cloudera-manager*.tar.gz -C /opt/cloudera-manager

2.创建用户cloudera-scm

1
[root@cdh1 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1
[root@cdh2 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1
[root@cdh3 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3.配置CM Agent

1
修改文件/opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini中
1
[root@cdh1 ~]# vi /opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini

1
server_host=cdh1

1
[root@cdh2 cloudera-manager]# vi /opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini

1
server_host=cdh1

1
[root@cdh3 cloudera-manager]# vi /opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini

1
server_host=cdh1

4.配置CM Server数据库

1
2
3
4
5
6
7
8
 拷贝mysql jar文件到目录 /usr/share/java/

 注意jar包名称要修改为mysql-connector-java.jar

[root@cdh1 software]# mkdir -p /usr/share/java/
[root@cdh1 ~]# mv mysql-connector-java-5.1.26-bin.jar mysql-connector-java.jar
[root@cdh1 software]# mv mysql-connector-java.jar /usr/share/java/
[root@cdh1 software]# cd /usr/share/java/


1
2
3
[root@cdh1 java]# mysql -uroot -p
mysql> grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;
mysql> flush privileges;

1
2
[root@cdh1 java]# cd /opt/cloudera-manager/cm-5.4.3/share/cmf/schema/
[root@cdh1 schema]# ./scm_prepare_database.sh mysql temp -h cdh1 -utemp -ptemp --scm-host cdh1 scm scm scm

1
格式：数据库类型.数据库.数据库服务器.用户名.密码. cm server服务器

5.创建Parcel目录

Server节点

1
2
[root@cdh1 schema]# mkdir -p /opt/cloudera/parcel-repo
[root@cdh1 schema]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

Agent节点

1
2
[root@cdh1 init.d]# mkdir -p /opt/cloudera/parcels
[root@cdh1 init.d]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

1
2
[root@cdh2 cloudera-manager]# mkdir -p /opt/cloudera/parcels
[root@cdh2 cloudera-manager]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

1
2
[root@cdh3 cloudera-manager]# mkdir -p /opt/cloudera/parcels
[root@cdh3 cloudera-manager]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6.制作CDH本地源

1
2
3
4
5
下载好文件CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel以及manifest.json，

将这两个文件放到server节点的/opt/cloudera/parcel-repo下。

[root@cdh1 schema]# cd /opt/cloudera/parcel-repo

1
2
打开manifest.json文件，里面是json格式的配置，找到与下载版本相对应的hash码，
新建文件，文件名与你的parel包名一致，并加上.sha后缀，将hash码复制到文件中保存。

7.启动CM Server. Agent

1
2
3
4
启动CM Server
做这个操作的时候，建立个快照
[root@cdh1 parcel-repo]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh1 init.d]# ./cloudera-scm-server start

1
2
cd /opt/cloudera-manager/cm-5.4.3/log/cloudera-scm-server
[root@cdh1 cloudera-scm-server]# tail -f cloudera-scm-server.log

1
2
3
4
Sever首次启动会自动创建表以及数据，不要立即关闭或重启，否则需要删除所有表及数据重新安装
启动CM Agent
[root@cdh1 ~]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh1 init.d]# ./cloudera-scm-agent start

1
2
[root@cdh2 cloudera-manager]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh2 init.d]# ./cloudera-scm-agent start

1
2
[root@cdh3 cloudera-manager]# cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
[root@cdh3 init.d]# ./cloudera-scm-agent start

8.访问：http://ManagerHost(主节点Ip):7180

1
2
3
在浏览器上输入：http://192.168.230.100:7180/cmf/login
若可以访问，则CM安装成功
用户.密码：admin

1
2
cat /proc/sys/vm/swappiness
echo 0 > /proc/sys/vm/swappiness
1
2
3
4
5
6
7
日志文件存放的位置
server和angent启动的日志
cd /opt/cloudera-manager/cm-5.4.3/log
安装服务的存放目录
cd /var/log
安装服务的配置文件存放目录
cd /etc/
本文作者 : Matrix
原文链接 : https://matrixsparse.github.io/2016/02/12/cdh安装与测试/
版权声明 : 本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明出处！

知识 & 情怀 | 二者兼得

微信扫一扫, 向我投食

微信扫一扫, 向我投食

支付宝扫一扫, 向我投食

支付宝扫一扫, 向我投食

标签: #大数据
留下足迹