Hadoop、hbase、zookeeper集群配置

栏目: 集群运维

Hadoop、hbase、zookeeper集群配置

一、环境配置:

1、  Hadoophbasezookeeperchukwa安装指定地址下载的软件:

Hadoop2.7.1http://192.168.5.30:8080/job/build-omms(win)/16/artifact/Hadoop/hadoop/

 

Hbase1.1.2http://192.168.5.30:8080/job/build-omms(win)/16/artifact/Hadoop/hbase/

 

Zookeeper3.4.6http://192.168.5.30:8080/job/build-omms(win)/16/artifact/Hadoop/zookeeper/

 

2、  部署PATH

运行环境需要Java环境,先安装jdk-7u60-linux-x64.tar.gz,解压之后重命名为java,放在/usr/local/目录下;

解压Hadoop(版本2.7)、hbasezookeeper,放在/usr/local/目录下;

自定义PATH环境:vim /etc/profile

export JAVA_HOME=/usr/local/java

export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib               

export HADOOP_DEV_HOME=/usr/local/hadoop                             

export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME}

export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}

export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}

export YARN_HOME=${HADOOP_DEV_HOME}

export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export HDFS_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export YARN_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export ZOOKEEPER_HOME=/usr/local/zookeeper                

export HBASE_HOME=/usr/local/hbase

export HBASE_CONF_DIR=${HBASE_HOME}/conf      

export

PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$HADOOP_DEV_HOME/bin:$HADOOP_DEV_HOME/sbin:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf:$HBASE_HOME/bin: $PATH

source /etc/profile,是自定义的环境生效

 

3、  修改/etc/hosts,本地无DNS,在hosts中相互绑定,最好/etc/sysconfig/network中的hostname也保持一致;

192.168.5.201 master

192.168.5.202 node1

192.168.5.203 node2

 

4、  实现masternode节点ssh无密码登录:

master上使用ssh-keygen –t rsa生产相应的公钥和私钥,进入~/.ssh目录,cp id_rsa.pub authorized_keys,然后将authorized_keys通过scp发送给节点服务器,存放在相同的目录下。

 

二、Hadoop部署:

1、  配置文件:修改hadoop配置文件,即各种site文件,文件存放在/usr/local/hadoop/etc/hadoop/下,主要配置core-site.xmlhdfs-site.xmlmapred-site.xmlyarn-site.xml,环境变量设置hadoop-env.shmapred-env.shyarn-env.sh这几个文件中的JAVA_HOME改为/usr/local/java/

core-site.xml配置:

<configuration>

<property>

    <name>fs.defaultFS</name>

    <value>hdfs://master:9000</value>

</property>

<property>

    <name>hadoop.tmp.dir</name>

    <value>/hadoop/tmp</value>

</property>

<property>

   <name>fs.file.impl</name>

      <value>org.apache.hadoop.fs.LocalFileSystem</value>

         <description>The FileSystem for file: uris.</description>

         </property>

 

<property>

   <name>fs.hdfs.impl</name>

      <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>

         <description>The FileSystem for hdfs: uris.</description>

         </property>

 

</configuration>

 

hdfs-site.xml配置:

<configuration>

<property>

  <name>dfs.namenode.name.dir</name>

    <value>/hadoop/dfs/name</value>

      <description>  </description>

       </property>

 

 <property>

   <name>dfs.datanode.data.dir</name>

     <value>/hadoop/dfs/data</value>

       <description> </description>

        </property>

 

 <property>

    <name>dfs.replication</name>

       <value>3</value> ---->根据集群大小来定

        </property>

 

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>node1:50090</value>

</property>

 

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

    </property>

</configuration>

 

mapred-site.xml配置:

<configuration>

  <property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

  </property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>master:10020</value>

</property>

 

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>master:19888</value>

</property>

 

</configuration>

 

yarn-site.xml配置:

<configuration>

 

<property>

  <name>yarn.nodemanager.aux-services</name>

  <value>mapreduce_shuffle</value>

</property>

 

<property>

  <name>yarn.nodemanager.recovery.enabled</name>

  <value>true</value>

</property>

 

<property>

  <description>The address of the applications manager interface in the RM.</description>

  <name>yarn.resourcemanager.address</name>

  <value>master:9030</value>

</property>

 

<property>

    <description>The address of the scheduler interface.</description>

    <name>yarn.resourcemanager.scheduler.address</name>

    <value>master:9031</value>

</property>

 

<property>

  <description>The address of the RM web application.</description>

  <name>yarn.resourcemanager.webapp.address</name>

  <value>master:9033</value>

</property>

           

<property>

  <description>The address of the resource tracker interface.</description>

  <name>yarn.resourcemanager.resource-tracker.address</name>

  <value>master:9035</value>

</property>

 

 <property>

   <name>yarn.nodemanager.resource.memory-mb</name>

   <value>1572</value>

 </property>

 

<property>

    <description>address of node manager IPC.</description>

    <name>yarn.nodemanager.address</name>

    <value>0.0.0.0:3999</value>

</property>

 

</configuration>

2、  集群配置

slaves文件中添加node节点主机名或者对应IP

node1node2

3、  启动Hadoop

scp –rmaster上的配置好的Hadoop同步到node节点上,并格式化namenode,在master节点上执行如下命令:/usr/local/hadoop/bin/hdfs namenode –format

                                             

看到successfully formatted,格式化成功。

使用/usr/local/hadoop/sbin/start-dfs.sh/usr/local/hadoop/sbin/start-yarn.sh启动Hadoop,在master上通过jps检查相关服务是否正常启动:

 

 

三、zookeeper部署:

1、  进入/usr/local/zookeeper/config目录中,将zoo_sample.cfg拷贝一份命名为zoo.cfg,修改配置:

tickTime=2000

dataDir=/var/lib/zookeeper/

clientPort=2181

initLimit=5

syncLimit=2

server.1=master:2888:3888

server.2=node1:2888:3888

server.3=node2:2888:3888

2、  创建myid:在zoo.cfg配置文件中的dataDir的目录下面创建myid,每个节点myid要求不一样:

masterecho 1 > /var/lib/zookeeper/myid

node1:  echo 2> /var/lib/zookeeper/myid

node2:  echo 3 > /var/lib/zookeeper/myid

四、hbase部署:

1、  配置文件:hbase-env.shhbase-site.xmlregionservers三个文件的配置

修改hbase-env.shJAVA_HOME=/usr/local/java

hbase-site.xml配置:

<configuration>

 

<property>

<name>hbase.rootdir</name>

<value>hdfs://192.168.5.201:9000/hbase</value>

<description>hbase data directory</description>

</property>

 

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

 

<property>

<name>hbase.master</name>

<value>192.168.5.201:6000</value>

</property>

 

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

 

<property>

<name>hbase.zookeeper.quorum</name>

<value>master,node1,node2</value>

</property>

 

<property>

<name>hbase.zookeeper.property.clientPort</name>

<value>2181</value>

<description>Property fromZooKeeper's config zoo.cfg.The port at which the clients willconnect.</description>

</property>

 

<property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/usr/local/zookeeper/data</value>

<description>Property fromZooKeeper's config zoo.cfg.The directory where the snapshot isstored.</description>

</property>

 

</configuration>

 

regionservers配置:

node1

node2

regionservers里面是节点服务器。

 

 

五、启动集群:

在启动之前,使用scp –rzookeeperhbase、发送到node1node2的相同目录

启动顺序依次:HadoopzookeeperhbaseHadoop的启动与停止:/usr/local/hadoop/sbin/{start-all.sh(start-dfs.sh,start-yarn.sh),stop-all.sh(stop-dfs.sh,stop-yarn.sh)

Zookeeper的启动与停止:/usr/loca/zookeeper/bin/{zkServer.sh startstop}

Hbase启动与停止/usr/local/hbase/bin/{start-hbase.sh,stop-hbase.sh},在启动hbase后,

 






曾梦想仗剑走天涯 看一看世界的繁华 年少的心总有些轻狂 如今你四海为家 曾让你心疼的姑娘 如今已悄然无踪影 爱情总让你渴望又感到烦恼 曾让你遍体鳞伤 Dilililidilililidenda Dilililidilililidada Dilililidilililidada 走在勇往直前的路上 Dilililidilililidenda Dilililidilililidada Dilililidilililidada 有难过也有精彩 每一次难过的时候 就独自看一看大海 总想起身边走在路上的朋友 有多少正在疗伤 Dilililidilililidenda Dilililidilililidada Dilililidilililidada 不知多少孤独的夜晚 Dilililidilililidenda Dilililidilililidada Dilililidilililidada 从昨夜酒醉醒来 每一次难过的时候 就独自看一看大海 总想起身边走在路上的朋友 有多少正在醒来 让我们干了这杯酒 好男儿胸怀像大海 经历了人生百态世间的冷暖 这笑容温暖纯真 每一次难过的时候 就独自看一看大海 总想起身边走在路上的朋友 有多少正在醒来 让我们干了这杯酒 好男儿胸怀像大海 经历了人生百态世间的冷暖 这笑容温暖纯真

关键字

最近访客