參考資料:
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/ClusterSetup.html
http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
下載 2.4.1 bin 包, 解壓好以后按照鏈接上配置各個(gè)配置文件, 啟動(dòng)時(shí)會(huì)遇到 "Unable to load realm info from SCDynamicStore" 的問(wèn)題, 這個(gè)問(wèn)題需要在 hadoop-env.sh 中加入如下配置(配置 HBase 的時(shí)候也會(huì)遇到這個(gè)問(wèn)題, 使用同樣的方法在 hbase-env.sh 中加入如下配置解決)
hadoop-env.sh(hbase-env.sh) 配置, 增加
export JAVA_HOME="/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home"export HBASE_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
最后自己寫(xiě)一下啟動(dòng)和停止腳本
hadoop-start.sh
#!/bin/bashHADOOP_PREFIX="/Users/zhenweiliu/Work/Software/hadoop-2.4.1"HADOOP_YARN_HOME="/Users/zhenweiliu/Work/Software/hadoop-2.4.1"HADOOP_CONF_DIR="/Users/zhenweiliu/Work/Software/hadoop-2.4.1/etc/hadoop"cluster_name="hadoop_cat"
# Format a new distributed filesystem
if [ "$1" == "format" ]; then
$HADOOP_PREFIX/bin/hdfs namenode -format $cluster_name
fi
# Start the HDFS with the following command, run on the designated NameNode:$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode# Run a script to start DataNodes on all slaves:$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode# Start the YARN with the following command, run on the designated ResourceManager:$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager# Run a script to start NodeManagers on all slaves:$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager# Start a standalone WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:$HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config $HADOOP_CONF_DIR# Start the MapReduce JobHistory Server with the following command, run on the designated server:$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
hadoop-stop.sh
#!/bin/bashHADOOP_PREFIX="/Users/zhenweiliu/Work/Software/hadoop-2.4.1"HADOOP_YARN_HOME="/Users/zhenweiliu/Work/Software/hadoop-2.4.1"HADOOP_CONF_DIR="/Users/zhenweiliu/Work/Software/hadoop-2.4.1/etc/hadoop"cluster_name="hadoop_cat"# Stop the NameNode with the following command, run on the designated NameNode:$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode# Run a script to stop DataNodes on all slaves:$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode# Stop the ResourceManager with the following command, run on the designated ResourceManager:$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager# Run a script to stop NodeManagers on all slaves:$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager# Stop the WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:$HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver --config $HADOOP_CONF_DIR# Stop the MapReduce JobHistory Server with the following command, run on the designated server:$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR
hadoop-restart.sh
#!/bin/bash./hadoop-stop.sh./hadoop-start.sh
最后是我的各項(xiàng)需要配置的 hadoop 配置
core-site.xml
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property></configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <!-- NameNode Configurations --> <property> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </property> <property> <name>dfs.datanode.datadir</name> <value>file:///Users/zhenweiliu/Work/Software/hadoop-2.4.1/data</value> </property> <property> <name>dfs.blocksize</name> <value>67108864</value> </property> <property> <name>dfs.namenode.handler.count</name> <value>100</value> </property> <!-- Datanode Configurations --> <property> <name>dfs.namenode.name.dir</name> <value>file:///Users/zhenweiliu/Work/Software/hadoop-2.4.1/name</value> </property></configuration>
yarn-site.xml
xml version="1.0"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><configuration> <!-- ResourceManager and NodeManager Configurations --> <property> <name>yarn.acl.enable</name> <value>false</value> </property> <property> <name>yarn.acl.enable</name> <value>false</value> </property> <!-- ResourceManager Configurations --> <property> <name>yarn.resourcemanager.address</name> <value>localhost:9001</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>localhost:9002</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>localhost:9003</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>localhost:9004</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>localhost:9005</value> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>CapacityScheduler</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> <!-- NodeManager Configurations --> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>8192</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>2.1</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>${hadoop.tmp.dir}/nm-local-dir</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>${yarn.log.dir}/userlogs</value> </property> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>10800</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/logs</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir-suffix</name> <value>logs</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- History Server Configurations --> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>-1</value> </property> <property> <name>yarn.log-aggregation.retain-check-interval-seconds</name> <value>-1</value> </property></configuration>
mapred-site.xml
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <!-- Configurations for MapReduce Applications --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>1536</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx1024M</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>3072</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx2560M</value> </property> <property> <name>mapreduce.task.io.sort.mb</name> <value>512</value> </property> <property> <name>mapreduce.task.io.sort.factor</name> <value>100</value> </property> <property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>50</value> </property> <!-- Configurations for MapReduce JobHistory Server --> <property> <name>mapreduce.jobhistory.address</name> <value>localhost:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>localhost:19888</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>file:////Users/zhenweiliu/Work/Software/hadoop-2.4.1/mr-history/tmp</value> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>file:////Users/zhenweiliu/Work/Software/hadoop-2.4.1/mr-history/done</value> </property></configuration>
復(fù)制 3 個(gè) ZK 實(shí)例文件夾, 分別為
zookeeper-3.4.5-1
zookeeper-3.4.5-2
zookeeper-3.4.5-3
每個(gè) ZK 文件下的 zoo.cfg 配置如下
zookeeper-3.4.5-1/zoo.cfg
tickTime=2000initLimit=10syncLimit=5dataDir=/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5-1/datadataLogDir=/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5-1/logsclientPort=2181server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2889:3889server.3=127.0.0.1:2890:3890
zookeeper-3.4.5-2/zoo.cfg
tickTime=2000initLimit=10syncLimit=5dataDir=/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5-2/datadataLogDir=/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5-2/logsclientPort=2182server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2889:3889server.3=127.0.0.1:2890:3890
zookeeper-3.4.5-3/zoo.cfg
tickTime=2000initLimit=10syncLimit=5dataDir=/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5-3/datadataLogDir=/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5-3/logsclientPort=2183server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2889:3889server.3=127.0.0.1:2890:3890
然后在每個(gè)實(shí)例的 data 文件夾下創(chuàng)建一個(gè)文件 myid, 文件內(nèi)分別寫(xiě)入 1, 2, 3 三個(gè)字符, 例如
zookeeper-3.4.5-1/data/myid
1
最后做一個(gè)批量啟動(dòng), 停止腳本
startZkCluster.sh
#!/bin/bashBASE_DIR="/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5"BIN_EXEC="bin/zkServer.sh start"for no in $(seq 1 3)do $BASE_DIR"-"$no/$BIN_EXECdone
stopZkCluster.sh
#!/bin/bashBASE_DIR="/Users/zhenweiliu/Work/Software/zookeeper/zookeeper-3.4.5"BIN_EXEC="bin/zkServer.sh stop"for no in $(seq 1 3)do $BASE_DIR"-"$no/$BIN_EXECdone
restartZkCluster.sh
#!/bin/bash./stopZkCluster.sh./startZkCluster.sh
參考資料:
http://abloz.com/hbase/book.html
實(shí)際上 HBase 內(nèi)置了 ZK, 如果不顯式指定 ZK 的配置, 他會(huì)使用內(nèi)置的 ZK, 這個(gè) ZK 會(huì)隨著 HBase 啟動(dòng)而啟動(dòng)
hbase-env.sh 中顯式啟動(dòng)內(nèi)置 ZK
export HBASE_MANAGES_ZK=truehbase-site.xml
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--/** * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */--><configuration> <!-- <property> <name>hbase.rootdir</name> <value>file:///Users/zhenweiliu/Work/Software/hbase-0.98.3-hadoop2/hbase</value> </property> --> <property> <name>hbase.rootdir</name> <value>hdfs://localhost:9000/hbase</value> <description>The directory shared by RegionServers.</description> </property> <property> <name>dfs.replication</name> <value>1</value> <description>The replication count for HLog and HFile storage. Should not be greater than HDFS datanode count.</description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>localhost</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/Users/zhenweiliu/Work/Software/hbase-0.98.3-hadoop2/zookeeper</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2222</value> <description>Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect. </description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property></configuration>
最后啟動(dòng) hbase
./start-hbase.sh
另外, hbase 需要大得 processes 數(shù)以及 open files 數(shù), 所以需要修改 ulimit, 我的 mac 下增加 /etc/launchd.conf 文件, 文件內(nèi)容
limit maxfiles 16384 16384limit maxproc 2048 2048
在 /etc/profile 添加
ulimit -n 16384ulimit -u 2048
如果 hbase 出現(xiàn)
2014-07-14 23:00:48,342 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695)
1. 查看 hbase master log, 發(fā)現(xiàn)
2014-07-14 23:31:51,270 INFO [master:192.168.126.8:60000] util.FSUtils: Waiting for dfs to exit safe mode...退出 hadoop 安全模式
bin/hdfs dfsadmin -safemode leave
master log 報(bào)錯(cuò)
2014-07-14 23:32:22,238 WARN [master:192.168.126.8:60000] hdfs.DFSClient: DFS Readorg.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1761102757-192.168.126.8-1404787541755:blk_1073741825_1001 file=/hbase/hbase.version檢查 hdfs
./hdfs fsck / -files -blocks
14/07/14 23:36:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableConnecting to namenode via http://localhost:50070FSCK started by zhenweiliu (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 14 23:36:33 CST 2014./hbase/WALs/192.168.126.8,60020,1404917152583-splitting/192.168.126.8%2C60020%2C1404917152583.1404917158940: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741842/hbase/WALs/192.168.126.8,60020,1404917152583-splitting/192.168.126.8%2C60020%2C1404917152583.1404917158940: MISSING 1 blocks of total size 17 B../hbase/WALs/192.168.126.8,60020,1404917152583-splitting/192.168.126.8%2C60020%2C1404917152583.1404917167188.meta: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741843/hbase/WALs/192.168.126.8,60020,1404917152583-splitting/192.168.126.8%2C60020%2C1404917152583.1404917167188.meta: MISSING 1 blocks of total size 401 B../hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741829/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001: MISSING 1 blocks of total size 372 B../hbase/data/hbase/meta/1588230740/.regioninfo: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741827/hbase/data/hbase/meta/1588230740/.regioninfo: MISSING 1 blocks of total size 30 B../hbase/data/hbase/meta/1588230740/info/e63bf8b1e649450895c36f28fb88da98: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741836/hbase/data/hbase/meta/1588230740/info/e63bf8b1e649450895c36f28fb88da98: MISSING 1 blocks of total size 1340 B../hbase/data/hbase/meta/1588230740/oldWALs/hlog.1404787632739: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741828/hbase/data/hbase/meta/1588230740/oldWALs/hlog.1404787632739: MISSING 1 blocks of total size 17 B../hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741832/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001: MISSING 1 blocks of total size 286 B../hbase/data/hbase/namespace/a3fbb84530e05cab6319257d03975e6b/.regioninfo: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741833/hbase/data/hbase/namespace/a3fbb84530e05cab6319257d03975e6b/.regioninfo: MISSING 1 blocks of total size 40 B../hbase/data/hbase/namespace/a3fbb84530e05cab6319257d03975e6b/info/770eb1a6dc76458fb97e9213edb80b72: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741837/hbase/data/hbase/namespace/a3fbb84530e05cab6319257d03975e6b/info/770eb1a6dc76458fb97e9213edb80b72: MISSING 1 blocks of total size 1045 B../hbase/hbase.id: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741826/hbase/hbase.id: MISSING 1 blocks of total size 42 B../hbase/hbase.version: CORRUPT blockpool BP-1761102757-192.168.126.8-1404787541755 block blk_1073741825/hbase/hbase.version: MISSING 1 blocks of total size 7 B.Status: CORRUPT Total size: 3597 B Total dirs: 21 Total files: 11 Total symlinks: 0 Total blocks (validated): 11 (avg. block size 327 B) ******************************** CORRUPT FILES: 11 MISSING BLOCKS: 11 MISSING SIZE: 3597 B CORRUPT BLOCKS: 11 ******************************** Minimally replicated blocks: 0 (0.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 0.0 Corrupt blocks: 11 Missing replicas: 0 Number of data-nodes: 1 Number of racks: 1FSCK ended at Mon Jul 14 23:36:33 CST 2014 in 15 millisecondsThe filesystem under path '/' is CORRUPT
執(zhí)行刪除
./hdfs fsck -delete
14/07/14 23:41:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableConnecting to namenode via http://localhost:50070FSCK started by zhenweiliu (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 14 23:41:46 CST 2014Status: HEALTHY Total size: 0 B Total dirs: 21 Total files: 0 Total symlinks: 0 Total blocks (validated): 0 Minimally replicated blocks: 0 Over-replicated blocks: 0 Under-replicated blocks: 0 Mis-replicated blocks: 0 Default replication factor: 3 Average block replication: 0.0 Corrupt blocks: 0 Missing replicas: 0 Number of data-nodes: 1 Number of racks: 1FSCK ended at Mon Jul 14 23:41:46 CST 2014 in 4 millisecondsThe filesystem under path '/' is HEALTHY
這時(shí)發(fā)現(xiàn) hbase 掛了, 查看 master log
2014-07-14 23:48:53,788 FATAL [master:192.168.126.8:60000] master.HMaster: Unhandled exception. Starting shutdown.org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'. at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:602) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:456) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:802) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615) at java.lang.Thread.run(Thread.java:695)
重建一下 hdfs/hbase 文件
bin/hadoop fs -rm -r /hbase
hbase master 報(bào)錯(cuò)
2014-07-14 23:56:33,999 INFO [master:192.168.126.8:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=192.168.126.8,60020,1405352769509, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on 192.168.126.8,60020,1405353371628 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2683) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4117) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3494) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20036) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168) at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39) at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111) at java.lang.Thread.run(Thread.java:695)
重建 region sever 節(jié)點(diǎn)
bin/hbase zkcli
rmr /hbase/meta-region-server
再次重啟 hbase, 解決
這些參數(shù)在 hbase-site.xml 里配置
1. zookeeper.session.timeout
這個(gè)默認(rèn)值是3分鐘。這意味著一旦一個(gè)server宕掉了，Master至少需要3分鐘才能察覺(jué)到宕機(jī)，開(kāi)始恢復(fù)。你可能希望將這個(gè)超時(shí)調(diào)短，這樣Master就能更快的察覺(jué)到了。在你調(diào)這個(gè)值之前，你需要確認(rèn)你的JVM的GC參數(shù)，否則一個(gè)長(zhǎng)時(shí)間的GC操作就可能導(dǎo)致超時(shí)。（當(dāng)一個(gè)RegionServer在運(yùn)行一個(gè)長(zhǎng)時(shí)間的GC的時(shí)候，你可能想要重啟并恢復(fù)它）.
要想改變這個(gè)配置，可以編輯 hbase-site.xml, 將配置部署到全部集群，然后重啟。
我們之所以把這個(gè)值調(diào)的很高，是因?yàn)槲覀儾幌胍惶斓酵碓谡搲锘卮鹦率值膯?wèn)題?！盀槭裁次以趫?zhí)行一個(gè)大規(guī)模數(shù)據(jù)導(dǎo)入的時(shí)候Region Server死掉啦”，通常這樣的問(wèn)題是因?yàn)殚L(zhǎng)時(shí)間的GC操作引起的，他們的JVM沒(méi)有調(diào)優(yōu)。我們是這樣想的，如果一個(gè)人對(duì)HBase不很熟悉，不能期望他知道所有，打擊他的自信心。等到他逐漸熟悉了，他就可以自己調(diào)這個(gè)參數(shù)了。
這個(gè)設(shè)置決定了處理用戶(hù)請(qǐng)求的線程數(shù)量。默認(rèn)是10，這個(gè)值設(shè)的比較小，主要是為了預(yù)防用戶(hù)用一個(gè)比較大的寫(xiě)緩沖，然后還有很多客戶(hù)端并發(fā)，這樣region servers會(huì)垮掉。有經(jīng)驗(yàn)的做法是，當(dāng)請(qǐng)求內(nèi)容很大(上MB，如大puts, 使用緩存的scans)的時(shí)候，把這個(gè)值放低。請(qǐng)求內(nèi)容較小的時(shí)候(gets, 小puts, ICVs, deletes)，把這個(gè)值放大。
當(dāng)客戶(hù)端的請(qǐng)求內(nèi)容很小的時(shí)候，把這個(gè)值設(shè)置的和最大客戶(hù)端數(shù)量一樣是很安全的。一個(gè)典型的例子就是一個(gè)給網(wǎng)站服務(wù)的集群，put操作一般不會(huì)緩沖,絕大多數(shù)的操作是get操作。
把這個(gè)值放大的危險(xiǎn)之處在于，把所有的Put操作緩沖意味著對(duì)內(nèi)存有很大的壓力，甚至?xí)?dǎo)致OutOfMemory.一個(gè)運(yùn)行在內(nèi)存不足的機(jī)器的RegionServer會(huì)頻繁的觸發(fā)GC操作，漸漸就能感受到停頓。(因?yàn)樗姓?qǐng)求內(nèi)容所占用的內(nèi)存不管GC執(zhí)行幾遍也是不能回收的)。一段時(shí)間后，集群也會(huì)受到影響，因?yàn)樗械闹赶蜻@個(gè)region的請(qǐng)求都會(huì)變慢。這樣就會(huì)拖累集群，加劇了這個(gè)問(wèn)題。
你可能會(huì)對(duì)handler太多或太少有感覺(jué)，可以通過(guò) Section 12.2.2.1, “啟用 RPC級(jí) 日志” ，在單個(gè)RegionServer啟動(dòng)log并查看log末尾 (請(qǐng)求隊(duì)列消耗內(nèi)存)。
聯(lián)系客服
