使用fuse掛載hdfs文件系統(tǒng)
本文參照了很多其他人的blog和文檔,首先表示感謝!
用戶空間文件系統(tǒng)(Filesystem in Userspace,FUSE)允許一些文件系統(tǒng)整合為一個Unix文件系統(tǒng)在用戶空間中執(zhí)行。通過使用Hadoop的Fuse-DFS分類模塊,任意一個Hadoop文件系統(tǒng)(不過一般為HDFS)都可以作為一個標準文件系統(tǒng)進行掛載。我們隨后便可以使用Unix的工具(如ls和cat)與這個文件系統(tǒng)交互,還可以通過任意一種編程語言使用POSIX庫來訪問文件系統(tǒng)。
Fuse-DFS是用C語言實現(xiàn)的,使用libhdfs作為與HDFS的接口。要想了解如何編譯和運行Fuse-DFS,可參見src/contrib./fuse-dfs中的Hadoop分布目錄。
環(huán)境:
centos5.5
hadoop-0.20.2
gcc自帶
Jdk:1.6.0.21
fuse-2.8.4.tar.gz
Automake
Autoconf
m4
Libtool
apache-ant-1.8.4-bin.tar.bz2
軟件包下載:
Ant下載地址http://mirror.bjtu.edu.cn/apache//ant/binaries/
Fuse下載地址sourceforge.net/projects/fuse/files/fuse-2.X/2.8.4/fuse-2.8.4.tar.gz/download
安裝:(如果安裝過程報錯提示某某command no found 請注意PATH,并且用rpm-qa | grep CMD (這個工具)查看是否缺少某些工具,我在安裝過程中少automake,果斷yum裝上,你懂得??!)
1、ant安裝
Tar jxvf apache-ant-1.8.4-bin.tar.bz2
Mv apache-ant-1.8.4 /usr/local/ant
2、fuse安裝
# tar zxvf fuse-2.8.4.tar.gz
# cd /usr/local/fuse-2.8.4/
# ./configure
# make
# make install
# modprobe fuse
3、環(huán)境變量設(shè)置(請結(jié)合自己的實際情況設(shè)置如下項目)
Vi /etc/profile
export OS_ARCH=i386
##i386/amd64
export OS_BIT=32
##32/64
export JAVA_HOME=/usr/java/jdk1.6.0_21
ANT_HOME=/usr/local/ant
export HADOOP_HOME=/home/hadoop/hadoop-append/hadoop-0.20.2
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:${HADOOP_HOME}/c++/Linux-$OS_ARCH-$OS_BIT/lib:/usr/local/lib:/usr/lib:
PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATH:$MAVEN_HOME/bin:$IVY_HOME:$ERLANG_HOME
保存退出
Source !$
編譯libhdfs
Cd $HADOOP_HOME
/usr/local/ant/bin/ant compile-c++-libhdfs -Dlibhdfs=1 -Dcompile.c++=1
ln -s c++/Linux-$OS_ARCH-$OS_BIT/lib build/libhdfs
/usr/local/ant/bin/ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
vi /home/hadoop/hadoop-append/hadoop-0.20.2/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh
if [ "$HADOOP_HOME" = "" ]; then
export HADOOP_HOME=/home/hadoop/hadoop-append/hadoop-0.20.2
fi
export PATH=$HADOOP_HOME/build/contrib/fuse_dfs:$PATH
for f in ls $HADOOP_HOME/lib/*.jar $HADOOP_HOME/*.jar ; do
export CLASSPATH=$CLASSPATH:$f
done
if [ "$OS_ARCH" = "" ]; then
export OS_ARCH=i386
fi
if [ "$JAVA_HOME" = "" ]; then
export JAVA_HOME=/usr/java/jdk1.6.0_21
fi
if [ "$LD_LIBRARY_PATH" = "" ]; then
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:/usr/local/share/hdfs/libhdfs/:/usr/local/lib
fi
./fuse_dfs $@
保存退出
掛載hdfs
$chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh
$chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs
$ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh /usr/local/bin
$ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs /usr/local/bin/
$mkdir /tmp/dfs
$cd $HADOOP_HOME/build/contrib/fuse-dfs
$./fuse_dfs_wrapper.sh dfs://192.168.64.99:9000 /tmp/dfs
卸載hdfs
Umount /tmp/dfs
開機自動掛載hdfs文件系統(tǒng)
vi /etc/fstab
fuse_dfs_wrapper.sh dfs:192.168.64.99:9000 /tmp/dfs fuse rw,auto 0 0