使用FUSE-DFS mount HDFS

编译环境
编译libhdfs
编译fuse-dfs
使用fuse_dfs
其它
- 路径及权限
- 读写测试

原文:http://jiangbo.me/blog/2012/10/23/mount-hdfs-with-fuse-dfs/

23 October 2012

Hadooop源码中自带了contrib/fuse-dfs模块，用于实现通过libhdfs和fuse将HDFS mount到*nux的本地。

编译环境

Linux: 2.6.18-164.el5 x86_64
JDK: 1.6.0_23 64bit
Hadoop: 0.19.1 下面假设源码目录为$HADOOP_SRC_HOME
Ant: 1.8.4
GCC: 4.1.2(系统默认)

实验环境:
- OS : CentOS 6.5
- Kernel: 2.6.32-358.23.2.el6.x86_64
- JDK : build 1.6.0_39-b04
- Hadoop: 1.0.4
- GCC : gcc-4.4.7-3.el6.x86_64
- Ant : ant-1.7.1-13.el6.x86_64

安装所需支持包

yum -y install ant libtool libsysfs libsysfs-devel openssl-devel
yum -y install subversion #更新源代码使用

编译libhdfs

修改configure执行权限

chmod +x $HADOOP_SRC_HOME/src/c++/pipes/configure
chmod +x $HADOOP_SRC_HOME/src/c++/utils/configure

修改Makefile，调整编译模式(Hadoop 1.0.4已不用修改)

64位机中，需要修改libhdfs的Makefile，将GCC编译的输出模式由32(-m32)位改为64(-m64)位

CC = gcc
LD = gcc
CFLAGS =  -g -Wall -O2 -fPIC
LDFLAGS = -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm -shared -m64(这里) -Wl,-x
PLATFORM = $(shell echo $$OS_NAME | tr [A-Z] [a-z])
CPPFLAGS = -m64(还有这里) -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/$(PLATFORM)

编译

在$HADOOP_HOME目录下执行

ant compile -Dcompile.c++=true -Dlibhdfs=true

Hadoop 1.0.0-1.0.4编择如下报错，网上搜索到bug和patch信息:https://issues.apache.org/jira/browse/MAPREDUCE-2127 ,但hadoop 1.0.4实际为缺少openssl-devel包。

     [exec] checking for pthread_create in -lpthread... yes
     [exec] configure: error: Cannot find libssl.so
     [exec] checking for HMAC_Init in -lssl... no
     [exec] /home/admin/tmp/hadoop-1.0.4/src/c++/pipes/configure: line 5234: exit: please: numeric argument required
     [exec] /home/admin/tmp/hadoop-1.0.4/src/c++/pipes/configure: line 5234: exit: please: numeric argument required

BUILD FAILED
/home/admin/tmp/hadoop-1.0.4/build.xml:2141: exec returned: 255

编译结果将生成libhdfs库，位于$HADOOP_SRC_HOME/build/c++目录下:

build/c++
└── Linux-amd64-64
    ├── include
    │   └── hadoop
    │       ├── Pipes.hh
    │       ├── SerialUtils.hh
    │       ├── StringUtils.hh
    │       └── TemplateFactory.hh
    └── lib
        ├── libhadooppipes.a
        ├── libhadooputils.a
        ├── libhdfs.la
        ├── libhdfs.so -> libhdfs.so.0.0.0
        ├── libhdfs.so.0 -> libhdfs.so.0.0.0
        └── libhdfs.so.0.0.0

在下载的hadoop 1.0.4压缩包c++目录中，已包括了Linux amd64/i386的编译包，也可直接使用该路径的libhdfs.so：

hadoop-1.0.4/c++/
├── Linux-amd64-64
│   ├── include
│   └── lib
└── Linux-i386-32
    ├── include
    └── lib

将libhdfs*复制到系统目录:/usr/local/lib/

cp build/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0 /usr/local/lib/
cd /usr/local/lib/
ln -sfT libhdfs.so.0.0.0 libhdfs.so 
ln -sfT libhdfs.so.0.0.0 libhdfs.so.0

编译fuse-dfs

安装fuse库:fuse-dfs依赖fuse库，可通过lsmod|grep fuse检查是否已经安装，如没有则安装相关依赖库。
```
yum -y install fuse fuse-devel fuse-libs
```

编译contrib/fuse-dfs模块：

ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1

编译完成将会生成$HADOOP_HOME/build/contrib/fuse-dfs/目录，内有：

build/contrib/fuse-dfs/
├── fuse_dfs             # 可执行程序
├── fuse_dfs_wrapper.sh  # 包含一些环境变量设置的脚本(通过export环境变量来指定参数即可)
└── test

在fuse_dfs_wrapper.sh中添加基本的环境参数和路径

export HADOOP_HOME=/opt/hadoop  #Hadoop work dir
export JAVA_HOME=/opt/java/jdk

if [ "$HADOOP_HOME" = "" ]; then
export HADOOP_HOME=/usr/local/share/hadoop
fi

export PATH=$HADOOP_HOME/contrib/fuse_dfs:$PATH

for f in ls $HADOOP_HOME/lib/*.jar $HADOOP_HOME/*.jar ; do
export  CLASSPATH=$CLASSPATH:$f
done

if [ "$OS_ARCH" = "" ]; then
export OS_ARCH=amd64
fi

if [ "$JAVA_HOME" = "" ]; then
export  JAVA_HOME=/usr/local/java
fi

# add local lib dir
if [ "$LD_LIBRARY_PATH" = "" ]; then
export LD_LIBRARY_PATH=$(dirname $0)/lib:$JAVA_HOME/jre/lib/$OS_ARCH/server:/usr/local/share/hdfs/libhdfs/:/usr/local/lib:
fi

# full path of fuse_dfs 
$(dirname $0)/fuse_dfs $@

使用fuse_dfs

主机环境

在使用fuse_dfs的主机安装支持包
```
yum -y install fuse fuse-libs
```

放置hadoop文件:将hadoop-1.0.4解压缩到/opt/hadoop目录中。

把fuse_dfs复制到指定目录(下载fuse_dfs for Linux amd64)：

/usr/local/fuse_dfs/
├── fuse_dfs
├── fuse_dfs_wrapper.sh
└── lib
    ├── libhdfs.so -> libhdfs.so.0.0.0
    ├── libhdfs.so.0 -> libhdfs.so.0.0.0
    └── libhdfs.so.0.0.0

其它

因为需要与DataNode通信，使用fuse_dfs的主机需要能够解析、并可访问所有NameNode、DataNode主机的名称和IP地址。

fuse_dfs版本

/usr/local/fuse_dfs/fuse_dfs_wrapper.sh --version
/usr/local/fuse_dfs/fuse_dfs 0.1.0

命令参数

/usr/local/fuse_dfs/fuse_dfs_wrapper.sh --help

USAGE: ./fuse_dfs [debug] [--help] [--version] [-oprotected=<colon_seped_list_of_paths] [rw] [-onotrash] [-ousetrash] [-obig_writes] [-oprivate (single user)] [ro] [-oserver=<hadoop_servername>] [-oport=<hadoop_port>] [-oentry_timeout=<secs>] [-oattribute_timeout=<secs>] [-odirect_io] [-onopoermissions] [-o<other fuse option>] <mntpoint> [fuse options]
NOTE: debugging option for fuse is -debug

挂载hdfs

测试mount

mkdir /tmp/dfs  #新建一个空目录
# 挂载dfs
# -d表示debug模式，如果正常，将-d参数去掉
# 需要root权限才能执行，fuse: failed to exec fusermount: Permission denied
# 172.16.33.151:9000为NameNode的IP和端口
/usr/local/fuse_dfs/fuse_dfs_wrapper.sh dfs://172.16.33.151:9000 /tmp/dfs/ -d

port=9000,server=172.16.33.151
fuse-dfs didn't recognize /tmp/dfs/,-2
fuse-dfs ignoring option -d
FUSE library version: 2.8.3
nullpath_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.13
flags=0x0000307b
max_readahead=0x00020000   
   INIT: 7.12
   flags=0x00000011
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, success, outsize: 40

unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56
getattr /
   unique: 2, success, outsize: 120
...

unmount
```
# 卸载dfs
fusermount -u /tmp/dfs
```

正常mount

/usr/local/fuse_dfs/fuse_dfs_wrapper.sh dfs://172.16.33.151:9000 /tmp/dfs
mount

port=9000,server=172.16.33.151
fuse-dfs didn't recognize /tmp/dfs,-2

# mount信息
fuse_dfs on /tmp/dfs type fuse.fuse_dfs (rw,nosuid,nodev,allow_other,default_permissions)

其它

路径及权限

现在的版本还不支持直接mount hadoop subdir，说是以后会考虑支持。

可以通过有些参数对某些路径实行保护

Fuse DFS takes the following mount options (i.e., on the command line or the comma separated list of options in /etc/fstab (in which case, drop the -o prefixes):

-oserver=%s  (optional place to specify the server but in fstab use the format above)
-oport=%d (optional port see comment on server option)
-oentry_timeout=%d (how long directory entries are cached by fuse in seconds - see fuse docs)
-oattribute_timeout=%d (how long attributes are cached by fuse in seconds - see fuse docs)
-oprotected=%s (a colon separated list of directories that fuse-dfs should not allow to be deleted or moved - e.g., /user:/tmp)
-oprivate (not often used but means only the person who does the mount can use the filesystem - aka ! allow_others in fuse speak)
-ordbuffer=%d (in KBs how large a buffer should fuse-dfs use when doing hdfs reads)
ro
rw
-ousetrash (should fuse dfs throw things in /Trash when deleting them)
-onotrash (opposite of usetrash)
-odebug (do not daemonize - aka -d in fuse speak)
-obig_writes (use fuse big_writes option so as to allow better performance of writes on kernels >= 2.6.26)

The defaults are:

entry,attribute_timeouts = 60 seconds
rdbuffer = 10 MB
protected = null
debug = 0
notrash
private = 0

读写测试

在5台物理服务器构造成的Hadoop集群(3台DataNode，千兆网口)上进行简单的读、写速度测试，操作在master主机上发起。
- 写入1GB文件，fuse_dfs进程CPU占用达到92-104%，速度22-23.7MB/s。
- 读取1GB文件，fuse_dfs进程CPU占用约在60-90%，速度79.4-96.0MB/s(达到测试点主机千兆网口限制)。
参考:2TB SATA硬盘读写速度：10GB文件写入平均速度179-195MB/s，读取平均速度：444MB/s。
在有一套ceph的环境(三台桌面机搭建)，100Mbps互联，初步测试写入18MB/s，读取22MB/s(可能受100Mbps网口带宽限制)，ceph客户端的CPU占用要低很多，大约在20%左右，并且测试用的客户端是虚拟服务器。

# HDFS挂载点：/tmp/dfs/tmp/
rm -f /tmp/dfs/tmp/1Gb.file;dd if=/dev/zero bs=1024 count=1000000 of=/tmp/dfs/tmp/1Gb.file
dd if=/tmp/dfs/tmp/1Gb.file bs=64k of=/dev/null