本文共 9912 字，大约阅读时间需要 33 分钟。

网上看到一篇KFS的文章，抽时间用一下。

 

转自： 

 

预装软件

 - zlib - bzip

 - glib

 - python - openssl

 - ncurses

 - Boost (preferably, version 1.34 or higher) - cmake (preferably, version 2.4.7 or higher) - log4cpp (preferably, version 1.0) - gcc version 4.3 or higher) - xfs-progs devel RPM on Linux 

一、安装依赖软件

1、安装log4cpptar xvzf log4cpp-1.0.tar.gzcd log4cpp-1.0./configuremakemake install

2、安装xfs

tar xvzf xfsprogs-3.0.1.tar.gzcd xfsprogs-3.0.1./configuremakemake installmake install-dev

3、安装cmake

tar xvzf cmake-2.6.4.tar.gzcd cmake-2.6.4./bootstrapmakemake install

4、安装boost

tar xvzf boost_1_39_0.tar.gzcd boost_1_39_0

5、安装fuse

最后检查ssh localhost命令是否需要密码。 如果没有，执行下列命令：

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsacat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

再试一下ssh localhost。如果还没有起作用那么你应该检查你的ssh-agent是不是有什么问题。

二、开始编译

本文假设KFS目录: ~/code/kfs. 为了支持FUSE(fuse version 2.7.3 or higher). 编辑 ~/code/kfs/CMakeLists.txt

SET(Fuse_LIBRARY_DIR "/usr/local/lib")

SET(Fuse_INCLUDE_DIR "/usr/local/include")

1. cd ~/code/kfs2. mkdir build3. cd build

选择下列三个其中一个

4. cmake ~/code/kfs/                                                                                       #-- will build DEBUG binaries by default4. cmake -DCMAKE_BUILD_TYPE:STRING="Release" ~/code/kfs           #-- will build Release binaries by default4. cmake -DCMAKE_BUILD_TYPE:STRING="RelWithDebInfo" ~/code/kfs    #-- will build reldbg binaries by default

5. gmake

6. gmake install

生成如下文件夹

Executables will be in: ~/code/kfs/build/bin

Libraries will be in: ~/code/kfs/build/lib

生成JAVA支持:

1. cd ~/code/kfs

2. ant jar

得到如下文件

 * ~/code/kfs/build/classes --- This will contain the Java class files * ~/code/kfs/build/kfs.jar --- The jar file containing the Java classes 将jar加到 CLASSPATH 环境变量

#export CLASSPATH=${CLASSPATH}:~/code/kfs/build/kfs-[version].jar

 

生成 Python 支持

1. cd to ~/code/kfs/src/cc/access

2. 编辑 kfs_setup.py ，设置include 路径       kfsext = Extension('kfs', include_dirs ['~/code/kfs/src/cc/', '<path to boost>'])3. python kfs_setup.py ~/code/kfs/build/lib/ build          #生成共享库文件_kfs.so_.4. python kfs_setup.py ~/code/kfs/build/lib/ install      #To install in site-packages for python:

(

安装在其它路径如： ~/code/kfs/build/lib python kfs_setup.py ~/code/kfs/build/lib install --home=~/code/kfs/build/lib

如果安装在其它路径，更新 PYTHONPATH 、LD_LIBRARY_PATH环境变量:

export PYTHONPATH=${PYTHONPATH}:~/code/kfs/build/lib/lib64/pythonexport LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/code/kfs/build/lib)

开始部署There are two modes of deployment: - Single machine setting: The metaserver/chunkserver run on the same, single machine - Distributed setting: The servers run on a cluster of machines.

For either type of server, the configuration file defines threevariables:

 * node: This defines the machine name where the binary should run

 * rundir: This defines the directory on the machine where KFS binaries will be installed.  * baseport: This port at which the metaserver/chunkserver process will listen for connection from clients  * loglevel: The level for outputting messages. Since KFS uses log4cpp, the values are INFO/DEBUG

For the metaserver, an additional variable is defined:

 * backup_path: This can be used to specify the (remote) location to which the metaserver checkpoint files should be backed up to.  * clusterkey : A key that is shared between metaserver/chunkserver.

For the chunkservers, there is are two additional variables that are defined:

  * space: The storage space exported by a chunkserver for storing chunks (units are 'G' for GigaBytes and 'M' for MegaBytes)   * chunkDir : The list of directories used to store chunk files on the chunkserver nodes.   For a JBOD configuration, this would be a space separated list of directory names.

By default,  - meta server's checkpoint/log files are stored in ${rundir}/bin/kfscp and ${rundir}/bin/kfslog - chunkserver's checkpoint/log files are stored in ${rundir}/bin/logs - chunkserver's chunks are stored in ${rundir}/bin/kfschunk.  This value is overridden when "chunkdir" variable is defined for a chunkserver.

NOTE: It is not advisable to change the default location for either

server's checkpoint/log files.  Changing them adversely affects theother helper scripts that are provided (such as, backing up the metaserver's logs/checkpoint files, periodically cleaning out oldcheckpoint/log files).

两种配置文件格式. 定义所有服务环境的配置文件：machines.cfg:

[metaserver]

node: machine1clusterkey: kfs-test-clusterrundir: /mnt/kfs/metabaseport: 20000loglevel: INFOnumservers: 2[chunkserver_defaults]rundir: /mnt/kfs/chunkchunkDir: /mnt/kfs/chunk/bin/kfschunkbaseport: 30000space: 3400 Gloglevel: INFO

例出所有节点的配置文件：machines.txt. (自己手工写)

10.2.3.1

10.2.3.210.2.3.3

To install the KFS binaries, perform the following steps:

1. cd ~/code/kfs/scripts

2. 设置好配置文件3. 运行：#--all the servers are on a single host --python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui -s#--When the servers are on multiple hosts --python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui

运行KFS

cd ~/code/kfs/scripts

python kfslaunch.py -f machines.cfg -m machines.txt -s

如果想单独关闭某个 chunkserver ，则可以在 chunkserver 中执行以下命令：

sudo scripts/kfsrun.sh -S -c -f bin/ChunkServer.prp这样只会关闭这个 chunkserver ，而不影响其他 server.

关闭后重新启动： scripts/kfsrun.sh -s -c -f bin/ChunkServer.prp

停止KFS

cd ~/code/kfs/scripts

python kfslaunch.py -f machines.cfg -m machines.txt -S

检查系统状态

cd ~/code/kfs/build/bin/tools

kfsping -m -s <metaserver host> -p <metaserver port>

The KFS package now includes a simple python-based web server that shows the set of servers that are currently connected to the metaserver.

The KFS web server runs on the same machine as the KFS metaserver. The web server's port is metaserver's baseport + 50. The KFS web server uses a file "all-machines.txt" to track where the chunkservers should be running. This file should be manually created and placed in ~/code/kfs/webui before deploying KFS binaries. The format of this file is the name of the chunkserver machines, one entry per line. For example.

10.2.3.1

10.2.3.210.2.3.3

When you open the page, , the web server will return a web page that lists:

where the chunkservers are currently running

where the chunkservers could not be started (due to ssh failures, etc.) where the chunkservers have failed

添加新chunkservers

It is not necessary to stop the KFS servers to add new chunkservers. Simply update the machines.cfg file and start the servers

删除KFS Binaries

cd ~/code/kfs/scripts

python kfssetup.py -f machines.cfg -m machines.txt -b ../build/bin -U

通过FUSE挂载KFS

Launch the KFS servers first

cd ~/code/kfs/build/bin

创建kfs.prp文件.the configuration file should have following entries:

metaServer.host = kfs1   #metaserver runs on host

metaServer.port = 20000

mkdir /tmp/kfs-fuse  # create a mount-point

kfs_fuse /tmp/kfs-fuse -f  # force the system to run the process in foreground

性能测试

测试环境:We provide preliminary performance numbers from a 9-node cluster using KFS version 0.2 code. The 9 nodes were on 3 different racks. The racks were connected by a GigE switch. Each machine had the following configuration:

CPU: Quad-core i386 processor operating at 2400 MHz

Operating system: SunOS 5.10 Generic_127128-11 Memory: 8GB Disk: 4TB of disk configured as a ZFS volume Network interface: 1GigE

Write Performance?For this test, we had a single client write 20GB of data to the servers. Each chunk was replicated 3-way. We used the KfsPerfWriter binary that is in the code repository. This tool opens a file and writes data to it sequentially.

[tests]$ ./KfsPerfWriter -p perf-test-machines.prp -m 20480 -f /perftest.1

Write rate: 412.244 (Mbps)Write rate: 51.5305 (MBps)

Read Performance?For this test, we had a single client read 20GB of data to the servers. We used the KfsPerfReader binary that is in the code repository. This tool opens a file and reads data from it sequentially.

[tests]$ ./KfsPerfReader -p perf-test-machines.prp -m 20480 -f /perftest.1

...Read rate: 833.8 (Mbps)Read rate: 104.225 (MBps)

 

KFS fsck 命令

KFS fsck command could be used to check the health of file system.

The command looks at meta server chunk map and collect stats about chunk and their state printing a summary about them in the end. fsck generates a request to meta server to dump its chunkmap. It then downloads the chunkmap and parses it to check health of file system.

Running fsck (fast mode)?

This mode of operation downloads the chunkmap and does basic checking of chunks and its replicas to verify the health of file system.

cd $KFS_HOME/scripts

python kfsfsck.py -f machines.cfgwhere $KFS_HOME would be something like ~code/kosmosfs. machines.cfg is the cluster config file. verbose option also lists chunks with two copies on same rack.

Running KFS with Replica Checker (slow mode)?

KFS fsck could also be used with replicachecker which allows us to find out the file which has missing blocks. In addition to it, we could also instruct it to delete or move such files to /lost+found directories to restore the system back to healthy state. Since replicachecker build namespace, this mode takes little more time than the fast mode of operation. To be able to run fast mode, we would have to make sure replicachecker and kfsshell are compiled.

Running fsck (slow mode)?

cd $KFS_HOME/scriptspython kfsfsck.py -f machines.cfg -r -b ../build -n ./network.defWhere $KFS_HOME would be something like ~code/kosmosfs. machines.cfg is the cluster config file. ../build is the directory which contains executables like "./bin/emulator/replicachecker and ./bin/tools/kfsshell" network.def contains list of chunk server nodes one per line in the format [chunkserver_ip chunkserver_port chunkserver_rack 0]

[foo@bar]$ head -4 network.def

192.345.1.1 30000 1 0192.345.9.2 30000 9 0192.345.4.3 30000 4 0192.345.1.4 30000 1 0

Validating replica sizes (very slow mode)?

Using --checksize option we could also check to see if sizes of individual replicas match. This operation would ping all chunk servers sequentially to get the size of replica to build the chunk map with replica sizes.

本文来自CSDN博客，转载请标明出处：