TWiki> UCSDTier2 Web>HdfsXrootd (revision 10)EditAttach

HdfsXrootd Development Twiki


Build Instructions

This section will use $PROJDIR as shorthand to denote a common area for the project source code to be located. Some optional instructions below are intended only if you plan to modify the Hadoop 2.0 extendable_client.patch and are separated in their own sections.

Obtain Dependencies

Follow steps at Install the Yum Repositories required by OSG to obtain OSG yum repo >= version 3.2.

Disable osg-release repo by setting enabled=0 in


Enable osg-testing repo by setting enabled=1 in


Install dependencies:

yum install java-1.7.0-openjdk-devel
yum install pcre-devel
yum install xrootd-client-devel
yum install hadoop-hdfs

NOTE RHEL >= 6.5 may break some OSG java 7 dependencies. If this happens, it can be fixed by running:

yum install osg-java7-compat

NOTE if you plan to rebuild hadoop with the patch as in the optional sections below you don't need to install hadoop-hdfs.

Checkout hdfs-xrootd-fallback repo:

svn checkout

Make sure JAVA_HOME is set:

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64

Obtain Dependencies for HDFS patch (Optional)

Install dependencies:

yum install maven3
yum install protobuf-compiler

Add mvn to path:

export PATH=$PATH:/usr/share/apache-maven-3.0.4/bin

Obtain cloudera hadoop tarball:

tar -xzf hadoop-2.0.0-cdh4.1.1.tar.gz

Patch hadoop to allow DFSInputStream inheritance:

cd $PROJDIR/hadoop-2.0.0-cdh4.1.1
patch -p1 < $PROJDIR/hdfs-xrootd/extendable_client.patch

Build hdfs-xrootd-fallback

cd $PROJDIR/hdfs-xrootd-fallback
make -f Makefile_rpm

This will generate the following files:

  • hdfs-xrootd-fallback-1.0.0.jar

Build hdfs-xrootd-fallback including patched hadoop (Optional)

cd $PROJDIR/hdfs-xrootd-fallback

This will generate the following files:

  • hadoop-hdfs-2.0.0-cdh4.1.1.jar
  • hdfs-xrootd-fallback.jar

Deployment for Testing

Assumptions - a hadoop cluster is available and a hadoop client node is properly configured to access it. The following instructions are only required on the client node.

Copy Binaries


Copy regenfs.jar to:
Copy to:


Append the following to /etc/hadoop/conf/core-site.xml:


Startup Fuse in Debug Mode

hadoop-fuse-dfs -oserver=regfs://,port=8020,allow_other,rw,rdbuffer=4096 -d /mnt/hadoop

Useful Test Commands

Put File In Hadoop

This example shows how to specify blocksize:

hadoop fs -Ddfs.blocksize=51200 -put xxx_for_jni_test /store/user/matevz

Use Hadoop fsck

To obtain useful info about block locations:

hdfs fsck /store/user/matevz/xxx_for_jni_test -files -blocks -locations

sample output:

Connecting to namenode via
FSCK started by jdost (auth:SIMPLE) from / for path /store/user/matevz/xxx_for_jni_test at Tue Jul 30 19:11:13 PDT 2013
/store/user/matevz/xxx_for_jni_test 117499 bytes, 3 block(s):  OK
0. BP-902182059- len=51200 repl=1 []
1. BP-902182059- len=51200 repl=1 []
2. BP-902182059- len=15099 repl=1 []

 Total size:   117499 B
 Total dirs:   0
 Total files:   1
 Total blocks (validated):   3 (avg. block size 39166 B)
 Minimally replicated blocks:   3 (100.0 %)
 Over-replicated blocks:   0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:      0 (0.0 %)
 Default replication factor:   1
 Average block replication:   1.0
 Corrupt blocks:      0
 Missing replicas:      0 (0.0 %)
 Number of data-nodes:      1
 Number of racks:      1
FSCK ended at Tue Jul 30 19:11:13 PDT 2013 in 18 milliseconds

The filesystem under path '/store/user/matevz/xxx_for_jni_test' is HEALTHY

Corrupt A Block

Run fsck as above to find the node and filename:
1. BP-902182059- len=51200 repl=1 []

The block filename in this example is: blk_-1470524685933485700

Look in /etc/hadoop/conf/hdfs-site.xml to find value of

Locate block:

find -name blk_-1470524685933485700

Temporarily rename the file with mv along with its associated meta file (ending in .meta) to "corrupt" it.

NOTE sometimes you have to restart the namenode to get hadoop to notice the block is "fixed" after replacing the moved block.


service hadoop-hdfs-namenode restart

-- JeffreyDost - 2013/07/25

Edit | Attach | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r10 - 2013/12/07 - 01:10:05 - JeffreyDost
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback