Difference: HdfsXrootd (11 vs. 12)

Revision 122013/12/08 - Main.JeffreyDost

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

HdfsXrootd Development Twiki

Line: 8 to 8
 

Build Instructions

Changed:
<
<
This section will use $PROJDIR as shorthand to denote a common area for the project source code to be located. Some optional instructions below are intended only if you plan to modify the Hadoop 2.0 extendable_client.patch and are separated in their own sections.
>
>
The sections below will use $PROJDIR as shorthand to denote a common area for the project source code to be located.
 

Obtain Dependencies

Line: 31 to 31
 NOTE RHEL >= 6.5 may break some OSG java 7 dependencies. If this happens, it can be fixed by running:
yum install osg-java7-compat
Deleted:
<
<
NOTE if you plan to rebuild hadoop with the patch as in the optional sections below you don't need to install hadoop-hdfs.
 Checkout hdfs-xrootd-fallback repo:
svn checkout https://svn.gled.org/var/trunk/hdfs-xrootd-fallback
Line: 41 to 39
 export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
Deleted:
<
<

Obtain Dependencies for HDFS patch (Optional)

Install dependencies:

yum install maven3
yum install protobuf-compiler

Add mvn to path:

export PATH=$PATH:/usr/share/apache-maven-3.0.4/bin

Obtain cloudera hadoop tarball:

cd $PROJDIR
wget http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz
tar -xzf hadoop-2.0.0-cdh4.1.1.tar.gz

Patch hadoop to allow DFSInputStream inheritance:

cd $PROJDIR/hadoop-2.0.0-cdh4.1.1
patch -p1 < $PROJDIR/hdfs-xrootd/extendable_client.patch
 

Build hdfs-xrootd-fallback

cd $PROJDIR/hdfs-xrootd-fallback
Line: 75 to 49
 
  • hdfs-xrootd-fallback-1.0.0.jar
  • libXrdBlockFetcher.so.1.0.0
Deleted:
<
<

Build hdfs-xrootd-fallback including patched hadoop (Optional)

cd $PROJDIR/hdfs-xrootd-fallback
make

This will generate the following files:

  • hadoop-hdfs-2.0.0-cdh4.1.1.jar
  • hdfs-xrootd-fallback.jar
  • libXrdBlockFetcher.so
 

Deployment for Testing

Assumptions - a hadoop cluster is available and a hadoop client node is properly configured to access it. The following instructions are only required on the client node.

Line: 120 to 83
 
hadoop-fuse-dfs -oserver=xfbfs://cabinet-10-10-3.t2.ucsd.edu,port=8020,allow_other,rw,rdbuffer=4096 -d /mnt/hadoop
Added:
>
>

Modifying Hadoop

The instructions here only need to be followed if you plan to modify the extendable-client.patch.

Obtain Dependencies for HDFS

Enable osg-development repo by setting enabled=1 in

/etc/yum.repos.d/osg-el6-development.repo

Install dependencies:

yum install maven3
yum install protobuf-compiler

Add mvn to path:

export PATH=$PATH:/usr/share/apache-maven-3.0.4/bin

Obtain cloudera hadoop tarball:

cd $PROJDIR
wget http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz
tar -xzf hadoop-2.0.0-cdh4.1.1.tar.gz

Patch hadoop to allow DFSInputStream inheritance:

cd $PROJDIR/hadoop-2.0.0-cdh4.1.1
patch -p1 < $PROJDIR/hdfs-xrootd-fallback/extendable_client.patch

Building HDFS

Currently the only jar file we modify is hadoop-hdfs-2.0.0-cdh4.1.1.jar. To build this, run:

cd $PROJDIR/hadoop-2.0.0-cdh4.1.1/src/hadoop-hdfs-project
mvn install -DskipTests

The jar will be located in:

$PROJDIR/hadoop-2.0.0-cdh4.1.1/src/hadoop-hdfs-project/target/hadoop-hdfs-2.0.0-cdh4.1.1.jar

To test, this jar should replace:

/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar

Creating Updated Patch File

Assuming changes have been made to $PROJDIR/hadoop-2.0.0-cdh4.1.1 and an original unmodded copy was untarred into $PROJDIR/hadoop-2.0.0-cdh4.1.1.orig, run the following to generate a patch file:

$PROJDIR/hdfs-xrootd-fallback/create_hdpatch.sh $PROJDIR/hadoop-2.0.0-cdh4.1.1.orig $PROJDIR/hadoop-2.0.0-cdh4.1.1

The new patch will be created in the current directory and be named extendable_client.patch.

 

Useful Test Commands

Put File In Hadoop

Line: 187 to 204
  Temporarily rename the file with mv along with its associated meta file (ending in .meta) to "corrupt" it.
Changed:
<
<
NOTE sometimes you have to restart the namenode to get hadoop to notice the block is "fixed" after replacing the moved block.
>
>
NOTE sometimes you have to restart the datanode to get hadoop to notice the block is "fixed" after replacing the moved block.
 
Changed:
<
<
On cabinet-10-10-3.t2.ucsd.edu:
>
>
On cabinet-10-10-8.t2.ucsd.edu:
 
Changed:
<
<
service hadoop-hdfs-namenode restart
>
>
service hadoop-hdfs-datanode restart
  -- JeffreyDost - 2013/07/25
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback