Configuration of a new UAF Node

Table of Contents

Create an Logical Volume from all spare disk space

First partition the disks accordinging and reboot so the machine picks up the current tables.

Create the Physical Volumes

pvcreate /dev/sda4 /dev/sdb4 /dev/sdc1 /dev/sdd1

Create the volume groups

vgcreate uaf_vg /dev/sda4 /dev/sdb4 /dev/sdc1 /dev/sdd1

Create the logical Volume

lvcreate -L 3550G -i 4 -I 256 uaf_vg -n uaf_lv

Create the file system

mkfs.ext3 /dev/uaf_vg/uaf_lv
tune2fs -m0 /dev/uaf_vg/uaf_lv

Add the file system to fstab

/dev/uaf_vg/uaf_lv      /data                   ext3    defaults        1 3

Mount the file system

mkdir /data
mount /data

Files to copy from an existing UAF Node

scp root@uaf-2.t2.ucsd.edu:.ssh/auth* /root/.ssh
scp root@uaf-2.t2.ucsd.edu:/etc/auto.* /etc/
scp root@uaf-2.t2.ucsd.edu:/etc/passwd root@uaf-2.t2.ucsd.edu:/etc/group root@uaf-2.t2.ucsd.edu:/etc/shadow /etc/
scp root@uaf-2.t2.ucsd.edu:/etc/profile.d/custom.sh /etc/profile.d/
scp root@uaf-2.t2.ucsd.edu:/etc/init.d/condor /etc/init.d/
scp root@uaf-2.t2.ucsd.edu:/etc/DIR_COLORS /etc/

Autofs configuration

chkconfig autofs on
/etc/init.d/autofs restart

Condor Configuration

WARNING!!: We have replaced this section with a GlideinWMS install to support a wider array of targets Please see T2CondorClient for details.

Copy the existing uaf configuration for the current node

Note: This step must be done on the central manager

cd /condor/release/etc
cp uaf-X.local uaf-NEW.local

Edit the uaf-NEW.local file and make the appropriate changes.

Create the following links

mkdir /etc/condor
ln -s /condor/release/etc/condor_config /etc/condor/
ln -s /condor/release/bin/* /usr/bin/
ln -s /condor/release/sbin/* /usr/sbin/

Create the following directory tree

mkdir -p /state/data/condor_local

Start Condor

chkconfig condor on
/etc/rc.d/init.d/condor start

Install the OSG Client

cd /data
export http_proxy=http://proxy-1.t2.ucsd.edu:3128
mkdir vdt
mkdir pacman
cd pacman
wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.29.tar.gz
tar zxvf pacman-3.29.tar.gz
cd pacman-3.29
source setup.sh
cd /data/vdt
export VDTSETUP_CONDOR_LOCATION=/condor/release
export VDTSETUP_CONDOR_CONFIG=/etc/condor/condor_config
export VDTSETUP_AGREE_TO_LICENSES=y
export VDTSETUP_ENABLE_CONDOR=
export VDTSETUP_INSTALL_CERTS=r
export VDTSETUP_EDG_CRL_UPDATE=y
export VDTSETUP_ENABLE_ROTATE=y
export VDTSETUP_CA_CERT_UPDATER=y

pacman -get http://software.grid.iu.edu/osg-1.2:client

Say yes to the questions

Install the Gridftp server

cd  /data/pacman/pacman-3.29
source setup.sh
cd /data/vdt
pacman -get http://vdt.cs.wisc.edu/vdt_200_cache:Globus-Base-Data-Server

Say yes to the questions

Enable the gftp server

source /data/vdt/setup.sh
vdt-control --on gsiftp

Configure the authentication by copying over /etc/grid-security/grid-mapfile, add a host key using certify

Setup the NTP server

ntpdate us.pool.ntp.org
scp uaf-2.t2.ucsd.edu:/etc/ntp.conf /etc/
/etc/init.d/ntpd restart
chkconfig ntpd on

Create the data tmp area

mkdir /data/tmp
chmod 1777 /data/tmp

--++ Add the TMPFS Area

Add the following to rc.local, create the swapfile1 if it does not exist, 20GB in size.

mkdir -p /data/tmpfs/
swapon /data/swap/swapfile1
mount -t tmpfs -osize=10G  mode=1777 /data/tmpfs/

for i in c d; do echo "32" > /sys/block/sd${i}/queue/iosched/quantum; done
for i in c d; do echo 256 > /sys/block/sd${i}/queue/read_ahead_kb; done

Adding the Compat packages

yum -y install compat* 

Add the i386 glibc

 yum -y install glibc-devel*

Install Java

Grab the JDK and install it

Add packages for CMS

yum -y install glibc coreutils bash tcsh zsh perl tcl tk readline openssl ncurses 
yum -y install e2fsprogs krb5-libs freetype fontconfig compat-libstdc++-33 
yum -y install libidn libX11 libXmu libSM libICE libXcursor libXrender libXpm mesa-libGLU

Adding iptables

scp uaf-5.t2.ucsd.edu:/etc/sysconfig/iptables /etc/sysconfig/iptables 
/etc/init.d/iptables restart

install Hadoop

rpm -ivh http://newman.ultralight.org/repos/hadoop/4/i386/caltech-hadoop-4-1.noarch.rpm
yum install hadoop hadoop-fuse

Add the following to the /etc/rc.local file

modprobe fuse
/usr/bin/hdfs -o server=proxy-1.t2.ucsd.edu,port=9000,rdbuffer=131072,allow_other /hadoop/

Install i386 and x64 versions of libz

yum -y install zlib-devel

KRB Config

Copy from another working UAF

/etc/krb5.conf

BWM-NG

Grab bwm-ng RPM for DAG and install it

http://www.gropp.org/?id=projects⊂=bwm-ng

Install some X applications

http://www.if-not-true-then-false.com/2010/install-adobe-acrobat-pdf-reader-on-fedora-centos-red-hat-rhel/

ViewVC?

Versions 1.0 is preferred as it uses highlight, at least for RHEL4

Limits.conf and Ulimit

Add the following to /etc/security/limits.conf

*       soft    rss 33554432
*       hard    rss 41943040
*       soft    nofile 2048
*       hard    nofile 2048

Add the following to /etc/pam.d/login

session    required     pam_limits.so

Add Gfortran

yum -y install gcc4-gfortran

Install Tetex

yum -y install tetex*

Setup the server server

• Edit /etc/httpd/conf/httpd.conf
• Comment out UserDir? disable * Uncomment UserDir? public_html
• Uncomment

<Directory /home/*/public_html>
    AllowOverride FileInfo AuthConfig Limit
    Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
    <Limit GET POST OPTIONS>
        Order allow,deny
        Allow from all
    </Limit>
    <LimitExcept GET POST OPTIONS>
        Order deny,allow
        Deny from all
    </LimitExcept>
</Directory>

• Add the Option Indexes to Directory /

Setup of XRootd HDFS

Make sure hadoop fs -ls works, and add the following (or confirm it) to /etc/sysconfig/hadoop

HADOOP_CONF_DIR=/etc/hadoop

Install the osg hadoop repo

 rpm -Uvh http://vdt.cs.wisc.edu/hadoop/osg-hadoop-1-2.el5.noarch.rpm

Currently you need to install xrootd from unstable

yum --enablerepo=hadoop-unstable install xrootd

JSON for CMS

Install JSON so that the users do not have to source CMSSW

wget http://download.fedora.redhat.com/pub/epel/5/x86_64/python-json-3.4-3.el5.noarch.rpm;rpm -ivh python-json-3.4-3.el5.noarch.rpm

-- TerrenceMartin - 2011/10/13