Configuration of a new UAF Node
Table of Contents
Create an Logical Volume from all spare disk space
First partition the disks accordinging and reboot so the machine picks up the current tables.
Create the Physical Volumes
pvcreate /dev/sda4 /dev/sdb4 /dev/sdc1 /dev/sdd1
Create the volume groups
vgcreate uaf_vg /dev/sda4 /dev/sdb4 /dev/sdc1 /dev/sdd1
Create the logical Volume
lvcreate -L 3550G -i 4 -I 256 uaf_vg -n uaf_lv
Create the file system
mkfs.ext3 /dev/uaf_vg/uaf_lv
tune2fs -m0 /dev/uaf_vg/uaf_lv
Add the file system to fstab
/dev/uaf_vg/uaf_lv /data ext3 defaults 1 3
Mount the file system
mkdir /data
mount /data
Files to copy from an existing UAF Node
scp root@uaf-2.t2.ucsd.edu:.ssh/auth* /root/.ssh
scp root@uaf-2.t2.ucsd.edu:/etc/auto.* /etc/
scp root@uaf-2.t2.ucsd.edu:/etc/passwd root@uaf-2.t2.ucsd.edu:/etc/group root@uaf-2.t2.ucsd.edu:/etc/shadow /etc/
scp root@uaf-2.t2.ucsd.edu:/etc/profile.d/custom.sh /etc/profile.d/
scp root@uaf-2.t2.ucsd.edu:/etc/init.d/condor /etc/init.d/
scp root@uaf-2.t2.ucsd.edu:/etc/DIR_COLORS /etc/
Autofs configuration
chkconfig autofs on
/etc/init.d/autofs restart
Condor Configuration
WARNING!!: We have replaced this section with a
GlideinWMS install to support a wider array of targets Please see
T2CondorClient for details.
Copy the existing uaf configuration for the current node
Note: This step must be done on the central manager
cd /condor/release/etc
cp uaf-X.local uaf-NEW.local
Edit the uaf-NEW.local file and make the appropriate changes.
Create the following links
mkdir /etc/condor
ln -s /condor/release/etc/condor_config /etc/condor/
ln -s /condor/release/bin/* /usr/bin/
ln -s /condor/release/sbin/* /usr/sbin/
Create the following directory tree
mkdir -p /state/data/condor_local
Start Condor
chkconfig condor on
/etc/rc.d/init.d/condor start
Install the OSG Client
cd /data
export http_proxy=http://proxy-1.t2.ucsd.edu:3128
mkdir vdt
mkdir pacman
cd pacman
wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-3.29.tar.gz
tar zxvf pacman-3.29.tar.gz
cd pacman-3.29
source setup.sh
cd /data/vdt
export VDTSETUP_CONDOR_LOCATION=/condor/release
export VDTSETUP_CONDOR_CONFIG=/etc/condor/condor_config
export VDTSETUP_AGREE_TO_LICENSES=y
export VDTSETUP_ENABLE_CONDOR=
export VDTSETUP_INSTALL_CERTS=r
export VDTSETUP_EDG_CRL_UPDATE=y
export VDTSETUP_ENABLE_ROTATE=y
export VDTSETUP_CA_CERT_UPDATER=y
pacman -get http://software.grid.iu.edu/osg-1.2:client
Say yes to the questions
Install the Gridftp server
cd /data/pacman/pacman-3.29
source setup.sh
cd /data/vdt
pacman -get http://vdt.cs.wisc.edu/vdt_200_cache:Globus-Base-Data-Server
Say yes to the questions
Enable the gftp server
source /data/vdt/setup.sh
vdt-control --on gsiftp
Configure the authentication by copying over /etc/grid-security/grid-mapfile, add a host key using certify
Setup the NTP server
ntpdate us.pool.ntp.org
scp uaf-2.t2.ucsd.edu:/etc/ntp.conf /etc/
/etc/init.d/ntpd restart
chkconfig ntpd on
Create the data tmp area
mkdir /data/tmp
chmod 1777 /data/tmp
--++ Add the TMPFS Area
Add the following to rc.local, create the swapfile1 if it does not exist, 20GB in size.
mkdir -p /data/tmpfs/
swapon /data/swap/swapfile1
mount -t tmpfs -osize=10G mode=1777 /data/tmpfs/
for i in c d; do echo "32" > /sys/block/sd${i}/queue/iosched/quantum; done
for i in c d; do echo 256 > /sys/block/sd${i}/queue/read_ahead_kb; done
Adding the Compat packages
yum -y install compat*
Add the i386 glibc
yum -y install glibc-devel*
Install Java
Grab the JDK and install it
Add packages for CMS
yum -y install glibc coreutils bash tcsh zsh perl tcl tk readline openssl ncurses
yum -y install e2fsprogs krb5-libs freetype fontconfig compat-libstdc++-33
yum -y install libidn libX11 libXmu libSM libICE libXcursor libXrender libXpm mesa-libGLU
Adding iptables
scp uaf-5.t2.ucsd.edu:/etc/sysconfig/iptables /etc/sysconfig/iptables
/etc/init.d/iptables restart
install Hadoop
rpm -ivh http://newman.ultralight.org/repos/hadoop/4/i386/caltech-hadoop-4-1.noarch.rpm
yum install hadoop hadoop-fuse
Add the following to the /etc/rc.local file
modprobe fuse
/usr/bin/hdfs -o server=proxy-1.t2.ucsd.edu,port=9000,rdbuffer=131072,allow_other /hadoop/
Install i386 and x64 versions of libz
yum -y install zlib-devel
KRB Config
Copy from another working UAF
/etc/krb5.conf
BWM-NG
Grab bwm-ng RPM for DAG and install it
http://www.gropp.org/?id=projects⊂=bwm-ng
Install some X applications
http://www.if-not-true-then-false.com/2010/install-adobe-acrobat-pdf-reader-on-fedora-centos-red-hat-rhel/
ViewVC?
Versions 1.0 is preferred as it uses highlight, at least for RHEL4
Limits.conf and Ulimit
Add the following to /etc/security/limits.conf
* soft rss 33554432
* hard rss 41943040
* soft nofile 2048
* hard nofile 2048
Add the following to /etc/pam.d/login
session required pam_limits.so
Add Gfortran
yum -y install gcc4-gfortran
Install Tetex
yum -y install tetex*
Setup the server server
- Edit /etc/httpd/conf/httpd.conf
- Comment out UserDir? disable * Uncomment UserDir? public_html
- Uncomment
<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>
- Add the Option Indexes to Directory /
Setup of XRootd HDFS
Make sure hadoop fs -ls works, and add the following (or confirm it) to /etc/sysconfig/hadoop
HADOOP_CONF_DIR=/etc/hadoop
Install the osg hadoop repo
rpm -Uvh http://vdt.cs.wisc.edu/hadoop/osg-hadoop-1-2.el5.noarch.rpm
Currently you need to install xrootd from unstable
yum --enablerepo=hadoop-unstable install xrootd
JSON for CMS
Install JSON so that the users do not have to source
CMSSW
wget http://download.fedora.redhat.com/pub/epel/5/x86_64/python-json-3.4-3.el5.noarch.rpm;rpm -ivh python-json-3.4-3.el5.noarch.rpm
--
TerrenceMartin - 2011/10/13