Instalação do dcache nos Thors SUN X4540

Description

Acrescentando storage para o Dcache utilizando o Thor X4540 da Sun

Cabeamento dos Thors

	THOR1	THOR2	THOR3
thor port Net0 Eth1	cx7.49	cx7.50	cx7.51
porta Switch Sprace	25	26	27
	cx6.41	cx6.42	cx6.43
thor port NetMng	cx7.52	cx7.53	cx7.54
porta Switch Mng GridUnesp	15	16	17
	cx1.73	cx1.74	cx1.75
thor port Net1 Eth2	cx1.05	cx1.06	cx1.07
porta Switch Sprace	7	8	9

Endereços IP e hostnames

Qual o endereço inicial do Thor ??? ----> Acesse o console serial para configurar o IP

login e senha (dãããã)

cd /SP/network
ls
set pendingipaddress=192.168.2.161
set pendinggateway=192.168.2.150
set commit pending=true

Fisicamente os Thor estão ordenados de baixo para cima, logo, o 1o thor de baixo é o No 1.

THOR	hostname	End. IP Net0	End. IP ILOM	End. IP Hexad	End. Externo
1	spstrg01	192.168.1.161	192.168.2.161	C0.A8.01.A1	200.136.80.11
2	spstrg02	192.168.1.162	192.168.2.162	C0.A8.01.A2	200.136.80.12
3	spstrg03	192.168.1.163	192.168.2.163	C0.A8.01.A3	200.136.80.13

Kickstart

/tftpboot/pxelinux.cfg

thor-install

[allan@spserv01 pxelinux.cfg]$ cat thor-install
default CentOS-5.4-install/vmlinuz
append initrd=CentOS-5.4-install/initrd.img method=http://200.145.46.3/CentOS ks=http://200.145.46.3/thor_sprace.ks ksdevice=eth1

IP decimal ---> IP Hexedecimal
192.168.1.161 = C0A801A1
192.168.1.162 = C0A801A2
192.168.1.163 = C0A801A3

CentOS-5.4-install

o ks utilizado para os thor está em http://200.145.46.3/thor_sprace.ks

ls -l
lrwxrwxrwx 1 root root 12 Mar 10 16:53 C0A801A1 -> thor-install
lrwxrwxrwx 1 root root 12 Mar 10 16:54 C0A801A2 -> thor-install
lrwxrwxrwx 1 root root 12 Mar 10 16:54 C0A801A3 -> thor-install
-rw-r--r-- 1 root root 525 Jan 7 09:36 default
-rw-r--r-- 1 root root 166 Mar 11 14:46 thor-install

Criação dos RAIDs de dados

• Criar o layout básico dos discos

echo "
n
p
2


t
2
fd
w
" > fdisk.expect

• Executar o fdisk:

for i in a b c d e f g h i j k l m n o p q r s t u v w x y z;do fdisk \
/dev/sd$i < fdisk.expect; done

for i in a b c d e f g h i j k l m n o p q r s t u v w x y z;do fdisk \
/dev/sda$i < fdisk.expect; done

• Depois de criar as partições devemos reiniciar a máquina

reboot

• Criar os RAIDs

mdadm --create /dev/md10 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sda2 /dev/sdi2 /dev/sdq2 /dev/sdy2 /dev/sdag2 /dev/sdao2

mdadm --create /dev/md11 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdb2 /dev/sdj2 /dev/sdr2 /dev/sdz2 /dev/sdah2 /dev/sdap2

mdadm --create /dev/md12 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdc2 /dev/sdk2 /dev/sds2 /dev/sdaa2 /dev/sdai2 /dev/sdaq2

mdadm --create /dev/md13 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdd2 /dev/sdl2 /dev/sdt2 /dev/sdab2 /dev/sdaj2 /dev/sdar2

mdadm --create /dev/md14 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sde2 /dev/sdm2 /dev/sdu2 /dev/sdac2 /dev/sdak2 /dev/sdas2

mdadm --create /dev/md15 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdf2 /dev/sdn2 /dev/sdv2 /dev/sdad2 /dev/sdal2 /dev/sdat2

mdadm --create /dev/md16 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdg2 /dev/sdo2 /dev/sdw2 /dev/sdae2 /dev/sdam2 /dev/sdau2

mdadm --create /dev/md17 --force --chunk=128 --level=5 --raid-devices=6 --spare-devices=0 /dev/sdh2 /dev/sdp2 /dev/sdx2 /dev/sdaf2 /dev/sdan2 /dev/sdav2

Benchmarking

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

yum install  xfsprogs bonnie++ 

mkfs.xfs /dev/md10 

mount /dev/md10 /mnt 
cd /mnt
mkdir teste
chown allan.allan teste
cd teste/
bonnie++ -f -n 0 -u 7833 -s 131072:131072

O 7833 é o gid para allan

Alguns comandos mdam:

mdadm --stop /dev/md10 para desmontar o raid ( antes, desmontar o ponto de montagem)

* Test A: Single disk

• • no RAID

rmount /dev/sdaa2 /mnt

• Test B: RAID5 (6 disks)
  • One disk from each controller

mdadm --create /dev/md10 --force --chunk=128 --level=0 --raid-devices=6 --spare-devices=0 /dev/sdc2 /dev/sdk2 /dev/sds2 /dev/sdaa2 /dev/sdai2 /dev/sdaq2

• Test C: RAID50 (48 disks)
  • A single RAID 0 with 8 RAID5, each one with 6 disks

mdadm --create /dev/md20 --force --chunk=512 --level=0 --raid-devices=8 --spare-devices=0 /dev/dm10 /dev/md11 /dev/md12 /dev/md13 /dev/dm14 /dev/md15 /dev/dm16 /dev/md17

• Test D: RAID0 (6 disks)
  • One disk from each controller

• Test E: RAID0 (24 disks)
  • 4 disks from each controller

• Test F: RAID0 (48 disks)
  • all disks

mdadm --create /dev/md10 --force --chunk=128 -e 1 --level=0 --raid-devices=48 /dev/sda2 /dev/sdaa2 /dev/sdab2 /dev/sdac2 /dev/sdad2 /dev/sdae2 /dev/sdaf2 /dev/sdag2 /dev/sdah2 /dev/sdai2 /dev/sdaj2 /dev/sdak2 /dev/sdal2 /dev/sdam2 /dev/sdan2 /dev/sdao2 /dev/sdap2 /dev/sdaq2 /dev/sdar2 /dev/sdas2 /dev/sdat2 /dev/sdau2 /dev/sdav2 /dev/sdb2 /dev/sdc2 /dev/sdd2 /dev/sde2 /dev/sdf2 /dev/sdg2 /dev/sdh2 /dev/sdi2 /dev/sdj2 /dev/sdk2 /dev/sdl2 /dev/sdm2 /dev/sdn2 /dev/sdo2 /dev/sdp2 /dev/sdq2 /dev/sdr2 /dev/sds2 /dev/sdt2 /dev/sdu2 /dev/sdv2 /dev/sdw2 /dev/sdx2 /dev/sdy2 /dev/sdz2

Resultados

test	sequential write (CPU)	sequential read (CPU)	random seek (CPU)
A	85MB/s (18%)	87MB/s (11%)	195.3MB/s (3%)
B	82MB/s (25%)	260MB/s (49%)	318.6MB/s (9%)
C	247MB/s (48%)	130MB/s (45%)	354.6MB/s (11%)
D	292MB/s (29%)	325MB/s ( 40%)	323.7MB/s (6%)
E	963MB/s (85%)	889MB/s (42%)	377.7MB/s (7%)
F	733MB/s (65%)	1153MB/s (88%)	391.5MB/s (7%)

Instalação do software para os pools:

O primeiro passo é instalar a versão do Java JDK, do site de SUN. Crie os diretórios:

mkdir /etc/grid-security
mkdir /etc/grid-security/certificates

Adicionar a seguinte linha ao /etc/fstab

osgce:/opt/osg-1.2.4/globus/TRUSTED_CA  /etc/grid-security/certificates           nfs     rw,auto,hard,bg,rsize=32768,wsize=32768,udp,nfsvers=3

O certificado para esta máquina foi solicitado à partir da osg-ce:

. /OSG/setup.sh
mkdir spstrg02
cd  spstrg02
cert-gridadmin -host spstrg02.sprace.org.br -prefix spstrg02 ca doegrids -affiliation osg -vo dosar -show -email mdias@ift.unesp.br
scp spstrg02* spstrg02.sprace.org.br:/tmp/.

Retorne à spstrg02 e instale os certificados de máquina:

mv /tmp/spstrg02cert.pem /etc/grid-security/hostcert.pem
mv /tmp/spstrg02key.pem /etc/grid-security/hostkey.pem
chown root: /etc/grid-security/hostcert.pem /etc/grid-security/hostkey.pem
chmod 400 /etc/grid-security/hostkey.pem
chmod 444 /etc/grid-security/hostcert.pem
openssl x509 -text -noout -in /etc/grid-security/hostcert.pem

Faça o download do pacote dCache-server

cd /tmp
wget http://www.dcache.org/downloads/1.9/dcache-server-1.9.5-9.noarch.rpm

Configuração:

cp /opt/d-cache/etc/dCacheSetup.template /opt/d-cache/config/dCacheSetup

As modificações feitas no arquivo acima foram:

serviceLocatorHost=osg-se.sprace.org.br
java="/usr/bin/java"
useGPlazmaAuthorizationModule=true
useGPlazmaAuthorizationCell=false
performanceMarkerPeriod=10

Outro arquivo de configuração:

cp /opt/d-cache/etc/node_config.template /opt/d-cache/etc/node_config

alterando:

vim /opt/d-cache/etc/node_config
SERVER_ID=sprace.org.br
NAMESPACE_NODE=osg-se.sprace.org.br
NODE_TYPE=pool
SERVICES=gridftp dcap gsidcap

Edite também:

vim /opt/d-cache/etc/dcachesrm-gplazma.policy
saml-vo-mapping="ON"
kpwd="ON"
saml-vo-mapping-priority="1"
kpwd-priority="2"
mappingServiceUrl="https://spserv01.sprace.org.br:8443/gums/services/GUMSAuthorizationServicePort"

Comente a linha iniciada com XACMLmappingServiceUrl. De um pool já em operação, copie os seguintes arquivos:

scp /etc/grid-security/storage-authzdb spstrg02:/tmp
scp /opt/d-cache/etc/dcache.kpwd spstrg02:/tmp

e mova na spstrg02 para os lugares adequados

mv /tmp/storage-authzdb /etc/grid-security/storage-authzdb
mv /tmp/dcache.kpw /opt/d-cache/etc/dcache.kpwd

Adicione a seguinte linha (se ela não estiver lá) ao começo do último arquivo:

version 2.1

Rode o script para instalação

/opt/d-cache/install/install.sh

A preparação dos pools é feita da seguinte forma, formatando os pools e preparando os sistemas de arquivos (verifique se o pacote = xfsprogs= está instalado):

/sbin/mkfs.xfs /dev/md1X
mkdir /raid{0,1,2,3,4,5,6,7}
for i in `seq 0 7`;  do mount /dev/md1$i /raid$i;done
/opt/d-cache/bin/dcache pool create 4654G /raid1/pool1

e assim vão sendo criados os pools /raidX/poo1 . Adicione os pools ao dCache:

 /opt/d-cache/bin/dcache pool add spstrg02_1 /raid0/pool1/
/opt/d-cache/bin/dcache pool ls

e inicie:

/opt/d-cache/bin/dcache start

-- AllanSzu - 12 Mar 2010