Instalação do CrabServer na access

Instalação:

Seguindo o seguinte documento https://twiki.cern.ch/twiki/bin/view/CMS/CrabServerInstallation

Requisitando um certificado para a máquina

. /OSG/setup.sh
cert-gridadmin -host access.sprace.org.br -prefix access ca doegrids -affiliation osg -vo dosar -show -email mdias@if.unesp.br
mv /home/mdias/accesscert.pem /etc/grid-security/hostcert.pem
mv /home/mdias/accesskey.pem /etc/grid-security/hostkey.pem
chown root: /etc/grid-security/host*
chmod 400 /etc/grid-security/hostkey.pem
chmod 444 /etc/grid-security/hostcert.pem

Instalando o voms server certificates

cd /tmp/
wget wget -O lcg-vomscerts-5.4.0-1.noarch.rpm http://glitesoft.cern.ch/EGEE/gLite/R3.1/generic/sl4/i386/RPMS.updates/lcg-vomscerts-5.4.0-1.noarch.rpm
rpm -ivh lcg-vomscerts-5.4.0-1.noarch.rpm

Criando a configuração básica e proxy delegation

# adduser crab
#addgroup cms
# su  - crab
cd /home/crab 
mkdir .globus
su
cp /etc/grid-security/hostcert.pem /home/crab/.globus/usercert.pem
cp /etc/grid-security/hostkey.pem /home/crab/.globus/userkey.pem
chown crab /home/crab/.globus/*
exit
export CVSROOT=:ext:mafd@cmscvs.cern.ch:/cvs_server/repositories/CMSSW
export CVS_RSH=/usr/bin/ssh 
cvs co CRAB/CRABSERVER/bin/DelegationRPMs.sh
mkdir sw_area 
export MYTESTAREA=/home/crab/sw_area
su
export MYTESTAREA=/home/crab/sw_area
CRAB/CRABSERVER/bin/DelegationRPMs.sh install -path $MYTESTAREA
exit
mkdir work
export PRODAGENT_WORKDIR=/home/crab/work
export SCRAM_ARCH=slc4_ia32_gcc345 
wget -O $MYTESTAREA/bootstrap.sh http://cmsrep.cern.ch/cmssw/cms/bootstrap.sh
sh -x $MYTESTAREA/bootstrap.sh setup -repository comp -path $MYTESTAREA -arch $SCRAM_ARCH
source $MYTESTAREA/$SCRAM_ARCH/external/apt/0.5.15lorg3.2-cmp/etc/profile.d/init.sh
apt-get update
apt-get upgrade
apt-cache search crab-server
apt-get install cms+crab-server+CRABSERVER_1_0_7-cmp
su
/opt/glite/etc/init.d/glite-proxy-renewald start
/opt/asap/etc/init.d/delegation start

É necessário cadastrar esta máquina na lista dos authorized_renewers da myproxy.cern.ch, enviando um e-mail com essa solicitação para pmendez@mailSPAMNOT.cern.ch com o subject da máquina, extraído da primeira linha de:

openssl x509 -text -subject -in /etc/grid-security/hostcert.pem

Logo depois é possível fazer o teste

[mdias@access ~]$ myproxy-init -s myproxy.cern.ch -c 504 -d -x -R "/DC=org/DC=doegrids/OU=Services/CN=access.sprace.org.br"
Your identity: /DC=org/DC=doegrids/OU=People/CN=Marco Dias 280904
Enter GRID pass phrase for this identity:
Creating proxy ................................................................................................................................................ Done
Proxy Verify OK
Your proxy is valid until: Thu May 21 13:00:52 2009
A proxy valid for 504 hours (21.0 days) for user /DC=org/DC=doegrids/OU=People/CN=Marco Dias 280904 now exists on myproxy.cern.ch.
[mdias@access ~]$ su 
Password: 
[root@access mdias]# myproxy-get-delegation -v -d -s myproxy.cern.ch -a /tmp/x509up_u537 
MyProxy v3.6 10 Aug 2006 PAM
using trusted certificates directory /etc/grid-security/certificates
server name: /DC=ch/DC=cern/OU=computers/CN=prod-pxsl4.cern.ch
checking that server name is acceptable...
server name does not match "myproxy@prod-pxsl4.cern.ch"
server name matches "host@prod-pxsl4.cern.ch"
authenticated server name is acceptable
A credential has been received for user /DC=org/DC=doegrids/OU=People/CN=Marco Dias 280904 in /tmp/x509up_u0.

Criando um source para variáveis do glite UI e setando as variáveis de ambiente

vim /usr/local/glite/3.1.25-0/etc/profile.d/cms_ui_env.sh
UITOP=/usr/local/glite
VER=3.1.25-0

if [ ! $GLITE_LOCATION ] ; then
    . $UITOP/$VER/external/etc/profile.d/grid-env.sh
    else
        echo "gLite environment already defined"
        fi

        #environment
        export PYTHONPATH=${PYTHONPATH}:${GLITE_LOCATION}/lib
exit
source /usr/local/glite/3.1.25-0/etc/profile.d/cms_ui_env.sh
export MYTESTAREA=/home/crab/sw_area
export SCRAM_ARCH=slc4_ia32_gcc345
CRABSERVER_version="CRABSERVER_1_0_7-cmp"
source ${MYTESTAREA}/${SCRAM_ARCH}/cms/crab-server/${CRABSERVER_version}/etc/profile.d/init.sh
export CRABSERVER_ROOT=$MYTESTAREA/${SCRAM_ARCH}/cms/crab-server/${CRABSERVER_version}
export PRODAGENT_WORKDIR=/home/crab/work
export PRODAGENT_CONFIG=$PRODAGENT_WORKDIR/CrabServerConfig.xml

Instalando o database

mkdir -p ${PRODAGENT_WORKDIR}/mysqldata
mysql_install_db --datadir=${PRODAGENT_WORKDIR}/mysqldata
mysqld_safe --datadir=${PRODAGENT_WORKDIR}/mysqldata --socket=${PRODAGENT_WORKDIR}/mysqldata/mysql.sock --skip-networking --log-error=${PRODAGENT_WORKDIR}/mysqldata/error.log --pid-file=${PRODAGENT_WORKDIR}/mysqldata/mysqld.pid &
mysqladmin -u root password 'XXXXX' --socket=$PRODAGENT_WORKDIR/mysqldata/mysql.sock

onde XXXXX é o password para o database. O usuário será "root".

Configurações iniciais

cd $PRODAGENT_WORKDIR
crabServer-new-config
crabServer-edit-config --component=ProdAgent --parameter=ProdAgentWorkDir --value=$PRODAGENT_WORKDIR
crabServer-edit-config --component=ProdAgentDB --parameter=dbName --value=CrabServerDB
crabServer-edit-config --component=ProdAgentDB --parameter=socketFileLocation --value=${PRODAGENT_WORKDIR}/mysqldata/mysql.sock
prodAgent-install-db
prodAgent-install-bosslite-db
crabServer-install-db
export myCacheDir=/home/crab/temp
mkdir -p $myCacheDir
crabServer-edit-config --component=CrabServerConfigurations --parameter=CacheDir --value="$myCacheDir"
crabServer-edit-config --component=CrabServerConfigurations --parameter=Protocol --value="srmv2"
crabServer-edit-config --component=CrabServerConfigurations --parameter=storagePort --value=8443
crabServer-edit-config --component=CrabServerConfigurations --parameter=storageName --value=osg-se.sprace.org.br
crabServer-edit-config --component=CrabServerConfigurations --parameter=storagePath --value="/pnfs/sprace.org.br/data/cms/crab"
crabServer-edit-config --component=CommandManager --parameter=Port --value=20081
crabServer-edit-config --component=ProdAgent --parameter=ProdAgentName --value=CrabServer@$HOSTNAME
crabServer-edit-config --component=JobKiller --parameter=KillerName --value=BossLiteKiller
crabServer-edit-config --component=JobStates --parameter=maxRetries --value=3
crabServer-edit-config --component=GetOutput --parameter=OutputLocation --value=SE
crabServer-edit-config --component=AdminControl --parameter=BotPeriod --value="04:00:00"
crabServer-edit-config --component=Notification --parameter=Notification_SMTPServer --value=localhost

O último comando é para setar os e-mails. Somente se você consegue receber o e-mail enviado abaixo, enviado via access:

mail -s "Subject" dest.address@cern.ch < mailText

OBS Não funcionaram

>crabServer-edit-config --component=GetOutput --parameter=skipWMSAuth --value=1
Error: Component GetOutput has no parameter named skipWMSAuth
Cannot edit non existent parameter
>crabServer-edit-config --component=CrabServerConfigurations --parameter=resourceBroker  --value=CERN
Error: Component CrabServerConfigurations has no parameter named resourceBroker
Cannot edit non existent parameter

Iniciando o serviço

crabServerd --start

Instalando alguns sensores

su
yum install sysstat
exit
crabServer-SysStat --start

Algum trabalho de pós instalação

encontrar em CrabServerConfigurations block section:

vim $PRODAGENT_WORKDIR/CrabServerConfig.xml
                
                
crabServerd --shutdown
crabServerd --start

Verifique que o serviço esta rodando em aqui

Operação

Desligar o serviço. Como root:

/opt/asap/etc/init.d/delegation stop
/opt/glite/etc/init.d/glite-proxy-renewald stop
su - crab
source variables.sh
crabServer-SysStat --shutdown
crabServerd --shutdown

Procure usando ps -aux os serviços mysql cujo user seja "crab" e use kill -9

ligar o serviço

/opt/glite/etc/init.d/glite-proxy-renewald start
/opt/asap/etc/init.d/delegation start
su - crab
source variables.sh
mysqld_safe --datadir=${PRODAGENT_WORKDIR}/mysqldata --socket=${PRODAGENT_WORKDIR}/mysqldata/mysql.sock --skip-networking --log-error=${PRODAGENT_WORKDIR}/mysqldata/error.log --pid-file=${PRODAGENT_WORKDIR}/mysqldata/mysqld.pid &
crabServerd --start
crabServer-SysStat --start

Updates

Marco em 29/04/2009

Alguns e-mails trocados para tentar colocar o crab server em atividade para o CMS, em hn-cms-t2@cern.ch, de Alessandra Fanfani:

> We deployed a CRAB server in our tier-2, but we have two questions:
> 1)Do you have an idea  how much space must be dedicated in our storage in
> order to make it available to all CMS community? We are planning a upgrade
> in our storage soon.

about 1-2TB. Few hundreds of GB are not enough to safely handle users 
sandboxes.
Is the storage area you are planning to use as CRAB server storage
a GridFTp server?

> 2)Do you know who we have to contact to put our new server configuration
> file at https://cmsweb.cern.ch/crabconf/files after that?

Did you already performed tests of the server with a local configuration 
file as suggested in:
https://twiki.cern.ch/twiki/bin/view/CMS/CrabServerInstallation#Testing_the_CrabServer
?
If yes, then you could ask to the crab development HN: 
  hn-cms-crabDevelopment@cern.ch

De Mattia Cinquilli:

>
> No, we are planning to use our dcache storage (srm) and allocate some
> space to CRAB jobs. About space, we will need to wait until our  
> upgrade to
> make it available to CMS community, for sure.
Here the problem is that the server can handle that protocol (and  
storage), but the WMS is not able to work with a dcache system (with  
different pools). So, actually, the solution is to have gridFtp  
installed on the same machine of the server with a dedicated disk/ 
partition, or (better performance) to have a dedicated machine with  
just gridFtp (as Legnaro setup).

Partimos para a configurar o servidor gridftp:

/usr/local/glite/3.1.25-0/edg/bin/edg-gridftp-ls -v gsiftp://osg-se.sprace.org.br/

deveria listar os diretórios. Entretanto temos problemas com permissões no dcache, que devemos contornar editando em *cada um * dos gridftp server da farm:

vim /opt/d-cache/etc/dcache.kpwd
mapping "/DC=org/DC=doegrids/OU=Services/CN=access.sprace.org.br" mdias
login mdias 503 503 / /pnfs/sprace.org.br/data /pnfs/sprace.org.br/data
        /DC=org/DC=doegrids/OU=Services/CN=access.sprace.org.br

onde mdias deve ser um usuário local nestes servers. Editando novamente o arquivo de configuração do crab: