Tags:
create new tag
, view all tags

Access to disk storage for LHC experiments at Cyfronet

Resources requiring valid grid credentials to access their resources

  • DPM server providing access to large disk arrays with some 240 TB total disk space available.
    In 2009 some 160TB is assigned for Atlas experiment.
    Server address: srm://dpm.cyf-kr.edu.pl/dpm/cyf-kr.edu.pl/home/atlas
  • Atlas space is divided into space token reserved areas:
    • Production space: ATLASMCDISK, ATLASDATADISK, ATLASPRODDISK
      namespace
      /dpm/cyf-kr.edu.pl/home/atlas/(atlasmcdisk)(atlasdatadisk)(atlasproddidsk)
      access only for atlas users with production role
    • Physics group space: ATLASGROUPDISK
      namespace
      /dpm/cyf-kr.edu.pl/home/atlas/(atlasgroupdisk)
      transfer requests allowed only via Web form
    • User space: ATLASSCRATCHDISK
      namespace
      /dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk
      user scratch space, cleaned automatically after 30 days
    • Polish user space: ATLASLOCALGROUPDISK
      namespace
      /dpm/cyf-kr.edu.pl/home/atlas/atlaslocalgroupdisk
      permanent space for users with polish group certificate: /atlas/pl'
      Replication to this space is requested via WebForm
    • Polish user space: ATLASLOCALGROUPDISK
      namespace
      /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk
      This namespace should be used for personal transfers using dq2 (lcg) client tools

Resources requiring only local authentication

  • local /home and /tmp directories on WNs
  • backed up home at /people/< user >
  • not backed up data storage at /storage/< user >
  • temporary scratch at /scratch-lustre/< user >

Grid authentication

  • set your proxy file to a fixed location, this will allow to use this authorization also from batch jobs
    export X509_USER_PROXY=/people/$USER/.globus/.vomsproxy
  • atlas voms proxy initilaization
    voms-proxy-init -valid 72:0 -voms atlas
  • or for proxy with "polish" group
    voms-proxy-init -valid 72:0 -voms atlas:/atlas/pl
  • then in batch jobs it is enough to define same X509_USER_PROXY in order to use the initialized proxy

Data handling commands on grid enabled storage elements - remote access

  • DPM SRM - dpm.cyf-kr.edu.pl - supports DPNS, RFIO, gridftp, srm protocols
  • Use rfio commands only for reading and storage cleanup
    In ~/.bash_profile
    export DPNS_HOST=dpm.cyf-kr.edu.pl
    export DPM_HOST=dpm.cyf-kr.edu.pl
    examples:
    rfdir /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08
    rfcp /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08/test.pool.root .
    quick remove of the whole directory tree:
    rfrm -fr /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08/
  • Users should write only to spacetoken areas with commands like srmcp, lcg-cp.
    Space outside spacetoken will be cleaned from time to time without a Warning!
  • Using dq2 client tool to replicate dataset to ATLASLOCALGROUPDISK
    dq2-get -L CYFRONET-LCG2_LOCALGROUPDISK -k ATLASLOCALGROUPDISK \
    -S srm://dpm.cyf-kr.edu.pl:8446/srm/managerv2?SFN=/dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08 \
    mc08.008801.Hijing_PbPb_5p5TeV_MinBias.simul.HITS.e379_s501_tid040379
  • srm examples:
    srmcp -space_token=ATLASSCRATCHDISK file:////tmp/hijing.1GB.root \
    srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
    srmcp srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1 file:////tmp/hijing.1GB.root

  • on srmV1 (port 8443):
    srm-get-metadata srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
    srm-advisory-delete srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
  • or on srmV2 (port 8446)
    srmls srm://dpm.cyf-kr.edu.pl:8446/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
    srmrm srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
  • lcg examples:
    lcg-cp --vo atlas --dst ATLASSCRATCHDISK file:////tmp/hijing.1GB.root \
    srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
    lcg-cp --vo atlas srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlassratchdisk/Andrzej.Olszewski/hijing.1GB.root.1 file:////tmp/hijing.1GB.root
LCG commands allow action on both storage and LFC catalogs that keep info about files on the Grid.

Data handling commands on grid enabled storage elements - direct access

XROOTD

Access to files in ATLAS global name space is described at FAX.

The xrootd access plugins have been installed on DPM servers at Polish Tier2 sites at Cyfronet and PSNC.

To use this mode of access:

  1. setup environment
    export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
    source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
    localSetupFAX
  2. STORAGEPREFIX will be set to local gateway
    at CYFRONET: STORAGEPREFIX=root://dpm.cyf-kr.edu.pl:1094
    at PSNC: STORAGEPREFIX=root://se.reef.man.poznan.pl:1094
    Global: export STORAGEPREFIX=root://atlas-xrd-eu.cern.ch:1094 (may not work yet)
  3. RUCIO_ACCOUNT should be set your ATLAS nickname (by default set to local account name which may be wrong)
  4. VOMS proxy should be valid

Example of use:

  1. check if dataset is available in FAX
    fax-is-dataset-covered data12_8TeV.periodH2.physics_Muons.PhysCont.NTUP_SMWZ.grp13_v01_p1067/
  2. listpath to files of the dataset
    fax-get-gLFNs data12_8TeV.periodH2.physics_Muons.PhysCont.NTUP_SMWZ.grp13_v01_p1067/
  3. copy file to local storage
    xrdcp root://dpm.cyf-kr.edu.pl:1094//atlas/rucio/data12_8TeV:NTUP_SMWZ.01014068._000133.root.1 /tmp/ao.test
  4. or use fax-get tool

-- AndrzejOlszewski - 1 June 2015

Topic revision: r11 - 2015-05-29 - AndrzejOlszewski
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback