Difference: Tier2StorageAccess (10 vs. 11)

Revision 112015-05-29 - AndrzejOlszewski

Line: 1 to 1
META TOPICPARENT name="CyfronetWeb"

Access to disk storage for LHC experiments at Cyfronet

Resources requiring valid grid credentials to access their resources

  • DPM server providing access to large disk arrays with some 240 TB total disk space available.
    In 2009 some 160TB is assigned for Atlas experiment.
    Server address: srm://dpm.cyf-kr.edu.pl/dpm/cyf-kr.edu.pl/home/atlas
  • Atlas space is divided into space token reserved areas:
      access only for atlas users with production role
    • Physics group space: ATLASGROUPDISK
      transfer requests allowed only via Web form
    • User space: ATLASSCRATCHDISK
      user scratch space, cleaned automatically after 30 days
    • Polish user space: ATLASLOCALGROUPDISK
      permanent space for users with polish group certificate: /atlas/pl'
      Replication to this space is requested via WebForm
    • Polish user space: ATLASLOCALGROUPDISK
      This namespace should be used for personal transfers using dq2 (lcg) client tools

Resources requiring only local authentication

  • local /home and /tmp directories on WNs
  • backed up home at /people/< user >
  • not backed up data storage at /storage/< user >
  • temporary scratch at /scratch-lustre/< user >

Grid authentication

  • set your proxy file to a fixed location, this will allow to use this authorization also from batch jobs
    export X509_USER_PROXY=/people/$USER/.globus/.vomsproxy
  • atlas voms proxy initilaization
    voms-proxy-init -valid 72:0 -voms atlas
  • or for proxy with "polish" group
    voms-proxy-init -valid 72:0 -voms atlas:/atlas/pl
  • then in batch jobs it is enough to define same X509_USER_PROXY in order to use the initialized proxy

Data handling commands on grid enabled storage elements - remote access

  • DPM SRM - dpm.cyf-kr.edu.pl - supports DPNS, RFIO, gridftp, srm protocols
  • Use rfio commands only for reading and storage cleanup
    In ~/.bash_profile
    export DPNS_HOST=dpm.cyf-kr.edu.pl
    export DPM_HOST=dpm.cyf-kr.edu.pl
    rfdir /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08
    rfcp /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08/test.pool.root .
    quick remove of the whole directory tree:
    rfrm -fr /dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08/
  • Users should write only to spacetoken areas with commands like srmcp, lcg-cp.
    Space outside spacetoken will be cleaned from time to time without a Warning!
  • Using dq2 client tool to replicate dataset to ATLASLOCALGROUPDISK
    -S srm://dpm.cyf-kr.edu.pl:8446/srm/managerv2?SFN=/dpm/cyf-kr.edu.pl/home/atlas/dq2localgroupdisk/Andrzej.Olszewski/mc08 \
  • srm examples:
    srmcp -space_token=ATLASSCRATCHDISK file:////tmp/hijing.1GB.root \
    srmcp srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1 file:////tmp/hijing.1GB.root

  • on srmV1 (port 8443):
    srm-get-metadata srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
    srm-advisory-delete srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
  • or on srmV2 (port 8446)
    srmls srm://dpm.cyf-kr.edu.pl:8446/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
    srmrm srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlasscratchdisk/Andrzej.Olszewski/hijing.1GB.root.1
  • lcg examples:
    lcg-cp --vo atlas --dst ATLASSCRATCHDISK file:////tmp/hijing.1GB.root \
    lcg-cp --vo atlas srm://dpm.cyf-kr.edu.pl:8443/dpm/cyf-kr.edu.pl/home/atlas/atlassratchdisk/Andrzej.Olszewski/hijing.1GB.root.1 file:////tmp/hijing.1GB.root
LCG commands allow action on both storage and LFC catalogs that keep info about files on the Grid.

Data handling commands on grid enabled storage elements - direct access


In ATLAS a global file access is being developed under name of FAX. This allows access of files without needing to know where the files are physically. This way of access is using a redirection mechanism available in xrootd and Scalla services of ROOT.
Access to files in ATLAS global name space is described at FAX.
The xrootd access plugins have been installed on DPM servers at Polish Tier2 sites at Cyfronet and PSNC. Until FAX is fully operational one can use direct access to files available on these sites using this xrootd protocol which is fast an reliable.
The xrootd access plugins have been installed on DPM servers at Polish Tier2 sites at Cyfronet and PSNC.
To get a path to any files on srm:
To use this mode of access:
  1. setup environment
    export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
    source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
  1. find site name of a dataset replica
    dq2-ls -r "dataset name"
  2. get path to files on srm
    dq2-ls -fp -L "site name" "dataset name"
  3. replace srm prefix by xrootd prefix
    CYFRONET: root://dpm.cyf-kr.edu.pl:1094/
    PSNC: root://se.reef.man.poznan.pl:1094/
  4. Example:
    dq2-ls -fp -L CYFRONET-LCG2_DATADISK user.ivukotic.xrootd.cyfronet-lcg2
    convert ->
    (note // after :1094)
To get a path to files from registered datasets:
  1. setup environment
    export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
    source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
  2. check site where dataset replica exists
    dq2-ls -r "dataset name"
  3. set STORAGEPREFIX according to site you want to access
    CYFRONET: export STORAGEPREFIX=root://dpm.cyf-kr.edu.pl:1094
    PSNC: export STORAGEPREFIX=root://se.reef.man.poznan.pl:1094
    Global: export STORAGEPREFIX=root://atlas-xrd-eu.cern.ch:1094 (may not work yet)
  4. get file paths
    dq2-list-files -g "dataset name"
    convert them adding manuallY $STORAGEPREFIX
  5. Example:
    export STORAGEPREFIX=root://dpm.cyf-kr.edu.pl:1094
    for f in `dq2-list-files -g user.ivukotic.xrootd.cyfronet-lcg2`; do echo $STORAGEPREFIX$f'; done
To use a file in ROOT:
  • setup environment
  • TFile::Open("xrootdpath")
In Athena job (probably, not tried):
InputCollections = [ 'xrootdpath' ]
  1. STORAGEPREFIX will be set to local gateway
    at CYFRONET: STORAGEPREFIX=root://dpm.cyf-kr.edu.pl:1094
    at PSNC: STORAGEPREFIX=root://se.reef.man.poznan.pl:1094
    Global: export STORAGEPREFIX=root://atlas-xrd-eu.cern.ch:1094 (may not work yet)
  2. RUCIO_ACCOUNT should be set your ATLAS nickname (by default set to local account name which may be wrong)
  3. VOMS proxy should be valid

Example of use:

  1. check if dataset is available in FAX
    fax-is-dataset-covered data12_8TeV.periodH2.physics_Muons.PhysCont.NTUP_SMWZ.grp13_v01_p1067/
  2. listpath to files of the dataset
    fax-get-gLFNs data12_8TeV.periodH2.physics_Muons.PhysCont.NTUP_SMWZ.grp13_v01_p1067/
  3. copy file to local storage
    xrdcp root://dpm.cyf-kr.edu.pl:1094//atlas/rucio/data12_8TeV:NTUP_SMWZ.01014068._000133.root.1 /tmp/ao.test
  4. or use fax-get tool
-- AndrzejOlszewski - 1 Dec 2013
 \ No newline at end of file
-- AndrzejOlszewski - 1 June 2015
 \ No newline at end of file
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback