Sending Remotely Produced Data Files to SAM

1. Introduction

Data files produced outside of the FNAL site can  be easily entered in to the sam/Enstore  system. If the data can be transferred to FNAL via network connections, this is encouraged since it is the most straightforward. However, in many cases the amount of data to be transferred is much larger than the network  bandwidth  allows, and exchangeable media transfer is required.  All data that is entered into the sam system must have a description file.  In some cases, an additional parameter file is referenced and should be sent with the description file.  When data is imported into the system via tape,  the tapes can be placed directly into the D0 robot if an additional enstore-metadata file is provided and a properly labeled and formatted tape is employed.  Please contact sam-design@fnal.gov if your institution has not imported data to this area before, and you will be sent information on file name conventions,  etc., and preparations will be made for your data.
 

2. Network Transfers

Some of our collaborators have  network connections which provide transfer of 1 to 2 GB per hour.  This makes sending data over the network very attractive.  If you are planning many large network transfers to Fermilab, please contact us first so we can warn the networking people. There will probably be security issues with this mode of transfer in the future related to the strong authentication computer security being implemented at Fermilab.  We will resolve these problems as we understand them. There is buffer space reserved on d0mino for these transfers and it is located at  /sam/cache1/import.  Transfer files to this area along with their description files and any parameter files referenced in the description files.

The procedure for sending data via the network:
 

  1. send the following to the /sam/cache1/import area on d0mino at fermilab via ftp:
  2.  change the protections on the files you put there so sam can remove them after they are properly stored: Use chmod 777 file_name.
Although the names on your files do not need to conform strictly to any particular rules, some conventions are very useful for storing and maintaining the data. Please
include the institution name or initials in the description, partition, and file names. It is useful to include a date (link 20000131_1655 for Jan 31, 2000 at 16:55)  at the end of the data file names to help avoid uniqness problems later on. There MUST NOT be any other "." in the description file name other than the one before the ".py". following are some examples:
 
sim.pmc02_01_nikhef.pythia.ttbar_mb1.1av_200evts.20000131_1655     for the data file
import_pmc02_01_nikhef_ttbar_mb1_20000131_1655.py       for the description file
pmc02_01_nikhef.ttbar_mb1.20000131_1655 .params    for the parameter file


There is a cron job which periodically looks for .py description files, and stores the associated data and parameter files into SAM.  Therefore, please  be sure that the .py file is the last file sent for each set of data files.  Files which are not successfully imported by the cron job will be moved into the  /sam/cache1/import/problem area until they are fixed.
 

3. Tape Transfers


Transfers of data into the system via tape are provided by physically shipping the tape to Fermilab and loading it into the d0 robot in The Feynman Computing Center. The tapes must be written using a utility provided by Enstore called enstore_tape , available from Fermi KITS. In some remote installations the rules and procedures  of the local computing center do not allow the direct use of such a utility, and special procedures need to be worked out with their computing staff. Please contact sam-design@fnal.gov if more information is needed.  The types of media supported in the robot are Exabyte Mammoth I, and DLT 7000 compatible cartridges. Only certain external labels are allowed in the Fermilab system and the bar codes must be compatible with the bar code reader in the robot.  D0 will supply bar coded labels or pre-labeled tapes for you to use for import.

The procedure for sending tape

  1. Initialize the tape using the enstore_tape init  utility.
  2. Write the tape using the enstore_tape write function.
  3. When the tape is full,  pack up the tape information using the enstore_tape dump-db function.
  4. Send the following meta-data files to the /sam/cache1/import_tape area on d0mino at fermilab via ftp:
  5. change the protections on the files you put there so sam can remove them after they are properly stored: Use chmod 777 file_name.
  6. Ship the tapes to: Lee Lueking, Fermilab, Box 500 (ms 357),Receiving Dept., Kirk & Wilson, Batavia, IL 60510. (This address may change when we get into heavy production).
The same conventions apply to the names as described in  section 3 above. In addition, the name for the enstore metadata file should look like the following example:
import_uta_PRF588.tape_meta
When the physical tape is received at Fermilab it will be inserted into the robot. The filenames  and other information about each file on each tape will be  added to the Enstore system from the tape metadata. Finally, the information in the description file will be declared to the sam system and the data will then be available for access.
 

4. Problems

If there are problems adding data to the sam/enstore system, the problem data will be moved into the problem directory under the import, or import_tape directories. If assistance is required by the administrator in charge of this area, you will be notified by mail. Please address problems to sam-design@fnal.gov.

=============================================================================
Project : SAM
Package : sam
$Id: SamRemoteDataGuide.html,v 1.2 1999/11/11 16:42:14 lueking Exp $

This work is part of a development project, called SAM, which consists of a
number of coordinated packages each named sam_xxxx .

Notice of authorship, copyright status, and terms and conditions, should
the software eventually become available for use outside Fermilab, can be
found in the README and LICENCE files in the top level directory of the main
sam package.

==============================================================================