How To Use SAM to Access CDF Data

Contents

• What is SAM?
• CDF Users Should be Aware of These Differences Between DFC and SAM
• SAM Registration
• Concepts and Definitions
• Preparing a SAM Job to Run on the CAF
• The CAF Shell Script (run.sh)
• The TCL Script
• CafGui and CafSubmit
• Remote CAF's
• Run a Test Job on One File on a Non-CAF Computer
• Recovery
• Monitoring

What is SAM

CDF Users Should be Aware of These Differences Between DFC and SAM

SAM Registration

Concepts and Definitions

Preparing a SAM Job to Run on a Fermilab CAF

There is a detailed CAF Users Manual with instructions on preparing jobs for the CAF. There is a link to it on the CAF Home Webpage . We do not repeat that information here, but only describe things required to use SAM.

Currently, we support general users running jobs on the CAF's at Fermilab using release 6.1.0. We also support small test jobs running on non-CAF machines. This will be expanded in the future. If you have an immediate need to use other releases or the remote CAF's and do not already know how to do that properly, contact cdfsam-admin@fnal.gov.

Currently, SAM administrators are recommending the following policy regarding access to large datasets from tape. If you are using SAM, you do not need to send requests to prestage data to the data handling group. If you receive an automated warning email about one of your jobs, forward it to cdfsam-admin@fnal.gov and let your job continue until you receive further instructions.

The CAF Shell Script (run.sh)

   source ~cdfsoft/cdf2.shrc
   setup cdfsoft2 6.1.0
   # as of 09SEPT2005
   setup diskcache_i -q GCC_3_4_3 v2_06_21
   setup dcap v2_32_f0408
   setup sam -q caf_prd

The TCL Script

1. Add this to the tcl script

     module disable ConfigManager
     module talk DHInput
       maxFiles set 5
       include dataset $env(SAM_DATASET)
       cache set SAM
     exit
   

   The parameter maxFiles is the maximum number of input files one segment will process and can be set to any integer greater than 0.
2. REMOVE from the tcl script anything that looks like this. This will overload the SAM servers and cause everyone serious problems. People will come looking for you if you do not.

     module talk DHInput
       include file file1.root
       include file file2.root
       include file file3.root
       include file file4.root
       include file file5.root
       include file file6.root
     exit
   

It is necessary to disable the "ConfigManager" to avoid database access problems.

For CAF jobs, the only "include" statements should be the one shown above, "include dataset $env(SAM_DATASET)". The design for CAF jobs is that the CAF submitter creates one project for many segments. This one project passes datafiles to many segments. If you have other commands of the form "include file", "include fileset", or "include dataset" in your TCL, then it will interfere with the proper deliver of data to your segments and create one project per include statement. There is a limit to how many projects the servers can support and they will rapidly be overloaded if many projects are started by one user.

maxFiles should be calculated as follows.

• maxFiles = (safety factor) * (queue time limit) / (time to process one file)

You will not use SAM if you do not include the "cache set SAM" statement. The default is to use the older DFC system.

CafGui and CafSubmit

To use the CAF Gui

• Enter "SAM" for the Method
• Enter the name of the dataset definition you wish to use for the Dataset
• Everything else is done the same whether you use SAM or not.

setup cdfsoft2 development

CafSubmit --dhaccess=SAM \
--tarFile=/cdf/scratch/YourUserName/tarFileWith_runshAndruntcl.tgz\
--outLocation=YourUserName@fcdflnxY.fnal.gov:/cdf/scratch/YourUserName/\$.tgz \
--procType=medium --group=common --dataset=xpmm0f \
--email=YourUserName@fnal.gov\
--start=1 --end=20 ./run.sh \$

The dataset definition can be one that already exists. In the future, we expect physics groups and individuals to create standard dataset definitions that are used by many people. In addition, for each "CDF dataset", a corresponding SAM dataset definition has already been created with the same name. We use "CDF dataset" to refer to the datasets that were defined before SAM. For example, the primary output datasets are "CDF datasets". Examples would be xpmm0f, or bhmu0d. There are instructions on how to create your own dataset definition here: How To Create SAM Dataset Definitions and Snapshots.

The number of segments is calculated as follows:

• Number of segments = (Total Number of Files) / (maxFiles from TCL)

Using Remote CAF's

Run a Test Job on One File on a Non-CAF Computer

Here is an example of how to create a dataset definition for a single data file for test purposes. How To Create SAM Dataset Definitions and Snapshots explains dataset definition in much more detail. Please, do not try to use this to create a dataset definition for each file in a large dataset with many files.

    sam create dataset definition \ 
        --defname="newDatasetDefinitionName" \
        --group=test \
        --defdesc="description" \ 
        --dim="FILE_NAME bd02d8e6.05bdhmu0"
  
    sam take snapshot --defname="datasetDefinitionName" --group=test

Alternative 1: Use environment variables

    export SAM_STATION=cdf-sam 
    export SAM_DATASET=datasetDefinitionName
    export SAM_PROJECT=yourUserName_`date +%Y%m%d_%H%M%S`

Alternative 2: Run a Script that Emulates the CAF Submitter

There is an alternative to the procedure described above which better emulates what really happens on the CAF. The major difference is that when you just set the environment variables, the DHInput module will automatically create the SAM project, however, the script below creates the project explicitly and DHInput just uses the existing project to get the files.

You need to copy the script below and run it. The script mimics the behavior of the CAF submitter. Be careful to not run this script on a CAF node, because the script creates a project. If a large CAF job was submitted and each of its segments created its own project, then resources would be overloaded. Further comments are given in the script. The script can be started with the command:

    thisScript.sh run.sh datasetDefinitionName > log 2>&1 &

#!/bin/sh

# checking the usage of the script
# no SAM specifics
script_name=${1}
if [ ! -x "${script_name}" ]
then
     echo "script name ${script_name} is not valid"
     echo "Usage ${0} script_name dataset_name"
     exit 2
fi

dataset_name=${2}

if [ -z "${dataset_name}" ]
then
     echo "The dataset name needs to be given!"
     echo "Usage ${0} script_name dataset_name"
     exit 3
fi

echo ${0} will run ${script_name} for dataset ${dataset_name}

source ~cdfsoft/cdf2.shrc

# setting up the SAM environment variables. 
# SAM specific needed for any interactive job
# either on the CAF or on a remote SAM station
set -x
echo "Setting SAM environment variables"
export SAM_STATION=cdf-sam
export SAM_DATASET=${dataset_name}
export SAM_GROUP=test
export SAM_PROJECT=${USER}_${SAM_STATION}_${SAM_DATASET}_`date +%s`_$$
export SAM_USER_NAME=${USER}
export SAM_QUALIFIER=caf_prd
export SAM_VERSION=
set +x

echo doing setup sam ${SAM_VERSION} -q ${SAM_QUALIFIER}
setup sam ${SAM_VERSION} -q ${SAM_QUALIFIER}
set -x

# starting the SAM project,
# this part is mimicing the behaviour of the CAF starting section,
# not needed for submission, e.g. on remote SAM stations.
echo starting project $SAM_PROJECT
sam start project --station=${SAM_STATION} --project=${SAM_PROJECT}\
--defname=${SAM_DATASET} --group=${SAM_GROUP} --retryMaxCount=200
SX=$?
echo return from sam start project $SX
echo Run section 1 on node $HOST by user $USER
set +x

export -n SAM_VERSION
# caf does not pass SAM_VERSION to the sections

# user job gets started
# needed for any job
${script_name} 1

# stopping the SAM project,
# this part is mimicing the behaviour of the CAF end section,
# not needed for submission, e.g. on remote SAM stations
set -x
echo ending project $SAM_PROJECT
sleep 10
sam stop project --force --station=${SAM_STATION}\
--project=${SAM_PROJECT} --retryMaxCount=20

Alternative 3: Modify your TCL

Do not use this approach on CAF. And do not use more than one "include" statement in your TCL. Each include will trigger the creation of a project. Users creating multiple projects will overwhelm the SAM system and possibly even crash it, upsetting everyone from management to the user next door.

The third alternative is to modify the TCL file. In some cases, this is the easiest and simplest approach, but it also tempting to try to use this to do business in the old DFC style by using "include file" many times. Do not do this.

In this approach, one modifies the TCL and then does not need to set environmental variables or do anything else. Just execute the script you would run on a CAF node. This also works interactively. The DHInput module will automatically create a project and generate its name.

    talk DHInput
      cache set SAM
      include dataset datasetDefinitionName
      maxFiles set 10
      sam
        station set cdf-sam
      exit
    exit

• The files that are delivered are those that satisfy the dataset definition constraints at the moment of interactive execution or CAF submission.
• DHInput will automatically take a snapshot and create a project.
• The sam station must be set by this tcl or by setting the environmental SAM_STATION.

    talk DHInput
      cache set SAM
      include file cd02d8df.00bddip0
      maxFiles set 1
      sam
        station set cdf-sam
      exit
    exit

Recovery

The SAM project keeps track of which files are successfully delivered and consumed. When a CAF job completes, it will send you an email which includes a "SAM Project Summary" as well as the report from CAF on the results of your job and its many segments. The section on monitoring explains how to get a "SAM Project Summary" if you are not running on CAF or lost the email. You can review these to determine which of your segments succeeded or failed. You also need to check the output of your job (log files and ntuples) to determine which segments succeeded or failed. Neither SAM nor CAF can be expected to catch all types of failures.

This part of the procedure needs further improvement and some people are working on this. Constructive input from users may help, although there are constraints inherent in SAM which will be impractical to change. Currently, there is a procedure that will allow you to recover failures which are detected by SAM. The script below will make a new dataset definition which will select the files that failed. If you run a new analysis job using this dataset definition, you can rerun your analysis on this data. Note that the segments will not have the same numbers as in the original job, nor will the assignment of files to segments be the same. Users need to deal with this bookkeeping issue on their own. This numbering issue is one of the constraints from SAM that it is impractical to change.

Currently, the only advice for rerunning segments that fail where SAM does not detect the failure is to rerun the entire job from scratch. SAM will not detect the following types of failures:

• The problem does not cause a crash or abort.
• The crash or abort occurs at the very end, after your job has sent a message to SAM saying the last file was successfully processed (consumed).
• The crash or abort occurs in the short interval between your process reporting to SAM that a file was consumed and requesting the next file.
• Sometimes but not always, SAM will not detect failed files delivered to the original proecess of a restarted CAF segment.

If the "SAM Project Summary" lists a file as delivered, then SAM delivered the file to the CAF segment, but the segment did not report back to SAM that all the events were successfully processed. This can be caused by a crash, abort, or CAF node restart.

One surprising behavior you might see. On rare occasions, there will be some CAF problem that causes a CAF segment to be stopped and restarted from scratch. SAM sees the restarted segment as a new additional segment. The files delivered and consumed by the original segment may or may not be considered to have failed. A new set of files is delivered to the restarted process. The restarted process will get a new additional CPID. You can detect this situation by counting the CPID's in the "SAM Project Summary". If there are more CPID's than the number of segments you started, then some restarts occurred during your job. In this case, you have to carefully count files to make sure all make it into the recovery dataset definition or start over from scratch."

    sam generate recovery project \
        --project="dataset_to_be_recovered" \
        --recoDefname="recovery_dataset"

In the script above, the "sam generate recovery project" command will create a list of all files which were consumed with the same Consumer Process ID as any of the single files that failed. That means that all files within the CAF segments which failed will get in the the recovery dataset definition. Usually that is the behavior you want to have. However, if you have a one to one relation between your input files and the output files you might not need to recover a whole CAF segment, but only get the individual files which failed. In this case you need to edit the recovery_by_file.sh script and change one line from:

  sam generate recovery project --project=$FAILED_PROJECT_NAME --printQuery > ${TMP_FILE_NAME}

  sam generate recovery project --ignoreConsumed --project=$FAILED_PROJECT_NAME --printQuery > ${TMP_FILE_NAME}

Monitoring

• You can use SAMTV to examine the progress of your job. It only updates one time per hour, but you can find the name of your project so that you can use it with other commands. At the bottom of the entry page is a help link that explains the information available there. Note that most CDF users will use the cdf-sam station.
• You can get a list of your projects and their names from the Database Browser. Start on this page . Then enter your user name in the "PROJECT_NAME" field, followed by "*" (a wildcard). Go to the "CREATE_DATE" row and click on the first "-" so that you get the most recent projects first. Then click on Submit. From there you can also find out if a project is running and get other information related to your project. Note that this assumes that you always put your username as the first part of your project name. This is true for CAF jobs. The SAM dataset definition names that you use on the CAF are also part of the project name so you can use wildcards around that to see all projects you ran on the CAF for a particular dataset.

• The SAM at a Glance webpage shows the status of the SAM stations, the dbservers and of the optimizer. This page can be used in order to diagnose problems which occur on these levels. A red dot in front of the links means that this service is unavailable. SAM stations with the monitor level critical are needed by many SAM users. Also SAM DB Servers will be used by many users and should not go out of service.

• Using the following command you can get a "SAM Project Summary",

  setup sam
  sam get project summary --project=<project-name>
  

  where you get the project name from SAMTV or the Database Browser. This gives you the ID of each consumer (each segment of your CAF job has a consumer ID, or CPID) and the number of files delivered, consumed, failed, etc. If you use the "-v" qualifier, you get a list of every file and to which consumer it went.