Administering and Using SAM Station

What is SAM station

Overview

1. Creating SAM Station
2. Starting SAM Station
3. Stopping SAM Station
4. Configuring SAM Station
5. Viewing the Station State
6. Connecting SAM Stations
7. Station options
8. Station environment variables
9. Tips and Tricks

1. Use sam_bootstrap
   Popular way of running station is use of sam_bootstrap package that automates actions like setting environment and running SAM executables ensuring persistence of the running state. Have ups set before proceeding.(See experiment specific instructions)
   1. Set up sam_bootstrap and locate server list file that is the instruction list file.
   2. Add line that would instruct sam_bootstrap to run station process in the format:

      station <station_environment> <station_version> <station_name> [ station options ]

      station_environment is the name of the sam_config environment you'd like to station to use.
      <station_version> <station_name> - self explanatory

      Example:
      station sm_central_analysis_prd v4_2_1_65 central-analysis 
      	    

   3. do "ups start sam_bootstrap"
   4. Check trace file ( $SAM_SERVER_HOME/station_*/trace ) for errors.
2. Start station directly.

   You'll need to set following packages to run station manually : sam_station , sam_config , sam_cp.

   Example:
   setup sam_station v4_2_1_65 -q GCC-2.95.2
   setup sam_config -q prd
   setup sam_cp
   smaster start [--name=<name>][--quiet|--verbose] [--nofork] [ other station options] 
   	

   If the name flag is omitted, the environment variable SAM_STATION must be set. 

1. smaster
   1. pmaster
   2. stagerng
      1. eworkerng
         1. samcp

If you want SAM to be started whenever the system reboots, you will need to make sure that your sysadmin has enabled automatic startup of UPS products (documented at http://www.fnal.gov/docs/products/ups/). Then, as a product maintainer, you will need to edit the file ${PRODUCTS}/.upsfiles/startup/<node>.products and insert commands similar to the examples shown below:

 
   echo "now invoking ups start sam_bootstrap" 
  /bin/su - sam -c \ 
     ". /usr/local/etc/setups.sh; \ 
       setup setpath; \ 
       export SAM_BOOTSTRAP_ENV=/fnal/ups/db/sam_bootstrap/sam_bootstrap_d0lxac1.env; \ 
       ups start sam_bootstrap" 
  echo "sam_bootstrap startup complete." 
        

ups stop sam bootstrap 
ps -ef | grep sam

The  add group command applies to a new group whereas the configure group command applies to a group which is already known to the station. The group itself must be known to SAM; use the SAM Web browsing tools to know about the valid groups (requests to create a new group should also be directed to sam-admin@fnal.gov  ). Note the difference between a group being known to the SAM system as a whole and a group being known to a particular station. If the station flag is omitted, the environment variable SAM_STATION must be set.

The max-disk flag specifies how much disk space the group may use on this station. The argument is the size allocated and a unit, for example 10G is 10 Gigabytes, and 10M is 10 Megabytes. The max-lock flag specifies the amount of disk space that is occupied by the files that the group group locks on disk (see the section on file locking). Note if you decrease the max-disk value station will uncache and erase files that belong to the group according to its cache replacement policy to match to the new group size. The max-projects flag specifies the maximum number of simultaneously run projects by the group. The  admin flag  specifies administrators of the group on the station (a group administrator on one station may not be an administrator of that group on another station. For example, a member of a group may setup a 'private' station on her desktop and dub herself an administrator of that group on the station). The value of the flag is a comma-separated list of UNIX user names of group administrators. The corresponding persons must be members of the group (from SAM's perspectives, see SAM Web browsing tools) in the first place.
 

Setting a Group's Cache Replacement Policy

$ sam set policy --group=<group_name> [--station=<station_name>] --policy=RANDOM|FIFO|LRU [--param=<policy-dependent-value>]

The policy determines what files are to be erased from disk when new deliveries are required. Note that the cache replacement policy does not affect files that group has locked on disk. Those files will not be removed until explicitly unlocked by the group (see the section on file locking). The currently implemented policies are Least Recently Used, First In First Out and Random. Some policies require a parameter whose meaning is policy-dependent.  Currently, only the Random algorithm requires a parameter which must be positive integer seed.
 

Locking of Files on Disk

$ sam lock|unlock file --file=<file_name> --group=<group_name> [--station=<station_name>]
 

Viewing the Station State

CDF Sam at a glance, D0 Sam at a glance : overview with little information about a special station.
CDF diagnostics, D0 diagnostics : get detailed information about one selected station.
CDF SAM TV, D0 SAM TV : get the current projects and deliveries to the stations.

These pages provide in principle the same information as the sam dump station command. Because the pages are updated every few minutes these pages could show that your station is down although you just restarted it. The other point is that if your station is in the yellow state it does not necessarily mean that your station has problems. It can also mean that your server is behind a firewall or some other network filter or server is temporarily busy processing another request and could not respond to a ping before timing out (3 seconds). In addition, the station's configuration, together with the information about currently cached files (and files being delivered, if any), is contained in a simple text utility:
 

$ sam dump station [--disks | --projects | --groups | --files={cached|requested|all} | --all [--station=<station_name>]
 

$ sam dump station --station=d0ppdg-wisconsin --groups --disks --projects --fsman
 *** BEGIN DUMP STATION d0ppdg-wisconsin version v4_2_1_57 running at diet.cs.wisc.edu 65 days 22 hours 20 minutes 55 seconds, admins: abaranov garzogli terekhov 
 Replica selection: prefer (d0mino enstore), avoid (empty)
 There are 0 authorized transfer groups
 Full delivery unit is enforced; external deliveries are unconstrained
 Excess consumer satisfaction: 1
 AUTHORIZED GROUPS:
  group dzero: admins: abaranov garzogli terekhov , swap policy: LRU, fair share: 1, quotas (cur/max): projects = 0/-1, disk: 44558656KB/52428800KB, locks:0B/52428800KB
 
 STATION DISKS:
 disk 1635 diet.cs.wisc.edu:/export/sam/cache, 7870143KB/52428800KB = 15% free 
 station disk total: 7870143KB/52428800KB = 15% free
    

 PROJECT MANAGER: fileReleaseTO = 1 days : maxConsumer Wait time = 1 days, max prefetched files : 5
 NO PROJECTS (746 already ended, 163 prematurely)
 FAIR SHARE MAN:
  Benefit weights: volumes mounted: 0.2, CPU: 0.2, KBytes transferred from MSS: 0.2, KBytes transferred inter-station: 0.2, files consumed: 0.2
 *** END OF STATION DUMP ***
    

After station is installed, disks , nodes, groups and samcp are configured you'll need to supply arguments that define station criteria to select source locations for files in your projects as well as configure global routing rules to benefit from other stations caches if protocols at your site do not allow immediate access to the experiment mass storage systems.

D0 installation must use global routing to gain access to enstore. Example :

 
--prefer-loc=enstore --routing-station=enstore::central-router --routing-group=dzero --routing-user=<your station name> 
    

1. Source location selection options
   1. --prefer-loc=str[,str2...]
      Source location prefer criteria that prioritizes all locations available for a file. Location should match *str* in order to be selected as a preferred location. The order of the preferred locations defines the order in which file locations are going to be tried.
   2. --omit-loc=str[,str2...]
      Avoid all locations that match *str* pattern
   3. --bad-loc-tolerance=<num> --bad-loc-expire=<min>
      "Bad location" is the source location that will be avoided when several replicas exist for the same file. Tolerance defines number of files that failed to be transfered from the particular location within expire time before this location is considered "bad" for all files that reside there. Default values are 3 and 4 hours respectively.
2. Cache fit related options
   1. --max-delivery-unit-size=<num> Default: 5
      If optimizer authorizes unit with more than --max-delivery-unit-size files this unit will be broken down to --max-delivery-unit-size chunk units.
      
   2. --min-delivery=<numKb> Default: Inifinite
      Normally, the station will deliver all or none of the files in the group of size set by --max-delivery-unit-size. If the entire group cannot be fit on disk and the min-delivery flag is set to a number in KBytes, then the station will attempt to deliver a fraction of the group but at least as much as the value of the flag. If the value of min-delivery is zero, there is no minimum unit of delivery (i.e., a single file may be delivered). Most of the remote station need this option to avoid cache stalls as big chunk deliveries do not make much sense.
      
   3. --opter-suffix=<string>
      Development option.
3. Routing
   If every station node had access to all data store locations then there would be almost no need for routing. In practice, there are reasons why stations will need to route files such as certain incoming transfers only go to a restricted set of nodes, limit the number of external transfer enabled nodes, or to route everything through your big cache, i.e.slow the cache turnover.
   1. --constrain-delivery=node1[,node2...]
      Unconditional route of all incoming files through node1,node2...
   2. --route=<location>::node1>[,<node2>...]
      Conditionally select all files whose selected source locations match location( regular expr.) to be routed through node1,node2,... Flag works on both external and internal transfers and overrides --constrain-delivery for relevant locations.
      Note: It is a valid case to route files that reside on a particular station node to itself. In such case intrastation transfers from such nodes will not be possible.
   3. --routing-station=<regexp::station_name> --routing-group=<group> --routing-user=<user>
      These are the options that take care of the situation when station does not have direct access to the group of files but allowed to request them on the behalf of other station. regexp is the regular expression that defines location pattern of the file group to be requested by the remote station "station_name". routing-group and routing-user are set of credentials under which request should be made. These options offer high level Global File Routing capabilities that obsolete administrative efforts to set up remote staging areas. Within the same station regular routing rules apply.
   4. --routing-station-metrics=<station::metrics> Default: 1
      Sets the number of files transfered concurrently from "station" via global routing.
      
   5. --routing-station-group=<station::group>
      Overrides default group used to authorize routing request within router.
   6. --routing-public-node=<node>
      Designates node that will be used by the remote stations to route through.
4. File transfer management
   1. --common-timeout=<min> Default: 60
      
   2. --intrastation-timeout=<min> Default: 8 hours
      Timeout values that define how much time transfer request is allowed to pend until station terminates it and declares an error. Applies to incoming and intrastation transfers respectively.
   3. --retry-attempts=<num> Default: 3
      This argument defines number of station retry attempts to deliver file from the single source. If file has multiple locations station will try each of them sequentially until delivery is successful otherwise delivery error is reported.
      
   4. --retry-interval=<num<s |m>> Default: 1 minute
      How much time should station wait until the next attempt to deliver file that had delivery failures.
      
5. Project resource management
   1. --file-release-timeout==<mins> Default: 24 hours
      The flag specifies the maximum time a project is allowed to hold files and not release any of them. The feature is targeted to reap seemingly abandoned projects (when the user abandons his/her project, the project still "uses" files given to it, if any, as far as the station is concerned).
      
   2. --consumer-wait-timeout==<min> Default: 24 hours
      The flag specifies the maximum time a project is allowed to wait and hold predelivered files before establishing consumption. In practice you set this time to a value that is the max job wait time in the queue.
      
   3. --max-prefetched-files=<num> Default: 5
      --max-prefetched-files allows station to balance delivery performance between projects. In particular it forces only configured number of external files to be staged by the station in parallel. This options is useful to ensure that no project jams the station delivery queue and underlying transport resources.
      
   4. --max-project-file-usage=<num> Default: Infinite
      The max-project-file-usage flag limits the number of files that a project can hold (use) at the same time. If more files are available in cache at the time the project is started, they also may be given to the project, however, no more files will be requested from HSM on the project's behalf once it is using "too many" files. The feature is designed to guard HSM from requests for files for projects that are abandoned or proceed to slowly to "deserve file deliveries".
      
   5. --excess-satisfaction=<num> Default: 1
      Defines project share in the cache by specifying how many files should be protected on the every project node from removal during cache turnover in addition to those that are already protected by the project's direct use. File is in use if it was given to any of its consumer processes. This option also defines the "size" of the file packet that is going to be requested when consumer process file need condition is detected.
      
   6. --aggressive-replication Default: off
      Turns off station mode in which replication within cache is done with respect to the immediate need for files by the project consumer processes.
      
   7. --pmaster-arg
      Passes any option to the pmaster process. Example --pmaster-arg=--consumption-map=fnal.gov::d0mino.

   8. Project master argument : --consumption-map=<regular expr>::node1,node2,node3
      This argument sets mapping between consumer process location and the list of locations accessible by the consumer process. Regular expression defines locations of the consumer processes option should be applied to. List of nodes defines locations where station will stage files to for those consumer processes that were matched by the regular expr.
6. General options
   1. --log-file==<log_file_name>
      Option sets the name of the log file to dump debug station output to. Log file is dated and closed at midnight , new one is created with the following date.
   2. --auto-stager=[yes|no]
      Whether station attempts to start stagers itself or not.
      
   3. --stager-arg="--max-transfers=5"
      This argument can be used to pass arbitrary option to the auto-stager. Auto-stager is a stager process spawned by the station automatically during startup.
7. Each non conflicting option has cumulative effect otherwise last option in the list takes precedence.
8. Please note that in regular expressions . or * symbols must be escaped with \ i.e. \. or \* when shell is used to run the SAM server.

1. Q: No file deliveries. Trace full of messages like constrained to: none
   A: check whether all disks are active including those involved in routing. Check whether routing options are not conflicting.
2. Q: No file deliveries. File is in cache but project consumer process is still waiting.
   A: There was communication problem between project master on the head node and consumer process. Restart process.
3. Q: Project gets error. Trace has messages like no valid locations
   A: Check whether there were delivery errors in the past. Station might have disabled those locations and does not try them anymore. Check whether samcp works by invoking it manually with -v -d flags and arguments you cut and paste from the station trace. Check whether you have set routing correctly so that your files that you request are available via protocols you support.
   To clear disabled location you may use sam undeclared bad location command on individual location or restart station as a whole.
4. Q: I've got No active disks at node errors.
   A: Check whether disks are active on the node your process running. Check whether you have consumption-map set so that processes are mapped onto the right consumption nodes.
5. Q: No deliveries. Station does not seem to be doing anything.
   A: Station delivers new files if projects release already delivered ones. Check if that is the case.
   A: There is DB communication failure that is reported every 10 minutes. Check the reason and wait or restart station if DB failure persists without apparent reasons.
   A: Station send request to optimizer but response has not been received. Check whether station is accessible (no firewall on station port) from optimizer machine.
   A: Station requested transfers but those didn't report back. Check transfer options to tune acceptable wait timings.
   A: Station disks are too small for projects lock their files there.
   A: The group you are running your project against is not configured with the adequate space.
6. Q: I'm running behind fire wall.
   A: Station requires a site to have bi directional access to the database,name and optimizer services. These services are centralized and well defined within SAM domain. You can use -OAhost -OAport arguments in the station config line to force it to bind machine public interfaces. In such case your cluster worker nodes do not have access to the outside world separate nameservice needs to be installed to gain access to the station interfaces at least.
7. Q: Something is not right. There are core files in the trace directory.
   A: Contact sam-admin@fnal.gov for support. Do not forget to attach relevant trace and core files, preferably compressed.

=============================================================================
Project : SAM
Package : sam
$Id: stationConfig.shtml,v 1.18 2004/07/15 19:27:24 abaranov Exp $

This work is part of a development project, called SAM, which consists of a
number of coordinated packages each named sam_xxxx .

Notice of authorship, copyright status, and terms and conditions, should
the software eventually become available for use outside Fermilab, can be
found in the README and LICENCE files in the top level directory of the main
sam package.

==============================================================================