SAM Installation: sam_bootstrap

Introduction

Before you install sam_bootstrap, you should install and tailor sam via:

  upd install sam -q -G -c
  ups tailor sam 

SAM Bootstrap requires a separate directory with write privileges for the user (usually sam) who will be running the servers. This directory is used for configuration files, server log files, etc. All servers controlled by the package should be listed in the server list file. Each line of this file controls one server, and each line must have the form:

server sam_configuration_qualifier ups_product_version [station_name] [server options]

Currently, the supported servers are:

Server	Ups Product	Notes
station*	sam_station	SAM station manager and cache management. There should be ONE station master per SAM station.
fss*	sam_station	File Storage server, required only for storing files into the SAM system. There should be at most ONE FSS per SAM station.
stager*	sam_station	The station manager will, by default, start its own stagers for communicating with all of the cache disks. Separate stagers need to be started if you are using additional disks for file storage (i.e., running an FSS). If you are running an FSS, stagers are required on all nodes (master and workers).
bbftp	sam_bbftp	SAM bbftp file transfer protocol daemon, required if you will be using sam_bbftp as a file transfer protocol.
gridftp	sam_gridftp	SAM gridftp file transfer protocol daemon, required if you will be using sam_gridftp as a file transfer protocol.
fcp	fcp	FCP server is used for limiting total number of incoming and outgoing file transfers that use kerberos rcp as a file transfer protocol.
nameservice	orbacus -q services	Central name and address registry for all SAM servers. At this time, the nameservice is only supported on d0db-prd, d0db-dev and d0db-int.
logger	sam_logger	Central logging service for all SAM servers; currently supported only on d0db-prd, d0db-dev, and d0db-int.
dbserver	sam_db_server	Central SAM database service
dbs	sam_db_srv (replacement for sam_db_server)	Central SAM database service after the v5_1_1 schema upgrade.
dim	sam_dimension_server_prototype	Central SAM dimension server, required for sam_db_srv post b6_1_0_0
pnfs	sam_pnfs_srv	Central SAM pnfs server, required for sam_db_srv post v6_4_0_1
optimizer	sam_optimizer	Central SAM optimizer, for ordering file transfer requests; currently supported only on d0db-prd, d0db-dev and d0db-int.

Servers which require a valid station name (i.e., station name that is known to the SAM database) are denoted by an asterisk (*). The "SAM configuration qualifier" should be one of the valid sam_config qualifiers (added during the ups tailor sam phase mentioned above), while the "ups product version" for the different servers must refer to their corresponding ups products given in the above table. Each server may be given one or more runtime options which will override the default ones.

A SAM station will typically contain one "master" node, which runs the SAM station master and is responsible for the cache management of all nodes operating within the cluster; and zero or more "worker" nodes. Sam_bootstrap will need to be tailored separately for master and worker nodes.

Important:
The master node runs the SAM station master and (optionally) FSS. There should be only one station master per SAM station. There should be at most one FSS per SAM station. SAM worker nodes run only SAM stagers. SAM_MAIL_RECIPIENT should always be set to an address local to the station, rather than to sam-auto@fnal.gov.

Typical Server Options:

Typical Remote SAM Server Options

Server	Option	Description
station	--constrain-delivery=d0mino.fnal.gov --min-delivery=1k	Files are routed through d0mino.
fss	<none>	Usally no options are required
stager	--with-fss --without-sm --max-transfers=N	MASTER node stager options
stager	--with-fss --with-sm --max-transfers=N	WORKER node stager options

Environment variables

   d0mino> setup sam_bootstrap
   d0mino> echo $SAM_BOOTSTRAP_ENV
   /usr/products/upsdb/sam_bootstrap/sam_bootstrap.env
   d0mino> cat $SAM_BOOTSTRAP_ENV
   SAM_SERVER_HOME=/home/sam/private
   export SAM_SERVER_HOME
   SAM_MAIL_RECIPIENT=local-admin@my.node
   export SAM_MAIL_RECIPIENT
   SAM_SERVER_LIST=node_based
   export SAM_SERVER_LIST
   SAM_ENCP_CONFIG="d0en"
   export SAM_ENCP_CONFIG

In the above example SAM Bootstrap uses directory /home/sam/private as its working directory, all e-mail notifications will be sent to local-admin@my.node, and the server list file is based on the node name, in which case

it is set to ${SAM_SERVER_HOME}/${HOST}_server_list.txt. The encp configuration is d0en.

Ups commands

ups tailor sam_bootstrap

This command should be called whenever you want to update the SAM bootstrap configuration. In particular, you should use this command immediately after you install the sam_bootstrap product onto your system (e.g., after executing upd install -G"-c" sam_bootstrap). It allows one to modify the above mentioned environment variables, and, if necessary, to create the server list file in the correct form. Once the server list file is created, or if it already exists, the tailor script will automatically install all ups products needed for running the given combination of SAM servers. Note that this command must be run under the products account (or a different account which has the necessary permissions to install products into the ups database).

Here is an example run on sammy (a Linux development node):

sammy:> ups tailor sam_bootstrap

$ ups tailor sam_bootstrap

Enter SAM_BOOTSTRAP_ENV [default: /fnal/ups/db/sam_bootstrap/sam_bootstrap.env]? 
Using /fnal/ups/db/sam_bootstrap/sam_bootstrap.env as sam_bootstrap environment file.

Enter SAM_SERVER_HOME [default: /home/sam/private]? 
SAM_SERVER_HOME is set to /home/sam/private.

Enter SAM_MAIL_RECIPIENT [default: sam-auto@fnal.gov]? lauri@fnal.gov
SAM_MAIL_RECIPIENT is set to lauri@fnal.gov.

Enter SAM_SERVER_LIST [default: /home/sam/private/sammy_server_list.txt]? 
SAM_SERVER_LIST is based on node and set to /home/sam/private/sammy_server_list.txt.

Enter appropriate ENCP configuration [default: d0en]? 
SAM_ENCP_CONFIG is set to d0en.


Server list file /home/sam/private/sammy_server_list.txt does not exist.
During configuration process temporary file /tmp/lauri_sammy_server_list.txt
will be created. This file will have to be moved by user sam to
/home/sam/private/sammy_server_list.txt.


Now determining default versions of server software, please stand by...
.....................

Ready to configure your servers.
Configure a station? [default: false]: y
Enter the environment? station_prd
Enter the sam_station product version: [default: v4_1_0_3]? 
Station name? d0-demo-station
Options to the server [default: None]? --constrain-delivery=d0mino.fnal.gov --min-delivery=1k

Configure another station? [default: false]: 

Configure a stager? [default: false]: y
Enter the environment [default: station_prd]? 
Enter the sam_station product version: [default: v4_1_0_3]? 
Station name [default: d0-demo-station]? 
Optional server configuration file [default: None]? 

Configure another stager? [default: false]: 

Configure a stager_old? [default: false]: n

Configure an fss? [default: false]: y
Enter the environment [default: station_prd]? 
Enter the sam_station product version: [default: v4_1_0_3]? 
Station name [default: d0-demo-station]? 
Optional server configuration file [default: None]? 

Configure another fss? [default: false]: 

Configure an optimizer? [default: false]: 

Configure a dbserver? [default: false]: 

Configure a logger? [default: false]: 

Configure an infoserver? [default: false]: 

Configure a nameservice? [default: false]: 

Configure fcp? [default: false]: 

Configure a sam_bbftp daemon? [default: false]: y
Enter the sam_bbftp product version: [default: f1_9_4c]? 

Configure a gridftp? [default: false]: y
Enter the environment? dev
Enter the sam_gridftp product version: [default: v1_6]? 
Options to the server [default: None]?

Configure another gridftp? [default: false]:

Using file /tmp/lauri_sammy_server_list.txt:

station station_prd v4_1_0_3 d0-demo-station 
stager station_prd v4_1_0_3 d0-demo-station 
fss station_prd v4_1_0_3 d0-demo-station 
bbftp station_prd f1_9_4c
gridftp station_prd v1_6

for installing needed ups products...

Installing encp -G "-c -q d0en"...
informational: encp v2_17 already exists on local node, skipping.
Installing orbacus -q sam_station -G "-c"...
informational: orbacus v3_3_4 already exists on local node, skipping.
Installing sam_bbftp f1_9_4c -G "-c"...
informational: beginning install of sam_bbftp.
informational: transferred
       /ftp/products/sam_bbftp/f1_9_4c/Linux+2/sam_bbftp_f1_9_4c_Linux+2
        from fnkits.fnal.gov to
        /fnal/ups/prd/sam_bbftp/f1_9_4c/Linux+2
informational: transferred
       /ftp/products/sam_bbftp/f1_9_4c/Linux+2/sam_bbftp_f1_9_4c_Linux+2/ups/.
        from fnkits.fnal.gov to
        /fnal/ups/prd/sam_bbftp/f1_9_4c/Linux+2/ups
informational: transferred
       /ftp/products/sam_bbftp/f1_9_4c/Linux+2/sam_bbftp_f1_9_4c_Linux+2.table
        from fnkits.fnal.gov:/ to
        /fnal/ups/db/sam_bbftp/f1_9_4c.table.new
DECLARE: A UPS tailor exists for this product
informational: ups declare succeeded
DECLARE: A UPS start/stop exists for this product
informational: ups declare succeeded
Installing sam_station v4_1_0_3...
informational: sam_station v4_1_0_3 already exists on local node, skipping.
Installing sam_user -G "-c"...
informational: sam_user v4_1_1 already exists on local node, skipping.
Execute the following to resolve chains: 
ups declare -f Linux+2.2 -q "" -g current sam_user v3_0_2 -z /fnal/ups/db

==================================================================

If you haven't already done so, you will need to tailor sam_bbftp.
To do this, from a product maintainer account:

     ups tailor sam_bbftp f1_9_4c

Then, from the sam account:

     ups installAsSam sam_bbftp f1_9_4c

You need to initialize the server_list file.
Please execute the following command as user sam:

     cp /tmp/lauri_sammy_server_list.txt \
         /home/sam/private/sammy_server_list.txt

In the above example the default values for many variables were accepted (note the empty response lines). The server list file did not exist, and therefore had to be created. The user is prompted for the necessary information for each type of server that might be needed. All required products are then automatically installed. (Any ups declare commands printed out after installation of the needed ups products can usually be ignored).

At the end of the configuration, the temporary server list file (owned by the product maintainer) must be copied into the actual server list file owned by user sam.

If sam_bbftp hasn't already been installed on this node, you will need to perform the tailoring and installAsSam steps:

$ id
uid=1275(lauri) gid=4525(products) groups=1530(g023),4525(products),4525(products)
$ 
$ ups tailor sam_bbftp

   The sam_bbftp product requires an authorization file
   that is owned by, and readable only by, user sam.

   This file should not be in an NFS-exported area.
   Ideally, it should be located in a secure area
   where only 'root' has privileges.  If this is
   not possible, a non-shared file system such as
   /tmp is satisfactory.

   NOTE, this file must exist and be readable by user
   sam from both sam_bbftp clients and servers.

Enter the full path to the sam_bbftp authorization file:
  (default: /tmp/.sam_bbftp/.sam_bbftp_cookie) 

   You should now create the sam_bbftp authorization file
   by using the command

        ups installAsSam sam_bbftp f1_9_4c

   You will need to know the appropriate authorization string
   to enter when requested.

   You must use an account which can create the 
   /home/sam/.sam_bbftp/.sam_bbftp_cookie file and set the file ownership
   to user sam.  (Usually, this is the sam account).

$ 
$ ksu sam
Authenticated lauri@PILOT.FNAL.GOV
ksu[10130]: 'ksu sam' authenticated lauri@PILOT.FNAL.GOV for lauri on /dev/pts/0
Account sam: authorization for lauri@PILOT.FNAL.GOV successful
ksu[10130]: Account sam: authorization for lauri@PILOT.FNAL.GOV successful
Changing uid to sam (7816)
sammy% 
sammy% ups installAsSam sam_bbftp
This is the sam_bbftp authorization procedure.
Please enter the sam_bbftp authorization string: 
Now creating sam_bbftp authorization file /tmp/.sam_bbftp/.sam_bbftp_cookie
Authorization is complete.

In case fcp server is used for controlling the maximum number of rcp transfers, the fcp ups product has to be configured. Note that this is only useful if fcp -c wrapper is used for sam_kerberos_rcp. Assuming that the fcp root directory is set to $FCP_ROOT the following commands have to be executed from a product maintainer account:


ups tailor -O $FCP_ROOT -f $OS_FLAVOR $FCP_PRODUCT_VERSION
ups tailor sam_kerberos_rcp (use "fcp -c" for rcp wrapper)

The necessary directories have to be created and configuration files have to be edited from the sam account:


mkdir -p $FCP_ROOT/cfg
mkdir -p $FCP_ROOT/bin
mkdir -p $FCP_ROOT/log
setup fcp $FCP_PRODUCT_VERSION
cp $FCP_DIR/cfg/fcp.cfg.example $FCP_ROOT/cfg/fcp.cfg

At this point edit the $FCP_ROOT/cfg/fcp.cfg file, and decide on is the maximum number simultaneous transfers. If logging isn't necessary, comment out the log_file variable, otherwise set it to $FCP_ROOT/log.

If sam_gridftp hasn't already been installed on this node, you will need to install it and it's dependencies, by following these temporary instuctions. This is due to a bug in ups.

INSTALLATION OF GLOBUS DATA HANDLING TOOLS AND
GLOBUS SECURITY INFRASTRUCTURE CONFIGURATION
---------------------------------------------
Each product gives instructions on what to do next.
Please follow instructions on the screen carefully.
The following steps describe the procedure
for a system that is installing globus for the first
time.

-------------------
Installation of the globus data handling server tools
- as products:
> upd install globus_dh_server -G-c
-------------------
Installation of the globus security infrastructure
- as root:
# setup globus_dh_server
# ${GLOBUS_LOCATION}/setup/globus/setup-gsi
-------------------
Installation of the globus data handling client tools
- as products:
> upd install globus_dh_client -G-c
-------------------
Installation of the globus security infrastructure
configuration tools
- as products:
> upd install sam_gsi_config -G-c
-------------------
Configuration of the globus security infrastructure
and request of the sam server certificate: ONLY THE
FERMILAB Kerberos CA IS TRUSTED.
-as root:
# ups tailor sam_gsi_config
# setup sam_gsi_config
# sam_cert_request
You will need to repeat the last 2 steps on each machine requiring 
gridftp server or client access.
-------------------
Configuration of the authorization file (grid-mapfile)
for the sam servers (putting the following line in
a cron job updates the list automatically)
-as sam
>  ups get_gridmap sam_gsi_config


---------------------------------------------------------
INSTALLATION PROCEDURE FOR SAM_GRIDFTP.
This procedure should be improved by listing sam_gridftp
in the tailoring script of sam_bootstrap

-------------------
Install the globus data handling tools and GSI (instructions
above)
-------------------
Install the new sam products that know about gridftp:
-as products:
>upd install sam_gridftp
Eventually this step will be done by sam_cp. Currently I prefer
to do it by hand, since a bug in upd (ticket 27906) may cause
the installation of globus (if not there already) to hang the
sam_cp installation.
>setup globus_location
>ups tailor sam_gridftp
(defaults OK)
>upd install sam_cp -G-c
>edit sam_cp_config file to include sam_gridftp as a protocol
 for your sam installation and include the new protocol, as
 by instructions on the screen (file to modify: 
 ${UPS_THIS_DB}/sam_cp_config/Config/sam_cp_config.py)

ups start sam_bootstrap

The start command is used for starting all SAM servers listed in the server list file. It has to be executed under the sam account. Note that server control scripts monitor running servers. In the case of the server failure an e-mail is sent to $SAM_MAIL_RECIPIENT (at most one mail per hour per server), server trace and core files are saved, and the server is restarted.

ups stop sam_bootstrap

The stop command will gracefully stop all SAM servers listed in the server list file. It must be executed from the sam account.

ups update sam_bootstrap

The update command can be used for updating servers after the product upgrade or after adding new or removing old servers. It will compare the most recently stored server list file to the current one to determine which servers have to be stopped, and which ones have to be started. Note that servers which do not change will not be affected. This command must be executed from the sam account.

ups restart sam_bootstrap

The restart command will stop all servers, then restart them. Use this for a fresh start of all servers on a particular machine. This command must be executed from the sam account.

ups inquire sam_bootstrap

This command is convenient for determining the SAM Bootstrap configuration, and can be run under any account. For example, on d0minowe have at the moment (see Section 2):

d0mino> ups inquire sam_bootstrap
SAM_SERVER_HOME is set to "/home/sam/private".
SAM_MAIL_RECIPIENT is set to "sam-auto@fnal.gov".
SAM_SERVER_LIST is based on node and set to "/home/sam/private/d0mino_server_list.txt".
SAM_ENCP_CONFIG is set to "d0en".

ups stopdbservers sam_bootstrap

For use on d0ora1 and d0ora3 only. This command will stop only the dbservers running on the particular node, leaving other services (such as the nameservice) alone. Use this command when the database is down but other services should still be functional.

ups startdbservers sam_bootstrap

For use on d0ora1 and d0ora3 only. This command will start only the dbservers running on the particular node, leaving other services (such as the nameservice) alone. Use this command when the database is brought back up.

Other commands

In addition to the above commands, one can also start or stop a particular server using the following procedure (as user sam):

    logon as sam
    setup sam_bootstrap
    ls ~sam/private/*.txt  (retrieves a text file)
    cat text_file found above
    from the text_file, find the version dbserver,logger, etc. is running
    exec run.sh ACTION SERVER QUALIFIER UPS_PRODUCT_VERSION [STATION_NAME] &

Here, "action" can be either start or stop, while the other arguments are the same as before. For example, in order to start the development DB server from the ups product sam_db_server v2_2_2 on d0ora3, one would login to d0ora3 as user sam, and execute:

<d0ora3> setup sam_bootstrap
<d0ora3> exec run.sh start dbserver dev v2_2_2 ""&

As another example, in order to stop the development station central-analysis on d0mino, from the ups product sam_station v2_2_3 one would do (again as user sam):

<d0mino> setup sam_bootstrap
<d0mino> run.sh stop station dev v2_2_3 central-analysis

Recommended standard procedures and examples

Server upgrades or modifications on D0 CAS (d0mino)

Updated 20Aug2002 to reflect new server list file configuration!!!

Node d0mino hosts the central-analysis sam station, the fss stagers for several remote sam stations, and the station master for the cab (central-analysis-backend) sam station. Each of these "types" of server are listed in separate server list files.

Before invoking any sam_bootstrap command, you must make sure that your SAM_BOOTSTRAP_ENV environmental variable is pointing to the correct server list environment:

central-analysis	What to set up:	$ setup sam
	$SAM_BOOTSTRAP_ENV:	/usr/products/upsdb/sam_bootstrap/CENTRAL-ANALYSIS_bootstrap.env
	Server_list file:	~sam/private/CENTRAL-ANALYSIS_server_list.txt
cab	What to set up:	$ setup -q cab sam
	$SAM_BOOTSTRAP_ENV:	/usr/products/upsdb/sam_bootstrap/CAB_bootstrap.env
	Server_list file:	~sam/private/CAB_server_list.txt
remote-stager	What to set up:	$ setup -q remote-stager sam
	$SAM_BOOTSTRAP_ENV:	/usr/products/upsdb/sam_bootstrap/REMOTE-STAGER_bootstrap.env
	Server_list file:	~sam/private/REMOTE-STAGER_server_list.txt

The default d0mino_server_list.txt file is a placeholder only, and should not contain any "real" server entries.

To upgrade the central-analysis station, use the following steps:

#
# as user sam:
#
$ setup sam                         # set default value for SAM_BOOTSTRAP_ENV
$ setup sam_bootstrap               # initialize default bootstrap environment
$ ups inquire sam_bootstrap         # determine location of server list file
$ <edit> server_list_file           # change product versions, add/remove servers, etc.

#
# as user products:
#
$ ups tailor sam_bootstrap          # install appropriate SAM products

#
# as user sam:
#
$ ups update sam_bootstrap          # stop obsolete servers, start new servers
  

Similar steps are used to update the other stations/servers, but make sure to set up the correct SAM_BOOTSTRAP_ENV environment first.

Server upgrades or modifications on farms

On farms there may be mutiple sets of SAM servers running from the same ups space. This example assumes that the SAM Bootstrap environment file for the IO node (d0bbin) and worker nodes are sam_bootstrap_d0bbin.env and sam_bootstrap_worker.env, located in /home/sam, respectively.

The IO node the upgrade procedure would be as follows:

#
# as user sam on node d0bbin:
#
$ setup sam_bootstrap               # initialize default bootstrap environment
$ export SAM_BOOTSTRAP_ENV=/d0farm/ups/db/sam_bootstrap/sam_bootstrap_d0bbin.env
                                    # use d0bbin bootstrap configuration
$ ups inquire sam_bootstrap         # determine location of server list file
$ <edit> server_list_file           # change product versions, add/remove servers, etc.
#
# as user d0flib on node d0bbin:
#
$ setup sam_bootstrap
$ export SAM_BOOTSTRAP_ENV=/d0farm/ups/db/sam_bootstrap/sam_bootstrap_d0bbin.env
$ ups tailor sam_bootstrap          # install appropriate SAM products

#
# as user sam:
#
$ setup sam_bootstrap
$ export SAM_BOOTSTRAP_ENV=/d0farm/ups/db/sam_bootstrap/sam_bootstrap_d0bbin.env
$ ups update sam_bootstrap         # stop obsolete servers, start new servers

For the worker nodes one cannot use the update command, since all nodes use the same startup log file, and the update command would work correctly only for the first node on which it would be executed. Because of that, one has to restart all servers on all worker nodes.

#
# as user sam on one of the worker nodes:
#
$ setup sam_bootstrap               # initialize default bootstrap environment
$ export SAM_BOOTSTRAP_ENV=/d0farm/ups/db/sam_bootstrap/sam_bootstrap_worker.env
                                    # use worker node bootstrap configuration
$ ups inquire sam_bootstrap         # determine location of server list file
$ <edit> server_list_file           # change product versions, add/remove servers, etc.

#
# as user d0flib on one of the worker nodes:
#
$ setup sam_bootstrap
$ export SAM_BOOTSTRAP_ENV=/d0farm/ups/db/sam_bootstrap/sam_bootstrap_worker.env
$ ups tailor    sam_bootstrap       # install appropriate SAM products

#
# as user sam on all worker nodes:
#
$ setup sam_bootstrap               # initialize default bootstrap environment
$ export SAM_BOOTSTRAP_ENV=/d0farm/ups/db/sam_bootstrap/sam_bootstrap_worker.env
                                    # use worker node bootstrap configuration
$ ups restart sam_bootstrap         # stop/start all services

Server installation on a new machine

The following procedure assumes that ups on the new machine has been configured properly (i.e., it has to allow qualifiers in the product directory names), and that this particular machine runs under one of the supported flavors of UNIX. Also, one has to make sure that the hostname, as well as the station names chosen for this particular machine, are known to the SAM database.This can easily be accomplished by sending e-mail to sam-admin@fnal.gov. Here are the steps needed to get SAM servers going:

#
# as user root:
#
- create sam account
- create sam working directory (typically, ~sam/private) and set ownership
   to user sam

#
# as user products:
#
$ setup upd
$ upd install sam                          # install sam_user portion
$ upd tailor sam                           # add needed configurations
$ upd install -G"-c" sam_bootstrap         # install sam server portion
$ ups tailor sam_bootstrap                 # follow the instructions

#
# as user sam:
#
- copy the temporary server_list.txt file into the
   actual server_list.txt file

$ setup sam_bootstrap
$ ups start sam_bootstrap

Note that, while the above steps are sufficient to start all configured SAM servers, in the case of SAM station one has to perform certain administrative tasks after station is up and running. These tasks include adding and configuring groups, adding disks to the station cache, etc.

Troubleshooting

Places to look for information:

• ~sam/private/log contains a log of sam_bootstrap activity (i.e., starting and stopping servers). It will list each server that is started and stopped.
• ~sam/private/server__node__env/: If a server does not seem to be starting correctly even though sam_bootstrap thinks it is started, look in the trace file in the appropriate server directory (e.g., to find the trace file for the dbserver running on d0ora3 for the dev environment, you'd look in ~sam/private/dbserver__d0ora3__dev). This might give clues, especially if the server is crashing immediately upon being started.

=============================================================================
Project  : SAM
Package  : $Name:  $
Revision : $Revision: 1.1 $
Modified : $Date: 2002/04/17 16:03:42 $ by $Author: lauri $

This work is part of a development project, called SAM, which consists
of a number of coordinated packages each named sam_xxxx.  Notice of
authorship, copyright status, and terms and conditions, should the
software eventually become available for use outside Fermilab, can be
found in the README and LICENCE files in the top level directory of
the main sam package.
=============================================================================