User Manual for Monthly.csh Programming Package

User Manual for Monthly.csh Programming Package

Overall Function of Programming Package

Monthly.csh is a programming package that invokes C Shell scripts, C program, and Kumac(PAW) that query the database for information on analysis projects, prints the information onto tables, performs mathematical manipulations, creates histograms, plots data, and creates an HTML page.

There are 2 basic sets of output from this package:

Shared data file usage diagrams
The plots produced are of the percentage of shared data files in a pair of analysis projects, between a given interval of time. And the total amount of analysis project pairs viewed between a given interval of time. The http link that displays the monthly.csh output is on the Sam Plots page under Data Access Statistics.
User and filename extracts
A secondary output (available 2/1/02) of monthly.csh is a set of files that can be used to perform further studies. They contain the following information for all analysis projects used in the plots/diagrams above:
1. users.csh
  (all data from the ANALYSIS_PROJECTS table)
  - project_id - analysis project id
  - person_id - person performing the analyis
  - work_grp_id - working group the person is associated with
  - create_date (MM/DD/YY 24HH:MM:SS) of the analysis project
  - start_time (MM/DD/YY 24HH:MM:SS) of the analysis
  - end_time (MM/DD/YY 24HH:MM:SS) of the analysis
2. files.csh
  - project_id (ANALYSIS_PROJECTS table) - analysis project id
  - kbyte_file_size (DATA_FILES table) - file size in Kbytes
  - file_name (DATA_FILES table) - file (dataset) name
The individual fields are whitespace delimited in the files.
Since this is run on a monthly basis and the data will be overlapping when a set of APLODAYS (see configuration file) are used, the user and filename extracts are performed only on the smallest interval (APLODAYS) in the monthly.config file. Currently, this is 30 days (as of 2/1/02).
These data files can be accessed here and are also available from the Sam Plots page. They are stored as tar balls and can be downloaded as needed (for instructions on opening these files click here. The naming convention used is similar to the one used for the diagrams and plots showing the low, high and midpoint project id used. This was done so one could relate back to the diagrams, if needed.

users.csh${APlo}-${APhi}_${data}_loc${APmid}.tbl

for example, users.cshl47903-50558_raw_loc47913.tbl

Monthly.csh Variables

Monthly.csh has one optional command line argument, a date (format MM/DD/YYYY). The statistics and data collected is based on offsets from this date. The default is current date.

(Note: This command line argument was added to allow for re-creating these statistics for periods other than current date. The time is set to 00:00:00 so, effectively, data is collected for the prior day (yesterday) and back.)

The other variables used to create these plots are:

APLO-lowest analysis project id calculated from the highest analysis project id, or the most current project,
APLODAYS-the number of days in the past the user would like to view,
RANGE - used to determine the APlo-APmid range of analysis projects.
APHI-the most recent analysis project found at the time the process
DATA-the type of data file (e.g. raw, reconstructed, special).

The variables RANGE, APLODAYS and DATA can be changed by the user in a configuration file used by monthly.csh.

Configuration File

Monthly.config is a configuration file read by monthly.csh. This text-only file is included in the same directory that contains the other files that make up the complete programming package. The user inputs key words (e.g. range, APloDays, no_data_tier) to define the variables. The user can only enter one range value, but multiple values for APloDays and no_data_tier.

The range value is used in monthly.csh to define the APlo-APmid range in analysis project. The APloDays is the number of days in the past the user is interested in viewing from the current date. And, no_data_tier are the data types that the user wants excluded from viewing.

Monthly.csh Environment variables

At the beginning of the file monthly.csh the environment variables for the locations of the file output directories and subdirectories are specified. The environment variables are:

ABSOLUTE-the parent directory,
ROOT_DIR- the directory that holds temporary files,
TABLE_DIR- the directory that holds all tables from sql queries,
FINAL_DIR- the directory that holds the output of the HTML page,
OUTPUT_DIR- a subdirectory of FINAL_DIR that holds the current HTML page which includes plots and tables,
SIZE_RPT- a subdirectory of FINAL_DIR that holds the page that is linked to each plot that holds the plot and table of size of data files,
PAST_RPT_DIR- subdirectory of FINAL_DIR where the past HTML pages are located, and
IMAGE_DIR- subdirectory of FINAL_DIR that holds all images used in HTML page.
TARBALL_DIR- subdirectory of FINAL_DIR that holds a zipped tar ball of the outputs for the users.csh and files.csh data.

Opening the tarballs

Opening the tarballs is a 2 step process. When created the original files were zipped, them put in a single tar file. This was done to reduce the space needed on the server they ran on. This also reduce the download time.

tar -xvf {filename}.tar
gunzip users.csh.....tbl.gz
gunzip files.csh.....tbl.gz

Change History:

7/24/02 (Weigand)
- Added the command line argument for date to allow for re-creation of plots and data. Previously, the process was driven off current date and required hacking to reconstruct a previous period.
- Modified many of the queries to speed the processing.
- Modified the table file for changes that had been made to sam_config which is what stopped this process from running since 5/1/2002.
- Modified monthly.csh to append 'sam_access_plots_{date/time}' to the ROOT_DIR variable for temporary files. This makes identifying (and cleanup) of these temp files easier to associate with a given execution.

If there are any problems, questions or concerns, please contact Gabriele Garzoglio at garzogli@fnal.gov or ext. 6470.