NEWS: The sam package is becoming an actual package! As of v7_0_0, sam is now a frozen executable image. The individual components are not distributed separately. This results in much improved performance, especially for NFS-mounted distributions. The sub-components that comprise the frozen sam package are: sam_user_pyapi: user interfaces sam_admin_pyapi: administrative interfaces sam_mis_pyapi: monitoring and info gathering utilities samgrid_batch_admin_pyapi: administration of batch configurations v8_2_2 - Change the identifier for 'sam get dataset' downloadPlugin from $ to @, since it's very hard to get sam_config to put a literal $ in an environment variable. - Allow sam store ot specify a relative source path on the command line. v8_2_1 - 'sam get dataset' can now be given a more flexible downloadPlugin specification, and the plugin and node name can be set by environment variables. - Commands that require the group to be set on the command line will now also accept it from the environment variable SAM_GROUP (except for a couple where this does not make much sense). - The web page generated by 'samadmin encp synch' now has a link to the enstore tape status web page, as this is the definitive list of noaccess/notallowed tapes. v8_2_0 - *BACKWARDS INCOMPATIBLE*: Move sam cancel file transfer request to samadmin. This command is not very safe and having it as a normal sam command just encourages its use. Any scripts calling this command will have to be changed. - Add sam get consumer process ids command. - Add new feature to Request System reconstruction handlers for D0 reprocessing (number of files) v8_1_6 - Add --extraCpidList option to sam generate recovery project Rewrite recovery project to add an option to check the process status. strict recovery project now just calls the standard recovery project command with the options set to use the process status _and_ the last file state. - Add the consumer process status to get project summary. - Add process description option to sam establish default process. - Fix an issue with sam get next file timeouts not ending the process like they are supposed to. v8_1_5 - Rewrite generate recovery project to not use a simple list of files, and so avoid the 1000 entries in an Oracle list limit - Add samadmin allow station project and disallow station project aliases (The only station admin commands which didn't have station in their names). v8_1_4 - Include email.* modules, which are required by smtplib, which is required by the sammis nameservice commands. v8_1_3 - sam generate recovery project now includes files that were seen by a process which completed successfully, but were not consumed for whatever reason (ie skipped files). - Add the count of the number of processes to sam get project summary - Add --minimalOutput flag to sam establish consumer, establish process, and get next file. This sends just the ID number or path to the new file to standard output, so shell scripts do not have to strip off the extraneous characters. - Exception handling of some station samadmin commands did not work correctly in v8_1_1 and v8_1_2. v8_1_2 - sam run project was broken in batch mode (but not interactive) - Fix Params struct to behave consistently depending on how it is created (from sam_common_pylib) v8_1_1 - Use python 2.4.4 & omniORB 4.1.0. - sam end file processing can now take a process description instead of an ID, and can also now end all consumers belonging to a single project. - sam translate constraints --summaryOnly uses a new db server interface which means it does not have to send the data for individual files to the client (requires sam_db_srv_pkg v8_1_0). - Add 'sam get dbserver database connection info' command (requires sam_db_srv_pkg v8_1_0). - Add option to 'sam establish process' to set the process description. - Disable retries by default for sam dump project: otherwise it takes a very long time for the command to terminate when trying to dump a finished project - Add HTML anchors to all headings in sam at a glance v8_1_0 - Build against sam_idl v8_1. (This is still compatible with v7_7 and v8_0 db servers, the only change is in the station interface.) - Add 'sam upload file' command. This command transfers data to a user specified location and upon success adds that location to the file replica location list. The command has the same interface as 'sam request file transfer' with the exception that you don't need to use station name. - New env. variable SAM_USE_FILE_UPLOAD has been defined to override sam store and sam request file transfer to use sam upload instead. The SAM_USE_FILE_UPLOAD env variable is in effect only if the destination is not /pnfs. The env. variable is there to avoid changes in d0_runjob. - Changes to sam dh get lite that make use of station interface to enable generation of transfer urls by SRM service. - Make sure the absolute path of the script directory is put on the import path. This matches the normal python behavior. - 'samadmin uncache station disk' now first tries the path as a disk mount point. If that fails it will fall back to assuming it is the full path of a data storage location (the old behavior). v8_0_0 - Use v8_0_0 of sam_idl_pylib (functionally identical to v7_7) - Added support to transfer/store files from the nodes which do not have sam stager running. User can specify a temporary staging location using temporaryTransferLocation to stage the files. This location needs to be accessible by stager. - Fixed a bug in samProject lite consumer that prevented getting more than one files from sam. - Add a check in sam request file transfer that the file name in the metadata actually matches the source file name. Before, if they didn't then if the given source file name existed it would ignore it and store the file name in the metadata, and if the source file didn't exist at all it would fail. Now it will always fail if there's a mismatch. - Add extra information to sam project summary: the dataset def for the project and the individual process descriptions. v7_7_1 - Fix pickle not working with frozen sampy - Add --ignore-noaccess flag to samadmin encp synch to skip noaccess tapes - Add documentation on using sampy v7_7_0 - Use v7_7 of sam_idl. - Fixes to SamTime for adding and subtracting months when some months are longer than others - samadmin uncache commands didn't work - if the station was running it told you to stop it; if the station was not running it just returned a CORBA TRANSIENT exception. - The current directory was not on the python import path. This has been changed to make it consistent with standard python behaviour. - Add environment variable SAMPYPATH (similar to PYTHONPATH). This allows you to add to the sam search path without exposing modules to any other version of python you might be using. (If you are using the -O python option then SAMPYPATH will _not_ add anything to the python search path.) - sampy startup prompt now displays the sampy version as well as the python version. v7_6_5 - Allow multiple threads in the nameservice mis commands (disabled by default, use --numThreads=n to activate). - sam calculate file crc no longer requires ecrc (or the encp product). - Remove encp and fileinfo from the table file. They should be set up internally as needed. (upd will no longer install them automatically, even if you do need them). - Remove gnuplot and ximagetools from the table file. I think the plotting tools will set up the required products internally if available. - Add the ability to block files below a certain size from being stored to pnfs. Set the environment variable SAM_MIN_PNFS_FILE_STORE_SIZE to enable this. Note that it is intended to catch careless users, not act as an absolute block, and is easy to bypass. - Add a --locations flag to 'sam get registered data disks' to allow displaying the data storage locations - --comment is now required by the command definition of 'sam modify file content status' (since it didn't work if you didn't include it) v7_6_4 - improved sam generate strict recovery project with --printQuery and --cpidList arguments v7_6_3 - Removed the dependancy of sam2Online of sam_user_pylib on environment variable "HOSTNAME"; - Use SAM.CRC_Data( SAM.CRC_UnknownValue, SAM.CRC_TC_Unknown_Old ) as the default CRC value in sam2Online so that the fss will update it after the transfer is done. v7_6_2 - Use updated sam_common_pylib to get RawEventInfo changes for D0 online - Add resize station disk command v7_6_1 - Fix online timeout (timeout was set to only 1 second) - Change traceback for most of the sam products to not to show the full path to the original source code as that is irrelevant to where it is now running. Only the relative package path will now appear. - Surpress python tracebacks when the user hits ctrl-c in a sam command. v7_6_0 - Add --summaryOnly flag to translate constraints - Modify the method for obtaining the version number: the old way didn't work when the frozen release was used with an external python interpreter - Port "sam dh get lite" from v5 sam_user_api - Add changes for raw data logging from Dehong - When resubmitting a file store into enstore, first check that there is no existing pnfs location v7_5_3 - Fix samadmin encp synch for the case where a tape has been deleted more than once v7_5_2 - Add noSummary option to sam translate constraints which makes it output a simple one file per line list of files. - Various sam_common_pylib changes (see separate release notes) - Use omniORB v4_0_7 - sam get dataset: add --node to override sam_download default consumer process name, add --downloadPlugin to override samcp with user specified command v7_5_1 - SamUpsMgr change broke encp synch. - Remove the host authorization checks for the nameservice cleanup. v7_5_0 - Convert sampy to use python 2.4.2 - Use omniORB v4_0_6 - Add -O python option to setup to add the necessary entries to PYTHONPATH and LD_LIBRARY_PATH to run with external python interpreter - If no script name is given and stdin isn't a terminal then sampy now tries to read code from it (as does proper python). Allows you to pipe python code into sampy. - Remove some of the complications in SamUpsMgr (made possible by the ability to read code from stdin) - In the nameservice object iterators were not being destroyed after use, nor were unbound contexts being destroyed. These both potentially cause memory leaks in the nameservice. - Add get dataset command v7_4_1 - Add dbserver_chooser to sam at a glance so you can browse logfiles - Fix sam store so that it works when descriptionFile option is used - Restore translate constraints output to old format and add option to get file sizes v7_4_0 - Build against sam_idl_pylib to gain access to new interfaces - added establish default consumer process command - added add consumption node command - added delete consumption node command - added commit consumer process command - added generate strict recovery project command v7_3_5: - Built against sam_common_pylib v7_3_3 for fix to user authentication by grid proxy - Bug fix to 'samadmin update user' command which didn't work at all v7_3_4: - Built against sam_common_pylib v7_3_2 to fix translate constraints bug when total size is larger than about 1e12 bytes. - changed run project to not use SAM_USER_PYAPI to locate GenericRunProjectScript: that variable is undefined in a batch job - modified interactive sam submit script to try and get the behaviour right with signals v7_3_3: - fixed quoting bug with cloning batch adapter configurations that have complex batch commands v7_3_2: - sam.translateConstraints now accepts new --countOnly option, calls different dbserver method in that case; will not be useable until v7_4_0 interface is adopted, however. - sam.translateConstraints now correctly handles the "--dim=help [--type=category]" syntax, even though this is Just Plain Wrong. (The translateConstraints interface should be used ONLY for returning a list of translated constraints in a DataFilePhysicalAttributesList structure; it should not be overloaded to return a string in the case of "--dim=help" -- use "get dimension info"). But who am I to decide these things? - build against sam_common_pylib v7_3_1 to gain the fix to DbServantConnectionIdentifier so that users running an interactive sampy session will show up with the appName as 'sampy' instead of blank. v7_3_1: - sam_user_pyapi sam.establishConsumer: calls new establishConsumer method in station (requires sam_station v6_0_1_13 or higher) passing full ApplicationFamily information; if not implemented in station, revert to the old establishNewConsumer method, which does not allow for appFamily (default is 'unknown' in this case). Fixes long-standing bug, reported most vociferously by Art Kreymer and Liz Buckley-Geer (MINOS). v7_3_0: - rebuild agains sam_idl_pylib v7_3_0, sam_common_pylib v7_3_0 for access to new interfaces - samAdmin disallowDisk: add 'keepCachedFiles' flag, call new 'disallowDisk2' interface when set to true. - samUser get dbserver connection count: new implementation v7_2_6: - Occasionally (no more often than once per day, it seems) the SamAtAGlance and/or nameservice polling scripts get exceptions: Unhandled exception in thread: Traceback (most recent call last): File "/fnal/ups/prd/python/v2_1/SunOS-5/lib/python2.1/threading.py", line 393, in __bootstrap self.__stop() File "/fnal/ups/prd/python/v2_1/SunOS-5/lib/python2.1/threading.py", line 399, in __stop self.__block.notifyAll() File "/fnal/ups/prd/python/v2_1/SunOS-5/lib/python2.1/threading.py", line 235, in notifyAll self.notify(len(self.__waiters)) File "/fnal/ups/prd/python/v2_1/SunOS-5/lib/python2.1/threading.py", line 217, in notify me = currentThread() TypeError: object is not callable: None With help from Sinisa, I think I have found the section of code that produces this (harmless but annoying) message in the output mail messages; the assumed fix is in this version. Only time will tell if it really fixes the problem. LATER RESULTS: no, this did NOT fix the bug. The bug cannot be fixed by us, it is a bug in the python threading library, fixed in python 2.3 (and beyond); reference bug tracking info at: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=754449&group_id=5470 v7_2_5: - rebuild against sam_common_pylib v7_2_5 so that all sam structs are reproducible from their string representation. - fix typo in table file reported by Robert Illingworth. v7_2_4: - optimize DB queries used in sam_mis incoming plots, consumption plots, per advice of Randy Herber. v7_2_3: - rebuild against sam_common_pylib v7_2_3 to improve retry algorithm - any sampy image will set the environmental variable __SAMPY_VERSION to the version of the code being run; the GNUmakefile sets it (via sam_util scripts) when cutting a release, otherwise set to 'development'. Implement 'sam get version', 'samadmin get version', 'sammis get version' and 'samgridBatchAdmin get version' methods. - sam_admin_pyapi: raise SamException from samDbAdmin.parseFullySpecifiedMountPoint if no nodename is specified. - sam_admin_pyapi: in samStationCacheAdmin, for the commands 'uncache station file', 'uncache station disk', and 'uncache entire station', which go directly to the DB server (rather than going through the station), force a SamException if the station is up and running, so that the station *must* be stopped in order to run these commands. This should help prevent confusing inconsistencies between station and db. v7_2_2: - fix typo in reporting of samadmin null volume synch, which did not affect the actual volume updates but caused the final report (when "-v" was specified) to barf with an AttributeError. v7_2_1: - add new 'samadmin null volume synch' command, which will find all files in sam with locations of type 'tape' but with null volume labels, query enstore for volume/fileSize/crc information, and update the SAM db with the actual data. (Very similar to 'samadmin dccp synch', but for null volume_id locations on tape rather than for volume_id that indicates the volume label is 'unknowndcachevolumelabel'). - rebuild against sam_common_pylib v7_2_1 to improve documentation (see sam_common_pylib release notes for details) v7_2_0: - rebuild against sam_idl_pylib v7_2_0, sam_common_pylib v7_2_0, so that we can use the new ServantAlreadyClosed exception. Retry call on a new servant if this exception is raised. - also, documentation now includes named anchors and references to them for easier navigation by documentationCategory. - bug fix in "sam request file transfer": descriptionFile option is only needed if --declareFile is specified, reported by Thomas Kuhr. - bug fix in NameServiceObj: catch ValueError when float(startTime) is funky. - bug fix in Autodest: fix autodestUpdatePnfsConfig to pass the library to the dbserver (it was collected but not passed), reported and fixed by Steve White. - remove "import xml" from the sampy.py script; the xml distributed with python 2.1 is incomplete and broken (as reported by Robert Illingworth and Alan Sill). v7_1_13: - exclude the SAM_DISABLE_CLOSE_SERVANT code (eventhough the testing has not been done yet) - bug fix in Autodest: fix call to addPnfsTapeLocation (had the wrong signature) - bug fix for "get registered dimensions" (NameError, did not import DbDerivedClient properly) v7_1_12_cdf: - in v7_1_11_cdf, code was introduced to disable the client "closeServant" call. This was in order to solve the problem where a client calls "closeServant" but the db server has already closed the servant; hence, a new servant is created just so that it can be closed. In v7_1_12_cdf, the code to disable the client "closeServant" method is still available via the configurable environmental variable $SAM_DISABLE_CLOSE_SERVANT (set to "true" to disable closing servants); HOWEVER, also introduced is what I think is a better bug fix: if the servant is _non_existent(), then do not bother to call the closeServant method. In this way, we will not create a new servant just to close it; but the client call to closeServant is not completely disabled. v7_1_11_cdf: - this release contains code that can disable client closeServant calls; it goes with the db server that closes connections after each call; the code is on the cvs branch (only sam_common_pylib changes) v7_1_11: - modify table file so that setup of sam_config includes the dependencies in the sam_config file (i.e., remove the "-j") - fix bug in table file: SAM_USER_PYAPI_DIR was pointing to the wrong directory. v7_1_10: - set SAM_USER_PYAPI_DIR in table file so that sam submit-ted processes have access to ${SAM_USER_PYAPI_DIR}/src/GenericRunProject.py v7_1_9: bug fix (typo) in 'sam simulate station restart' v7_1_8: - add 'sam simulate station restart' for timing studies - rebuild against sam_common_pylib v7_1_8 v7_1_7: - rebuild against sam_common_pylib v7_1_6 for improvements in performance for marshalling/unmarshalling items. v7_1_6: - add appropriate cdfnodes to authorized nodes for unbinding from nameservice (sam_mis_pyapi/lib/NameServicePlugin.py). - add RELEASE_NOTES to web pages. v7_1_5: - rebuild against sam_common_pylib v7_1_5 to incorporate bug fixes in GnuPlot and ExecSubprocess. - add setupOptional('gnuplot'), setupOptional('ximagetools') to table file v7_1_4: - rebuild on sam_common_pylib v7_1_4 to incorporate bug fixes in SamUpsMgr. v7_1_3: - update to sam_common_pylib: new SamUtility/SamUpsMgr so that we can setup required packages inside of the python code - fix to sam_mis_pyapi: make sure that 'gnuplot' is in our path, else "setup gnuplot" before trying to generate plots; ditto for 'convert' and ximagetools. (Bug noticed by Jean Paul Martin, an astute d0-sam-shifter who reported that the netstat pages had not been updated; also affected the other sammis plots). - add "sam delete dataset definition", "sam update dataset definition" to allow for modification to definitions from the command line. NOTE that if snapshots have been taken, the only modification allowed is to add new keywords. v7_1_2: - add "sam get registered station monitor levels" "sam get registered station life cycle statuses" interfaces (requested by Art Kreymer) - add "sam close servants" interface (requested by CDF) - add "samadmin uncache station file" "samadmin uncache station disk" "samadmin uncache entire station" interfaces (requested by CDF) - add "samadmin purge zombie projects" interface (requested by Robert Illingworth) - add "samadmin delete data disk" interface (requested by Liz Buckley-Geer) - add "sam get registered data disks" interface (requested by anybody who has had to add a new disk location and could not find the original mount point) v7_1_1: - add global -s (retrySilently) option to all sam interfaces to turn off retry messages - add global -t (SamTimer) option to all sam interfaces, to print useful timestamps; implement timestamps at informative checkpoints in SamServerProxyRetryHandler (begin remote method call, begin unmarshalling of return arguments, return to caller) - implement sam.getRawAncestors, sam.getFileDescendants command interfaces. v7_0_2c: - freeze using sam_freeze v2_1_1, which should alleviate "Fatal Python Error" when using the frozen release in the D0Release world. - add version to the generated documentation v7_0_2b: - fix typo in sampy_startup.py v7_0_2a: - fix typo in www/index.html - add more examples (GetRawAncestors.py, GetFileDescendants.py, GetProjectDescendants.py) - more efficient building of documentation (generate them in $SAM_DIR/www rather than copying them from the individual sub-packages) v7_0_2: - new build process, uses sampy_startup.py script for setting the environment. - sampy_startup.py adds all PYTHONPATH elements to the target sys.path (so that external modules in the user's PYTHONPATH can be imported) v7_0_1a, b: - changes to table file required for SunOS so that the PATH is correct for sampy to be able to launch itself v7_0_1: - add setup of samgrid_batch_adapter to table file v7_0_0: - initial release of frozen sam distribution This is the SAM package, which is only a wrapper for setting up the necessary pieces of client code. v6_0, required after SAM DbServer schema change upgrade, now sets up and uses sam_user_pyapi rather than sam_user_api or sam_user. (And, correspondingly different sam_common_pylib, sam_idl_pylib rather than sam_common). v5_0: now sets up and uses sam_user_api rather than sam_user. v4_x and previous: sets up sam_config and sam_user. This package should contain GNUmakefile only.